Spaced Scheduling for Large Language Model Training
Authors: Amine El hattami
Nicolas Chapados
Christopher Pal
Venue: Accepted by TMLR ()

Abstract: Recent breakthroughs in deep learning have accelerated progress toward increasingly capable large language models (LLMs), even sparking discussions about the path to Artificial General Intelligence (AGI). Yet, current LLM training pipelines continue to depend on heuristics and human-driven empirical analysis to curate data. In practice, more sophisticated data selection methods often incur high costs, exhibit limited adaptability, or do not consistently surpass simple random baselines across various models and datasets. In this work, we propose Spaced Scheduled Training (Sst), a novel adaptive data selection strategy that prioritizes training examples based solely on per-example perplexity computed from the model’s own evolving parameters. By obviating the need for external reference models, Sst customizes data selection to the model’s unique characteristics, including its pre-training data composition, and eliminates biases commonly introduced by these external models. Extensive experiments on seven LLMs (0.5B to 32B parameters) in the instruction-finetuning (IFT) setting show that Sst consistently outperforms representative state-of-the-art selection approaches like Deita and InsTag on the Open LLM Leaderboard. For instance, with Qwen2.5-32B and a 30k examples data budget, Sst achieved a 42.75% Open LLM Leaderboard score, exceeding a leading data-selection baseline (38.56%) and the full-100k dataset baseline (39.58%). We further present a theoretical framework to assess computational overhead of model-based selection methods, showing that Sst remains efficient in practical scenarios, and propose strategies to mitigate the overhead in worst-case scenarios. Our findings underscore the potential of model-informed dynamic data selection, offering an efficient, adaptable, and cost-effective approach. We release our training code, trained models, and data mixes in our public repository.

Published in Transactions on Machine Learning Research (06/2025)
Spaced Scheduling for Large Language Model Training
Amine El Hattami
ServiceNow Research, Mila, Polytechnique Montréal
Nicolas Chappados
ServiceNow Research
Christoper Pal
ServiceNow Research, Mila, Polytechnique Montréal, Canada CIPAR AT Chair
Reviewed on OpenReview: https: // opw.reveww. net/ formm? id= poxTOY2288
Abstract
Recent breakthroughs in deep learning have accelerated progress toward increasingly capa- ble large language models (LLMs), even sparking discussions about the path to Artificial General Intelligence (AGT). Yet, current LLM training pipelines contimuc to depend heuristics and human-driven empirical analysis to curate data. In practice, more sophisti- cated data selection methods often incur high costs, exhibit limited adaptability, or do not consistently surpass simple random basclines across various models and datasets. In this work, we propose Spaced Scheduled Training (SST), a novel adaptive data selection strategy that prioritizes training examples based solely on per-example perplexity computed from the model's own evolving parameters. By obviating the need for external reference models, SST customizes data selection the model's unique characteristics, including its pro-training data composition, and eliminates biases commonly introduced by these external models. Ex- tensive experiments on seven LLMs (0.5B to 32B parameters) in the instruction-finetuiin (IFT) setting show that SST consistently outperforms representative state-of-the-art selec- tion approaches like DEITA and INSTAG on the Open LLM Leaderboard. For instance, with Qwen2.5-32B and 30k examples data budget, SST achieved a 42.75% Open L.M Leaderboard score, exceeding a leading data-selection baseline (38.56%) and the full-100k dataset baseline (39.58%). We further present theoretical framework to assess compu- tational overhead of model-based selection methods, showing that SST remains efficient in practical scenarios, and propose strategies to mitigate the overhead in worst-casc sccnarios. Our findings underscore the potential of model-informed dynamic data selection, offering an efficient, adaptable, and cost-effective approach. We release our training code, trained models, and data mixes in our public repository
1 Introduction
Recent advances in large language models (LLMs) have transformed natural language processing, enabling breakthroughs in applications ranging from artificial agents to scientific discovery. While scaling the model size and the training data has driven much of this progress, data quality increasingly recognized as a bottleneck, particularly when hardware and computational budgets constrain further scaling. Data selection methods can be broadly categorized into two approaches: static and dynamic. Static approaches pre-select data offline using heuristics external "oracle" models. These methods often incur substantial computa-
tional cost, either from training specialized evaluation models (Liu et al. 2024
Lu et al. 2023 or from using
https://github. com/ Amin3e/nnn

---

Published in Transactions on Machine Learning Research (06/2025)
Avonue Open L.M. Lovkerhonnd Store
Ser (OURS)
BANDOM DATASET
Iss TAG
38-
RRoLoss
DEITA
34
300
200
300
400
Training Theration
Figure 1: Performance comparison on Qwen2.5-32B (30k instruction budget for 2 epochs). SST significantly outperforms strong random baseline, competitive Gha- methods (INSTAG, DEITA) and another adaptive method (Rho-Loss) Despite Ow often being considered less sensitive to sampling ap- proachos, SST achieves higher Open LLM Ieaderboard be throughout training and displays lower variance across four random seeds. The late-stage uptrend suggests further training may extend SST's ' lead. All meth- ods draw 30k examples from the same 100k-instructio pool.
costly commercial LLMs for data scoring (Chen et al. 2024). Moreover, oracles can introduce biases in the evaluation, such as the documented verbosity bias in ChaiGPT (Saito et al. 2023). This bias is particularly problematic when the same oracle is used for benchmarking, as is the case with MT-bench (Zheng et al. 2023 While these methods can yicld reasonable performance, they lack adaptability: the dataset remains fixed regardless of the trained model's characteristics (e.g., size, or pre-training data composition). Dynamic approaches, in contrast, integrate data selection during training (Mindermann et al. 2022 Jiang et al. 2019 Loshchilov & Hutter 2015), which can, in principle, adapt the curation to the model''s evolving state. Yet many of these techniques are expensive to run at scale or fail to provide consistent performance. For instance, RHO-LOSS [Mindermann et al. 2022) requires training a proxy model, performing a forward pass on the entire training dataset using this proxy model, and conducting additional forward passes for batch selection. Further, studies have shown that many dynamic methods fail to consistently outperform simple random selection baselines (Kaddour al.) 2023 Liu et al., 2024). As a result, recent works like Tülu 3 (Lambert et al. 2024) still rely hcuristics and large-scale empirical analysis to refine data. Recent rescarch high- lights additional challenges in data selection, especially when dealing with diverse data sources. Determining the optimal mixing ratio is difficult and depends on the specific model being trained. Further, relying on external oracle models can introduce unintended biases, as they may overemphasize data that aligns well with their own output. For instance, DEITA (Liu et al.] 2024h sampled a 6k-cxample subsct from a 300k- example pool comprising three sources (ShareGPT (Chiang et al. 2023), UltraChat (Cui el al. 2024), and WizardLM (Xu et al. 2023)); ); however, this subset exclusively contained examples from ShareGPT. Recent LLM pre-training work, such Llama 3 (Dubcy al., 2024 suggests that adjusting mixing ratios multiple times during training is beneficial. This observation highlights the importance of dynamic approaches and reinforces the need for more adaptive methods.
To address these challenges, we propose a novel adaptive and efficient data selection strategy called Spaced Scheduled Training (SsT) that dynamically adjusts the training dataset based on a model's evolving learning state. Unlike approaches that rely on external scoring models, SST prioritizes training examples based on per-example perplexity, a computationally efficient and reliable proxy for example difficulty. Our work builds prior work in static selection (Marion et al.) 2023) but incorporates dynamic scheduling to continuously adjust the dataset mix throughout training. SST also differs from existing work in that it: (i) eliminates the need for costly external oracle models; (ii) tailors selection to the target model's unique characteristies (c.g., size and pre-training data composition), avoiding biases from external scoring models; and (iii) adapts the dataset mix continuousl through its "spaced scheduling" mechanism, allowing different models to emphasize data that is most beneficial at each stage of training. We evaluate SST across seven LLMs with sizes ranging from 0.5B to 32B parameters from four distinct model families, including Llama 3.1 (Grattafiori al.
2024a), Ilama3.2 (Dubey et al. 2024), Gemma (Team al. 2024), and Qwen 2.5 (Yang et al. 2024 Our
2

---

Published in Transactions on Machine Learning Research (06/2025)
empirical analysis, using the recent Open LLM Leaderboard (Fourrier et al. 2024 demonstrates that SsT delivers consistent performance across architectures and model sizes. As illustrated throughout training in [ SST outperforming all selection baselines, including those relying on ChatGPTT-based selection <Liu ct al. 2024 Lu et al. 2023). Furthermore, we introduce a thcoretical framework, grounded in LLM scaling laws Kaplan et al. 2020), to quantity the scaling of overhead for data selection methods, including SST.
Figure
This framework allows for a more principled comparison of different approaches. Through this framework, we show how inference-oplimized backends allow SST to maintain low computational overkead, enabling large-scale use. We summarize our key contributions follows:
Novel Dynamic Data Selection Algorithm (Sst): We introduce Spaced Scheduled Training (SST), a novel approach that relies on per-example perplexity to prioritize training examples and dynamically adjust the data mix throughout training ($ g..
Comprehensive Empirical Evaluation: We demonstrate SST's effectiveness in the IFT setting on seven models (0.5B to 32B parameters). SST outperforms all basclines on five out of seven models, including the full 100k dataset baseline. On the remaining two models, SST achieves average Open LLM Leaderboard scores within 1.14 absolute points of those from the full 100k baseline ($ (5.
Scalability and Thcorctical Overhcad Analysis: We introduce a framework, based on LLM scaling
laws research to analyze the overhead scaling of data selection methods, including SST. We demonstrate
how to implement SST efficiently for large-scale training ($ 4.2)
Practical Insights for LLM Data Selection: We analyze perplexity- based signals to provide foun- dational insights and guidelines for future work in data selection. Our findings show that perplexity sampling is influenced by factors such as subset size and pre-training data composition. Furthermore, it is beneficial to delay perplexity sampling until training stabilizes, and a dynamic policy for selecting data leads to improved results ($ 3
2 Related Work
Data selection plays a crucial role in training large language models (LLMs). It aims to prioritize informative examples or remove non-useful or noisy data that might degrade performance. Data selection approaches fall into two main categories: static and dynamic methods. This section reviews key research in data selection, highlighting their strengths and limitations, and motivating our proposed SST approach.
Static methods pre-select the training data offline, before training begins, independent of the specific model being trained. Marion et al. (0023 proposed static pruming method relying on perplexity (PPL) scores computed with an external reference model. They demonstrated that PPL is more effective than more complex metrics, such as Error L2-Norm (EL2N). By pruning examples with low PPL, they achieved similar performance using only 30 % of the data. [Sorccher et al. (2022) explored pruning in computer vision, using proximity to the decision boundary as a difficulty measure in a teacher-student perceptron setup. They showed that the examples chosen for pruning depend on the initial dataset size and identified the conditions for aggressive pruning. They emphasized that the success of pruning methods relies on the quality of the pruning metric, noting that most metrics they tested are costly to compute, making them impractical for large-scale use. [Sachdeva et al. (2024) explored pruning for pre-training T5 models (Raffel et al. 2020), introducing two scoring methods: (I) DENSITY, which estimates whether similar examples
have been sampled, and tries to maximize coverage; (2) AsK-LLM evaluates example quality by prompting an instruction-tunet LLM (FLAN-T5; see Chhug et al. (0022) to predict whether it contains informative signals (a ges/no question). The study demonstrated that DENSITY performs comparably to using the full dataset. In contrast, AsK-LLM outperforms full-data training by rejecting 90 % of the data and converging 70% % faster. Recent instruction fine- tuning (IFT) work leverages state-of-the-art commercial models to score and select examples (Chen et al.
2024 Liu et al. 2024 Zhao et al. 2023 Xu et al. 2023). ALPAGASUS Chen
ct al. (2024) uses ChatGPT with a handcrafted prompt to predict scores from 0 to 5, pruning the examples below certain threshold. DEITA (Liu el al. 2024), building on [u et al. 2023). adds diversity as a selection criterion and proposes a two-level scoring system: Evolve Complexity c and Evolve Quality q, to compute a single score s = c x q. Initially, ChatGPT is used for scoring, but Liu et al. (2024) later trains model to replicate ChatGPT's scoring, substantially reducing the cost. DEITA shows that mere 10,000 examples
can outperform models trained on ten times as many. Similarly, INSTAG (Lu et al. 2023 uses intention
3

---

Published in Transactions on Machine Learning Research (06/2025)
tags as metrics for instruction diversity and complexity. It utilizes ChatGPT to assign one more tags to each instruction example. To ensure high-quality tags, the tags are normalized using frequency filtering and aggregation (e.g., semantic aggregation). INSTAG prioritizes complex queries with the highest number of tags, while maintaining diversity by selecting examples that expand tag coverage. To reduce the computational cost of lag assignment, the authors created INSTAGGER, a distilled LLM that mimics CPaiGPT annotation capabilities. We compare our approach with DEITA and INSTAG in our experimental work.
Model-based static approaches currently produce the best results. However, despite reducing manual effort associated with empirical approaches which involve extensive human intervention, they often incur high computational costs. When training specialized evaluation models, the cost of these models can be offset by repeated use, but this advantage diminishes in practice. For instance, DETTA @Liu et al. 2024) requires training two specialized scoring models that have a maximum sequence length of 2,048; handling longer sequences requires re-training new evaluation model. APT-based methods, like ALPAGASUS (Chen et al., 2024), are susceptible to prompt design issues and biases such as the verbosity bias inherent in GPT mod- els (Saito et. al., 2023), leading to imbalanced data selection, as ralust by the over-representation of single source dataset in DEITA'S curated 6k examples subset containing 100% of the examples from the ShareGPT dataset. These issucs are particularly problematic when the same models are used for both data curation and benchmarking (e.g., MT-bench: Zheng et al. 2023)) as performance can be artificially inflated. Beyond computational overhead, static dataset selection methods suffer from a fundamental limitation: they fail to account for the model's unique pre-training data composition. For instance, if a model is pre-trained on 70% code-related data, an effective selection strategy would need to down-weight code-related tasks. This also applies to the data complexity categorization, where an example requiring 10-step chain-of-thought rcasoning process may be considered casy for 70B model but intractable for 1B model.
Dynamic methods integrate data refinement directly into the training process. For instance, online batch selection aims to optimize training by selecting examples for each batch, often by scoring and ranking a large batch to select the top-k examples (Loshchilov & Hutter 2015). [iian ct al. (2019) proposed Selective- Backprop, which prioritizes examples with high loss. However, Mindermann el al. (2022) challenged this approach, arguing that high-loss examples can be noisy or mislabeled. They introduced the Reducible Holdout loss selection (RHO-LOSS) (Mindermann et al., 2022) which reduces the impact of noisy data by weighting down their losses. RHO-LOSS uses proxy model trained on holdout set to select examples that minimize the holdout loss by approximating the reducible holdout loss objective. While RHO-LOSS improved accuracy and training speed, it incurs significant overhead due to the cost of training the proxy model, a forward pass on the entire training data using the proxy model, and the additional forward passes for the batch selection. Dynamic methods attempt to overcome the rigidity of static approaches by selecting data during training. However, despite their adaptability, these methods often fall short of ttate-of-the-art static model-based approaches. For instance, Kaddour et al. (2023), found that RHO-LOSS (Mindermann) et al. 2022) fails to outperform simple random selection baselines consistently. Moreover, dynamic methods also suffer from algorithmic complexity that prohibits their use at scale. The aforementioned overhead of RHO-LOSS, for example, makes it impractical at scale.
In both categories, when the initial data pool contains multiple sources or categories, determining the opti- mal mix ratio creates an additional complexity for any data selection method. Finding the optimal mixture remains underexplored and currently relics heavily on heuristics and extensive empirical analysis to select the mixing ratio. Further, recent work in Llama models (Dubey et al.] 2024) shows that adjusting the mix ratio multiple times during pre-training can be beneficial, as it enables the model to better adapt to its evolving needs. The shortcomings discussed above underscore the necessity for an adaptive data selection strategy that dynamically tailors the training data to the evolving needs of the model while preserving com- putational efficiency at scale, forming the foundation for our proposed method, Spaced Scheduled Training (SST), detailed in $ SST leverages the target model itself to guide the data selection process. By elim- inating the need for costly external models, SST adapts both data categorization and selection criteria to the characteristics of the trained model. Our approach uses per-example perplexity as a computationally efficient and reliable proxy for example difficulty, premise rigorously examine in $ and Appendix D with its overhead further analyzed in 4.2 By exlending the static selection method of Marion et al. (2023) with a dynamic mechanism that continuously adjusts the dataset mix throughout training, SST overcomes
4

---

Published in Transactions on Machine Learning Research (06/2025)
the limitations of previous methods and better aligns the training data with the model's evolving state. These improvements enable SST to outperform the best existing methods (INSTAG and DEITA) and even baselines utilizing significantly more training data, as shown in $
3 Preliminary Analysis
This section presents the key findings of our preliminary analysis of perplexity-based data selection in the IFT setting. These insights motivale and lay the groundwork for our proposed adaptive method in $
We analyze perplexity- based data selection within the Instruction Fine-TTning (IFT) setting, extending the approach of Marion al. (2023) with several key distinctions. IFT Sctting: We investigate perplexity- based data selection in the context of IFT rather than pre-training- Target Model as Reference: Unlike approaches using external models, we use the target model itself to guide the selection of its training data, Broad Analysis Scope: We evaluated models ranging from 0.5B to 32B parameters across state-of-the- art architectures (Llama3.1 (Grattafiori et al. 2024a), Llama3.2 (Grattafiori et al., 2024a),, Qwen2.5 (Yang et al. 2024h, and Gemma2 (Team et al. 2024)),, offering a significantly broader evaluation than the two models used in Marion et al. (2023), totaling 248 training runs across different sampling settings and seeds, We compare static perplexity- peplexxitybbased sampling and random selection (using 10%, 30% and 50% of pool of 100,000 examples) against a baseline trained the full dataset, with performance measured using the Open LLM Leaderboard benchmarks. We sample from a dataset collection D (D,) where D; is an instruction dataset. The collection consists of 15 datasets from the TÜLU 3 mixture (Lambert et al. 2024). We compute the per-example perplexity values PPL(()) (e) on the target tokens only, to align with the IFT setting as follows:
PPL(C) exp (12 Σ ELL(())) exp(Le), [cl
tyEe
where NLL(t;) denotes the negative log likelihood of output token tj, and Le is the target loss of the example e. Detailed experimental settings and further analysis are provided in Appendix [ The key findings of our analysis are threefold. First, heuristic-based curation is not sufficient: as illustrated in Figure 2 simple random sampling can sometimes outperform training on all 100,000 examples, suggesting that even carcfully curated data mix [Lambert al. 2024 contains redundant loss beneficial examples.
Random Per-Dataset
Keep Bottom Per-Datasct
Keep Middle Per- Dut-Dttastt
Keep Top Per-Datases
15%
Relatvve Periemass 0800
10%
0%
-10%
-15%
Subaut Size
-20%
10k
.25%
30k
-30%
50k
0.5
3
8
9
14
27
32
0.5
14
27
32
0.5
8
14
27
32
0.5
1
3
8
9
14
27
32
Model Size
Figure 2: Performance of static per-dataset perplexity sampling: Middle-segment tends to improve smaller models (<8B), while top-segment benefits larger models (e.g., Qwen2.5-32B) Only certain static perplexity- based configurations surpass the baselines, motivating the adaptive approach introduce later. Relative performance change AR is calculated as AR =R=(Smethod (Smethod S //tnee/i/ssee (Equation 7 where Smethod and Shaseline are the average Open LLM Leaderboard score of the method and 100% baseline. Points below the red dashed line indicate performance drops compared to baseline. Error bars represent standard error over two seeds. Detailed experimental setup is provided in Appendix D
Second, the best-performing perplexity sampling criteria vary with model size and subset size: smaller models (< &B) tend to benefit from "middle" perplexity ranges, while larger models (> 14B) often

---

Published in Transactions on Machine Learning Research (06/2025)
gain more from high-perplexity examples (Figure 2j. Further, this criterion varies with subset size. Figure shows that using examples from the bottom segment is not beneficial across models. When selecting large subset (e.g., 50% of the data) for a model that typically benefits from mid-perplexity examples, choosing these examples predominantly from the top perplexity segment can be more advantagcous than attempting to draw such a large fraction from only the middle segment. The latter approach could incorporate too many low-perplexity (casy) examples from the broadened middle segment, thereby negatively impacting performance. This trade-off also applies when selecting & smaller subset, such as 10% of the examples. Finally, aligning data complexity with training progress proves beneficial. As shown in Figure B different perplexity segments are most useful at various training stages, where the models benefit from beginning with easier examples and then transitioning to more challenging ones, which mirrors curriculum learning principles (Bengio et al. 2009). Further, we found it beneficial to start training with randomly selected data per dataset until a set training iteration threshold reached. After this point, we compute the perplexity values once and initiate perplexity sampling. This delay ensures that perplexity sampling starts after the model's data shift adaptation in early training, ensuring that the sampling uses more meaningful perplexity values. As shown in Figure 4
the delay varies across models and decreases model size increases, and delaying beyond this threshold harms performance due to changes in the perplexity distribution, as we describe in more detail in Appendix D.3
Relatvve Peforazze Clange AR (N)
60%
Keep Bottom Per- Dalaset
Keep Top Per Dalasel
Keep Middle Par-Dalaset
Keep Middle Per
40%
Owen2.5-322
20%
0%
Laama3.1-88 1-8B
25%
75%
100%
0%
50%
75%
Train Iteration Fraction of 2 Epoch Training (%)
Figure 3: Performance comparison of static perplexity segments during training for Llama3. 1--BB (left) and Qwen2.5-32B (right). Models benefit from starting with casier examples (lower perplexity) before transitioning to harder ones (higher perplexity); motivating dynamic selection stralegies akin curriculum learning (Bengio et al.] (009). The figure displays the best-performing and second- best performing static configurations identified for cach model (Appendix D) Performance is mcasured as relative change AR (Equation
7 of Open LLM Leaderboard mean scores compared to the base pre-trained model. Standard error over two seeds shown.
20%
Qwen2.5-0.5B
Llama1.1-BB
Qwan2. 5-32B
10%
-10%
-20%%
11%
155%
25%
50%
75%
0%
1.5%%
25%
50%
75%
05%
15%
25%
50%
75%
Perplexity Sampling Start Iteration (%)
Figure 4: Efffec of delaying static perplexity- data selection. Delaying perplexity sampling improves performance up to a threshold that varies by model and generally decreases with model size. Smaller models (e.g., Qwen2.5-0.5BB benefit from a longer delay (25% of training), while larger models (e.g., Qwen2.5-32B) degrade as early as 10%. AR (Equation 7 is the relative performance change percentage compared to using delay (using the pre-trained model for data selection). Results show mean scores and standard error across two runs with different random seeds.
Together, these results motivate our dynamic data selection strategy that delays perplexity sampling until the model starts producing meaningful and reliable perplexity values and refines which perplexity segments

---

Published in Transactions on Machine Learning Research (06/2025)
are most beneficial as training progresses, rather than relying on single static approach. Taken together, these preliminary findings highlight the limitations of static perplexity-based selection methods, emphasizing that no single static strategy consistently benefits all models across training stages. Smaller models benefit from different perplexity segments compared to larger models, and the optimal choice varies dynamically during training. Furthermore, delaying perplexity-based selection until models stabilize yields significant performance improvements, underscoring the need for an adaptive approach. Motivated by these observa- tions, we propose Spaced Scheduled Training (SST) ($4) to mitigate the limitations of static perplexity- baseed selection methods.
4 Spaced Scheduled Training
We propose Spaced Scheduled Training (SsT), an efficient and adaptive method for selecting and scheduling training examples based on their perplexity. SST, as shown in Figure [] builds on insights presented in 3 where we demonstrate that static selection strategics often fail to deliver consistent improvements. contrast, SST provides a dynamic approach to perplexity-based data selection, failoring this selection to the characteristics of the target model (i.e., the model being trained), such as its size and its unique- and often obscure pre-training configuration. SST utilizes dataset-aware selection, unlike methods such DEITA [Liu et al. 2024) INSTAG (Lu et al. 2023), which disregard implicit dataset categorizations during the selection process. By focusing on examples most likely to improve the model's learning, SST enhances performance while optimizing resource utilization. It consistently outperforms static and heuristic- methods, offering a scalable and practical solution for elficient data selection. Moreover, SST avoids reliance on external reference models that are costly or have restrictive data licenses (e.g., ChatGPT),, additional scoring models that become outdated (Lu et al. 2023 Liu et al.] 2024). or complex selection techniques that are impractical at scale (Mindermann et al.) 2022 Liu al. 2024 making it easily adaptable to existing training pipelines.
pet
prd
Dataset Collection
Sampled Collection D
Target Model
D
D2
D
P.
(a) selection window
p
prd
prt
pPd
C
Bn (b) Selection window shifts (lowards easier examples)
5:
P..
(c) Selection window shifts right (towards harder examples) decroasing
Losos inpwsiing
Figure 5: A visualization of the adaptive phase of our SST algorithm,, using a dataset collection D (Da), composed of d = 1... datasets. (a) SST initializes the reference perplexity percentiles pref for each dataset Da to the 50th percentile. Tt then selects the initial collection D' based on the initial dataset ratios 8a, such that the total size of D' is Pz] Then, SST trains the model on D' and monitors the loss curve. (b) If the loss increases significantly, the selection window shifts loward easier examples (bot segment), suggesting that the current data configuration is too difficult for the model's current state. (c) If the loss decreases significantly, indicating that the target model can Iearn from more complex examples, SST shifts the selection window toward harder examples (top segment).. If the loss curve is stable, SST continues training without changes. At each shift, SST adjusts the dataset ratios using Ba (Fquation 2 prioritizing datasets with higher PPLp50 (d) values. This process continues until the end of training.
7

---

Published in Transactions on Machine Learning Research (06/2025)
4.1 Method
SST dynamically selects and schedules training examples at each stage of training, using per-example per- plexity as a measure of example utility. SsT operates on two levels: (1) adjusting dataset ratios to prioritize datasots and (2) filtering examples to focus on the most impactful training examples within cach dataset. This dual-level approach allows SST to continuously optimize the training process, enhancing convergence speed and improving generalization for the target model. The algorithm consists of two main phases: a warm-up phase and an adaptive scheduling phase, as shown in Algorithm
Algorithm 1 Spaced Scheduled Training (Ssr) (full version in Algorithm
Require: 0: initial model parameters; D = (Di...., Dat: collection of d datasets; Amax: maximum training iterations; ky E (0, 1]:
warm-up slope window ratio; maximum warm-up slope evaluation retry count. P. (0, 1/: glohal subset. ratio; (0,
window-shift ratio; é > 0: slope threshold for loss slope evaluation;
Ensure: Updated model parameters
1: Warm-up phase
2: f, WARSIUF(8, D, bmax kw)
3: Adaptive Scheduling Phase
4: & -uu/tmax
5: P = CosPunIPPL(e,
Compute per-example perplexities PPL(e) for each example D
6: Compute
each datasct Da using Equation
7: pest + 50 for each dataset Da
Initialize window reference 50th percentile
"SuusmTraaDDnasssttssssss Ps. 9a, pret)
9: Initialize loss buffer Lbut + [I (capacity [/s Emuna)
Main training loop
11: for fw 1 to fmax do L,L, TRAINSTEP (
Train model batch, update batch perplexities
13:
Append La to Lbul
14:
if [batl [R [max] then
13:
elope + COSPUTESLOPE(Lemn))
e.g., via simple linear regression
Clear Lhat
17:
D. 1- Recompute datasct mixing ratios 2- Decide window shifts harder easier examples Loss is decreasing, shift to more complex examples
Update Ba
dataset Da using Equation
19:
20:
if slope < -5 then
21:
paf +- mimi peot x (1 7), 100 1000
slope >
23:
prs + max prot x (1 7), 24x100
D Loss is increasing, shift to easier examples D. Loss is stable, keep the current window 3- Select updated subset based on new mixing ratios and reference perplexity percentiles SeaecTPPDDDtteeSSSaSatr Pa: Sd. P.
else
23:
no change to paot
27:
29:
end if
30:
and for
Final trained model parameters
Table 1: Overview of SST-specific hyperparameters used during the warm-up phase. Values are based on empirical observations Generic hyperparameters (e.g., Imax or Ps) common to data filtering methods are omitted.
Description
Value
kw
Warm-up slope evaluation window size (Iraction of (mse). Used to detect early loss trend stabilization. Warm-up maximum retry count for slupe evaluation. Safeguard prevent an overly long warm-up plase.
0.1
Fw
Slope evaluation threshold. Ensures robustness minor fluctuations. Selection window adjustment factor. Controls the magnitude of the selection window shift.
10
0.1
The warm-up phase stabilizes training dynamics and ensures the model generates mean- ingful perplexity values before transitioning to adaptive scheduling. During this phase, the model trains on uniformly sampled data across all datasets, while SST monitors the training loss curve using a rolling window of size kw Emaz, where tmaz is the maximum training iterations and hw E (0, is a hyperparameter controlling the window's proportion relative to the training duration. The warm-up ends when one of the following conditions met: (1) the loss curve stabilizes, indicated by a nearly constant slope after the initial sharp decline in loss typical of early training; (2) maximum retry count Tar is reached, which acts
Warm-up phase:

---

Published in Transactions on Machine Learning Research (06/2025)
safeguard to limit the duration of the warm-up phase. The hyperparameter rww is particularly useful for smaller models (fewer than 8B parameters), as they may not exhibit a clearly identifiable loss stabilization point. Choosing appropriate values for kw and Iw is critical: kw should allow sufficient iterations to detect changes in the loss trend, while Iw prevents the warm-up from taking a significant proportion of the overall training, reducing the benefits of SST. Based on empirical observations ($3), we found setting ky S 0.25 and Tw so that the warm-up phase takes no more than 30% of the total training duration to be effective. In our experimentation, we set Aw = 0.1 and Iw 3. This phase helps SST converge more rapidly toward better selection window by deferring the start of adaptive scheduling until the model produces reliable perplexity values, which might otherwise be unstable during early training due to data distribution shifts or chat template adaptation in IFT.
SST performs the following during its adaptive scheduling phase: (1) Com- pute the example perplexity values PPI.(e) (Equation [ across the entire dataset collection D a single time. (2) Compute the datasct mix ratios Sd. For cach datasct d E DD the ratio given by:
Adaptive Scheduling Phase:
(d)
= x EvED PPLpsold'')
(2)
where (d) is the 50th percentile (median) of the perplexity distribution of dataset d, and pa is the global subset ratio. These ratios satisfy EdeD 822 = Ps and are used to form the initial dataset collection D' for training, prioritizing datasets with higher median perplexity values. During training, the PPLp500 (d) values are efficiently updatod using examples from the current batch, avoiding additional forward passes. The updated values adjust the dataset ratios Sa, increasing the proportion of datasets with higher PPL,so (d) and thereby shifting SST's focus towards more challenging tasks. (3) Define selection window parameters and initialize reference percentiles. For each datasct D₂ in the collection D, the reference perplexity percentile, part, is initialized to the 50th percentile. This Prof denotes the center of the dynamic selection window for Da- The width of this selection window, in percentile points of that dataset's perplexity distribution, is given by Wa = Ba x 100. Consequently, SST selects examples from Da whose perplexities fall within the percentile range [Prof Wa/2, Paot + Wa/2). During training, prof is adaptively modified. It is constrained by the bounds [Wa/2, 100- - Wa/2] (i.c., [ 100 - 8, xxx100)) to ensure the selection window remains entirely within the 100th percentile range. This initialization strategy ensures that, at the beginning of the adaptive phase, the selection window focuses on the median difficulty examples within each dataset before any dynamic adjustments are made based on the training loss. (4) Set k = twr/tmax, where the is the actual number of itcrations completed during the warm-up phase. This value k determines the rolling window size mm-x iteralions) for monitoring the loss curve during the adaptive scheduling phase. (5) Select a subset collection D' from D using 3a and the reference perplexity percentiles paot The main training loop runs for the remaining fuax - the itcrations, ensuring that the total number of training iterations is tmax: which also allows for a fair comparison with other methods and baselines. During this loop, SST tracks the training loss using a rolling window of size k Imax and updates the perplexity values of the examples in the current batch. Then, at every k- fmax steps, SST carries out: (1) Recalculate Ba (Equation 2). (2) Adjust the selection window based on the slope of the training loss curve. If the slope is negative (indicating decreasing loss), the current data configuration is likely not challenging enough. In this case, SST updates pree by multiplying it by (1 TT) (subject to the upper bound), shifting focus to harder examples. Conversely, if the slope is positive (indicating increasing loss), SST updates pref by multiplying it by (1 (1-r) (subject to the lower bound), shifting focus to easier examples. Otherwise, SST continues training without making any changes. (3) When pref and Aa change, SST samples D' using the updated values. When evaluating the slope, SST uses a small = 10-3 to account for numerical instability. Further, SST caps pref to ensure it remains within valid ranges. The value balances adaptabiltty and stability. It is similar to learning rate in optimization algorithms, controlling the rate of change in the selection window, In our experiments, we find = 0.1 effective. We use a weighted sampler to dynamically prune examples by assigning their weight to O. This approach enables the efficient exclusion and reintegration of examples into the training pool without incurring additional overhead from data loading. We present the Spaced Scheduled Training (SST) algorithm, providing a simplified overview in Algorithm [ and the complete version in Algorithm 2 Key hyperparameters are listed in Table I and Table 5 summarizes the notations used throughout the description.
9

---

Published in Transactions on Machine Learning Research (06/2025)
4.2 Overhead Analysis and Mitigation
As with any model-based data selection method, SST introduces computational overhead required to score the data. This overhead is often a concern in practice, especially at scale, as it can diminish a method's real-world usefulness. Training time alone is an insufficient metric for comparing overhoad across methods because it depends on specific hardware and implementation details and may not fully capture scaling behavior related to the cost. of scoring examples. Here, we propose theoretical framework based on LLM scaling laws (Kaplan et al.] (220)) to quantify the scaling of the overhead with the data and model sizes as a principled way to compare model- data selection methods, and provide time measurements to provide practical guideline for using SST.
Following (Kaplan et al. 2020), given a dataset D, the computational cost C(Ilrana) of training a model under a random sampling policy Handd for N epochs can be approximated as
((ITrand) R N IDI (Crorward + Cbackwara) R N ID] Soowardd: where Cbackward & 2 2forward:
(3)
with Clorward and Cbackward being respectively the computational costs of a single forward and backward pass. Next, consider a model-based data selection method llselect requiring a single forward pass to evaluate the data for filtering (e.g., static pruning or SST). Its training cost ((Igee))) can be written as:
C(IIselect) ~ (IDI Clorward + Cauisc) + (p. IDI - - Ctorward) s ('(lssotoc)) + Ps C(IIran));
(4)
selection cost
training cost filtered data
where Ps E [0, 1) is the subset ratio, Goorward is the forward-pass cost of the reference model used for evaluation, C'misc covers any additional overhead such as adjusting selection windows when using SST, and C' (Illee))))) is the overhead cost when using a model-based method Thclect- data selection method is considerod computationally efficient if its overhead (the first term in Equation is offset by the reduced training cost from using only Pa|P] examples. Contrasting Equations and we conclude that Nsclect is efficient if C(lleetect) S C(llrand). Therefore, SST is computationally efficient provided that
1 3N
('(lsceee)) + Ps C(Ilrand) S C(Ilrand)
Ps S
with Chnise Rd O.
(5)
SST uses the same model for reference and training, so Clorward Crorwadd For simplicity, we assume Gniso R O. This is because the miscellancous costs (c.g., calculating the loss slope, updating dataset ratios Ba) are typically negligible compared to the cost of the initial perplexity computation for the entire dataset (IDI Ooorward). Equation 5 also incorporates the warm-up phase ($ 4 (cf. Algorithm 2 Under these assumptions, we choose P. = 0.3 for the experiments in $ 5 comfortably satisfying Eqqation for N = 2, ensuring SST is computationally efficiont. In SST, the selection cost C'([ssrr) representing the overhead of the method, scales with the target model size, unlike methods relying on a constant-sized reference model (Ilconst), such as INSTAG (Lu et al. (202)), where the selection cost ('(loooost) is constant regardless of the target model size. To mitigate this overhead, inference-optimized frameworks can be used to accelerate perplexity computation.
To evaluate the potential speedup, we experimented with vLLM (Kwon et al. 2023 which provides 1.8x to 2.7x speedup with Bfloat16, and further speedups with 8-bit or 4-bit quantization using model sizes from 0.5B to 32B and contrasted the speedup with INSTAG using an 8B reference model using the ratio C'(TssT)/CC (oar))/C"TTmnaaaaa (m))) Since the downstream performance is likely to be affected when using lower precision, we also evaluate the performance using the evaluation setup described in $3 and contrasted each with the performance when using Bfloat16 precision without vLLM. We provide the details of the experimental setup in Appendix C The results of this study in Figure G show that using an optimized inference framework like vLLM is able to offsct the overhead introduced by SST at larger model sizcs. Specifically, we observe that when 8-bit) as it balances the overhead and performance trade-off. In contrast, using 4-bit precision (SST+ vLLM 4-bit) introduces non-negligible performance degradation. The main experimental results use a bascline environment (no vLLM) for fairness since some of the performance speedup in vLLM are highly dependent the hardware configuration, but Figure 0 is an additional ablation.
(CTmm))) 2 1 it becomes beneficial to use vLLM with 8-bit precision (SsT+ vLLM
10

---

Published in Transactions on Machine Learning Research (06/2025)
Selection Cost Relative to IncTag
InsTag (Reference)
Perforunnce Relative InsTag
1.25
SST Bi16
3
SST vLLM B116
SST vLLM 8-bit
SST vLLM 4-bit
2-
0.58
7B
14B
32B
0.58
38
718
14B
32H
Model Size (B) (log scale)
Model Size (B) (log scalc)
Figure Overhead and performance analysis of SST using vLLM with different floating -poii precisions SST compared to INSTAG using fixed 8B parameters reference model: SST introduces significantly lower overhead for models smaller than 8B parameters even without vLL.M (green region). For larger models (orange region), vLLM effectively offsets SST's overhead, with 8-bit precision (in red) providing the best balance between overhead reduction and performance retention. (Left) Shows the scaling of the selection cost ratio (C' ([IsssTT))CC ([/eeT) defined above) with the model size. (Right) Shows the performance ratio of SST with vLLM different precision levels relative to InsTag.
Finally, compare wall-clock times for training with and without SST across three runs on three different model sizes (Appendix C For instance, training the Qwen2.5-32B model on 100k examples takes about 36 hours, whereas using SST to evaluate the 100k then train on only 30k examples takes about 17 hours. This 17-hour runtime includes approximately 34% overhead from SST. Yet, still cuts total training time by approximately 50% compared to the 36-hour baseline. These figures provide rough estimate of the overhead introduced by SST in practice. Moreover, the overhead is acceptable when Ps satisfies the condition in Equation 5 and the performance Q(OnssT) is either significantly better than Q(01).ana) (as shown in 5 or remains comparable if compute resources are limited. In addition to the computational overhead, SST requires storage for perplexity values over the dataset, ie. O(DD). In our experimental setup (100k examples, Bfloa116 precision), this memory footprint is under 0.2MB.
5
Experiments
This section outlines the experimental design used evaluate SST's effectiveness in the IFT setting. We de- scribe the experimental setup ($5.1). present the results and discussion (55.2). and conclude with limitations (5.3).
5.1 Experimental Setup
We the data, training settings, and evaluation procedures described in $ consistent with the method- ology of Lambert et al. (2024) (see Appendix A for details). Each method selects 30k examples from 100k data pool, as our findings in $3 indicate that this subset is sufficient to match the performance of using the full dataset. We compare our method to DETTA and INSTAG, the current best-performing methods. We exclude methods like ALPAGASUS, which require commercial models that prohibit rigorous experimentation due to cost constraints. Additionally, we do not compare against dynamic methods, as prior work (Liu et al.) 2024 Lu et al.] 2023; Kaddour al. 2023) demonstrated that LLM-based static methods outperform existing dynamic methods in the IFT setting explored here. To ensure fairness, restrict the pool to examples with fewer than 2048 tokens, DETTA and INSTAG baselines rely external tagging models with this token limit. We compare our method (SsTauk) against: (1) UNIFORMg0K- which samples data uniformly across the dataset; (2) INSTAGaDE, which uses the instruction-Lagging method of Lu el al. (202);; (3) DELTAдoR: which follows Liu et al. (2024) with complexity and quality scoring, and embedding based diversity selection; (4) RHO-Lossaoe- which follows Mindermann al. (202)): (5) FULLjook. which uses the entire pool without selection. For SSTsok; we set To 3, Ps 0.3, 7 0.1, and kn 0.1 in all experiments, and compute
perplexity only on target tokens to align with the IFT setting. For INSTAGanE and DELTAGOR, we use the
11

---

Published in Transactions on Machine Learning Research (06/2025)
Table 2: Performance comparison of SsT30k with baseline methods across various model sizes and bench- marks. Results demonstrate that SST consistently outperforms baseline methods, including those using significantly more training data (FUL1100k), in most scenarios. Notably, SsT achieves superior performance larger models (c.g., Qwen2.5-32B) and challenging benchmarks (e.g., MMLC-PRO), while maintaining competitive results on smaller models and diverse tasks.
Method
Avg
MATH Lvl
IFEval
MMLU-PRO
GPQA
MUSR
BBH
Qwen2. 5-32B
FUL1-100k
39.58
34.67
73.9
48.03
16.33
16.52
48.06
UNIFORMaOk
40.84
35.8
74.02
49.22
19.35
19.14
47.51
INsTAGaon
38.21
35,42
70.01
46.51
17.11
15.43
11.74
DEITÀGOS
36.49
34.44
71.78
48.29
9.06
20.7
34.67
RHo-Lossaax
38.56
30.31
72.10
48.03
16.00
16.93
48.00
SSTaak (ours)
42.76
36.1
76.03
53.25
18.46
24.79
48.8G
G2-27h
FULL:100k
32.34
21.08
72.26
32.18
11.41
16.84
39.08 08
UNIFORMaok
19.86
70.16
28
12.08
15.24
37
INsTAcians
31.24
22.66
68.01
31.38
10.63
17.77
36.98
DErTÀgoÉ
32.07
23.11
71.93
35. 97
8.5
16. 78
3G 11
RHoo-Lossook
20.45
S8T3ox (ours)
32.89
20.92
70.28
34.28
12.64
19.66
39.55
G2-9b
FULL-1006
28.43
15.26
66.74
29.52
8,95
18.58
31.56
UNIFORMaok
28.85
11.93
65.36
30.8
9.96
21.56
33.47
INSTACook
28.58
12.39
66.97
29.14
17.59
DErTÀgOé
28. U8
14.27
67.58
28.
8.95
17.26
32.11
LosSoo
13.00
SSTsok (ours)
29.17
11.48
66.39
30.5
10.29
23.33
33.02
Llama3.
FULL100k
29
UNIFORMgoR
21.11
5.29
61.85
21.8
5.7
15.32
16.71
INSTAGaak
24.
14.31
25.2
DErLÀaOÉ
18.0
4.91
59.27
20.77
0.0
9.0
14.15
Ruo-Losssok
19.77
4.76
48.89
22.93
4.60
14.17
23.27
SSTa0s (ours)
22.45
4.53
59.28
23.84
4.25
14.63
28.19
Llama3.
FULLf.1008
14.88
46.93
15.58
UNIFORM3eR
15.52
1.89
49.95
14.33
3.91
9.03
13.99
INSTAGgok
13.08
1.81
43.43
13.24
1.23
9.16
9.62
DRITA
11.73
1.66
46.54
12.08
0.0
2.73
7.36
Ruo-Lossaak
10.73
2.00
39.39
9.13
2.68
3.17
8.03
SSTaok (ours)
16.33
1.59
16.26
3.58
16.2
Llamaa3.22 3-11
FULLiook
7.45
0.45
1.66
0.0
3.62
4.01
UNIFORM304
6.59
0.08
33.56
1.0
0.0
2.38
2.55
INSTAGaaR
6.8
0.23
31.61
1.52
0.0
3.17
4.3
DErLAAoR
6.46
0.38
29.51
1.22
2.93
4.76
Ruo-Losssak
5.74
0.60
27. 64
1.29
0.0
2.14
2.76
Taok (ours)
6.56
1.49
Qwen2 550.5B
FULL1100
7.32
0.91
07
5.88
0,22
1.27
4.56
UNIFORMaok
7.05
6.65
4.14
Ins LAGqok
6.94
1.21
28 88
6.37
0.45
1.11
3.66
DEITÀ
RHO- RHo-Lossaas
5.92
1.81
24.00
5.00
0.00
1.02
3.67
SSTsok (ours)
7.52
29.18
6.56
3.02
1.27
4.12
original scoring models from the respective works. For RIOO-LSSS30R, we adapt the original method to the IFT setting and set the number of training iterations to match the number of training steps in SSTyok We report all results in Table
12

---

Published in Transactions on Machine Learning Research (06/2025)
5.2 Results and Discussion
The results in Table 2 demonstrate the clear advantages of our proposed method over baseline approaches. Using only 30% of the training data, SST304 achieves significant performance improvements across different models. It surpasses other baselines, including the one trained on the full 100k dataset in five out of seven tested models and closely matches the performance on the remaining model (Llama3.2-1l compared to methods utilizing the same amount of data. These results highlight. Sst's effectiveness optimizing com- putational resources. The observation that SST, utilizing only data subset (e.g., SsT30k), can outperform baselines trained the full dataset (FULL1008) in several cases (Table 2 extends beyond merely filter- ing erroneous examples and relates to data efficiency. Even curated instruction mixtures like the TüLU 3 mixture (Lambert et al. 2024) may contain rodundancics or examples offering diminishing returns, making subsets potentially more compute-effective Indeed, Lambert et al. (2024) demonstrated that subsets (e.g.. 25-75% this mix often performed comparably to the full set within a fixed budget, echoing findings where smaller, highly curated sets proved highly effective (Zhou et al.) 2023a). Furthermore, SST's adaptive na- ture allows tailor data selection the target model's specific pro-training composition and evolving state, unlike static mixes which might oversample lopies already mastered during pre-training. SST may also implicitly filter examples that are overly complex relative to model's capabilities (e.g., complex reason- ing for smaller models) or out-of-distributin relative to pro-training or evaluation data (e.g., low-resource languages not present in benchmarks, ef. Table ), thereby focusing training on more learnable and rele- vant content. Our dynamic data scheduling approach ensures a balanced exposure to examples of varying complexity tailored to the target model, enabling robust generalization across diverse tasks and models. contrast, methods like DEITA3OK: which involve an elaborate data selection process INSTAG300> which tries to replicate state-of-the-art models filtering behavior, fail to adapt to the model's evolving needs during training. These methods produce inconsistent results across the different model sizes and families. The results indicate that the adaptive baseline (Ruo-LLssaox)) exhibits inconsistent. performance across different models and benchmarks compared to other methods. This general inconsistency aligns with previous find- ings regarding RHO-Loss (addour et al. 2023). Interestingly, on the Math Lvl benchmark specifically, it displays a noticeable trend of achieving higher scores on smaller models 1llama322-3B3 Llama3.2- 2-BBB and Qwen2.5-0.5B). Despite this specific strength, RHO-LoSSeae significantly out performmeed by the proposed by SST3OA on average across all models and benchmarks. is worth noting that while the RHO-Loss method adapted for the IFT sctting in this study, its obscrved inconsistency echoes the performance limitations reported by Kaddour et al. 2023 in the original context.
(a) Static Methods
(b) RHoLossae
(c) SeTaw
100
60
40
Fullowk
INSTacie
DISEAAE
0%
25%
30%
70%
100%
0%
26%5
30%
75%
100%
Noltobota
OpeaMethhhstro122
Alzebxa
Tulud Presoun
WilCChat Other
Dool CodeAppocs
NommaMath TIR
SaRIFF
Tabil Penoon Code
Tuhn3 Presoun MATII
FLAN22
Open/edistaut
Tibt Penona OSM
Figure 7: Comparison of data mixture composition for static and dynamic selection methods using Qwen2.5- 32B. The figure shows the percentage distribution of datasets within the training data mixture for different selection basclines. Other represents the datascts that account for less than 1% of the total data. (a) Illustrates the fixed composition for static methods. (b) and (c) Depict the evolving data mixture composition for the dynamic methods RHO-Tosssan and SsTasn, respectively, various training progress points (0%, 25%, 50%, 75%, and 100%).
13

---

Published in Transactions on Machine Learning Research (06/2025)
Figure 7 reveals distinct dynamic data selection patterns. RHO-Loss (7b) increasingly sacrifices diversity, becoming heavily dominated by MeteMMathnsstucci and Tulu3 Persona MATH towards the end; this skewed focus potentially links to its overall inconsistency despite its observed tendency for better Math Lvl 5 per- formance specifically on smaller models. In contrast, SST (7c) appears to maintain a broader representation of datasets throughout training. This sustained diversity likely contributes to SST's more robust and consis- tently superior performance (Figure Table ) contrasting with RHo-LLsss's inconsistency stemming from its lack of diversity, a crucial factor in IFT as shown in DETTA and INSTAG. Conversely; SsT's mid-training shift towards Aya and FLAN V2 coincides with the temporary performance dip seen in Figure I Crucially, SsT's subsequent recovery from this dip highlights its strong adaptability, demonstrating its ability to navi- gate transiently challenging data mixes and effectively refine selection away from the initial static distribution towards a more tailored mix, which likely underlies its sustained upward performance Irend lowards the end of training, reinforcing the SsT's argument that adaptive selection based on the model's state leads to more effective data compositions than static approaches.
Sst shows significant improvement using large models: In $ we showed that models with more than 20B parameters are less sensitive to static data selection methods. In contrast, our results show the efficacy of SST in leveraging smaller, carcfully scheduled data subsets to maximize performance even for larger models. Using Qwen2.5-32B, SST30* delivers an average performance of 42.75%, surpassing all baselines including the one using more than threefold the amount of data (FULL1102)) ). Notably, the method improves results on challenging tasks such as MMLU-PRO, where it achieves a score of 53.25%, outperforming the FULL bascline by 5.22%. The G2-27B model further illustrates the benefits of our method, where SST30k achieves an average performance of 32.89% outperforming DETTA and INSTAGg0R while maintaining competitive results against FULL1ooK- This trend persists across multiple tasks, with SST showing resilience and adaplability in both general benchmarks like GPQA and domain-specifie tasks such as MATH LvL 5. Using Llama3.1-8B SST30k outperforms all baselines with comparable data sizes but falls behind FULL100K- This gap may be due to the model's weaker instractionnfollooinng abilities, as seen in its lower IFEval benchmark scores. The earlier Llama3.1 likely lacked sufficient exposure to instruction-like examples during late-stage pre-training- a stralegy used in newer models like Tlama3.2 and Qwen2.5. To test this hypothesis, we increased the sample size from 30k to 50k examples. As shown in Table 3 SSTsow outperforms all methods, including FULL100k- This suggests that Llama3. benefits from additional data for optimal performance. While increasing the subset size improves results, an ideal approach would determine the optimal subset size rather than treating it as a fixed hyperparameter. We leave this exploration for future work.
Figure [ shows the performance of SS compared to other methods on Qwen2.5-32B throughout the training process across four random seeds (42, 123, 456, 789), using a multi-sedd evaluation strategy similar to Lambert et al. (2024). The figure highlights that the performance advantage of SSTY04 is not merely a final outcome but is maintained consistently throughout the training process compared to all baseline methods. Notably, SSTSok exhibits lower variance compared to the other methods, indicating more stable and reliable training dynamics. Furthermore, the distinct upward trend observed for SST308 in the later training stages suggests that its lead might increase further with extended training beyond the two epochs evaluated. Further examination of the data mixture compositions, as depicted in Figure 7, reveals distinct behaviors among the selection strategies. Static methods like DEITA exhibit a fixed, less diverse subset, dominated by a few datasets compared to the Full
In analyzing the Llama-3. results (Table - Table 3), we note that while Ssr significantly improves the average leaderboard score compared to the base model, its performance on specific benchmarks (GPQA, MMLU-PRO) is lower. This aligns with the known challenge of knowledge degradation or 'catastrophic forgetting' sometimes observed after SFT (Zheng et al., 2025). SFT may inadvertently overwrite pre-trained knowledge, particularly if the SFT dataset lacks sufficient explicit reinforcement of that knowledge. Supporting this, degradation on knowledge benchmarks (e.g., TruthfulQA, MMLU, GPQA) was observed when using larger SFT data fractions in Lambert et al. (2024). While SST exhibits this effect, similar degradation occurs with other sampling methods, and it may be pronounced here as our initial data pool did not specifically include tasks designed to reinforce this type of pre-trained knowledge.
14

---

Published in Transactions on Machine Learning Research (06/2025)
Table 3: Performance comparison of SST with the same baselines as in Table 2 on Llama3.1-81 using 50k
examples instead of 30k. SST outperforms all baselines, including FULLI0OR- This suggests that, for some models, the subset size is a critical hyperparameter for data selection.
Method
Avg
MATH Lvl
1FEvl
MMLU-PRO
GPQA
MUSR
BBH
FULL-1okk
23.59
5.44
62.41
23.39
4.81
15.97
29.53
UNiFORMaok
21.35
5.37
61.23
22.22
5.70
15.00
18.55
TNsT
22.08
5.00
57.90
24.12
4.41
14.96
26.10
DEITASOS
18.07
4.76
60.12
21.78
0.00
8,00
13.77
SST (ours)
23.85
5.37
62.71
24.82
5.64
14.97
29.60
Using intermediate tagger models can be detrimental: Both DEITA and INSTAG rely on tagger models to select data to replicate the behavior of much larger models (e.g., ChaiGPT). While this reduces evaluation costs, our results reveal significant performance drawbacks. For instance, DEITA30R and TAGTo lag behind the simple UNIFORMy0 baseline Qwen2.5-32B, because their tagger models struggle with highly complex examples (e-g. NutinaMath datasets) or sequences approaching the 2048-token limit-the the limit of all tagging models proposed by DEITA and INSTAG. This limitation is more noticeable for DEITA with Qwen2.5-32B, likely because it relies on two taggers and an embedding model as opposed to a single tagger used by INSTAG, which can further exacerbate the issue. On G2-27B, both methods show a comparable performance to other baselines, however, our results in show that this model is particularly less sensitive data selection methods, suggesting, in this setup, that the performance decreases with the complexity of the data selection approach, matching the findings Marion et. al. 2023) in the context of pre-training. This effect is also noticeable smaller models such Llama3.2-33. where UNIFORM30k outperforms INSTAG330 and DEITA by 2.44 and 3.79 points, respectively. Our analysis suggests that the data selected by DEITABOK and INSTAG3oR contains a significant number of complex examples for such models when contrasting their selection with perplexity- based categorization.
Table 4: FLAN V2 example in Malayalam selected by DEITA. While the example is a good quality, it may be detrimental for models not exposed Malayalam data during pre- sretrriining, where such an example falls within the high-peeplexity range. SST dynamically adjusts the selection window away from these examples.
Inst: You statement written Malayalamn (..) Output the word from option.
<ASK> noñee0d 1959 ancooloyymoen woo datrB ausinmaaag
Gar ajomanaam allammominomele eVvaasssooym (...)
A: arilnovol B: aieeveeos' C: aloychloeeaarr D: swnkmottag
Further, lagger models face other limitations, such their context window limitation, when the tagger training data distribution differs from the distribution of the data that needs to be selected. The latter is particularly problematic in our experiment when evaluating multi-lingual data, shown in Table In both cases, the lagger weaknesses introduce additional noise in the data selection process, leading to suboptimal performance. Solving the issues would require training newer tagging models, inducing a significant cost, which questions the practicality of such methods.
5.3 Limitations and Future Work
SST remains effective long as each dataset's perplexity distribution has nontrivial overlap with the selection window. However, if the data is extremely unbalanced, simpler approaches (e.g., per-dataset random sampling) may perform better. Observations in $ shoo that skewed perplexity distribution can lead to many noisy examples in the selection, and excluding them improved performance. Therefore, one could develop an SST variant that ignores these outliers or uses a different usefulness metric along with our modulation method. We did not explore such variations, leaving them for future rescarch. Although we focused on IFT, SsT could benefit pre-training as well. In this context, two challenges arise: (1) is unclear if our warm-up triggers suitable for pre-training, and (2) the overhead may be prohibitive with larger
15

---

Published in Transactions on Machine Learning Research (06/2025)
datasets. A potential solution is to pre-compute perplexities offine using external model-a standard approach in pre-training data cleaning (Penedo et. al.) 2023] -ndd then apply adaptive selection after a warm-up phase. Using 4-bit precision for perplexity evaluation could further reduce scoring costs, but its impact on selection quality in pro- rro-training remains unclear. Nonethcless, the results of Marion et al. (2023) in pre-training, alongside our own findings IFT, suggest that SST may offer significant gains in pre-training as well. We leave an in-depth investigation of these trade-offs to future work. In $ 4.2 we assumed Cmisc R 0 since we found this term negligible in our single-node experiments by syncing the gathering of perplexity values with gradient updates, to avoid additional inter-GPU synchronization. However, communication and synchronization overhead can grow significantly especially if high-speed interconnects like InfiniBand are not available or if the implementation does not take into account these factors.
6 Conclusion
Our proposed Spaced Scheduled Training (SST) framework provides an adaptive, efficient, and model-specific approach to data selection, eliminating reliance on external oracle models. We demonstrate that continuously adjusting the dataset composition based on real-time perplexity signals improves performance. Through extensive evaluations on seven LLMs (0.5B-32B) and thcoretical overhead analysis grounded in scaling laws, SST consistently achieves performance gains across architectures and scales efficiently to large training regimes. These results offer robust. empirical evidence and practical insights for enhancing data quality in LLM instruction fine-tuning.
Broader Impact Statement
Data selection methods like Spaced Scheduled Training (SST) enhance Large Language Model (LLM) per- formance and efficiency. However, any data sampling method, including SST, carries the risk of introdue- ing or amplifying biases and ethical concerns (Mchrabi et al., 2022: Wang et al. 2025 Guo et al.) 2024). LLMs are trained on extensive text datasets that often reflect societal biases (e.g., gender, race). Sampling methods, even those based on model confidence, may inadvertently favor or reinforce patterns from domi- nant groups in the data, potentially amplifying these biases (Wang et al.) 2025). This can result in unfair outcomes, such as marginalizing perspectives in content generation or producing discriminatory results in decision-making systems. Defining and measuring fairness and bias remains challenging (Wang et al., 2025 particularly since bias is often deeply embedded in training data (Guo et al.) 2024). Benchmarks like Helm Safety (Liang et al. 2023) can identify some issues but have limitations. Model performance is sensitive to benchmark design choices (e.g,, prompt format, example order) (Grattafiori et al. 2024b), and evaluation data may be contaminated by pre-training data, potentially inflating scores and complicating accurate bias assessment (Graltafiori et al. 2024bb. Effectively analyzing data contamination remains an open research challenge (Grattafiori et al.) 2024b). In practice, LLMs are components of larger systems. Practitioners should implement system-level safety measures alongside existing benchmarks, often using external models to supplement model-level safeguards like Llama Guard (Grattafiori el al.] 2024b). We acknowledge the importance of conducting a dedicated, large-scale analysis of the fairness implications and potential biases introduced or mitigated by perplexity-based sampling. We leave this exploration for future work.
References
Edward Beeching, Clémentine Fourrier, Nathan Habib, Sheon Han, Nathan Lambert, Nazneen Rajani, Omar Sanseviero, Lewis Tunstall, and Thomas Wolf. Open Ilm leaderboard (2023-2024). htttss//huggingface co/spacces/ppen 11m leaderboard old. open_ 11m 1ll_lllmmmeeeeeeeeee leaderboard 2023.
Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston. Curriculum learning. In Proceedings of the 26th annual international conference on machine learning, pp. 41 48, 2009.
Lichang Chen, Shiyang Li, Jun Yan, Hai Wang, Kalpa Gunaratna, Vikas Yadav, Zheng Tang, Vi- jay Srinivasan, Tianyi Zhou, Heng Huang, and Hongxia Jin. Alpagasus: Training a better alpaca with fewer data. In The Twelfth International Conference on Learning Representations, 2024. URL
16

---

Published in Transactions on Machine Learning Research (06/2025)
Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ton Stoica, and Fric P. Xing. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, March 2023. URL https://1msys. orghb///..g///og//
3023-03-30-vicuna)
Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuczhi Wang, Mostafa Dehghani, Siddhartha Brahma, Albert Webson, Shixiang Shane Gu, Zhuyun Dai, Mirac Suzgun, Xinyun Chen, Aakanksha Chowdhery, Alex Castro-Ros, Marie Pellat, Kevin Robinson, Dasha Valter, Sharan Narang, Gaurav Mishra, Adams Yu, Vincent Zhao, Yanping Huang, Andrew Dai, Hongkun Yu, Slav Petrov, Ed H. Chi, Jeff Dean, Jacob Devlin, Adam Roberts, Denny Zhou, Quoc V. Le, and Jason language models, 2022. URL httph:htttsaaaa
Wei. Scaling
.rg. abt// 11416
Ganqu Cui, Lifan Yuan, Ning Ding, Guanming Yao, Wei Zhu, Yuan Ni, Guotong Xie, Zhiyuan Liu, and Maosong Sun. Ultrafeedback: Boosting language models with high-quality feedback, 2024. URL https:
Tri Dao. FlastAntet:ion-2:: Faster attention with better parallelism and work partitioning. In International Conference on Learning Representations (ICLR), 2024.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv preprint wrXiv:2407.21780 2024.
Clémentine Fourrier, Nathan Habib, Alina Lozovskaya, Konrad Szafer, and Thomas Wolf. Open Ilm leader- board v2. htttps:/huggingfaaee
11m leaderboard 2024.
Leo Gao, Jonathan Tow, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Kyle McDonell, Niklas Muennighoff, Jason Phang, Laria Reynolds, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. A framework for few-shot language model evaluation, September 2021. URL httthhtts:///o/ org 10. 5281/zenodo. ..o 5371628
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al- Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zhang, Aurelien Rodriguez, Austen Gregerson, Ava Spataru, Baptiste Roziere, Bethany Biron, Binh Tang, Bobbic Chern, Charlotte Caucheteux, Chaya Nayak, Chloc Bi, Chris Marra, Chris McConnell, Christian Keller, Christophe Touret, Chunyang Wu, Corinne Wong, Cristian Canton Ferrer, Cyrus Nikolaidis, Damien Allonsius, Daniel Song, Danielle Pintz, Danny Livshits, Danny Wyatt, David Esiobu, Dhruv Choudhary, Dhruv Mahajan, Diego Garcia-Olano, Diego Perino, Dieuwke Hupkes, Egor Lakomkin, Ehab AlBadawy, Elina Lobanova, Emily Dinan, Eric Michael Smith, Filip Radenovic, Francisco Guzmán, Frank Zhang, Gabriel Synnaeve, Gabrielle Lee, Georgia Lewis Anderson, Govind Thattai, Graeme Nail, Gregoire Mialon, Guan Pang, Guillem Cucurell, Hailey Nguyen, Hannah Korevaar, Hu Xu, Hugo Touvron, Ilyan Zarov, Imanol Arricta Ibarra, Isabel Kloumann, Ishan Misra, Ivan Evti- mov, Jack Zhang, Jade Copet, Jaewon Lee, Jan Geffert, Jana Vranes, Jason Park, Jay Mahadeokar, Jeet Shah, Jelmer van der Linde, Jennifer Billock, Jemny Hong, Jenya Lee, Jeremy Fu, Jianfeng Chi, Jianyu Huang, Jiawen Liu, Jie Wang, Jiecao Yu, Joanna Bitton, Joe Spisak, Jongsoo Park, Joseph Rocca, Joshua Johnstun, Joshua Saxe, Junteng Jia, Kalyan Vasuden Alwala, Karthik Prasad, Kartikeya Upasani, Kate Plawiak, Ke Li, Kenneth Heafield, Kevin Stone, Khalid El-Arini, Krithika Tyer, Kshitiz Malik, Kuenley Chiu, Kunal Bhalla, Kushal Lakhotia, Lauren Rantala- Rantala-Yeary, Laurens van der Maaten, Lawrence Chen, Liang Tan, Liz Jenkins, Louis Martin, Lovish Madaan, Lubo Malo, Lukas Blecher, Lukas Landzaat, Luke de Oliveira, Madeline Muzzi, Mahesh Pasupuleti, Mannat Singh, Manohar Paluri, Marcin Kardas, Maria Tsimpoukelli, Mathew Oldham, Mathicu Rita, Maya Pavlova, Mclanie Kambadur, Mike Lewis, Min Si, Mitesh Kumar Singh, Mona Hassan, Naman Goyal, Narjes Torabi, Nikolay Bashlykov, Nikolay Bogoychev, Niladri Chatterji, Ning Zhang, Olivier Duchenne, Onur Çelebi, Patrick Alrassy, Pengchuan Zhang, Peng- wei Li, Petar Vasic, Peter Weng, Prajjwal Bhargava, Pratik Dubal, Praveen Krishnan, Punit Singh Koura, Puxin Xu, Qing He, Qingxiao Dong, Ragavan Srinivasan, Raj Ganapathy, Ramon Calderer, Ricardo Sil- veira Cabral, Robert Stojnic, Roberta Raileanu, Rohan Maheswari, Rohit Girdhar, Rohit Patel, Romain
17

---

Published in Transactions on Machine Learning Research (06/2025)
Sauvestre, Ronnie Polidoro, Roshan Sumbaly, Ross Taylor, Ruan Silva, Rui Hou, Rui Wang, Saghar Hos- seini, Sahana Chennabusappa, Sanjay Singh, Sean Bell, Seohyun Sonia. Kim, Sergey Edunov, Shaoliang Nie, Sharan Narang, Sharath Raparthy, Sheng Shen, Shengye Wan, Shruti Bhosale, Shun Zhang, Simon Vandenhende, Soumya Batra, Spencer Whitman, Sten Sootla, Stephane Collot, Suchin Gururangan, Syd- ney Borodinsky, Tamar Herman, Tara Fowler, Tarek Sheasha, Thomas Georgiou, Thomas Scialom, Tobias Speckbacher, Todor Mihaylov, Tong Xiao, Uijwal Karn, Vedanuj Goswami, Vibhor Gupta, Vignesh Ra- manathan, Viktor Kerkez, Vincent Gonguet, Virginie Do, Vish Vogeti, Vitor Albiero, Vladan Petrovic, Weiwei Chu, Wenhan Xiong, Wenyin Fu, Whitney Meers, Xavier Martinet, Xiaodong Wang, Xiaofang Wang, Xiaoqing Ellen Tan, Xide Xia, Xinfeng Xie, Xuchao Jia, Xuewei Wang, Yaelle Goldschlag, Yashesh Gaur, Yasmine Babaei, Yi Wen, Yiwen Song, Yuchen Zhang, Yue Li, Yuning Mao, Zacharie Delpierre Coudert, Zheng Yan, Zhengxing Chen, Zoe Papakipos, Aaditya Singh, Aayushi Srivastava, Abha Jain, Adam Kelsey, Adam Shajufeld, Adithya Gangidi, Adolfo Victoria, Ahuva Goldstand, Ajay Menon, Ajay Sharma, Alex Bocsenberg, Alexei Bacvski, Allic Feinstein, Amanda. Kallet, Amit Sangani, Amos Too, Anam Yunus, Andrei Lupu, Andres Alvarado, Andrew Caples, Andrew Gu, Andrew Ho, Andrew Poulton, Andrew Ryan, Ankit Ramchandani, Annie Dong, Annie Franco, Amuj Goyal, Aparajita Saraf, Arka- bandhu Chowdhury, Ashley Gabriel, Ashwin Bharambe, Assaf Eisenman, Azadeh Yazdan, Beau James, Ben Maurer, Benjamin Leonhardi, Bernie Huang, Beth Loyd, Beto De Paola, Bhargavi Paranjape, Bing Liu, Bo Wu, Boyu Ni, Braden Hancock, Bram Wasti, Brandon Spence, Brani Stojkovic, Brian Gamido, Britt Montalvo, Carl Parker, Carly Burton, Catalina Mejia, Ce Liu, Changhan Wang, Changkyu Kim, Chao Zhou, Chester Hu, Ching-Hsiang Chu, Chris Cai, Chris Tindal, Christoph Feichtenhofer, Cyuthia Gao, Damon Civin, Dana Beaty, Daniel Kreymer, Daniel Li, David Adkins, David Xu, Davide Testuggine, Delia David, Devi Parikh, Diana Liskovich, Didem Foss, Dingkang Wang, Duc Le, Dustin Holland, Ed- ward Dowling, Eissa Jamil, Elaine Montgomery, Eleonora Presani, Emily Hahn, Emily Wood, Erie-Tuan Le, Erik Brinkman, Esteban Arcaute, Evan Dunbar, Evan Smothers, Fei Sun, Felix Kreuk, Feng Tian, Filippos Kokkinos, Firat Ozgenel, Francesco Caggioni, Frank Kanayet, Frank Seide, Gabriela Medina Flo- rez, Gabriclla Schwarz, Gada Badcer, Georgia Swee, Gil Halpern, Grant Herman, Grigory Sizov, Guangyi, Zhang, Guna Lakshminarayanan, Hakan Inan, Hamid Shojanazeri, Han Zou, Hannah Wang, Hanwen Zha, Haroun Habeeb, Harrison Rudolph, Helen Suk, Henry Aspegren, Hunter Goldman, Hongyuan Zhan, Tbrahim Damlaj, Igor Molybog, Igor Tufanov, Ihis Lcontiadis, Irina-Elena Veliche, Itai Gat, Jake Weiss- man, James Geboski, James Kohli, Janice Lam, Japhet Asher, Jean- Baatiste Gaya, Jeff Marcus, Jeff Tang, Jennifer Chan, Jenny Zhen, Jeremy Reizenstein, Jeremy Teboul, Jessica Zhong, Jian Jin, Jingyi Yang, Joe Cummings, Jon Carvill, Jon Shepard, Jonathan McPhie, Jonathan Torres, Josh Ginsburg, Junjie Wang, Kai Wu, Kam Hon U, Karan Saxena, Kartikay Khandelwal, Katayoun Zand, Kathy Matosich, Kaushik Veeraraghavan, Kelly Michelena, Keqian Li, Kiran Jagadeesh, Kun Huang, Kunal Chawla, Kyle Huang, Lailin Chen, Lakshya Garg, Lavender A, Leandro Silva, Loe Bell, Lei Zhang, Liangpeng Guo, Licheng Yu, Liron Moshkovich, Luca Wehrstedt, Madian Khabsa, Manav Avalani, Manish Bhatt, Martynas Mankus, Matan Hasson, Matthew Lennie, Matthias Reso, Maxim Groshev, Maxim Naumov, Maya Lathi, Meghan Keneally, Miao Liu, Michael L. Seltzer, Michal Valko, Michelle Restrepo, Mihir Patel, Mik Vyatskov, Mikayel Samvelyan, Mike Clark, Mike Maccy, Mike Wang, Miquel Jubert Hermoso, Mo Mctanat, Moham- mad Rastegari, Munish Bansal, Nandhini Santhanam, Natascha Parks, Natasha White, Navyata Bawa, Nayan Singhal, Nick Egebo, Nicolas Usunier, Nikhil Mehta, Nikolay Pavlovich Laptev, Ning Dong, Nor- man Cheng, Oleg Chernoguz, Olivia Hart, Omkar Salpekar, Ozlem Kalinli, Parkin Kent, Parth Parekh, Paul Saab, Pavan Balaji, Pedro Rittner, Philip Bontrager, Pierre Roux, Piotr Dollar, Polina Zvyag- ina, Prashant Ratanchandani, Pritish Yuvraj, Qian Liang, Rachad Alao, Rachel Rodriguez, Rafi Ayub, Raghotham Murthy, Raghu Nayani, Rahul Mitra, Rangaprabhu Parthasarathy, Raymond Li, Rebekkah Hogan, Robin Battey, Rocky Wang, Russ Howes, Ruty Rinott, Sachin Mehta, Sachin Siby, Sai Jayesh Bondu, Samyak Datta, Sara Chugh, Sara Hunt, Sargun Dhillon, Sasha Sidorov, Satadru Pan, Saurabh Mahajan, Saurabh Verma, Seiji Yamamoto, Sharadh Ramaswamy, Shaun Lindsay, Shaun Lindsay, Sheng Feng, Shenghao Lin, Shengxin Cindy Zha, Shishir Patil, Shiva Shankar, Shuqiang Zhang, Shuqiang Zhang, Sinong Wang. Sneha Agarwal, Soji Sajuyigbe, Soumith Chintala, Stephanie Max, Stephen Chen, Steve Kehoe, Steve Satterfield, Sudarshan Govindaprasad, Sumit Gupta, Summer Deng, Sungmin Cho, Sunny Virk, Suraj Subramanian, Sy Choudhury, Sydney Goldman, Tal Remez, Tamar Glaser, Tamara Best, Thilo Kochler, Thomas Robinson, Tianhe Li, Tianjun Zhang, Tim Matthews, Timothy Chou, Tzook Shaked, Varun Vontimitta, Victoria Ajayi, Victoria Montanez, Vijai Mohan, Vinay Satish Kumar, Vishal Mangla,
18

---

Published in Transactions on Machine Learning Research (06/2025)
Vlad Ionescu, Vlad Poenaru, Vlad Tiberiu Mihailescu, Vladimir Ivanov, Wei Li, Wenchen Wang, Wenwen Jiang, Wes Bouaziz, Will Constable, Xiaocheng Tang, Xiaojian Wu, Xiaolan Wang, Xilun Wu, Xinbo Gao, Yaniv Kleinman, Yanjun Chen, Ye Hu, Ye Jia, Ye Qi, Yenda Li, Yilin Zhang, Ying Zhang, Yossi Adi, Youngjin Nam, Yu, Wang, Yu Zhao, Yuchen Hao, Yundi Qian, Yunlu Li, Yuzi He, Zach Rait, Zachary DeVito, Zef Rosnbrick, Zhaoduo Wen, Zhenyu Yang, Zhiwei Zhao, and Zhiyu Ma. The llama 3 herd of models, 2024a. URL https://arxiv.. wwwrriii....ovieeooo httg//bs//400.... 2407 21783
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al- Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llaaa 3 herd of models. arXiv preprint arXiv:2407.21788 2024b.
Yufci Guo, Muzhe Guo, Juntao Su, Zhou Yang. Mengqiu Zhu, Hongfei Li, Mengyang Qiu, and Shuo Shuo Liu. Bias in large language models: Origin, evaluation, and miligation, 2024. URL httpss//arxiv.. org/
abs/2411 10915
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Mcasuring mathematical problem solving with the math dataset, 2021. URL https:
7/arxiv. org. wwtggabs//10330055333
Angela H Jiang, Daniel L.-K Wong, Giulio Zhou, David G Andersen, Jeffrey Dean, Gregory R. Ganger, Gauri Joshi, Michael Kaminksy, Michael Kozuch, Zachary C Lipton, et al. Accelerating deep learning by focusing on the biggest losers. arXiv preprint Xiv: 00762, 2019.
Jean Kaddour, Oscar Key, Piotr Nawrot, Pasquale Minervini, and Matt Kusner, No train gain: Revisiting efficient training algorithms for transformer-bass language models. In Thirty-sewenth Conference on Newral Information Processing Systems, 2023. URL https://openreview..r wwt.://oforum//n-hbbb.guugll
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXic preprint arXiv:2001.08361 2020.
Andreas Köpf, Yannic Kilcher, Dimitri von Rütte, Sotiris Anagnostidis, Zhi Rui Tam, Keith Stevens, Abdul- lah Barhoum, Due Minh Nguyen, Oliver Stanley, Richárd Nagyfi, Shahul Es, Sameer Suri, David Alexan- drovich Glushkov, Arnay Varma Dantuluri, Andrew Maguire, Christoph Schuhmann, Huu Nguyen, and Alexander Julian Mattick. OpenAssistant Conversations Democratizing Large Language Model Align- ment. In Thérty-Seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track, November 2023.
Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonza- lez, Hao Zhang, and lon Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the 29th Symposium on Operating Systems Principles, pp. 611 626, 2023.
Nathan Lambert, Jacob Morrison, Valentina Pyatkin, Shengyi Huang, Hamish Ivison, Faeze Brahman, Lester James V. Miranda, Alisa Liu, Nouha Dziri, Shane Lyu, Yuling Gu, Saumya Malik, Victoria Graf, Jena D. Hwang, Jiangjiang Yang, Ronan Le Bras, Oyvind Tafford, Chris Wilhelm, Luca Soldaini, Noah A. Smith, Yizhong Wang, Pradeep Dasigi, and Hannaneh Hajishirzi. Tülu 3: Pushing frontiers in open language model post-training. 2024.
Jia LI, Edward Beeching, Lewis Tunstall, Ben Lipkin, Roman Soletskyi, Shengyi Costa Huang, Kashif Rasul, Longhui Yu, Albert Jiang, Ziju Shen, Zihan Qin, Bin Dong, Li Zhou, Yann Fleureau, Guillaume Lample, and Stanislas Polu. Numinamath dataset and report. /ttps://gitee httpuub..com/pojcct-nmmaa.. datasaadattttttttaea pdf 2024.
aimo httpr/rrrre/e/ooooa//a//rrrrrrrr
Peng Li, Yeye He, Dror Yashar, Weiwei Cui, Song Gc, Haidong Zhang, Danielle Rifinski Fainman, Dongmei Zhang, and Surajit Chaudhuri. Table-GPT: Table-tuned GPT for Diverse Table Tasks, October 2023.
19

---

Published in Transactions on Machine Learning Research (06/2025)
Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Kumar, Benjamin Newman, Binhang Yuan, Bobby Yan, Ce Zhang, Christian Alexander Cosgrove, Christopher D Manning, Christopher Re, Diana Acosta-Navas Navas, Drew Arad Hudson, Eric Zelikman, Esin Durmus, Faisal Ladhak, Frieda Rong, Hongyu Ren, Huaxiu Yao, Jue WANG, Keshav Santhanam, Laurel Orr, Lucia Zheng, Mert Yuksekgonul, Mirac Suzgun, Nathan Kim, Neel Guha, Niladri S. Chatterji, Omar Khattab, Peter Henderson, Qian Huang, Ryan Andrew Chi, Sang Michael Xie, Shibani Santurkar, Surya Ganguli, Tatsunori Hashimoto, Thomas Icard, Tianyi Zhang, Vishrav Chaudhary, William Wang, Xuechen Li, Yifan Mai, Yuhui Zhang, and Yuta Koreeda. Holistic evaluation of language models. Transactions on Machine Learning Research, 2023. ISSN 2835-8856. URL Featured Certification, Expert Certification.
Wei Liu, Weihao Zeng, Keqing He, Yong Jiang, and Junxian He. What makes good data. for alignment? comprehensive study of automatic data selection in instruction tuning. In The Twelfth International Conference on Learning Representations, 2024. URL https:// wwwt/eeuummeeeeeee het//forumm1d0BTAAell.0
Shayne Longpre, Le Hou, Tu Vu, Albert Webson, Hyung Won Chung, Yi Tay, Denny Zhou, Quoc V. Le, Barret Zoph, Jason Wei, and Adam Roberts. The flan collection: Designing data and methods for effective instruction tuning, 2023.
llya Loshchilov and Frank Hutter. Online batch selection for faster training of neural networks. arXiv preprint drXic:1511.06343 2015.
Keming Lu, Hongyi Yuan, Zheng Yuan, Runji Lin, Junyang Lin, Chuanqi Tan, Chang Zhou, and Jingren Zhou. #InsTag: Instruction Tagging for Analyzing Supervised Fino-tuming of Large Language Models. 2023. URL
Ziyang Luo, Can Xu, Pu Zhao, Qingfeng Sun, Xiubo Geng, Wenxiang Hu, Chongyang Tao, Jing Ma, Qingwei Lin, and Daxin Jiang. Wizardcoder: Empowering code large language models with evol-instruct, 2023.
Max Marion, Ahmet Üstün, Luiza Pozzobon, Alex Wang, Marzich Fadace, and Sara Hooker. When less is more: Investigating data pruning for pretraining 1lms at scale. arXiv preprint Sft::2399.04666 2023.
Ninareh Mehrabi, Fred Morstatter, Nripsuta Saxena, Kristina Lerman, and Aram Galstyan. A survey on bias and fairness in machine learning, 2022. httpss//////x httpg//abb//1000665555
Sören Mindermann, Muhammed Razzak, Mrinank Sharma, Jan M. Brauner, Winnie Xu, Andreas Kirsch, Aidan Gomez, Benedikt Höltgen, Sebastian Farquhar, and Yarin Gal. Prioritized training on points that are learnable, worth learning, and not yet learned, 2022. URL httpa////ppereeeeeeeeeefrrruuuue YOcGpgUhSvp
Guilherme Penedo, Quentin Malartic, Daniel Hesslow, Ruxandra Cojocaru, Alessandro Cappelli, Hamza Alobeidli, Baptiste Pannier, Ebtesam Almazrouei, and Julien Launay. The refinedweb dataset for falcon I1:: outperforming curated corpora with web data, and web data only. arXiv preprint 011100 2023.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lec, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21 (1440):1166 67, 2020.
Nazneen Rajani, Lewis Tunstall, Edward Beeching, Nathan Lambert, Alexander M. Rush, and Thomas Wolf. 2023.
No robots. hatp:://hhugginnaaacccccoaaatats
Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. Zero: Memory optimization towards training A trillion parameter models. CoRR, abs/ abs/1910.02054, 2019. URL http:://aaaw.ooo///aaaaaaa org/abs/1910.
02054
David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Potty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R. Bowman. Gpqa: A graduato-level google-proof q&aa benchmark, 2023.
URL httt:::/tt:::ttttte www.aawwwww...raaaa.aaaa.a 12022
20

---

Published in Transactions on Machine Learning Research (06/2025)
Noveen Sachdeva, Benjamin Coleman, Wang- Cheng Kang, Jianmo Ni, Lichan Hong, Ed H. Chi, James Caverlee, Julian McAuley, and Derek Zhiyuan Cheng. How to train data-efficient 1ms,, 2024. URL
Keita Saito, Akifumi Wachi, Koki Wataoka, and Youhci Akimoto. Verbosity bias in preference labeling by large language models. Xiv preprind urXiv:2310.1000 2023.
Shivalika Singh, Freddie Vargus, Daniel Dsouza, Börje F. Karlsson, Abinaya Mahendiran, Wei- Yin Ko, Herumb Shandilya, Jay Patel, Deividas Mataciunas, Laura OMahony, Mike Zhang, Ramith Hettiarachchi, Joseph Wilson, Marina Machado, Luisa Souza. Moura, Dominik Krzemifiski, Hakimeh Fadaci, Irem Ergün, Ifeoma Okoh, Aisha Alaagib, Oshan Mudannayake, Zaid Alyafeai, Vu Minh Chien, Sebastian Ruder, Surya Guthikonda, Emad A. Alghamdi, Sebastian Gehrmann, Niklas Muennighoff, Max Bartolo, Julia Kreutzer, Ahmet Üstün, Marzieh Fadaee, and Sara Hooker. Aya dataset: An open-access collection for multilingual instruction tuning, 2024.
Ben Sorscher, Robert Geirhos, Shashank Shekhar, Surya Ganguli, and Ari Morcos. Beyond neural scaling laws: beating power law scaling via data pruning. Advances in Neural Information Processing Systems, 35:19523-19536, 2022.
Zayne Sprague, Xi Ye, Kaj Bostrom, Swarat Chaudhuri, and Greg Durrett. Musr: Testing the limits of chain-of-thoughl with multistep soft reasoning, 2024. URL
Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc V. Le, Ed H. Chi, Denny Zhou, and Jason Wei. Challenging big-bench tasks and whether chain-of-thought can solve them, 2022. URL httph:t///aatt
wtrg/ab//2110.0000
Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivière, Mihir Sanjay Kale, Juliette Love, et al. Gemma: Open models based on gemini rescarch and technology. arXiv preprint or arXiv:2403.00 008.09955 2024.
08295,
Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, Tianle Li, Max Kn, Kai Wang, Alex Zhuang, Rongqi Fan, Xiang Yue, and Wenhu Chen. Mmlu-pro: A more robust and challenging multi-task language understanding benchmark, 2024. URL https: //atxi: www.aariivvoorra/////a.a org/abs/2406. 2tabb//2406005574
Ze Wang, Zekun Wu, Jeremy Zhang, Xin Guan, Navya Jain, Skylar Lu, Saloni Gupta, and Adriano Koshiyama. Bias amplification: Large language models as increasingly biased media, 2025. URL
bttps://arxiv..rgg// h4w.aab//a222aaee 15234
Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Feng, Chongyang Tao, and Daxin Jiang. WizardIm: Empowering large language models to follow complex instructions. arXiv preprint arXiv:230,.12244 2023.
An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang. Le Yu, Mci Li, Mingfeng Xue, Pei Zhang, Qin Zhu, Rui Men, Runji Lin, Tianhao Li, Tingyu Xia, Xingzhang Ren, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yu Wan, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, and Zihan Qiu. Qwen2.5 technical report. Xiee preprint or Xiv:2 12. 15115, 2024.
Wenting Zhao, Xiang Ren, Jack Hessel, Claire Cardic, Yejin Choi, and Yuntian Deng, Wildchat: 1m chat interaction logs in the wild. In The Twelfth International Conference on Learning Representations, 2024. URL httpps://operrevieer wttlleeweeeeeeeeee
Yingxiu Zhao, Bowen Yu, Binyuan Hui, Haiyang Yu, Fei Huang, Yongbin Li, and Nevin Zhang, A preliminary study of the intrinsic relationship between complexity and alignment. arXiv preprint w:Xim:338.056000 2023.
21

---

Published in Transactions on Machine Learning Research (06/2025)
Junhao Zheng, Xidi Cai, Shengjie Qiu, and Qianli Ma. Spurious forgetting in continual learning of language models. In The Thirleenth International Conference on Learning Representations, 2025. URL https: //openreview.net/// httt:/ooruummssscr/111..
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena, arxiv hus/2306.05685 (2023). URL: htlps://upi. semanticscholar. prg/CorpusDD, 259129398, 2023.
Chunting Zhou, Pengfei Liu, Puxin Xu, Srini Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, Lili Yu, et al. Lima: Less is more for alignment. Xiv preprint arXiv:2305. 1 12006, 2023a.
Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou, and Le Hou. Instruction-followte evaluation for large language models, 2023b. URL https : http///axiv..rr///// org/ abs //1///21110991111
22

---

Published in Transactions on Machine Learning Research (06/2025)
A Experimental Setup Details
Training Data Mixture: We use a stratified subsample of 100k examples from the recent Tulu 3 SFT
Mix Lambert et al. 2024) containing 15 datasets actoss diverse tasks and domains. These include FLAN
v2 (Longrre et al. 2023), No Robots (Rajani et al. 2023), OpenAssistant (Köpf et al.) 2023). Tulu 3 Persona
MATH, Tulu 3 Persona GSM, Tulu 3 Persona Python, Tulu 3 Persona Algebra, Tulu 3 Persona IF (Lambert
et al. 2024). NuminaMath-
(LI et al., 2024). Aya (Singh et al.) 2024), WildChat GPT-4 (Zhao et al, 2024), TableGPT (Li et al., 2023), SciRIFF (Köpf et al. 2023), Evol CodeAlpaca (Luo et al.] 2023). Using 10% of the full mixture allows us to perform rigorous experimentation across multiple models while requiring
reasonable amount of compute, allowing for reproducibility by future research. Further, (Lamberl el. al.
2024 showed minimal average performance drop even with a 5% subset of the Tulu 3 mixture.
Training and Evaluation Setup: We use the same training setup and code base proposed by (Lambert
et al.
2024). We perform full parameter training for two epochs with an effective batch size of 128, a learning (LR) of 5e - 06 using a linear LR scheduler with 3% warm-up ratio. We set the maximum sequence length to 2048. All models were trained on 8 NVIDIA H100 GPUs using FlashAttention (Dao 2024 and
rate
DeepSpeed Stage 3 (Rajbhandari et al., 2019) For models larger than 20B we use the 32bit paged Adam optimizer. We provide the full training sctup our public repository We evaluate our models using the Open LLM Leaderboard 2 (Fourrier et al. 2024) tasks. It addresses performance saturation issues from the earlier version (Beeching et al. 2023) by introducing harder and less contaminated benchmarks. This update enables more meaningful evaluation result, particularly for recent LLMs which is crucial for our experiment. It includes IFEval (Zhou et al., 2023b), BBH (Suzgun et al., 2022], MATH LvL 5 (Hendrycks) et al. 2021). GPQA (Rein et al. 2023),, MuSR (Sprague et al. 2024), and MML(-PRO (Wang el. al., 2024). We ensure reproducibility by using the same average normalized scores across all benchmarks as in (Fourrier ct al. 2024)).
malluatiionhaarnes (Gao et al.
2021 version and report the
Methods and Baselines: Similar to Marion et al. 2023 we train models on 10,, 30,, and 50% of the data from the bottom, middle, and top segments of the per-example perplexity distribution. For instance, to create a 10% middle subsample, we select examples between the 45th and 55th percentiles. For each subset size, we create two datasets one using the overall perplexity distribution of the mixture and the other using the per-dataset perplexity distribution. To select data, use either the pre-trained version of the target model as reference, or checkpoints of the samn model trained on same subset al different train iteration. Specifically, we select data using checkpoints at 0.25, 0.5, 1, and 2 epochs. When using the pre-trained model as reference, we use a simple chat template proposed by (Lambert et al. 2024) to encode the chat data, where we prepend each turn content with its role (e.g., "User: ") and separate the turn with a new line. This allows us to avoid adding special tokens to the tokenizer which will require resizing the model's embedding layer. We compare the methods to various baselines where we train the same pre-trained model on: (1) the full data mixture (100%), (2) a random subset of the same size (Random), (3) and random subset drawn uniformly for each dataset (Uniform). For the random and uniform baselines, show the average performance across the two random sceds (123, and 42) and report the standard error.
To better study the impact of the perplexity-base data selection across different model charac- teristics, and to ensure that our findings are generalizable, we use a diverse set of models. Specifically, we use different model families including Llama3.1 (L3.1), Llama3.2 (L3.2), Qwen2.5 (Q2.5), and Gemma2 (G2). This allows us to study how pre-training factors, such as the pre-training data composition and training setting affect the data selection performance. Further, we compare our models a cross different models sizes ranging from 0.5B to 32B parameters, to study the influence of model size. In total, we trained seven models each with 31 different training configuratons, resulting in 248 training runs for this experiment.
Models:
4 Available after double-blind review.
23

---

Published in Transactions on Machine Learning Research (06/2025)
B
Method Details
Algorithm 2 Spaced Scheduled Training (SST)
Require: 0: initial model parameters; D = (Di...., Dat: collection of d datasets; Amax: maximum training iterations; kw E (0. 1|:
warm-up slope window ratio; Tw: maximum warm-up slope evaluation retry count. Pa (0, global subeet ratio; 7 (0, window-shift ratio; > O: slope threshold for loss slope evaluation;
Updated model parameters
1: Algorithm: Ssr(6, D, bmaxi Par Ta Pirs Rw)
2: Warm-up phase
3: f WARMUUP(8, D, fmax: Ter
4: Adaptive Schoduling
5: k -uu/fmax
+ COxPUTEPPL((s,.
Compute per-example perplexities PPL(e) for each example e 'D
7: Compute 8a for each datasct Dy using Equation
8: past + 50 for each dataset Da
Initialize window reference to the 50th percentile
9: D' SBRUECTPRRDATARRTTSSSSSSS
Pa. (Bu), (P)r))
10: Initialize loss buffer Lbut + (capacity [A )
Main training loop
12: for fu + to fmax do L, P TRAINSTEP ((.AAMPLBAACC((D)))
13:
Train model batch, and update perplexities
Append Ln Lbut
15: if [barl = [k [k-fwax]] then
slope + CoMPUTESL.PE(Leut)
e.g., via simple linear regression
17:
Clear Lour
P. 1) Recompute datasct mixing ratios
Update each dataset using Equation
20:
2) Decide if window shifts to harder or easier examples
slope -2
22:
prd + mmin prot x (1 +1, 100
Fa
D Loss is decreasing, shift to more complex examples
23:
else if slope >> then
D Loss is increasing, shift to easier examples b Loas stable, keep the current window 3) Select updated subset based on new mixing ratios and reference perplexity percentiles D' +- SeunecTPaaaATAeeTSSSeTTT Pu: Ra: Pact)
PA + max Px x (1 7), 100
25:
else
no change to Pat end if
27:
28:
29:
30:
end if
31:
end for
return
trained model parameters
33:
function WARMUPP(,, Emaxs Purs km))
34:
+-
35:
r 2--
36:
Initialize losa buffer Lbut (capacity Lkw (max))
37:
while retry < P'n do
38:
0, Ln TRAINSTEP (0, UNIFORSSAMPLE(D)
Append I.g Aout
40:
If [Laut] Lkw fanax] then
slope +- COXPUTESSLOP((Daa))))
42:
Clear
43:
if -2 E slope & then
b. Loss stabilized Eud warm-up early
45:
end if
47:
tu -1 r r 1-++1
end while
50:
return fw
end function
function Struee PunDAATAseSSuusss(.... P.. Prot) for d to m do
53:
Pew pr x100 Phigh +- par + 20x000
D' + E Da Pun PPL(c) S Poinn
Kiep examples whose perplexities fall within [lo,, high]-th percentile
56:
end for
D'
return
59:
end function
24

---

Published in Transactions on Machine Learning Research (06/2025)
To clarify the methodology presented, we now define the key notations used throughout our description of the Spaced Scheduled Training (SST) algorithm in Table which provides a comprehensive summary of these symbols and their corresponding definitions for easy reference.
Table 5: Summary of key notations and symbols used in the Spaced Scheduled Training (SST) algorithm.
Notation
Description
General Training Parameters
Imax
Maximum number training iterations.
Px
Global subset ralio (fraction of data to select).
D
Collection of datasets { (D1,..., Da}-
D
Selected subset collection from D used for training.
Perplexity Calculation
PPL(e)
Per-example perplexity for example e. The 50th percentile (median) perplexity for dataset Da.
(d)
Warm-up Phase Parameters
fw
Number of iterations spent the warm-up phase.
kw
Warm-up slope evaluation window size ratio (fraction of lmax).
Tur
Maximum warm-up retry count for slope evaluation.
Adaptive Phase Selection Window Control
Ba
Dataset mix ratio for dataset Da.
Pref
Reference perplexity percentile for dataset Da- Lower/upper percentile bounds for selection window. Selection window adjustment factor (shift ratio).
Plow Phig
k
Rolling window size ratio for loss monitoring (fraction of bmax).
Slope threshold for loss stability evaluation.
C
Overhead
C.1 Wall-Clock Time Comparison
In this section, we present empirical measurements of the overhead introduced by SST compared to random sampling. We use the 100k dataset described in Section 3 For random sampling, we train on all 100k examples, whereas for SST, we select 30k subset out of the same 100k examples and train on that subset following the same training setup as in Section To quantify the overhead, define the wall-clock time for each method as T(.) and estimate the relative overhead ratio
T(IlsTT T(Ilaandoo) T T(IlST
Toverhead
(6)
where T(IlsSTOo.) is the measured time to train using SsT on the 30k subset, and T(Iranddoo) O.3T((Iraniia the time taken by random sampling on 30k examples. We approximate T(Irandgov) by scaling down the measured 100k run time, assuming per-example costs remain roughly constant.
Table 6 compares the training times 100k examples (random sampling) versus SST 30k subset. Although the overhead of evaluating and filtering the data is significant, the training time reduction is substantial ial: for instance, training Qw322.5-32B is reduced from 36.7 hours to 16.8 hours on our hardware. These numbers are highly dependent on hardware configuration and implementation details, but they provide a rough estimate of SsT's overall savings. On multi-node setup, this overhcad may increase due to additional communication costs required to synchronize tensor updates across nodes, but the exact impact will vary based specific in infrastructure and networking capabilities (e.g., whether InfiniBand is used).
25

---

Published in Transactions on Machine Learning Research (06/2025)
Table 6: Wall-clock time comparison between SST and random sampling on different model sizes. Although SST introduces evaluation overhead, the overall time reduction remains substantial because of the smaller training subset (30k vs. 100k). The values are averaged over 3 runs, and the standard error is shown.
Time (hrs)
Method
Num GPUs
Random 100k
SST
SST Overhead
Q2.5-0.5B
2.15 + 0.05
1.20
0.07
40%
L3.1-8B
2.70 +: 0.06
2.10 + 0.04
60%
Qwen2.5-32B
8
36.70 + 0.01
16.80 + 0.03
34%
C.2 Optimized Inference
This sections describes the experimental setup used to analyze how inference-optimizee backends, such as vLLM, can reduce the overhead introduced by SST, as presented in $4.2 The experiments follow the same setup detailed in $ However, we restrict the models to the ones from the Qwen 2.5 family to ensure consistent comparison across different settings. To evaluate the improvement, we track any additional overhead introduced by SST during training. To use vLLM (Kwon et al.) 2023), we implemented a custom training loop based on (Lambert et al.) 2024) where, we pause training, save the training state (model, optimizer, dataloader, etc.), unload the model and optimizer to free GPU memory, and then evaluate the 100k dataset with vLLM. Once evaluation is complete, the training state is reloaded to resume training. This approach introduces overhead from saving and loading the trainer stale that is non-negligible for large model states, but it used solely for this analysis and not in the main experiments in $5 We evaluated vLLM with Bfloat16, 8-bit, and 4-bit procision, using the results to Bfloat16 without vLLM comparison reference. The findings are presented in Figure
D Detailed Analysis on Static Perplexity Sampling
We start by introducing results on the effectiveness and limitations of static perplexity-based data selection in IFT. These findings motivate and lay the groundwork for our proposed adaptive method, introduced in
The work of Marion et al. (023)) demonstrates that simple perplexity-based data selection outperforms more complex metrics. However, that analysis is limited to pre-training, with no comment on its broader applicability to downstream tasks. Additionally, their study was limited to two models (124M and 1.5B parameters), leaving open questions about the generality of these findings across different model sizes and architectures (e-g., Llama vs. Gemma). Building on their methodology, we extend the analysis with several key distinctions: IFT Setting: We investigate perplexity-based data selection in the context of IFT rather than pre-training. Target Model Reference: We don't rely external reference models and use the target model to guide the selection of its training data. Broad Analysis Scope: We evaluated models ranging from 0.5B to 32B parameters across three state-of-the-art architectures, offering a significantly broader evaluation than the two models used in Marion et al. (2023). Through this analysis, we aim to address the following key questions:
Performance and Consistency: Docs perplexity- data selection perform well in the IFT selting, and how does its effectiveness vary across model sizes and architectures? Impact of Training on Selection Performance: Does the performance of perplexity- based data selection improve with training?
Selection Criteria Across Training Stages: Is the criteria for selecting data based on perplexity consistent throughout, training, does it need to be adapted to achieve consistent performance?
26

---

Published in Transactions on Machine Learning Research (06/2025)
D.1
Experimental Setup
We conduct experiments using a stratified 100k subsample of the Tulu 3 SET Mix datasel Lambert et al. 2024), which spans 15 diverse and recent datasets (e.g,, No Robots (Rajani et al.] 2023). Aya (Singh et al, 20241, NuminaMathh-TIR (LI et al. 2024)). We chose this subsample to allow for rigorous experimentation given the computational resources available, while ensuring it represents the full mixture, informed by the sampling analysis in (Lambert et al. 2024). We use models from different architectures (Tlama3.1 (Grattafiori iet al. 2024a). Llama3.2 Grrattaiori et al. 2024a), Qwen2.5 (Yang et al.) [024), and Gemma2 (TTam et al, 2024 ranging from 0.5B to 32B parameters to better understand the impact of different model charac- teristics (e.g,, pre-training data composition, size) on perplexity- data selection performance. We use full-parameter training for two epochs using the sctup proposed by (Lambert et al. 2024). We assess per- formance on the newer Open LLM Leaderboard 2 (Fourrier et al. 2024). which includes more challenging benchmarks compared to the earlier version (Beeching et al.) 2023). including IF (Zhou et al.) 2023b),
BBH Suzgun et al. ),22), MATH LvL (Hendrycks et al. 2021). GPQA (Rein et al. 2023), MuSR (Sprague et al. 2024), and MMLU-PRO (Wang et 2004)). We provide detailed data, training, evaluation, and data
selection setups in Appendix [ and in subsequent sections, provide the methods and baselines used to address each key questions outlined above.
D.2 Performance and Consistency
This section investigates the effectiveness of static perplexity-based data selection for IFT across diverse models architectures and sizes. The primary objective is to assess the effectiveness of static perplexity-based selection for IFT and to analyze how its impact changes with model size and pre-training characteristics.
Methods: For this analysis we consider four data selection strategics. Random: from (1) the full 100k mixture or (2) per-dataset (Random, Random Per-Dataset). Static perplexity-based: from the bottom, middle, and top segments of (3) the overall mixture (Keep Bottom, Keep Middle, Keep Top) or (4) per- dataset (Keep Bottom Per-Dataset, Keep Middle Per-Dataset, Keep Top Per-Dataset). For each method, we select subsets of 10%, 30,, and 50% of the full 100k mixture, following (Marion et al. 2023 Given the large-scale nature of our experiments (248 training runs), we limit the evaluation of random selection strategies to two independent runs with different seeds (123 and 42) shown to be effective by (Lambert itt al. 2024). This maintains computational practicality while still capturing some measure of variance. Each method is compared against the full 100k data mixture baseline using its average score across all benchmarks ($D.1) Baseline: We contrast the above strategies against a baseline that uses the full 100k data, mixture
(100%). Models: To ensure the generalizability of our findings, we evaluate a diverse set of models varying in size, pro-training data composition, and architectural design. Specifically, we use Qwen2.5 0.5B, Llama3.2 IB, Llama3.2 3B, Llama3.1 8B, Gemma2 9B, Qwen2.5 14B, Gemma2 27B, and Qwen2.5 32B. We selected these models to study the impact of unique model characteristics on static perplexity- based data selection performance. In total, we trained 8 models, each with 31 different training configurations, resulting in 248 training runs for this experiment alone.
Figure 2 shows the strongest-pefforiing basclinc, Random Por-Datasct with the top-performing static perplexity-based strategies: Keep Bottom Per-Dataset, Keep Middle Per-Dataset, and Keep Top Per- Dataset. Full results, including all methods and baselines, and per-benchmark results, are detailed in Appendix F The relative performance change AR is calculated as:
Smethod - SSasselnne
AR
(7)
Sbaseline
where Smethod and Spaseline are the average Open LLM Leaderboard score of the method's and the baseline respectively. This metric allows consistent evaluation across models with varying bascline performance. These results highlight several key insights:
Heuristic-based data selection is insufficient for consistent performance. Some models, even with naive random selection, outperform the 100% bascline, indicating that while the extensive heuristic and supirical-based approach of Lambert et al.] (2024) is effective, further refinements could yield even greater performance gains simply by optimizing training resource allocation. It also suggests opportunities
27

---

Published in Transactions on Machine Learning Research (06/2025)
Random
Keep Bottom
Keep Middle
Keep Top
15%
Perbomance AR (Ng
10%
5%
0%
-5%
-10%
Subset Size
-20%-
10k
25%
30k
-30%
50k
0.5
1
8
27
32 0.5
9
32
0.5
27
0.5
27 32
Model Size (B)
Figure 8: Performance of static perplexity sampling compared to random selection from all the mixutre. Relative performance change AR calculated as AR mSethhod - Spaseline) / baseline where Smethod and Sbsseline are the average Open LLM Leaderboard score of the method's and 100% bascline. Points below the red dashed line indicate performance drops compared to the baseline, and error bars show the standard error over two random seeds.
to enhance computational efficiency, reduce training costs, and improve scalability for larger models. Dataset-aware selection produces the best results. Per-dataset selection consistently outperforms full-mix sampling across all models for both the random baseline and perplexity- baseed strategies. This effect is particularly evident in complexiyy datasets like ours, where FLAN v2 examples are easier than more reasoning- cassoing-intensive tasks such NuminaMath. This issue amplified when selecting from the overall mix using complexity-aware method, like DEITA, INSTAG, static perplexity- based strategics, they skew the subset (oward harder examples. However, IFT models require exposure to diverse range da datasets domains, including easier ones, and dataset-aware sampling preserves this balance.
Static perplexity sampling improves on bascline but lacks consistency. In Figure we observe that keeping the bottom segment consistently underperfornı naive random selection, suggesting that the most useful data lies the top 50% of the perplexity distribution. This likely due to data leakage (Tambert) let al.) 2024h, where models may have encountered these examples during pre-training, which could explain why the effect is more pronounced in some models (e.g., Llama3.2 1B and 3B) than others. Another possible explanation is that modern LLMss, like those in our study, are more capable due to training trillions of tokens Yang et al. 2024). In both cases, training compute is allocated inefficiently leading to suboptimal performance. In contrast, per-dataset middle-segment selection tends to improve performance for models smaller than 14B parameters, while keeping the top segment benefits larger models, such as, Qwen2.5 S.ven2.5-3BB, which matches the performance of the 100% baseline using only 10% of the data. However, for models smaller than 27B, Random Per-Dataset remains strong baseline, suggesting that static perplexity sampling alone is insufficient for consistent performance. Moreover, the best perforrero configuration varies across models and subset sizes, highlighting the need for adaptive data selection strategy for robust results.
D.3 Impact of Training on Selection Performance
The work of Marion al. (0233 showed that data selection performance improves with better reference models, cither larger size or trained on better data. In this section, we investigate whether similar behavior occurs when using the target model the reference it trains on more data. Specifically, we aim to identify when in the training process data selection performance peaks and how it evolves with additional training.
We select three models (Qwen2.5-0.5B. Llama3. 1-8B, and Qwen2.5-32B) and use the perplexity sampling configurations from the previous section: Keep Top Per-Datasel for Qwen2.5-32B and Keep Middle Per-Dataset for the other two models. We first train each model on data randomly sampled per dataset, following the best-performing random strategy from the previous scction (Random Per-Datasct). At specific points during the two-epoch training process (5%, 10%, 15%, 25%, 50%, and 75%), we compute the perplexity values and apply perplexity sampling to select 30% of the 100k dataset. We then continue training the model
Methods:
28

---

Published in Transactions on Machine Learning Research (06/2025)
for the remaining iterations to complete two full epochs. For each method, we conduct two runs with different random seeds (123, 42), and report the average score and standard error across the runs. Baseline: We use results from the previous experiment, where we selected 30% of the 100k mixture using the pre-trained model as the reference at the start of training. Figure shows the results of this analysis.
Data selection performance does not always improve with training. As shown in Figure 4 per- formance improves when perplexity sampling is delayed to certain point in training but degrades beyond this threshold, which varies across models we tested and generally decreases with model size. For instance, Qwen2.5-0.5B benefits from delaying selection until 25% of the training process, while Qwen2.5-32B starts to degrade as early as 10%. Llama3.1-8B and Qwen2.5-32B exhibit significant performance drops after the performance peak, with Qwen2.5-32B being particularly affected. Our analysis (Figure 9 attributes this behavior to changes in the perplexity distribution. For Llama3. the distribution becomes heavy-tailed as the model trains on more data, which shifts focus toward overly challenging examples. For Qwen2.5-32B3, the distribution becomes narrower and skews loward the top segment, emphasizing overly complex (e.g.. Table 7 and noisy examples (e.g., Table S). In contrast, Qwen2.5-0.5B is less influenced by these distribu- tional changes
show in Figure [] where the distribution remains relatively stable, explaining why it benefits from longer delay in perplexity-based selection. We also observe that these changes also aligns approximately with major trends changes in the overall training loss (Figure 9 that we describe in more details in $ These findings suggest that an adaptive data selection strategy, which delays perplexity-basee selection to an optimal point that varies by model, is essential for achieving consistent performance.
D.4 Selection Criteria Across Training Stages
In this section, we investigate whether using different segments of the perplexity distribution at various training stages can improve the final performance of perplexity-based data selection. The goal is to determine if modulating the selection window is necessary to achieve consistent performance.
Methods: For this experiment, we use Llama3.1 1-BB and Qwen2.5-32B and compare the performance of the best and second-best static perplexity sampling configuration from the previous section. We use Keep Middle and Bottom Per-Dataset for Llama3. 1-8B, and Keep Top and Middle Per-Dataset for Qwen2.5-32BB We omit Qwen2.5-0.5k from this analysis, as its performance variation does not show meaningful comparisons.
The best perplexity distribution segment varies during training. Figure g shows that using dif- ferent perplexity segments at different training stages affects performance. For example, both Qwen2.5-32B and Llama3.1-8B benefit from starting with easier examples (i.e., using a lower perplexity segment) before transitioning to more challenging examples. When contrasting this behavior in Figure 3 to the performance peaks observed in Figure 4 we find that these transitions align approximately with the performance peaks. Our analysis suggests that using easy examples earlier in training (i.e., lower perplexity segments) stabilizes learning by allowing enough training iterations to adapt to the data distribution shift that occurs early in training before handling more complex examples effectively. This aligns with early research on curriculum learning (Bengio et al.) 2009). which demonstrated that starting with easier examples can enhance learning efficiency. These results raise the question of whether an adaptive selection strategy, capable of modulat- ing the selection window dynamically, can offset the initial performance drop and perhaps improve final performance (we show in Table that it does).
29

---

Published in Transactions on Machine Learning Research (06/2025)
200
Buldieeeffanne waweeeeeanneeeeeeeeeeee Cangee
Qwen2.5-0. 5-0.58
Llama3.3-4 3-1-88
QwanZ 5-328
-10%
200%
25%
75%
155%
15%
Perplexity Sampling Start Iteration (%)
(a)
1.4
1.4-
1.4
127
1.0
1.0
0.8
0.8
0.8
0.8
25%
35%
15%
Prggess
Propress
(b)
1.2
Qwan2.5-0.58
Uama331-88
Qwen2.5-328
1.0
0.6
0%
0.4
50%
75%
10*
20-
10F
Perplexity (log scale)
(c)
Figure 9: Effect of delaying perplexity sampling on model learning. Performance peaks (a) approximately when the loss begins to flatten (b). Further delay in perplexity sampling degrades performance due to a long- tail perplexity distribution, causing the subset to include disproportionately many complex or noisy examples (high perplexity). (a) Repeats Figured for easy comparison of training loss and distribution behaviors. (b) Displays overall training loss over time. (c) Shows perplexity distribution different training iterations.
30

---

Published in Transactions on Machine Learning Research (06/2025)
E Detailed Results
E.1 Intractable Examples
In this sections, we present examples of intractable examples that represent mislabeled or noisy data that we identified during our analysis.
31

---

Published in Transactions on Machine Learning Research (06/2025)
Table 7: First 50 out of 133 lines from extremely hard example found during analysis
Instruction
Create program evaluate postfix expression
Python program to evaluate value of a postfiz eapression Class to convert the expression class Conversion:
Constructor to initialize the class variables def __ (self, capacity):
self.top -1
self capacity capacity
# This array is used a stack self array []
# Precedence setting self output []
self precedence ( 1-:11 "*::2, 1/222, :0)
# check if the stack is empty def isEmpty (self):
return True if self top == -1 else False
# Return the value of the top of the stack def peek return self [-1ay[-1]
Pop the element from the stack def pop(self):
Output
if not self isEmpty (): aelf.to. 1
return self array pop ()
e else:
return
Push the element to the stack def push (self, op): self top +- self array. append
utility function to check the given character # is operand
def is0perand (self, ch): return ch ch.isalpha
Check if the precedence of operator is strictly less than top of stack or not def notGreater (self, i): try:
self precedence
b self precedence
return True if <= else False
except KeyError:
return False
32

---

Published in Transactions on Machine Learning Research (06/2025)
Table Ambiguous example found during our analysis. The target output contains place holders suggesting ChatGPT generated data.
Inst: Gen a sunmary of an article containing more than 500 words.
Out: The article is about
It discusses the subject in depth,
exploring various aspects of the topic. It provides a comprehensive overview of different perspectives on the topic and presents evidence to support its conclusions. In conclusion, the article provides a by offering a balanced and
comprehensive evaluation of unbiased assessment of the subject.
33

---

Published in Transactions on Machine Learning Research (06/2025)
Table 9: Performance of static perplexity sampling compared to random selection and 100% baseline for Qwen2.5-0.5B.
Method
Beed
Subxet
AR
Avg
MATH Lvl &
IFEval
MMLU-PRO
GPQA
MUSB
BBH
1.76
Middle
Middle Pai-Dutasek
1.27
Random Darastt
123
0.04
30-71
7.03
0.36
4.33
Rondom Por-Detaset
123
Keep Top
0.01
7.94
0.95
29.81
5.32
1.79
2.02
4.71
Baseline
1.0
0.0
7.87
0.91
31.07
0.22
1.27
4.66
Randam
123
0.1
0.8
7.83
1.81
26.
7.03
1.9
1.82
3.
Keep Middle Par-Dutasel
42
0.3
0.01
7.76
1.96
25.53
3.9
0.04
1.27
0.22
Rendom Par-Datasel
42
0.3
0.03
7.64
0.91
29.09
0.60
0:11
1.48
4.14
Keup Top Per- Datamet
42
0.5
0.03
7.64
0.91
26.98
6.50
0.45
615
4.6
Random
42
0.1
0.03
7.8
2.04
24.94
6.33
1.9
2.75
3.28
Random
123
0.3
0.06
7.36
1.28
27.08
5.03
0.56
2.0
0.04
Keep Per-Dacaset
0.3
1.00
2.37
23
7-43
26.08
1001
Keep
9.06
3.15
Keep Top
Datnntt
Keep
Keep
Per-Dniaset
1.02
Keep Bottom Per-Dataeet
0.
5.53
Keep Bottom Per-Dutastt Randam
00.08
7.25
1.36
26.91
6.38
0.11
1.48
3.88
0.5
0.09
7.17
1.28
27.27
4.74
1.12
1.48
4.79
Keep Middle Per-Dutasel
42
0.1
0.09
7.19
1.50
24.6
0.07
1.84
1.76
3.97
Keep Bottom Datawet
12
0.3
-0.11
7.03
1.51
26-43
6.24
0.0
0.94
4.22
Random
42
0.3
-0.14
79
1.06
25.08
4.75
1.45
1.8
4.25
Random Por-Dasaset
123
0.1
0.15
68
1:96
24.
5.78
0.0
0.84
Keep Top
0.1
0.16
6.02
0.98
20:36
4.02
0.92
1.0
53
Table 10: Performance of static perplexity sampling compared to random selection and 100% baseline for Llama3.2-1B.
GPQA
Random Par-Darasst
123
0.5
0.08
5.57
0.45
36. 28
1.71
0.0
4.39
3.
Bascinn
42
1.0
0.0
7.94
0.45
33.96
1.06
0.0
3.62
4.01
Rondom Por-Detasel
42
0.5
0.0
7. 7.93
0.38
34.08
1.31
0.0
3.29
4.52
Keep Middle Per-Dataset
42
0.5
0.0
7. 722
0.30
34 18
1.49
0.0
3.52
5.01
Keep Bottom
42
0.5
-0.03
7. 77
0.38
32. 75
1.43
0.0
4.05
8.2
Random
42
0.5
-0.04
7-61
0.3
30.57
1.36
0.0
3.78
3.
Random Par-Daraset
123
0.3
-0.04
7-65
0.3
33.3
1.45
0.0
3.6
3.13
Keep Bottom Per-Datanct
42
0.5
-0.05
7.05
0.3
34 14
2.15
0.0
15
4.72
Random
123
0.5
-0 -0.06
7:48
0.08
34
1.08
0.0
0. 89
3.3
Keep Middle
42
0.5
-0.07
7.37
0.6
31
1.61
0.0
2.8
5.82
Keep Top
42
0.5
-0.08
7. 26
0.3
32, 16
1.27
0.0
2.5 59
7.35
Keep Bottom
42
0.3
-0.08
7.29
0.68
30.08
2.43
0.0
3.3
4.15
Keep Top
42
0.3
-..09
7.25
0.35
3T.57
1:22
8.0
2.81
3.54
Keep Middle Par-Dataset
42
0.3
-..09
7.22
0.43
30.91
1.41
6.0
3.34
3. 3.48
Random Par-Daraset
42
0.3
-0.1
7. 13
0.3
31.2
1.14
0.0
3.0
3.12
Random
123
0.3
-0.1
7.16
0.45
32.02
1.64
0.0
1.31
3.34
Keep Bottom Der-Dstavet
42
0.3
-0.11
7.04
0.53
30 28
1.93
0.0
2.45
3.82
Keep Middle
42
0.3
-0.14
6.83
0.38
20 18
0.88
0.0
3.7
2.95
Keup Top Pur-Datane
42
0.3
-0.15
6.72
0.15
29.51
1:35
6.0
2.29
3.72
Keep Top Par-Dntnuct
42
0.5
-0.10
6.89
0.23
30.51
1.25
0.0
1.2
3.44
Random Par-Darasst
123
0.1
-0.17
6.58
0.53
28 18
1.81
0.0
2.32
3.05
Random
42
0.3
-0.18
6.54
0.3
29.8
1.39
0.0
1.43
3.82
Random
42
0.1
-0.19
6.42
0.45
26.58
0.92
0.0
4.05
3.16
Random Por-Dacaset
42
0.1
-02
6.38
0.53
27.54
1.86
6.0
1.95
2.84
Randam
123
0.1
-0.23
8.1
0.8
26.11
1.42
0.0
2.35
2.80
Keep Top
42
0.1
00.24
6.02
0.53
26.20
1.26
0.0
3.06
2.67
Keep Top Pa--uinset
42
0.1
0.24
6.07
0.6
26.29
1.38
0.0
3.09
3.44
Keep Battom Per- Dxtivtt
42
0.1
-0.25
5.93
0.8
26.2
2.11
0.0
0.72
3.07
Keep Battom
42
0.1
-0.27
5.78
0.38
24.02
1.78
0.0
2.17
2.84
Keep Middle Per-Dataset
42
0.1
0.28
5.88
0.53
25.11
1.55
0.0
3.14
3.8
Keep Middle
42
0.1
0.32
5.38
0.38
22.35
1.58
0.0
2.06
3.15
F Additional Results for Static Perplexity Data Selection
In the section we present additional results for the static perplexity- data selection experiment described in
34

---

Published in Transactions on Machine Learning Research (06/2025)
Table 11: Performance of static perplexity sampling compared to random selection and 100% baseline for Llama3.2-3B.
Mothod
Seed
Subiet
4 R
Avg
MATH Lvl
IFEval
MMLU-PRO
GPQA
MOSR
BBE
Keep Per-Dataset
0.02
3.58
Buscline
0.9
1.44
2.01
Raudom Per-Datunct
2.13
Ratdou Por-Dalancs
2.24
Keep Top
Per-Dataset
Per-Datasct
Keep Middle
0.06
1.12
14.96
14.62
44
16.6
Keep
14.72
47.
1.70
85
13.94
Keep Bottom Pur-Dataset
0.5
14.8
1.86
47.94
15.87
0.45
8.1
19.63
Random
42
0.5
-0.00
14.
1.80
47.3
18.81
1.12
9. 0.47
16.52
Keep Top Par-Datamet
12
0.5
-0 09
14.07
1.50
06.51
15.72
2.8
6 72
10:17
Random
123
0.5
-0.1
14.35
1.89
48.31
11.29
1:23
6. B1
17.74
Keep Bottom Par-Datasot
12
0.3
-0.15
14.22
1.28
47.04
14.26
2.01
6.5
11.95
Keep Middle
42
0.3
-0.1 12
14:21
1.74
96.54
10.0
1.45
10.45
13.0
Keep Middle Per-Dataset
42
0.3
-0 13
13.02
1:36
46.1
10.
3.02
14:00
Random Per-Dataset
133
0.1
13
13-74
0.81
11-26
3.02
10.31
Keep Top Per-Dataset
0-3
1/5
13-54
1.58
3.55
Keep Botiom
2.24
Keep
Keop
Keep Top Per-Dataset
12.41
38.88
12:35
2:35
6.37
19.87
Keep Middle Per-Blataee
11.96
10.79
1:34
3.88
13.1
Keep Bottom
-0.27
11.89
1.28
37.81
2.91
7.18
9.67
Random
125
-0.3
11.22
2.04
39.02
9.36
2.24
3.22
8.05
Keep Botsom Per-Dataset
42
0.1
-0.3
11.32
1.36
36.07
9.49
2.35
5.31
8.77
Random
42
0.1
-0.31
11.09
1.44
36.60
7.50
1.9
5.02
0.53
Table 12: Performance of static perplexity sampling compared to random selection and 100% baseline for Llama3.1-8B
Mathod
Seed
Subeet
AR
Ave
MATH Exsl
IPRvv
MMLLU-PRO
GPQA
MUSI
Keep Top Par-Dutaset
0.08
T.38
Keep Per-Datasct
0.03
22.0
4.25
Keep
Keep Per-Datases
Keep Top
Keep Bottom Per-Datasst
4.58
24.52
Random Per-nataaet
21.
61.65
21.41
0.6
10.
16.12
Random
0.5
21.
58.73
21.17
10.18
21.35
Keep Top
42
0.3
0.02
21. 78
4.88
60.07
24. 66
6.7
13.
24.47
Keep Botsom Par-Dutasel
42
0.3
0.03 03
21.
2.
07.08
24
4.25
16 42
26.57
Random
125
0.3
-0 03
21.47
6.12
80.25
22. 644
3.91
14 35
28 09
Random Per-Dataset
125
0.1
- 03
21.47
6.34
76.73
22:17
B.57
18.15
24.84
Keep Bottom
42
0.5
0.03
21.46
2.02
58.50
21.00 05
4.14
15.60
26.79
Random Per-Dataset
42
0.1
0.04
21.27
4.76
64.42
24.65
0.26
18. 18.87
23.35
Random
123
0.5
58.93
5.37
Keep Top
21.13
51.52
24.25
6.71
77.
20.74
Keep Top Per-Dataset
4.38
8.33
Keep Top
Keep
20.
Keep Middle
0.08
20.45
17.44
Keep Middle Per-Datases
-0.11
19.74
22.
17.02
Random
125
0.1
-0.12
19.46
4.48
46.21
22.77
1.6
15.25
29.25
Keep Bottom
42
0.3
-0.12
19.48
0.42
61.50
20.
6.15
12.08
26.31
Keep Bottom
42
0.1
0.12
19.52
5.36
45.88
23. 45
8.28
14.62
26.2
Keep Bottom Par-Dutaset
42
0.1
0.13
19.16
5,29
46.07
22.08
6.15
14.62
28.08
35

---

Published in Transactions on Machine Learning Research (06/2025)
Table 13: Performance of static perplexity sampling compared to random selection and 100% baseline for Gemma2-9B.
Method
Avg
GPQA
Raudom Per-Datanes
42
0.5
0.00
20 10.10
12.00
66.96
29. 64
10.4
20.07
30 35.37
Keep Top
42
0.3
0.03
28.51
10.0
09: 46
33. 34
11:74
17.63
36.8
Keep Top Per-Dataset
42
0.3
0.03
25 55
12:10
67: 95
31 06
7:83
23 23.75
30.50
Random
123
0.5
D-02
25 36
13. 07
09.47
30.41
10.4
18. 18.56
35 93-99
Keep Middle Per-Ilataset
12
0.5
D-02
26
11. 88
06:36
30.3
10.
23 33
30.02
Keep Bottom
82
0.3
0.02
28. 14
20.17
61.43
32-13
8.95
19.49
32.54
Random
125
0.1
0.01
28 01
15. 16
62.9
31. 19
13.09
18 566
35.30 :59
Keep Top Pur-Dataret
42
0.1
0.0
27. 86
13. 44
01:54
35.27
6.71
22.00
$4.09
Buseline
42
1.0
0.0
27.81 81
15. 20
06.74
62
8.95
18.68
$1.50
Random Per-Dateact
42
0.3
0.0
27 92
11.98
66.36
30.5
9.98
21.66
38.41
Keep Per-Ilataset
0-3
0.0
12.3
Keep Top
0.3
10.3
67.81
Random Per-Dataset
7-61
Random
Pa-Duiasee
17.
Botiom Pa-Doitsee
64.97
15.75
Random
0.5
29
69.04
8.51
11.85
$3.59
Random Per-Dataret
26,
48
88.72
7.72
24.
Keep Bottom
42
0.1
-0.00 04
26. 56
18.45
63.02
30.
12.08
18.75
36.51
Keep Hottom Per-Datasot
02
0.1
-0 04
96 82
15. 63
59:04
331
12.08
13.54
36.11
Keep Middle
12
0.5
-0 04
26 87
12.30
66.51
27 195
99.51
16.88
31.97
Randow
42
0.1
0.05
26 44
15.71
09.97
30. 88
9.62
16.33
32.78
Keep Middle Pey-Datasck
42
0.1
DG
26 24
12. 22.02
00:36
76
6..5
17.18
38.45 43
Keep Top Per-Detaset
42
0.5
25
10.8
68. 18
7.49
16.25
31.76
9.28
14. 14.08
Keep Top
Keep
Table 14: Performance of static perplexity sampling compared to random selection and 100% bascline for Gemma2-9B.
Mothod
Seed
Subiet
AR
Avg
MATH Lvl
IFEval
MMLU-PRO
GPQA
MUSR
BBE
Keep Top Per-Dataeet
0.5
D.01
45.38
17:44
21.92
Bazeline
1.0
0.0
70.41
15.95
14.54
Random Per-Dataset
0.5
0.0
42.3
Keep Top Par-Dataset
17:53
21.
Per-Datases
0.05
Random
43.81
15.21
17.
Random
Random
67
64.
41.
15:55
45.36
Random Per-Dataset
123
43.75
14.21
20.9
05
Keep Top Par-Dataset
235
28
85.77
15:55
15. 15.57
40.52
Randou Per-Datencs
42
0.1
08
82.
20. 47
60.42
43.08
14.08
21.3
38.70 76
Random
123
0.3
-0.00
82.85
23.87 87
64.6
45.11
15.1
16.00
43.24
Keep Middle Per-Dataset
42
0.1
-0.08
32.95
28. 072
58:48
44.05
17.58
18.29
41.78
Keep Bottom Pur-Dutaect
42
0.5
-0.08
32.93
28.4
03.91
44.4
17.0
11.06
42.45
Keep Bottom Pa-Duiaset
42
0.3
0.1
82.29
28.4
02.75
44.98
15.1
10.19
30.77
Keep Middle Per-Datwees
42
0.5
0.1
81.05
22. 06
65.39
42. 17.07
14.88
14.14
41.35
Keep Middle Per-Dataset
0.3
31.13
62.74
19.99
Keep Bottom Par-Dataset
0.1
-0.11
31.08
63.05
45.56
14.77
36

---

Published in Transactions on Machine Learning Research (06/2025)
Table 15: Performance of static perplexity sampling compared to random selection and 100% bascline for Gemma2-27B.
Mothod
Seed
Subset
4 R
Avg
MATH Lvl
IFEval
MMLU-PRO
GPQA
MOSR
BBE
Keep Top PataDet
42
0-5
D.08
32.
82
70.
20
38.7%
Random Per-Dataset
0-3
0:06
32.41
68.37
Bandom Per-Dataset
0-3
0.04
311
10.0
Keep Top Per-Dataset
0-3
b.03
69.
10.
Keep Top
Keep Par-Dutusel
Buseltue
47
Keep Middle Per-Dataset
0.3
0.
300
18
67.85
12.53
17.
$3.95
Random
125
0.1
-0.01
30.
20. 47
09.28
19
12.75
18.
38 83
Keep Bottom Par-Datasct
42
0.3
-0.01
30.
22 61
71.04
33.
13.53
10.0
40,80
Random
02
0.1
-0.00 01
30 29
21 71
68.31
33.0
19.20
17:21
39.
Keep Bottom
12
0.5
-0 02
29 83
23 87
67.55
31 77
8.722
17.7
41:13
Keep Botiom Par-Duiasel
42
0.5
20. 02
20. 60
70.08
67
11:07
12.68
37.15
Keep Middle
42
0.5
03
29.
17. 07
73.
11.44
13.05
38 52
Random Per-Dataset
42
0.1
-
29
20 62
61.96
12:5
18 47
38 38
Keep Botsom
42
0.1
29
24 88
53.30
12:1
17 17.34
Keep Middle
0-3
8-72
10.00
Keep Botiom
Keep Per-Dataset
Keep
Middle Per-Datauci
Keep
17
9.4
18.52
Random
125
0.5
-0.12 12
26
21.05 07
65.7
9.28
13.06
32.2
Random Per-Dataret
0.5
0.14
26.
86
07.58
27.20
4.47
12:47
$2.08
Keep Middle
0.1
-0.16
25.58
12.08
05.54
27.77
7.61
18.9
38.89
Table 16: Performance of static perplexity sampling compared to random selection and 100% baseline for Qwen2.5-32B.
Mothod
Seed
Subuct
R
Avn
MATH Lvl
IFEval
MMLU-PRO
GPQA
MUSR
BBH
Keep Top Par-Dataset
0.5
0.1
41.53
75.03
Random Per-Dataset
0.3
0:07
74.57
Random
0.08
Random
0.05
72.3
17.9
Random Per-Datunct
0.05
Random Per-Datuees
0.04
Randou
0.04
25
Keep
Random Per-Dataset
Per-Datacet
47.65
Keep Middle Per-Dataoct
0.5
Keep Top PartDettset
0.3
0.02
38.
33. 11
75. 23
192
19.02
18.
48 49.68
Random Per-Dataset
42
0.1
0.02
38.
31.5
49.8
10.53
22 06
48.09
Keep Top Par- Datastt
42
0.1
0.01
25
34 52
64
50. 00
18:37
20 25
44.01
Keep Top
12
0.3
D.01
235 13
32 18
73.06
51. 77
17-45
17.00
49.5 51
Keep Top
12
0.1
0.0
337 73
30 29
66.01
51 61
17.78
22. 594
49.85
Baseliac
42
1.0
0.0
87,
34. 07
73.9
10.38
16.62
48 06
Keep Middle
42
0.5
87,
30 80
75.58
17.70
15.51
46.16
Keep Per-Dataset
0.3
18.
Keep
Keep
Per-Dataict
17.62
Keep
Botiom Paz-uiasee
Bottom Por-Dotaset
0.05
86
50.2
17.34
Keep Hottom
0.5
36
07.3
16.99
16.78
45.75
Keep Middle
0.1
00.07
36.2
31.6
05.23
45.62
17.28
18. 18.41
40.25
Keep Botsom
42
0.1
0.07
35.12
25.23
61.02
49.09
17.07
21.71
46.60
Keep Bottom
12
0.3
-0.14
32.33
15.99
64:65
18.37
16.67
18.96
45.24
37


Reviews:
[{"review_id": "OYDsP34P4Q", "reviewer": "Reviewer_Rpij", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n\n1. The discussion of related work is comprehensive and beneficial, helping the general audience quickly understand this field.\n\n2. The experiments on static methods are thorough, and the analysis is insightful. These detailed experiments will support future work in this area.\n\n3. The proposed method demonstrates good performance and computational efficiency.\n\n**Weaknesses**\n\n1. Given the dynamic nature of the proposed method, it would be beneficial to include dynamic method baselines. (Please refer to Change 1)\n\n2. A detailed analysis of Tables 1 and 2 would be insightful for readers. (Please refer to Changes 2, 3)\n\n3. The impact of the paper could be enhanced by applying it to domains where less data is more advantageous. (Please refer to Change 4)", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "1. This paper offers a thorough empirical evaluation of static data selection methods. The analysis is exhaustive and is expected to inspire future work in this area.\n\n2. The proposed method demonstrates strong performance, with performance gains that increase with model size. Additionally, the computational efficiency is commendable.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. It is recommended that the authors include some dynamic methods from related works in Table 1.\n\n2. The training dataset is a mixture of many datasets. It would be beneficial to illustrate the ratio of selected data for each sub-dataset for both the proposed method and the baselines.\n\n3. In Table 1, each benchmark includes several subjects, such as math, physics, chemistry, etc. It would be helpful to illustrate the performance differences compared to full data on each subject to better understand the detailed performance gains or losses.\n\n4. Currently, the major benefit of the proposed method is the reduction in training cost. Although there are some performance gains on certain models, they are not significant. In many domains, less data can yield significantly better performance [Ref 1]. It is highly recommended to test the proposed method on this particular dataset.\n\n[Ref 1]  LIMO: Less is More for Reasoning", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "3nA56JdFbk", "reviewer": "Reviewer_TeJJ", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength:\n1. The experiment settings are clear and reasonable. The paper investigates different sampling strategies for models of different sizes and structures.\n2. The paper proposed an adaptive sampling strategy, which can adjust the sample selection over a mixture of datasets and within each dataset.\n\nWeakness:\n1. On the variance of the runs. The paper reports variance only with two runs for Sec. 3. This is insufficient to show any statistical result. \n2. Based on the large variance in Sec. 3, it is hard to justify whether the results in Sec. 4 demonstrate SST's better performance over other cases. \n3. Sec. 4.1 is unclear and hard to understand, please consider putting Alg. 1 into this section.\n4. It is insufficient to claim aligning data complexity with training progress proves beneficial based on Fig. 2. The paper should provide the comparison between adjusting sampling PPL from high to low, from low to high, fixed low, fixed middle, fixed high. Also, why Llama 3.1-8B misses Keep Top, and Qwen2.5 misses Keep Bottom?\n5. Does the method consider the change of sample PPL during training? Or does it use fixed PPL throughout training? If it requires computing $\\beta_d$, I presume it requires computing PPL for all data for multiple times. If that is the case, computation in Sec. 4.2 does not make sense to me. The selection cost in eq(4) should be multiplied by the number of adjustments of $\\beta_d$.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper investigates the perplexity-based sampling method in IFT on LLMs. First, numerical experiments with different sampling strategies are conducted using a static sampling method. The paper argues that for various models, keeping samples of different percentiles is required, and changing the sampling percentile is more beneficial. Then, the paper proposed an adaptive sampling strategy, which has two folds. First, the method assigns datasets with different weights based on its median PPL; then, within each dataset, the method adjusts the percentile based on the loss change of that dataset. Numerical comparisons show that the proposed strategy outperforms other sampling strategies in most scenarios.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "In section 4, it missed a space after the word SST.\n\nPlease address the weakness above.", "broader_impact_concerns": "Sampling based on perplexity for LLMs (or other models) may introduce implicit bias and unfairness, which can raise ethical concerns and require further investigation.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "NdFSJEq6QB", "reviewer": "Reviewer_eqf2", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. A novel method for dataset selection for the instruction tuning stage, and the motivation for the method is clear, and the proposed method is sound.\n2. The experimental results demonstrate the effectiveness of the proposed method. It has a large improvement compared to the baseline method. The method is evaluated under 7 models with different model sizes\n\n\nWeaknesses:\n1. Although the author begins with the motivation that heuristic methods for data selection are unreliable, the proposed method itself is also heuristic in nature. A theoretical analysis of the data selection strategy, rather than just the computational overhead, would be preferable and would help strengthen the justification for the approach.\n2. It's a little bit surprising that the proposed method can even largely surpass the performance of the full dataset. If we disregard the presence of erroneous data in the dataset, the experimental results become somewhat unconvincing that a subset can even surpass the full set. I checked the open LLM leaderboard with the results. For example, for Llama-3.1-8B, the performance for GPQA and MMLU-Pro is 8.05% and 25.42%, which are both higher than the finetuned SST results in this paper (4.25%, 23.84%). I'm not sure if I interpreted correctly with your results, please point it out if I misunderstood something.\n\n\nQuestion:\n\nFor Section 4.1 Adaptive Scheduling Phase, compute the examples perplexity values PPL(e) (Equation 1) once: Which model is used to compute the ppl? Is this the untrained one or the trained model after warmup? Would the selection / the stage of the model impact the performance, since it directly impacts the distribution of the PPL for that dataset?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes a dynamic data selection strategy for training datasets. The core motivation is that aligning data complexity with the model’s training progress can lead to improved performance. Preliminary experiments with a PPL-based heuristic search method reveal that its effectiveness is influenced by both model size and the chosen data subset. To address this, the authors introduce a method that dynamically adjusts the PPL selection range based on feedback from the training loss and the dataset mixing ratio. They also analyze the computational overhead introduced by this selection strategy. Experimental results demonstrate that the proposed method significantly outperforms baseline approaches and even surpasses the performance achieved using the full dataset.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. Please add one space after the method name. Currently, the method name and the following words are stacked together. Also, please check the paper writing carefully, there are a lot of typo mistakes like: `but incorporating incorporates`\n2. Please rearrange the structure of the paper for the figures. Reading this paper requires frequent back-and-forth referencing of tables and figures. For example, Figure 2 is never mentioned anywhere in the main paper. Some words below Figure 2 refer to Figure 6, which appears only in the appendix. Additionally, some notations are not rigorously defined—for instance, $P_{\\text{ref}}^d$ lacks a clear explanation.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a technique for data selection in large language models that makes use of the per-example perplexity assigned by the model itself. Overall, the idea is straightforward, and the results are convincing - it outperforms various data selection baselines across various settings. The reviewers all recommended acceptance. My only suggestion would be to better clarify (in the title or introduction or wherever) that you primarily focus on data selection for instruction tuning in this work.
Decision: Accept (poster)