EvoLM: In Search of Lost Training Dynamics for Language Model Reasoning
Authors: Zhenting Qi
Fan Nie
Alexandre Alahi
James Zou
Himabindu Lakkaraju
Yilun Du
Eric P. Xing
Sham M. Kakade
Hanlin Zhang
Venue: NeurIPS 2025 oral ()
Keywords: Language Models
Training Dynamics
Pretraining
Post-training
Primary area: evaluation

Abstract: Modern language model (LM) training has been divided into multiple stages, making it difficult for downstream developers to evaluate the impact of design choices made at each stage.
We present EvoLM, a model suite that enables systematic and transparent analysis of LMs' training dynamics across pre-training, continued pre-training, supervised fine-tuning, and reinforcement learning. 
By training over 100 LMs with 1B and 4B parameters from scratch, we rigorously evaluate both upstream (language modeling) and downstream (problem-solving) reasoning capabilities, including considerations of both in-domain and out-of-domain generalization. 
Key insights highlight the diminishing returns from excessive pre-training and post-training, the importance and practices of mitigating forgetting during domain-specific continued pre-training, the crucial role of continued pre-training in bridging pre-training and post-training phases, and various intricate trade-offs when configuring supervised fine-tuning and reinforcement learning. 
To facilitate open research and reproducibility, we release all pre-trained and post-trained models, training datasets for all stages, and our entire training and evaluation pipeline.

EvoLM: In Search of Lost Training Dynamics for Language Model Reasoning
Zhenting Qi'
Fan Nie
Alexandre Alahi
James Zou2
Himabindu Lakkaraju'
Yilun Du
Eric Xing
Sham Kakade' Hanlin Zhang' +CMU
HHavaard
Stannfodd
'EPFL
Abstract
Modern language model (LM) training has been divided into multiple stages, mak- ing it difficult for downstream developers to evaluate the impact of design choices made at each stage. We present EvoLM, a model suite that enables systematic and transparent analysis of LMs' training dynamics across pre-training, continued pre-training, supervised fine-tuning, and reinforcement learning. We train over 100 LMs with IB and 4B parameters from scratch, and evaluate both upstream (language modeling) and downstream considerations of both in-domain and out-of-domain generalization. Key insights highlight the diminishing returns from excessive pre-training and post-training the importance and practices of mitigating forgetting during domain-specifee continued pre-training, the crucial role of continued pre-training in bridging pre- pee-training and post-training phases, and various intricate trade-off's when configuring supervised fine-tuning and reinforcement learning. To facilitate open research and repro- ducibility, release all pre- trained and post-trained models, training datasets for all stages, and our entire training and evaluation pipeline.
mroolemm-solvingg capabilities, including
Model Suite
Datasets
Code
Model
Training
Evaluation
Sumples
Sumpies
Upetoean
(Preplexitt)
(Pass11) Passlfic
Llama architecture, (1B, 4B)
Pretrain
CPT
SFT
RL
Iodemaain
ARC
Figure 1: Overview of EvoLM, transparent model suite for studying language-mose model training dynamics across pre-training, continued pre-training (CPT), supervised fine-tuning (SFT), and reinforcement learning (RL.. The framework evaluates both upstream (language modeling) and downstream (problem-solving) performance across in-domain (e.g., math) and out-of-dommin (e.g., code, logic) settings, enabling systematic analysis of design trade-offs and scaling behaviors.
Introduction
Scaling up language models has been paradigm that enables various downstream applications [8. 52, 32]. One approach to understanding scaling- and enabling more efficient resource allocation-is is through scaling laws, which characterize the quantitative relationship between pre-training log-loss and compute [22, 27, 23, 21J. I. In part due to vast design space [34] and the complex interactions of several training phases such as pre- vreirrann and post-training [13, 70] for open- weight models [17], remains challenging to clearly identify which decisions consistently lead to reliable downstream performance gains.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Although progress has been made in understanding how models learn during training [59, 53, 43, 16, 66], accurately forecasting downstream problem-solving performance remains challenging due the vraiing--iffrence inference mismatch in auto-regressive generative models [46] and the non-smooth nature of downstream performance improvements [45]. Existing studies often rely on checkpoints with limited transparency regarding training details, which can introduce potential confounding factors, including (I) dependence on opaque analyses from post-training studies that utilize off-the-shelf base models, often without strict control over key variables such model size, pre-training data size, and data components [42, 10, 61], and (2) evaluations based on intermediate checkpoints [59, 51], which may have sub-optimal downstream performance due to incomplete learning rate decay [48, 24, 54, 67, 30], thereby complicating fair comparisons.
In this work, we establish an end-to-end development pipeline using open toolkits [1, 71, 49] and open data sources [38, 63, 55, 31] to systematically and transparently investigate language models' reasoning capabilities throughout their lifecycle, covering phases of pretraining, continued pretraining, supervised fine-tuning, and reinforcement learning. We introduce EvoLM, a model suite comprising 100+ decoder-only autoregressive LMs with IB and 4B parameters, each trained from scratch with complete learning rate decay across various configurations of model size and dataset scale. Pre-trained publicly available corpora Web [38] only, our base models achieve competitive performance on English-only language modeling tasks compared with other open- Open-weight models with significantly more pretraining compute (Table 4). For example, our 1B and 4B models, pre- trained on 320B tokens, perform competitively with TinyLlama- IB and Qwen1.5 1.5-4B respectively, despite their significantly more pre-training data (2T and 3T tokens). We evaluate both upstream language modeling performance (measured by perplexity) and downstream practical problem-solving capabilities (assessed through generative rollout performance) both in-domain (ID) math reasoning and out-of-domain (OOD) general reasoning tasks. Through extensive controlled and transparent experiments, our study addresses several critical gaps in understanding LM training dynamics, provides insights into model behaviors, and identifies open research directions in recent literature. In summary, our contributions include:
Systematic analyses of language model capabilities across their entire lifecycle-from pre-training RL post-training-wiie evaluation on reasoning-intensinn upstream cloze tasks and downstream generative tasks, considering both in-domain and out-of-domain generalization.
Open-sourcing 100+ LMs trained from scratch with 1B and 4B parameters and their training data for all stages, enabling the research community to build upon our findings.
Open-sourcing comprehensive, transparent, and reproducible training pipeline and evaluation framework, facilitating further research into scaling laws, training dynamics, and evaluating upstream and downstream capabilities of language models.
2 Experimental Settings
2.1 Training Setup
We initialize all models using the LLaMA-2 [56] architecture with 1B and 4B parameters. Our training pipeline consists of four sequential stages:
Pre-training: Conducted on FineWebEEdu [38]. Guided by the Chinchilla scaling law [23] that recommends a compute-optime ratio of approximately 20 tokens per model parameter, we pre-train models across token budgets ranging from the optimal 20x model size to 320B tokens to investigate the effects of mild over-training (> 1x Chinchilla, <16x Chinchilla) and excessive over-training (> 16x Chinchilla) on task performance.
Continued Pre-training (CPT): Performed on FineMath [2] with token budgets from 2B to 42B. To mitigate catastrophic forgetting of general-domain knowledge, also incorporate pre-training data replay strategies [25, 41, 4, 62].
Supervised Fine-Tuning (SFT): Applied to a dataset of QA pairs augmented from GSM8K [12] and MATH [20], collected from a mixture of MetaMathQA [63], OpenMathlnstruu122 [55], and
NutinaMath [31]. We filter out low-quality prompts using model correctness consistency [39], discarding samples with zero inter-model consensus.

---

Reinforcement Learning (RL): Conducted using Proximal Policy Optimization (PPO) [47], with a binary verifiable reward. The RL stage uses the same data sources SFT but ensures no overlap with the SFT dataset.
We use a compact model signature to denote the configuration of each model across training stages. For example, IB- 18--60BBT--- 1----4281-11000pp-------------
16 represents model with the following setup:
IB: A model with 1 billion parameters.
160BT: Pretrained on 160 billion tokens from FineWeb-EEd.
8+42BT: Continued pretrained with billion tokens of replayed general comaral data (Fine Web- Edu) and 42 billion tokens of domain-specific data (FineMath).
100Kepl: Supervised fine-tuned on 100K examples for 1 epoch.
100Kep16: Reinforcement learning fine-tuned on 100K examples for 16 epochs.
For all configurations, we train models with complete learning rate scheduling and only take the final checkpoints as subjects of study. More training details can be found at Section C.2.
2.2 Evaluation Protocol
Upstream Cloze Tasks These tasks assess models' language modeling capabilities via next-token prediction, without requiring conversational abilities. We evaluate pretrained and continued-pretrainn models the following datasets, reporting average O-shot accuracy across them: HellaSwag [65],
Winogrande [44]. PIQA [6], OBQA [36], ARC-Easy/Challenge [1]].
Downstream Generative Tasks These tasks evaluate models' problem-solving abilities a genera-
tive, conversational setting. We test supervised fine-tuned and RL-finetuned models on: 1) In-Domain
Tasks (math reasoning): GSM8K-Platinum [57] (a revised version of the full GSM8K [12] test
set that minimizes label noises) and MATH [20]. 2) Out-of-Domait Tasks: CRUXEval [19] (code reasoning). BGQA [28] (logical reasoning). TabMWP [35] (table reasoning). and Strategy QA [18] (commonsense reasoning). We evaluate models in a zero-shot manner by prompting them to gen- erate full solutions in response to problems and report average performance for ID) and OOD tasks.
More evaluation details including dataset descriptions, sampling parameters, and standard errors are reported in Section C.3. Evaluation metrics include:
Accuracy: We measure accuracy under four prompting schemes: 1) Pass@ 1: Temperature
A single deterministic response is generated. The problem is marked correct if this response is correct. 2) Maj@ 16: Temperature = 1. Sixteen responses are sampled, and the majority answer evaluated for correctness. 3) RM@ 16: Temperature = 1. Sixteen responses are sampled; the one with the highest ORM score evaluated for correctness. 4) Pass@ 16: Temperature responses are sampled: the problem is marked solved if any one of the responses is correct.
Sixteen
For all these settings, final answers are extracted from model outputs and compared against ground- truth solutions to determine correctness. We additionally report Correct Ratio: In the response groups that have least one correct solution, we compute the ratio of the number of correct solutions to the total number of solutions (16).
We use an outcome reward model- Serl--Ssyooo---eeee Serkk--eward--La ama-3. 3--.B...1B--BB-- 1--8B-0.. [33]-t0 assign scalar scores generated solutions, based on input problems and responses. This metric serves as proxy for solution quality.
ORM Score:
3 Scaling Studies Across Three Training Stages
3.1 Scaling Up Pre-training Compute
To quantify how varying the total amount of pre- pre-training compute affects language modeling perfor- mance, we pre-train 0.5B, IB, 4B models on token budgets ranging from 10 B up to 320 B tokens. As shown in Figure 2, performance on upstream tasks improves steadily with more pre-training tokens, but with rapidly diminishing returns beyond around 80x to 160x model size. For example, the model's s average accuracy increases from roughly 46% at 20 BT to 52% at 80 BT, yet gains shrink to
3

---

less than a percentage point when moving from 80 BT to 160 BT. The larger 4B model continues to benefit slightly longer but also plateaus by 320 BT.
Greedy
Maj@16
RM@
Pass@16
ORM (avg@16)
Correct Ratio@16
(.)
0.4
630
000
0.4
-26
03
02
80
160
320
80
360
320
80
160
320
80
1.00
320
80
160
320
80
160
320
Pretraining Tokens (B)
Figure 3: Downstream task performance number of pretraining tokens models:
-FFT: [B-BT.. 40BT, 80BT. 160BT, --0BT1-8-42BT-1000cpp
SFT+RL: 1B- (B-BT,,,, 40BT, 80BT, 160BT,
We further assess how these pre-training budgets translate to downstream capabilities for both SFT and SFT+RL models. Figure 3 shows all six met- rics ID and OOD downstream tasks from 20BT to 320BT pretraining budgets for IB models. Both SFT and SFT+RL variants exhibit strong initial gains up to 80BT, but performance saturates thereafter: For instance, ID Maj @ 16 accuracy of SFT model rises sharply from 8% at 20 BT to 15% at 80 BT, yet only inches up to 17% at 320 BT. RL yields a consistent uplift over pure SFT, but likewise shows negligible benefit from over-training beyond 80BT. Moreover, Maj @ 16, RM@ 16, and Pass @ 16 accuracies OOD tasks decrease after 160BT budget, and such degra- dation is also amplified by drop in ORM score, showing the overall generation quality decreases to a certain amount. These patterns reveal that excessively large pre-training budgets also lead to diminishing returns on downstream performance and might even cause degradation. This finding is consistent with previous work [51], which points out that scaling up pre-training does not always improve SFT, and we further complete the studies by showing that 1) such performance gain stagnation is also reflected on downstream generative reasoning tasks and 2) RL finetuning is also constrained by overtraining.
0.575
6.50
0.550
JB
Awg Acc.
0.525
0.500
0.475
0.450
0.425
Pretraining Tokens (B)
Figure 2: Upstream task performance vs. pre- training tokens models (0.5B, 1B, 4B)- (10BT, 20BT, 40BT, 8OBT, 160BT, 320BT).
even hurt LMs' performance after
Takeaway 1. Excessive general-domad pre-training does not always improve domain-specipp post-training and might even cause performance degradation on some downstream tasks (saturation happens around 80x 160x model size our study).
We further look into how model size interplays with scaling up pre-training. As Table illustrates. under fixed pre- pre-training compute budget (IB-320BT vs. 4B-80BT), the smaller IB model even outperforms the 4B model across both SFT and SFT+RL settings. When matching pre-training tokens, we see the same trend at lower budgets: at 80B tokens the IB-80BT and 4B-80BT models perform comparably, with the smaller model slightly ahead. However, once the budget rises to 160B tokens, the BB-160BT model "unlocks" its scale: For example, the 4B SFT model jumps to an ID Maj @ 16 of 26.4% (vs. 14.2% of IB counterpart) and the 4B SFT+RL model jumps to 34.8% (vs. 22.5% of IB counterpart), demonstrating that only after reaching the saturation regime pre-training does model size translate into substantial gains in post-training performance.
Takeaway 2. Under limited pre-training budgets, smaller post-trained models can even out- perform larger counterparts. Conversely, once pre-training tokens reach the saturation regime, increasing model size enables clear improvements in both in in-domain performance and OOD general- ization.

---

Acc. (SFT/ / SFT+RL)
OOD Acc. (SFT SFT+L
Base Model
Greedy
Maj@ 16
Pass @ 16
Greedy
Maj@16
Pass@16 16
Same Pretraining Compute
IB-320BT-8-42BB
14.1/20.1
16.1
36.0/494 34.2/ 43.0
28.3/28.3 24.8/28.2
24.8/29.9 52.2/60.2 23.4/29.6
54.4/62.6
4B-80BT-8+42BT
11.3/15.7
13.2/20.0
Same Pretraining Tokens
IB-8OBT-8+42BT
12.1/18.0
14.1/21.4
35.1/45.4
25.4/27.5
24.6/31.0
55.7/65.3
4B-80BT-8+42BT
11.3/15.7
13.2/20.0
44.2/43.0
24.8/28.2
52.2/60.2 23.4/29.6
IB-160BT-8+42BT
12.5/17.5
14.2/22.5
34.5/45.J
23.8/28.2
25.6/31.6 26.0/33.2
55.3/64.9
4B-160BT-8+42BT
22.0/27.8
26.4/34.8
47.6/ 58.4
29.6
57.3/ 66.2
Table 1: Comparison between IB and 4B SFT / SFT+RL models under fixed pre-training compute/- tokens.
3.2 Scaling Up Continued Pre-training Compute
We investigate the impact of continued pretrain- ing (CPT) compute by varying the total CPT tokens from 0 (no CPT) to BT, using IB- 160BT pretrained model as the base. As shown in Figure 4, increasing CPT compute gradually degrades upstream task performance, indicating catastrophic forgetting [15]. To mitigate this issue, adopt a simple "replay" strategy [25] by randomly interleaving a small amount of pre- training data during CPT. Figure 4 demonstrates that the model with 8 BT replay consistently maintains higher upstream accuracy than the no- replay baseline across all CPT budgets. We then apply SFT on the CPT models on 100K exam- ples for one epoch to investigate the impact of replay on downstream performance. Table 2 reports for each CPT mix. Pure FineMath CPT (50 BT) achieves 19.27%, whereas mix of 8 BT Fine replay with 42 BT FineMath tokens even yields better result 21.01%. %. Configurations with either too little (1.6+48.4 BT) or too much (16+34 BT) replay perform worse, highlighting that a modest replay budget (around 5%) optimally balances retention of general-domain knowledge with adaptation to downstream generative tasks.
CPT Config
Acc.
No CPT
6.04
FineMath SOBT
19.27
FineWeb 1.6BT + FineMath 48.4BT
16.21
Fin 8BT + FineMath 42BT
21.01
FineWeb 16BT FineMath 34BT
15.22
Table 2: GSM8K-Slatiigg performance (Pass @
accuracy) of pretrained model 1B-160BT contin- ued pretrained with various configurations and then finetuned using 100K SFT examples with epoch.
Pass accuracy on
" Takeaaay 3. CPT domain-spectio data induces catastrophic forgetting of pre-trained knowledge which could harm both upstream and downstream performance, while incorporating small replay budget (e.g. 5%) could effectively mitigate this degradation.
pretrained replay
wro replay replay 160
replay 1.68
Avg. Avg. ACC.
0.52
0.50
In Figure 5, we plot downstream performance of both SFT and SFT+RL. models as function of CPT budget (with fixed BT replay budget). All variants improve steadily with more domain- specific tokens up around 32 BT and then plateau by 42 BT. For instance, the ID greedy accuracy of the SFT model rises from about 5% at 2 BT to 12% at 32 BT before leveling off.
0.48
10
20
40
Total CPT tokens (B)
Figure 4: Upstream task performance vs. CPT
tokens on models:
Pretrained: 1B-160BT,
Such a trend is also observed in OOD metries.
CPT: IB- 1.--168BTT--//BT.....2 42BT),
Across the CPT range, RL finetuning consis-
CPT: 1B-160BT-0+1100T..... 50BT).
tently outperforms pure SFT; notably, without
CPT: 1B-160BT-11166-44441,, 16+344BT),
CPT, RL can even underperform SFT (as seen
5

---

in Maj@I 16, RM@ 16, and @ 16), yet the gain brought by RL tends to strengthen as CPT tokens increase.
---
SFT
SFT+RL
Greedy
Maj@16
RM@11
Pass@16
ORM (avg@16)
Correct Ratio@116
0.4
0.3
131
03
03
04
-00
04
0.28
044
415
CPT Tokens (B)
Figure 5: Downstream task performance vs. continued pre- pre-training tokens on models:
SFT: IB-160BT- 1B-160BT-++ ... 42BT) 100Kkep
SFF+RR: IB- 10-10BB--100eep--100
IB-160BT-8+(2BT, 42BT (2BT)-100Kepp 100kp-000K)
** TTkeeaay 4. Donuin-specific post-training should be supported by adequate domain-specifee CPT data: without it, SFT performance remains suboptimal and RL can even degrade such perfor- mance.
As domain-specifee CPT data increase, in-domain downstream improves and the SFT models could benefit more from RL finetuning.
Takeaway 5.
derfosrrrampeffoomanncc. performance steadily
Takeaway With sufficient specific CPT data, post-training on in-domain tasks not only improves in-domain performance but also generalizes effectively to OOD tasks.
3.3
Scaling Up SFT Compute
SFT+RL
Greedy
Maj@16
RM@16
Pass@16
ORM (avg@16)
Correct Ratio@16
6A
-006
62
03
32
6.35
6.6
04
2.6
000
-22
Epochs
Figure 6: Downstream task performance vs. number of SFT epochs for models: SFT::
SFT+RL: 1B- 1B-160BT-8+42BT--1 118
To evaluate how downstream performance responds increased SFT compute, we conduct two complementary studies using IB-160BT-8+42BB as the base model.
Varying SFT epochs. Holding SFT examples fixed 100K, finetune the base model for {1, 2, 4, 8, 16, 32) epochs. As shown in Figure 6, ID metrics increase steadily with more epochs and saturate around epochs, reflecting increased memorization of solving in-domain problems. In contrast, OOD performance peaks at 2-4 epochs before declining, indicating that over-specialization hinders generalization. These findings also validate commonly chosen SFT hyperparameter of approximately 3 epochs. Moreover, the marginal gains from downstream RL finetuning shrink on over-trained SFT models: once the model has excessively memorized the supervised data, there little room for RL to improve.

---

SFT
SFT-RL
Greedy
Maj@16
RM@
Pass@16
ORM (avg@16)
Correct Ratio@16
631
-18
000
0.4
6.4
04
3.4
100
200
100
400
100
200
3.0
400
1.00
200 100
460
100
200
300
400
100 200
300
400
100 200
300
400
Examples
Figure 7: Downstream task performance number of SFT examples for models: SFT:
100K, 150K, 400K/epl -FF++RL: IB- IB-160BT-8-42BT-(50KK 100K, 150K, ...
Varying SFT dataset size. As proposed by previous study [42]. post-training performance for downstream tasks follows power-law relationship with SFT dataset size, but the conclusion is drawn from experiments conducted on up 10K examples. We further scale that budget by varying the number of SFT examples from 50K to 400K, holding epochs fixed at one to minimize memorization. As illustrated Figure 7, performance improves monotonically with more examples, confirming that additional SFT compute consistently improves performance on in-domain tasks. However, OOD metrics fluctuate and can even decline with larger datasets. Similarly as scaling up epochs, the incremental benefit from RL diminishes as the model learns more SFT examples.
Takeaway Excessive SFT improves ID performance with diminishing returns but does not necessarily improve and can even degrade 0OD performance.
Takeaway Excessive SFT, especially overly large epochs, could limit farther RL improvements.
3.4 Scaling Up RL Compute
Similarly. to evaluate how downstream performance responds to increased RL compute, we also vary either epochs or dataset size, using incorporate O epochs/examples to indicate the SFT baseline. More experiment results and findings regarding RI. can be found at Section B.2.
base model. We additionally
Groedy
Majg16
RM@16
Passgl6
Greedl
Naj816
RMg
Pass 16
Accuracy
ORM (avg@16)
Correct Ratio@16
Accuracy
ORM (avp@166
Correct Ratio@16
-325
023
-14
-260
-15
-38
0OD
00D
13
12
4.8
16
32
4.8
32
4.
32
100200 300400
100200300400
100200300400
Epochs
Examples
(a) Performance number of RL epochs for models
(b) Performance number of RL examples for models 1 18-160973-42277-000kc.. 50K, 100K, 150K, 400K eepl.
4, 8,
16, 32).
Figure 8: Downstream task performance under different RL scales.
7

---

Varying RL epochs. We apply RL across another 100K examples (disjoint from the SFT dataset) for 10, 1, 2, 4, 16, 32) epochs. As shown in Figure 8a, for both ID and OOD tasks, greedy, Maj @ 16, and RM@ 16 accuracies peak at around 8-16 epochs and then saturates thereafter. We also notice that while the correct ratio keeps increasing, @ass@ 16 accuracy greatly degrades beyond 4 epochs, indicating that RL. primarily sharpens confidence in already-corres effectively expanding the set of solvable samples. This is also reflected by results in Table 1: For IB and 4B SFT models, Maj @ 16 accuracy could sometimes underperform greedy accuracy, indicating that low- low-quality solutions take the majority. However, after RL is conducted on the SFT models, all Maje @ 16 accuracies are higher than greedy accuracies.
outputs rather than
Varying RI dataset size. Given a fixed epoch of 8, we vary the RI. dataset size from O to 400K examples. Figure 8b shows that for both ID and OOD metrics, greedy, Maj @ 16, and RM @ 16 accuracies continue to increase from more data up to around 150-200K examples, after which gains flatten and fluctuate. In contrast, Pass @ K saturates much earlier and starts to degrade, while the correct ratio keeps increasing, similar to the finding in scaling up RL epochs. This finding is in line with observations by concurrent work [64] that similarly conclude that RL mainly boosts the confidence of existing correct outputs rather than enhancing the fundamental reasoning capabilities of LMs. We further expand this insight by illustrating the precise trade-offs for both RL epochs and dataset size. Additionally, we notice drastic performance drop at 350K and 400K examples, and training results show that during the final RI. steps, both models learn to greatly increase response length and their generations often exceed their predefined context window lengths, thus causing the performance drop. However, RL with overly large epochs is much more stable and such collapse caused by response length scaling is not observed.
Takeaway 9. RL with excessive epochs or examples improves downstream performance both ID and OOD tasks but with diminishing returns (saturation happens at 4-8 epochs 50-100K examples in our study).
18
4B
Greedy
Pass 16
0.55
0,200
0.50
" Takeaway 10. Beyond saturation regime,
0.175
RL primarily increases the probability of sam- pling high quglituaitty rollouts but does not neces- sarily improve models' fundamental reasoning capabilities.
0.150
0.45
0.125
0.30
0.700
0.675
To further investigate how to configure SFT and RL data allocation in fata-constrained scenarios, we subsample 100K examples from the entire 500 K dataset and evaluate five SFT/RL splits: (10 / 90, 30/ / 70, 50/ / 50, 70 30, 90 / 10) K and conduct either SFT RL for 4 epochs. We choose 100K because it is around the saturation regime of both ID and OOD performance (Fig- ure 8b). As shown in Figure 9, ID accuracy (greedy and Pass @ 16) increases with the pro- portion of SFT data, plateauing beyond around 70 K. whereas OOD metries are driven by RL allocation, peaking at 10K SFT (i.e. 90K RL). These trends hold across both the IB and 4B models.
0.28
0.650
0.26
0.625
10 30
50
70
90
10
30
50 70 90
SFT/RL Data Allocation (K examples)
Figure 9: Downstream task performance for (1B, 4B 1111600T88422B-(1000) ... (oKK)ep4-(900) 10K Jep4. Darker green/blue denotes data allocation to SFT/RL. The total number of post- training samples fixed 100K.
Takeaway 11. Under a constrained downstream data budget, allocating more examples to SFT maximizes in-domain guins at the expense of weaker OOD generalization, while allocating more to RL improves OOD performance.
4 Additional Studies and Discussions
Given that we find post-training interacts non-trivially with ena.rrainnng--neessttttiiin sophisticated training recipe-dooe does downstream performance scale smoothly or predictably? This section provides one example illustrating why our comprehensive study is essential to fully grasp how training
8

---

dynamics shape downstream performance in LMs, and another example where metric could correlate with downstream problem-solvee performance.
4.1 Intermediate Checkpoints May Not Be Reliable Surrogates
In reality, practitioners usually train each de- sired model through the full learning-rate sched-
Model
U Upstream
Dowastream (Greedy / Pass 16)
ule and exhaust the available pre-training data, rather than taking intermediate checkpoints final models. To mimic the real-world work- flow of training models from scratch for 20B or 40B tokens, we compare those standalone runs against the checkpoints extracted at the same token counts (20B and 40B) from a longer 160B- token pre- trainiiinng run. After each model sees 20B or 40B tokens, we further apply single epoch of SFT 100K examples to deliver basic conversational grounding, and evaluate the models on two easiest subsets of the MATH dataset.
Moth Level
Math Level
205T full 2007 inc.
46.43
2.75 17.85
13.61
46,07
2.52/11.44
1.50/ 12.64
1OST full 40BT int.
09.38
2,97 19.9/1.96
3 14,88
49.06
1.37 /9,38
2.68 /.77
Table 3: Performance on Upstream tasks and MATH (Level 1 and 2) under different pretrain- ing configurations. "xBT full"" refers to a complete pre- pre-training run BT, while "xBT int." refers to an intermediate checkpoint taken during training to 160B tokens, corresponding to x BT seen far.
As Table 3 shows, the intermediate checkpoints consistently lag behind their dedicated 20B and 40B counterparts on both upstream task accuracy and math reasoning performance. This gap arises because earlier stopping points points-captures captured before learning-rate decay and data repetition-omi the full optimization trajectory that smaller runs complete. In other words, simply slicing out a 40B-token checkpoint from longer schedule does reproduce the benefits of training model exclusively for 40B tokens.
These results caution against using such intermediate checkpoints proxies for studying and understanding fully trained smaller models. When interpreting training dynamics, it is essential to compare like-for-like runs-each with its own complete schedule-ratiee rather than relying mid-course snapshots that understate true model capability.
4.2 Correlating Downstream Task Performance with ORM Score
While perplexity across domains sometimes shows strong correlations, downstream task accuracy may not be consistently correlated, largely because post-trained models are miscalibrated and thus lower validation perplexity does not necessarily indicate better generative performance. experiments, we found that the correlation between ORM scores and downstream task accuracy presents a clear relationship. In Figure 10, we plot ORM score (avg@ @ 16) versus Maj @ 16 accuracies for all post-trained model variants starting from base model scores exhibit consistently strong predictive power, evidenced by high correlation coefficients ranging approximately from 0.62 to 0.84 across both ID and OOD tasks. While observe that the correlation low for StrategyQA, this might arise because 1) Strategy emphasizes more about commonsense knowledge rather than explicit deductive reasoning, or 2) the reward model used is less suited to the specific problem distribution of this dataset.
our
and find that ORM
OPN (900246)
GSMBK-P (2-0)221)
MATH (336
BCQA (1779
STODA (ré-g 06
TabMWP (7200733)
CRUXEval (2-01339)
-10
-20
Maj@16
Figure 10: Correlation between accuracy and ORM score across different tasks. Each subplot represents one dataset, where each point corresponds to a model variant. A dashed line indicates the linear trend, and the Pearson correlation coefficient is reported in each title.
The non-trivial correlation between ORM scores and downstream accuracies suggests that scores produced by large ORMs can serve as reliable unsupervised proxy metrics for assessing generation quality during post-training phases. For example, ORM scores can particularly useful in data- constrained scenarios where collecting sufficient high-quality test examples is challenging. ORM

---

scoring is also advantageous when direct testing is impractical, such as in tasks where final answers inherently difficult automatically extract and verify. Moreover, the generalizability of ORMs enables practitioners to train them on existing reasoning tasks and apply to other data-constraint reasoning tasks. Under such cireumstances, ORM scores enable effective validation and iterative refinement of models without the reliance extensive labeled evaluation datasets.
** Takeaway 12. ORM score could be more reliable unsupervised validation metric that helps predict downstream task performance during post-training. compared to validation loss. Notably, ORM scores from an 8B reward model correlate well with problem-solving accuracies of 1B models on many downstream reasoning tasks.
5 Concluding Remarks
In this work, systematically studied how factors such training tokens and model size influence language models' upstream and downstream performance. Our study revealed scaling trends, dimin- ishing returns from excessive training, and the importance of carefully managing domain-specifer continued pretraining to prevent forgetting. Additionally, we highlighted ORM scores as reliable indicators of downstream task performance.
acknowledge several limitations in our study. First, we focused on qualitative analyses of models up to 4B parameters. Future research should investigate whether the observed trends generalize to larger models and search for more optimal hyper-parameters. Second, our focus on reasoning-centre post-training objectives leaves unexplored dynamics for objectives like safety aligmment, instruction- following, tool-calling, and coding tasks. Lastly, our RL experiments employed only Proximal Policy Optimization (PPO) with verifiable rewards. Exploring alternative reinforcement learning methods could offer broader insights into their effeets on downstream capabilities.
Broadly, we advocate open-source research to enhance transparency, enabling better understanding, controlling, and responsibly managing machine learning models through community efforts.
10

---

References
[1] L. AL. Litgpt. http:://ggthub..cccoiggggnnggn//
2023. 2
[2] L. B. Allal, A. Lozhkov, E. Bakouch, G. M. Blázquez, G. Penedo, L. Tunstall, A. Marafioti,
H. Kydlfček, A. P. Lajarín, V. Srivastav, J. Lochner, C. Fahlgren, X.-S. Nguyen, C. Fourrier,
B. Burtenshaw, H. Larcher, H. Zhao, Zakka, M. Morlon, C. Raffel, L. von Werra, and T. Wolf.
Smollm2: When smol goes big data-centric training of a small language model, 2025. 2
[3] Bai, Bai, Y. Chu, Z. Cui, K. Dang. Deng, Y. Fan, W. Ge, Y. Han, Huang, al. Qwen technical report. arXiv preprint arXiv:2309. 16609, 2023. 17
[4] L. Bethune, D. Grangier, D. Busbridge, E. Gualdoni, M. Cuturi, and Ablin. Scaling laws for forgetting during finetuning with pretraining data injection. arXiv preprint arXiv:2502.06042, 2025.
[5] S. Biderman, H. Schoelkopf. Q. G. Anthony, H. Bradley, K. O'Brien, E. Hallahan, M. A. Khan, S. Purohit, U. S. Prashanth, E. Raff, et al. Pythia: A suite for analyzing large language models across training and scaling. In International Conference on Machine Learning, pages 2397-2430. PMLR, 2023. 17
[6] Y. Bisk, R. Zellers, R. L. Bras, J. Gao, and Choi. Piqa: sense in natural language. In Thirty-Fourth AAAI Conference on Artificial Intelligence, 2020.
Reasoning about physical common-
3
[7] D. Brandfonbrenner, N. Anand, N. Vyas, E. Malach, and S. Kakade. Loss-to-loss prediction: Sealing laws for all datasets. arXiv preprint arXiv:2411.12922 2024.
[8] T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
[9] D. Busbridge, A. Shidani, F. Weers, J. Ramapuram, E. Littwin, and R. Webb. Distillation scaling laws. arXiv preprint arXiv:2502.08606, 2025. 17
[10] T. Chu, Y. Zhai, J. Yang. S. Tong, S. Xie, D. Schuurmans, Q. V. Le, S. Levine, and Y. Ma. Sft memorizes, rl generalizes: comparative study of foundation model post-training. arXiv preprint arXiv:2501.17161, 2025. 2, 17
[11] Clark, Cowhey, O. Etzioni, T. Khot, A. Sabharwal, Schoenick, and O. Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. ArXiv, 2018.
[12] K. Cobbe, V. Kosaraju, M. Bavarian, M. Chen, H. Jun, L. Kaiser, M. Plappert, J. Tworek, Hilton, R. Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110 10. 14168, 2021. 2, 21
[13] R. Saminguez-Olmedee F. E. Dorner, and M. Hardt. Training on the test task confounds evaluation and emergence. arXiv preprint arXiv:2407.07890, 2024. 1
[14] Z. Du, A. Zeng, Y. Dong, and J. Tang. Understanding emergent abilities of language models from the loss perspective. arXiv preprint arXiv:2403.15796, 2024. 17
[15] R. M. French. Catastrophic forgetting in connectionist networks. Trends in cognitive sciences, 3(4):128-135, 1999.
[16] S. Y. Gadre, G. Smyrnis, V. Shankar, S. Gururangan, M. Wortsman, R. Shao, J. Mercat, A. Fang, J. Li. S. Keh, et al. Language models scale reliably with over-training and on downstream tasks, arXiv preprint arXiv:2403.0885.
2024. 2, 17
[17] K. Gandhi, A. Chakravarthy, A. Singh, Lile, and N. D. Goodman. Cognitive behaviors that enable self-improving reasoners, or, four habits of highly effective stars, 2025.
11

---

[18] M. Geva, D. Khashabi, E. Segal, T. Khot, D. Roth, and J. Berant. Did aristotle use a lap- top? a question answering benchmark with implicit reasoning strategies. Transactions of the Association for Computational Linguistics, 9:346-361, 2021. 3, 21
[19] A. Gu, B. Rozière, H. Leather, A. Solar-Lezama, G. Synnaeve, and S. I. Wang. Cruxeval: A benchmark for code reasoning, understanding and execution. arXiv preprint arXiv:2401 .03065, 2024. 3, 21
[20] D. Hendrycks, C. Burns, S. Kadavath, A. Arora, S. Basart, Tang, D. Song, and J. Stein- hardt. Measuring mathematical problem solving with the math dataset. arXiv preprint arXiv:2103.03874, 2021. 2, 21
[21] D. Hernandez, T. Brown, т. Conerly, N. DasSarma, D. Drain, S. EI-Showk, N. Elhage, Z. Hatfield-Dodds, T. Henighan, T. Hume, et al. Scaling laws and interpretability of learning from repeated data. arXiv preprint arXiv:2205. 10487, 2022. 1
[22] J. Hestness, S. Narang, N. Ardalani, G. Diamos, H. Jun, H. Kianinejad, M. M. A. Patwary, Yang. and Y. Zhou. Deep learning scaling predictable, empirically, arXiv preprint arXiv:1712.00409 2017.
[23] Hoffmann, S. Borgeaud, A. Mensch, E. Buchatskaya, T. Cai, E. Rutherford, D. d. L. Casas,
L. A. Hendricks, J. Welbl, A. Clark, et al. Training compute-optimal large language models. arXiv preprint 2iii:22033115566 2022. 2, 17
[24] S. Hu, Y. Tu, X. Han, C. He, G. Cui, X. Long, Z. Zheng. Y. Fang, Y. Huang, W. Zhao, et al. Minicpm: Unveiling the potential of small language models with scalable training strategies. arXiv preprint arXiv:2404.06330 066395 2024. 2
[25] A. Ibrahim, B. Thérien, K. Gupta, M. L. Richter, Q. Anthony, T. Lesort, E. Belilovsky, and
I. Rish. Simple and scalable strategies to continually pre-train large language models. arXiv preprint arXiv:2403.0877 2024. 2,5
[26] J. Jin, V. Syrgkanis, S. Kakade, and H. Zhang. Discovering hicrarchical latent capabilities of language models via causal representation learning, 2025. 17
[27] J. Kaplan, S. McCandlish, T. Henighan, T. B. Brown, B. Chess, R. Child, S. Gray, A. Rad- ford, J. Wu, and D. Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001 080661 2020. 1, 17
[28] M. Kazemi, Q. Yuan, D. Bhatia, N. Kim, X. Xu, V. Imbrasaite, and D. Ramachandran. Boardgameqa: A dataset for natural language reasoning with contradictory information. Advances in Neural Information Processing Systems, 36:39052-39007 2023. 3, 21
[29] W. Kwon, Z. Li, S. Zhuang, Sheng, L. Zheng, C. H. Yu, J. E. Gonzalez, H. Zhang, and I. Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles, 2023.
21
[30] H. Li, W. Zheng, J. Hu, Q. Wang, H. Zhang, Z. Wang, S. Xuyang, Y. Fan, S. Zhou, X. Zhang, et al. Predictable scale: Part i-optimal hyperparameter scaling law in large language model pretraining. arXiv preprint MiXXv:5503.0411. 2025. 2
[31] LII Beeching, L. Tunstall, B. Lipkin, R. Soletskyi, S. C. Huang, K. Rasul, Yu, A. Jiang, Shen, 7. Qin, B. Dong, I.. Zhou, Y. Fleureau, G. Lample, and S. Polu. Numina- math. [https:/ //gggs/gghgig co /tt///AANO///mmiaaaaaaaaaaaaa project- numina/ haapnngreammmm-oossseeeeee httpg//rweppprreeooooo////a//rre 2024. 2,
(https: / com/ dataset. pdf),
[32] A. Liu, B. Feng, B. Xue, B. Wang, B. Wu, C. Lu, C. Zhao, C. Deng, C. Zhang, C. Ruan, et al. Deepseek-v3 technical report. arXiv preprint a134 2024.
[33] C. Y. Liu, L. Zeng. Liu, R. Yan, He, C. Wang, S. Yan, Y. Liu, and Zhou. Skywork-reward: Bag of tricks for reward modeling in 1lms. arXiv preprint 1:Xv::2410.1845,, 2024. 3
12

---

[34] E. Liu, A. Bertsch, L. Sutawika, L. Tjuatja, P. Fernandes, L. Marinov, M. Chen, S. Singhal, C. Lawrence, A. Raghunathan, et Not-just-scaling laws: Towards a better understanding of the downstream impact of language model design decisions. arXiv preprint arXiv:2503.0366) 2025.
[35] P. Lu, I.. Qiu, K.-W. Chang, Y. N. Wu, S.-C. Zhu, T. Rajpurohit, P. Clark, and A. Kalyan. Dynamic prompt learning via policy gradient for semi-structured mathematical reasoning. arXiv preprint arXiv:2209. 14610, 2022. 3, 22
[36] T. Mihaylov, P. Clark, T. Khot, and A. Sabharwal. Can a suit of armor conduct electricity? a new dataset for open book question answering. In EMNLP, 2018. 3
[37] N. Muennighoff, A. Rush, B. Barak, T. Le Scao, N. Tazi, A. Piktus, S. Pyysalo, T. Wolf, and C. A. Raffel. Scaling data-constrained language models. Advances in Neural Information Processing Systems, 007550358-03776 2023. 17
[38] G. Penedo, H. Kydliček, A. Lozhkov, M. Mitchell, C. A. Raffel, L. Von Werra, T. Wolf, et al. The fineweb datasets: Decanting the web for the finest text data at scale. Advances Neural Information Processing Systems, 37:308 1-30849, 2024. 21
[39] Z. Qi, M. Ma. J. Xu, L. L. Zhang, F. Yang, and M. Yang. Mutual reasoning makes smaller Ilms stronger problem-solverss, arXiv preprint Siti::20080019955 2024. 2
[40] Z. Qin, Q. Dong, X. Zhang, L. Dong, X. Huang, Z. Yang, M. Khademi, D. Zhang, H. H. Awadalla, Y. Fung. et al. Scaling laws of synthetic data for language models. arXiv preprint arXiv:2503. 19551, 2025. 17
[41] H. Que, J. Liu, G. Zhang, C. Zhang, X. Qu Y. Ma, Duan, Bai, J. Wang, Y. Zhang, et al. D-cpt law: Domain -spacii continual pre-training scaling law for large language models. Advances in Neural Information Processing Systems, 37:90318-90354, 2024. 2, 17
[42] M. Raghavendra, V. Nath, and S. Hendryx. Revisiting the superficial alignment hypothesis. arXiv preprint arXiv:2410.03717, 2024. 2, 7, 17
[43] Y. Ren and D. J. Sutherland. Learning dynamics of Ilm finetuning. 1XXi:2247711099 2024.
arXiv preprint
[44] K. Sakaguchi, R. L. Bras, C. Bhagavatula, and Y. Choi. Winogrande: An adversarial winograd schema challenge at scale. Communications of the ACM, ((())99-1066, 2021.
[45] R. Schaeffer, B. Miranda, and S. Koyejo. Are emergent abilities of large language models mirage? Advances in Neural Information Processing Systems, 36:55565 -55581, 2023. 2
[46] F. Schmidt. Generalization in generation: A closer look at exposure bias. arXiv preprint
arXiv:1910.00292. 2019. 2
[47] J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv: 7707066377 2017. 3
[48] Y. Shen, M. Stallone, M. Mishra, G. Zhang, S. Tan, A. Prasad, A. M. Soria, D. D. Cox, and R. Panda. Power scheduler: A batch size and token number agnostic learning rate scheduler. arXiv preprint arXiv:2408. 13359, 2024.
[49] G. Sheng, C. Zhang, Z. Ye, X. Wu, W. Zhang. R. Zhang, Y. Peng, H. Lin, and C. Wu. Hybridflow: flexible and efficient rihf framework. arXiv preprint arXiv: 2409.19256 2024. 2
[50] C. Snell, E. Wallace, D. Klein, and S. Levine. Predicting emergent capabilities by finetuning. arXiv preprint arXiv:241 16003, 2024. 17
[51] J. M. Springer, S. Goyal, K. Wen, T. Kumar, X. Yue, S. Malladi, G. Neubig, and A. Raghunathan. Overtrained language models are harder to fine-tune. arXiv preprint arXiv:2503. 19206, 2025.
2, 4, 17
13

---

[52] G. Team, R. Anil. S. Borgeaud, J.-B. Alayrac, J. Yu. R. Soricut, J. Schalkwyk, A. M. Dai, A. Hauth, K. Millican, et al. Gemini: family of highly capable multimodal models. arXiv preprint arXiv:2312. 1805, 2023.
[53] K. Tirumala, A. Markosyan, L. Zettlemoyer, and A. Aghajanyan. Memorization without overfitting: Analyzing the training dynamics of large language models. Advances in Neural Information Processing Systems, 15:38274-38290 2022. 2
[54] H. Tissue, V. Wang, and L. Wang. Scaling law with learning rate annealing, arXiv preprint arXiv:2408. 1029, 2024.
[55] S. Toshniwal, W. Du, I. Moshkov, B. Kisacanin, A. Ayrapetyan, and I. Gitman. 0penmathinstruct-2: Accelerating ai for math with massive open-source instruction data. arXiv preprint arXiv:2410.01560, 2024. 2,
[56] H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozière, N. Goyal, E. Hambro, F. Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302. 13971, 2023. 2, 17
[57] J. Vendrow, E. Vendrow, S. Beery, and A. Madry. Do large language model benchmarks test 2025. 3, 21
reliability? arXiv preprint
[58] Y. Wang. Q. Yang, Z. Zeng, L. Ren, L. Liu, B. Peng, H. Cheng, X. He, K. Wang, J. Gao, et al. Reinforcement learning for reasoning in large language models with one training example. arXiv preprint arXiv:2504.20571, 2025. 18
[59] M. Xia, M. Artetxe, C. Zhou, X. V. Lin, R. Pasunuru, D. Chen, L. Zettlemoyer, and Stoyanov. Training trajectories of language models across scales. arXiv preprint arXiv:2212.09800, 2022.
2
[60] A. Yang, B. Zhang, B. Hui, B. Gao, B. Yu, Ci, D. Liu, Tu, Zhou, J. Lin, et al. Qwen2. 5-math technical report: Toward mathematical expert model via seff-improvement. arXiv preprint arXiv:2409. 12122, 2024. 21
[61] E. Yeo, Y. Tong, M. Niu, G. Neubig, and X. Yue. Demystifying long chain-of of-thought reasoning in llms. arXiv preprint arXiv:2502.03372 2025. 2, 17
[62] Ç. Yıldız, N. K. Ravichandran, N. Sharma, M. Bethge, and B. Ermis. Investigating continual pre- training in large language models: Insights and implications. arXiv preprint arXiv:2402.17400. 2024. 2
[63] L. Yu, W. Jiang, H. Shi, J. Yu, Z. Liu, Y. Zhang, J. T. Kwok, Z. Li, A. Weller, and W. Liu. Metamath: Bootstrap your own mathematical questions for large language models. arXiv preprint arXiv:2309. 12284, 2023. 2, 21
[64] Y. Yue, Z. Chen, R. Lu, A. Zhao, Z. Wang, S. Song, and G. Huang. Does reinforcement learning really incentivize reasoning capacity in Ilms beyond the base model? arXiv preprint arXiv:2504. 13837, 2025. 8, 17
[65] R. Zellers, A. Holtzman, Y. Bisk, A. Farhadi, and Y. Choi. Hellaswag: Can a machine really finish your sentence? arXiv preprint 19iv:1905.07830, 2019. 3
[66] B. Zhang, Z. Liu, C. Cherry, and O. Firat. When scaling meets Ilm finetuning: The effect of data, model and finetuning method. arXiv preprint arXiv:2402.17192. 2024. 2, 17
[67] H. Zhang, D. Morwani, N. Vyas, J. Wu, D. Zou, U. Ghai, D. Foster, and S. M. Kakade. How does critical batch size scale in pre-training? In The Thirteenth International Conference Learning Representations, 2025. 2
[68] P. Zhang, G. Zeng, T. Wang, and W. Lu. Tinyllama: An open-source small language model. arXiv preprint arXiv:2401.0223
2024. 17
[69] S. Zhang, S. Roller, N. Goyal, M. Artetxe, M. Chen, S. Chen, C. Dewan, M. Diab, X. Li,
x. V. Lin, et al. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.00 10255,0100000 01068, 2022. 17
14

---

[70] R. Zhao, A. Meterez, S. Kakade, C. Pehlevan, S. Jelassi, and E. Malach. Echo chamber: RI post-training amplifies behaviors learned pretraining. arXiv preprint arXiv:2504.07912, 2025.
1.
[71] Y. Zheng, R. Zhang, J. Zhang, Y. Ye, Z. Luo, Z. Feng, and Y. Ma. Llamafactory: Unified efficient fine-tuning of 100+ language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations), Bangkok, Thailand, 2024, Association for Computational Linguistics. 2
[72] C. Zhou, P. Liu, P. Xu, S. Iyer, J. Sun, Y. Mao, X. Ma, A. Efrat, P. Yu, L. Yu, et al. Lima: Less more for alignment. Advances in Neural Information Processing Systems, 36:55006-55021, 2023. 17
15

---

Appendices
Related Work
17
Additional Experiment Results
17
B.1 Observational Comparison of Pre-trained Models
17
B.2 Scaling Up RL compute...
18
B.3 Post-trained Models are Miscalibrated for Language Modeling Tasks
19
Reproducibility
20
C.1 Model Architectures
20
C.2 Training Details
20
C.2.1 Hyperparameters
20
C.2.2 SFT/RL Template
................................
20
C.2.3 Training Data
21
C.3
Evaluation Details
21
C.3.1
Benchmarks and Sampling Parameters
C.3.2 Statistical Significance
22
C.3.3 Example Model Outputs
24
16

---

A Related Work
Studying Language Models Across Training Stages. Recent research has explored how different training stages shape downstream capabilities of language models. Observations by recent study [16] indicate that extensively pre-trained language models scale reliably on downstream tasks, though their conclusions predominantly address pre-trained models evaluated via top-I error, leaving open questions regarding models subjected to additional post- training. In contrast, "catastrophic overtraining" is identified by recent work [: Prolonged pre-training beyond certain point actually impairs downstream fine-tuning by increasing sensitivity to parameter updates and exacerbating forgetting. Complementing this, researchers [66] have derived multiplicative joint scaling law for fine-tuning, showing performance gains depend more on scaling model size than pretraining data, with optimal approaches depending critically on task and data regimes.
Pre-training Drives Post-training. Recent success in LM post- post-training has led to research investi- gating how post-training is affected by pre-training. Recent research [26] applies causal inference on observational data, finding general upstream capabilities strongly correlate with base model FL.Ps, influencing specialized abilities like math reasoning. Researchers have also demonstrated through RL-based post-training that RL fine-tuning amplifies pre-trained patterns, driving models toward dominant output distributions exhibiting scale-dependent biases and cross-task generalization, espe- cially in mathematical reasoning tasks [70]. Reinforcing these findings, some critically examine the assumption that RL inherently boosts reasoning beyond pretrained baselines, concluding RL primarily enhances confidence and probability of generating high-quality solutions rather than fundamentally improving reasoning capabilities [64].
Scaling Laws for Language Models. Early scaling work [23, 27] established fundamental relation- ships linking training loss to model size, data quantity, and compute. Recent studies have extended this framework in several ways. A dual-axis scaling law has shown reliable loss predictions even in highly over-trained regimes, significantly beyond traditional optimal compute points [16]. Addition- ally, new quantitative models predict emergent behaviors in model accuracy either through explicit loss thresholds or by probing with targeted finetuning [50, 14]. Cross-distribution transferabilityy has also been modeled, allowing accurate extrapolations of loss curves between different datasets from minimal pilot data [7]. Further refinements address data-limited contexts, deriving optimal epoch allocation when unique training data is scarce [37], and revealing similar scaling patterns for synthetic data with clear diminishing returns [40]. Moreover, scaling laws now capture continual pre- trainininn dynamics, guiding mixing domain- specific and general data, and quantifying forgetting effects during domain adaptation with replay data [41]. Finally, research into compute allocation has developed scaling relationships specifically for distillation, determining precisely when distillation methods surpass direct pre-training efficiency [9].
Post-training for Reasoning. Recent research has investigated the impact of post-training strategies on the reasoning capabilities of LLMs. One study challenges the "Superficial Alignment Hypothesis" [72], demonstrating that SFT post-training performance scales with the number of fine-tuning examples, akin to pre-training scaling laws [42]. Moreover, RL post-training has been shown to amplify behaviors acquired during pre-training, particularly in tasks requiring advanced mathematical reasoning and coding [70]. A comparative study indicates that while SFT tends to memorize training data, RL foster better generalization [10]. Investigations into the mechanics of reasoning have demystified long ofain-of-ttought Iearned through RL, identifying factors that enable the generation of extended reasoning trajectories [6]]. Conversely, critical examination questions whether RL truly incentivizes reasoning capacities beyond what is already learned during pre-training, suggesting that RL may not elicit fundamentally new reasoning patterns [64].
B
Additional Experiment Results
B.1
Observational Comparison of Pre-trained Models
Table I compares our pre-trained models against several ween- models including OPT [69], Pythia [5], Tama [68], Llama [56], and Qwen [3]. Our models, pretrained on a significantly smaller number of tokens (320B tokens for our 1B and 4B models), demonstrate competitive performance with other state-of-the-art small models such as Tinyl.lama- IB (trained on 2T tokens) and Dwen1.5-4B (trained on 3T tokens).
17

---

Model Name
Tokens
H/S
W/G
PIQA
OBQA
ARC-E
ARC-C
Avg.
OPT 1.3B
300B
53.65
59.59
72.36
33.40
50.80
29.44
49.87
Pythia IB
300B
47.16
53.43
69.21
31.40
48.99
27.05
46.21
Pythia 1.4B
300B
52.01
57.38
70.95
33.20
54.00
28.50
49.34
TinyLlama IB
2T
61.47
59.43
73.56
36.80
55.47
32.68
53.23
Llama3.2 IB
9T
63.66
60.46
74.54
37.00
60.48
35.75
55.31
Qwen3 1.7B
36T
60.46
61.01
72.36
36.80
69.91
43.26
57.30
42.25
$1.30
67.85
32.80
$4.80
29.61
46.44
40B
47.53
$4.62
69.59
36.20
58.08
30.29
49.38
IB (ours)
80B
51.05
53,59
70.78
37.20
62.71
35.92
51.88
160B
52.30
53,99
71.71
36.60
63.09
36.09
52.30
320B
53.86
53.51
71.93
37.20
62.29
36.18
52.49
Pythia 6.9B
300B
63.89
61.17
76.39
37.20
61.07
35.15
55.81
OPT 6.7B
300B
67.18
65.35
76.50
37.40
60.06
34.73
56.87
Qwen wwen1.5.. 4B
3T
71,45
64.09
77.10
39.60
61.41
39.51
58,86
Qwen2.5 3B
18T
73.61
68.51
78.89
42.00
73.23
47.18
63.90
Qwen3 4B
73.71
70.64
77.75
41.00
76.22
51.88
65,20
Ilama 3.2
73.63
69.69
77.53
43.20
71.76
45.90
63.62
48.84
54.38
69.91
35.80
59.68
32.68
50.22
4B (ours)
160B
56.49
55.88
72.63
40.20
66.67
39.93
55.30
320B
61.38
57.46
74.27
41.80
67.55
39.16
56.94
Table 4: Upstream benchmark comparison across various small-size LMs. All scores are percentages. We highlight our base model performance in bold font, models with performance at comparable scale in red, and excessively over-trained models with similar performance in green.
Specifically, despite TinyLlama- IB and Qwen1.5-44 1.54BB models being trained with 6.25x and 9.38x more tokens respectively, our 1B and 4B models achieve similar slightly better results across standard benchmarks like HellaSwag (H/S), Winogrande (W/G), PIQA, OBQA, ARC-Easy (ARC-E), and ARC-Challenge (ARC-C). This empirical observation is consistent with experimental findings in Section 3.1, highlighting diminishing returns from excessive pretraining: beyond a certain optimal compute threshold, additional pretraining leads minimal incremental gains in general domain upstream task performance.
B.2 Scaling Up RL compute
To further look into effective practice for scaling up RL. compute, we plot results in "example-epochs"
units (#examples x #epochs, in 10y) Figure 11. We use the same configurations as Section 3.4. Under a fixed compute budget. allocating more epochs on moderate dataset (e.g.. 100K 800K example-epochs) typically yields higher ID and OOD performance than spreading compute over larger dataset with fewer epochs, and RL with excessive training examples could sometimes lead to collapsed performance due to overly long and unfinished responses (shown by the crosses Figure 11 and response length in Figure 12), while we do not observe such problems when conducting RL with excessive training epochs (shown in Figure 13). This demonstrates that deeper policy optimization sample is more cost-effective than broader data coverage for RL scaling, which is consistent with findings proposed by [58] showing that RI using even only one training example could be effective in incentivizing the mathematical reasoning capabilities of LLMs.
18

---

Baseliee
Ifoed 1OOK examples)
RI (fbed epechs)
Greedy
Maj@16
RM@
Pass@16
ORM (avg@16)
Correct Ratio@16
024
0.05
-160
038
0.23
632
776
633
3.5
0.20
025
Compute (x10* example-epochs)
Figure 11: Downstream task performance RL compute. A cross mark indicates models that tend to generate responses longer than their context window limits.
response_length/ment
response_ lepponselengtt/mman
900
800
800
700
600
600
500
400
Step
Whwa
500
1k
1.5k
2k
2.5k
3k
500
1k
1.5k
2k
2.5k
3k
Figure 12: Response length versus training step when tuning 400Kep8.
Figure 13: Response length versus training step when tuning 1B-16003T-&+422T110000-, 100Kep32.
B.3 Post-trained Models are Miscalibrated for Language Modeling Tasks
Our upstream evaluations indicate that post-trained LMs exhibit significant miscalibration when assessed through validation PPL. We evaluate PPL on the validation set (disjoint from the training set) for each post-trained model. As illustrated Figure 14, we observe negligible correlations between validation perplexity and downstream task accuracy across various datasets. Specifically, the Pearson correlation coefficients remain close to zero, reinforcing that low perplexity does not reliably predict enhanced generative reasoning performance. This contrasts sharply with the strong predictive capability exhibited by ORM scores, as discussed in Section 4.2. While validation perplexity is conventionally used monitor model quality, it is insufficient for post-training phases, particularly when evaluating generative reasoning tasks. In practice, relying solely on perplexity as a validation metric could misguide resource allocation decisions during training.
0C0
Regnession
Porpbokts
GOMAI-P (rfe6 019)
MATH (ffm6.21))
RCQA (ele0.000)
STODA (.00
MWP (rle6 000)
CRUXEVAl (P] (22000011 0011
04
0.1
02
5.30
0.35
0,475
0.500
0.525
0.500
0.35
0.30
0.00
0.30
3.55
Maj@16 Acc.
Figure 14: Correlation between accuracy and validation PPI across different tasks. Each subplot represents one dataset, where each point corresponds post-trained model variant. A dashed line indicates the linear trend, and the Pearson correlation coefficient is reported in each title.
19

---

C Reproducibility
C.1 Model Architectures
We show model architecture details for 0.5B, 1B and 4B models in Table 5.
Model Size
Hüäden Size
Intermodiate Size
Vocab Size
Contest Length
Heads
Layers
Query Groups
0.58
3216 4806
33000
2048
32
IR
2048
52000
2048
32
22
4095
7792
32000
2048
32
28
Table 5: Model architecture details.
C.2 Training Details
C.2.1 Hyperparameters
Hyperparameters for pretraining/continued pretraining, SFT, and RL are shown in Table 6, Table 7, Table 8. respectively. We use the Adam optimizer and up to 32 NVIDIA H100 80GB HBM3 GPUs for all training stages. For pretraining, continued pretraining, and SFT, we use standard warmup-cosine-deaay strategy for the learning rate schedule. For RL, we apply warmup-constant learning rate schedule.
0.5B
1B
4B
precision
bf16-mixed
precision
bí1 bf16-mixed
precision
6
global_ bat__aathhhhheee size
512
global_hatch_se batch_atciee size
512
global bat___athh___aee size
1024
max_seq max_seq_lengee
2048
max_seq_lengie length
2048
max_seq se__se__geegghhhhh
2048
Ir warmup ratio
Ir warmup ratio
0.1
warmup ratio
0.1
max norm Ir
max norm Ir min_Ir weight_decay
1
max norm Ir min_Ir weight_decay
I
0.00025
0.0002
0.00015
min_Ir
0.000025
0.00002
0.000015
weight_decay
0.1
0.1
0.1
betal
0.9
betal
0.9
betal
0.9
beta2
0.95
beta2
0.95
beta2
0.95
epoch
epoch
epoch
Table 6: Hyperparameters for pre-training/comiineed pre-traing pre-training.
1B
4B
cutoff
2048
cutoff len
2048
batch_sizi
128
batch_size
256
learning rate rate
0.00001
learning rate
0.0000075
Ir scheduler _tpp
cosine 0.1
Ir schhdullr type
cosine
warmup ratio
warmup v
0.1
Table 7: Hyperparameters for supervised finetuning.
C.2.2 SFT/RL Template
We use the following template for SET and RL tuning:
Human: {query}
Assistant: [response]
20

---

1B
4B
actor Ir critic_ Ir
2.00E-06
actor Ir
1.00E-06
2.00E-05
critic_It kl
1.00E-05
0.0001
0.0001
batch_sizz
1024
wain_batch. size
2048
max_prompt_length
1024
max_prompt_length
1024
max_response_leng
1024
max_response_lengy
1024
ppo_mini_batee sas__bathh_si__ size
1024
ppo_mini_batch_
2048
ppo_micro_bateed aatch_size_perr__e
32
ppo_micro_baae
settio_bahhssseeeeeeeee sine__ssie__er_gpu
16
log _es_prob__iieee micro_ microo_btchhhat_taii batch size_per__g___ gpu
64
se__pro__mm_ batch_aahhhh__ee pize_per__P_
32
warmup_ste steps_ratte
0.1
warmup_ steps_tetti
0.1
Table Hyperparametees for reinforcement learning (PPO).
C.2.3 Training Data
FineWeb-Edu [38]: An extensive educational dataset sourced from web content, specifically designed for pretraining language models on high-quality academic and educational text. There are ~ ~13 trillion tokens in total.
curated dataset of mathematical texts, problems, and solutions, intended to enhance language models' mathematical knowledge. There are ~50 billion tokens in total.
FineMath [38]:
Open Instruct2 [55], MetaMathQA [63], NuminaMath [31]: Instruction-tuning datasets containing mathematical questions paired with step-by-step solutions and explanations, designed improve the mathematical reasoning capabilities of LLMSs. The responses corresponding the
prompts from these datasets are collected by prompting the Qwen2.5- 5-re22.-7BB-- 7B-Math- -a-ttut--nnstuuct model [60].
C.3 Evaluation Details
C.3.1 Benchmarks and Sampling Parameters
For all test datasets and all models, directly ask the models the corresponding questions applying the same prompt template used for SFT/RL. We set the temperature to 0 for greedy decoding and for decoding with randomness (the number of generations being 16), and set the repetition penalty to 1.1. We use the vLLM framework [29] for inference. Details of each test dataset are as follows.
MATH [20] is large-scale benchmark designed to evaluate mathematical reasoning. It contains 12,500 challenging problems sourced from math competitions, categorized into seven topics including Algebra, Geometry, Calculus, and Number Theory, and divided into 5 difficulty levels. Each problem requires generating detailed, step-by-step solutions rather than simple numerical answers, emphasizing comprehensive reasoning skills and logical deduction.
GSM8K-PPatinum [57]
manually cleaned and denoised version of GSM8K [12] which is a math benchmark that consists of 8.5K high-quality. linguistically diverse grade-school math word problems designed for multi-step reasoning (2 steps). Solutions involve elementary arithmetic
operations and require concepts beyond early algebra. Its test set contains 1319 unique problems.
BoardgameQA [28] is logical reasoning benchmark designed to evaluate language models' ability to reason with contradictory information using defeasible reasoning, where conflicts are resolved based on source preferences (c.g., credibility or recency). Its set contains 15K unique problems.
CRUXEval [19] is a benchmark for evaluating code reasoning, understanding, and execution, featuring 800 Python functions (3-13 lines) with input-output pairs for input and output prediction tasks. Given a function snippet and an input example, LLMs are tasked to generate the corresponding outputs. Its test set contains 800 unique problems.
StrategyQA [18] is commonsense question-answerte benchmark designed for multi-hop reasoning where the necessary reasoning steps are implicit and must be inferred using strategy. Each of the 2,780 examples includes a strategy question, its step-by-step decomposition, and supporting Wikipedia evidence.
21

---

TabMWP [35] is benchmark introduced to evaluate mathematical reasoning over tabular data. It contains around 38,000 math word problems, each associated with relevant tables, spanning diverse mathematical reasoning types like arithmetic operations, comparisons, and aggregation tasks.
C.3.2 Statistical Significance
In Section C.3.2, we show the standard errors model performance on ID and OOD tasks for all main results in Section 3, encompassing Maj@ 16, Pass@ 16, RM@ 16, ORM scores, and correct ratios. The results are obtained from four experiments with different random seeds.
22

---

Table 9: Standard errors of main results for scaling up pretraining, (Section 3.1), CPT (Section 3.2), SFT (Section 3,3), and RL (Section 3.4). Statistics of Maj@ 16. 16.
Cancer Redkettt
E30TTS
SENETION
$10-3-000
0100anny
ES50000
8100500
AME NENWOOD
000
250000
024000
0 K310000
TONDON
$33002
3.2000
536001
SROW
514004
E300)
2-035
$3000
511071
830002
$10000
2300S
11000
$30000
4383200
14000000
450000
ALO00060
08000
Coreet Masteele
Soila tyo7
W0000011
453
4100
1915
410483
4003
(B3 mmghti
DERACTIO
10
02400000
0011000
6
16, 16, RM@ 16, and Correct Ratio 16 are in percentages.
023000
MASSO
OImES
GIBEO
Sigetto
010
015311
0155
01228
41520
6820
012821
68
0362
0323
0008
03028
0,5
01525
I IHE MIDE CNNEIIINN INEE See
HIE WIDE SUNCH (Rl
HIEE WIDE oo
100 M MESE LORa 1O vERU
SE-EE SES XESOI
HOST BEGOI
SINE SETERU unseed
HAREE CONESH
M HIRE HIDE INISH
AIEE
3 H0D
Pass@
23

---

C.3.3
Example Model Outputs
We show 3 ouputt examples. The outputs are collected from the model 1B- 13-160BT-8+42BT 100Kepl-100Kep8.
(GSMSKPlatinum)
[Model Input]
Human: While on vacation Bali, Thea bought a hat from a craftsman worth $70. she gave the craftsman four $20 bills, how much change did she get?
Assistant:
[Model Output]
To determine how much change Thea got, need follow these steps: Calculate the total cost of the hat. 2. Subtract the amount she paid for the hat from the amount she received.
First, the cost of the hat $70. She gave the craftsman four $20 bills. Therefore, the total amount she gave is: 4 x 20 80 dollars.
Next, we subtract the amount she paid for the hat from the total cost of the hat: 70
Performing the subtraction: 70 80 -10 So, Thea got 10 dollars in change. The final answer is: 10 dollars. The change Thea got is 10 dollars. Therefore, the answer is: 10
dollars. The final answer is: 10 dollars.
(CRUXEval) [Model Input]
Human: You are given the following function:
def f (array):
new new_array = array copy
new_array reversed (new_aree (new_array)
return [x*x for x in new_array]
You are also given the following input (the order of the input is the same as the order of the input variables in the function):
[1, 2, 1]
With the given function and input, what would be the output? Assistant:
[Model Output] To find the output of the function f, need to apply the function step by step. The function starts with an array called "array" and copies its elements to a new array called mnww_arayy Then, it reverses the order of the elements in "new_arre" Finally, it returns a list containing the squares of each element "hew_array" The given input is:
[1, 2, 1]
So. when we apply the function, we get:
f([L, 2, 1]) [12,22 22 12]
Now, the output is:
[14.1]
Thus, the output is: [1,
(BoardGameQA)
[Model Input]
Human: Solve the following problem. Your final answer should "True", "False", or "Uncer- tain".
few players are playing a boardgame. The current state of the game is as follows. The ant hides the cards that she has from the owl. The chihuahua borrows one of the weapons of the worm. The husky reveals a secret to the worm. The owl has six friends. The worm has 8 friends. The worm has football with radius of 16 inches. And the rules of the game are as follows. Rulel: worm has a football that fits 26.7 x 34.4 x 38.5 inches box, then the worm

---

does not swear to the german shepherd. Rule2: If the husky reveals a secret to the worm and the chihuahua borrows one of the weapons of the worm, then the worm manages to convince the monkey. Rule3: If the owl has fewer than nine friends, then the owl shouts at the dolphin. Rule4: Here an important piece of information about the worm: it has more than friends then does not swear to the german shepherd for sure. Rule5: If you are positive that you saw one of the animals dances with the bee, you can be certain that it will not manage to convince the monkey. Rule6: Be careful when something manages to convince the monkey but does not swear to the german shepherd because in this case it will, surely, hide her cards from the mouse (this may may not be problematic). Rule7: If at least one animal shouts at the dolphin, then the worm does not hide her cards from the mouse. Rule5 is preferred over Rule2. Rule6 is preferred over Rule7. Based on the game state and the rules and preferences, does the worm hide the cards that she has from the mouse? Assistant:
[Model Output]
To determine whether the worm manages to convince the monkey, we need to follow the given rules and preferences step by step.
1. *ule I* If the worm has a football that fits in a 26.7 x 34.4 x 38.5 inches box, then the worm does swear the german shepherd.
This rule is satisfied.
*ulue 2* If the husky reveals secret to the worm, then the worm manages convince the monkey.
This rule is satisfied.
3. **Rule ***; If the owl has fewer than nine friends, then the owl shouts at the dolphin.
-This rule is not satisfied.
4. **Rule 4+ Here is an important piece of information about the worm: if it has more than 2
friends then does not swear to the german shepherd for sure.
This rule is satisfied.
**Rule 5* If you are positive that you saw one of the animals dances with the bee, you can be certain that it will not manage to convince the monkey.
This rule is satisfied.
6. **Rule 6**; Be careful when something manages to convince the monkey but does not swear to the german shepherd because this case it will, surely, hide her cards from the mouse.
This rule is not satisfied.
Thus, the worm manages to convince the monkey. Therefore, the final answer is: True
25

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Section 3, Section 4.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Section 5.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
26

---

Justification:
There is no theoretical result in our paper.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: Section C.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
27

---

Answer: [Yes]
Justification: In supplementary materials.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips. cc/
onnilicc))))) for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: / //nips.co.. wwtppplic/gguccieecccceoooocccce gwtc///ggee///eee CCodeSubmiieeeeece
httooi///uuddeesodeeeuubbiiiiiii iwaibbmmssiioooolllioiooo (eionnolccy) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Section C.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: Section C.3.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, at least for the experiments that support main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
28

---

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of enrors is not verified.
For asymmetric distributions, the authors should be careful to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative eтror rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of exccution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Section C.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: //te:r//nnuui.///eee http///wwpp..pppuulpieepepeeceee
Answer: [Yes]
Justification: Our research conforms to the Code of Ethics of NeurIPS and see Section 5 for discussions on Broader Impact.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g.. if there is special consid- cration due to laws or regulations their jurisdiction..
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Section 5.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
29

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Question:
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Answer: [Yes]
Justification: Our models are trained with fully open ingredients, and training datasets are already filtered and validated by producers.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: In supplementary materials.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
30

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: Section C.2 and README file in supplementary materials. Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/codeemmodll as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: Our work does not include any crowdsourcing experiments or research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: Our work does not include any crowdsourcing experiments or research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, not include any information that would break anonymity (if applicable), such the institution conducting the review.
31

---

16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, originality of the research, declaration is not required.
Answer: [NA]
Justification: We use LLMs to fix grammar mistakes and polish paper writeups.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components. httpp//es.../oooeeeeeeeeeeeee/ee wwt.cc/////o// Cctte////ooooeeeeeee/22222
Please refer to L.LM policy (https: http://eeurrppp.pe for what should or should not be described.
32


Reviews:
[{"review_id": "yFB3JI1Gzk", "reviewer": "Reviewer_D5SP", "summary": "This paper performs a systematic study of LLM training dynamics across 4 common phases of training: pretraining, continued pretraining, supervised finetuning, and reinforcement learning. To target improvments on math benchmarks, the authors start with pretraining on FineWeb-Edu, perform CPT with FineMath, and finally use a dataset of QA pairs combined from MetaMathQA, OpenMathInstruct2, and NuminaMath for SFT and RL. The paper is organized as a series of takeaways from scaling up each phase of training, and the resulting models from the experiments will be released to the research community.", "questions": "1. For the CPT phase, did you start with an annealed checkpoint and warm-up the learning rate or did you just change the dataset during the original anneal.\n\n2. Did you explore using ORM scores from a smaller reward model to predict downstream reasoning tasks? It would be interesting to see the scale of reward model required for your observed correlation to hold.", "limitations": "Yes, the authors have adequately discussed the limitations, primarily the scale at which the experiments are conducted.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n\nThe paper is clear and focused with well-motivated experimental design. The setup of the paper enables the authors to explore important interactions between different phases of training.\n\nFrom the scaling general-domain pretraining, the authors observe that upstream performance saturates around a token duration of 80x - 160x model size; in addition, by looking at how SFT performs on top of models at different durations, they observe that improvements from domain-specific post-training also tend to stop after this saturation point. Continued pretraining is found to boost SFT and RL finetuning, but a small replay budget was needed to avoid catastrophic forgetting. For both SFT and RL, the authors found points where improvements saturated. Finally, the authors provided some evidence that outcome reward model scores could be used to predict performance on downstream tasks.\n\nOverall, while the observations generally match how LLM training is currently done in practice, the paper adds a methodical and open study of LLM training dynamics at the 1B and 4B scale.\n\n**Weaknesses**\n\nThe paper does not have a discussion of how the hyperparameters (like learning rate) were chosen and if the learning rate was swept for any of the post-training phases. Could the authors provide details on this?", "was_revised": true, "final_justification": "The paper is clear and has a well-motivated experimental design to explore important interactions between different phases of training. The reproducibility of the paper has been improved by including the details of training hyperparameter. Overall, I recommend acceptance.", "rebuttal": "# Response to Reviewer D5SP\n\nWe thank the reviewer for the thoughtful and insightful comments that make our paper better. Below, we address each raised point individually. \n\n> W1: The paper does not have a discussion of how the hyperparameters (like learning rate) were chosen and if the learning rate was swept for any of the post-training phases. Could the authors provide details on this?\n\nThank you for the question. We used a consistent and carefully tuned set of hyperparameters across all experiments to isolate the effects of scaling compute, model size, and training stages. For pretraining, we adopted a cosine learning rate schedule with warmup, and tuned the base learning rate and batch size separately for 1B and 4B models to ensure stable convergence and optimal upstream task performance. For post-training stages (SFT, RL), we selected hyperparameters based on preliminary grid searches, guided by prior work on hyperparameter scaling and training stability in LLMs, including Wortsman et al. [1] and Zhang et al. [2]. To improve transparency, we will add a table of all key hyperparameters to the Appendix in the newer version, including learning rates, batch sizes, optimizers, and scheduling details for each training phase.\n\n[1] Small-scale proxies for large-scale transformer training instabilities\n\n[2] How does critical batch size scale in pre-training?\n\n> Q1: For the CPT phase, did you start with an annealed checkpoint and warm-up the learning rate or did you just change the dataset during the original anneal.\n\nThanks for the question. We start with an annealed pretraining checkpoint and warmup the learning rate again.\n\n> Q2: Did you explore using ORM scores from a smaller reward model to predict downstream reasoning tasks? It would be interesting to see the scale of reward model required for your observed correlation to hold.\n\nWe did not explore using ORM scores from smaller reward models in this study, as we focused on a strong 8B reward model to ensure high signal quality when correlating with downstream task accuracy. However, we agree that understanding how the scale of the reward model affects its predictive power is an important and promising direction. Future work could investigate whether smaller ORMs can still reliably approximate downstream quality, and whether certain tasks require different levels of capacity in the reward model."}, {"review_id": "f8q7vTrCu8", "reviewer": "Reviewer_nh5Y", "summary": "The paper studies the impact of pretraining data selection on large language model (LLM) performance. It introduces multiple data filtering strategies and evaluates their effectiveness across a wide range of upstream and downstream tasks, model scales, and post-training phases including continued pretraining (CPT), supervised fine-tuning (SFT), and reinforcement learning (RL). The goal is to identify data characteristics that most strongly influence model performance under various resource constraints.", "questions": "- The paper examines scaling of CPT, SFT, and RL individually. While this may be outside the current scope, it would be helpful to hear the authors’ thoughts on budget allocation across PT, CPT, SFT, and RL when total compute is fixed.\n- In Section 3.3 (Figures 5 and 6), why does scaling the dataset size appear to have more impact than scaling the number of epochs? Any insights into this effect?\n- In Section 4.1 (lines 264–273), clarification is needed: does it mean the difference of using intermediate checkpoint is no learning rate decay? Do the authors use EMA checkpoints, and if so, how do they influence performance? Also, any insights on \"continue pretraining\" to include more tokens (different from CPT)?\n- Are the filtering strategies equally effective across all model scales? Some trends (e.g., impact of data entropy) could vary depending on model size, would be good to confirm.\n- Do filtering strategies benefit certain task types more than others (e.g., reasoning vs. factual recall)?", "limitations": "Limitations are discussed in the paper. No societal concerns are noted.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "## Strength\n- The paper is well written, clearly structured, and easy to follow.\n- The experiments are comprehensive and systematic, covering multiple model sizes and training phases.\n- The direction is practically important: understanding how to best select or filter training data can directly benefit LLM training efficiency and downstream performance.\n- The authors’ analysis offers useful insights for practitioners interested in optimizing compute/data tradeoffs for pretraining and post-training.\n\n## Weakness\nI am not an expert in this specific subfield, but I find the contribution solid overall. I don't see any real weakness, a minor presentation nit: the use of “—” is inconsistent in lines 268–269 and 278–279. Please see some clarification questions in question section below.", "was_revised": true, "final_justification": "I thank the authors for their rebuttal, I will keep my score and stay positive.", "rebuttal": "# Response to Reviewer nh5Y\n\nWe are grateful to the reviewer for the thoughtful and insightful comments to make our paper better. Below, we address each raised point individually. \n\n> W1: I am not an expert in this specific subfield, but I find the contribution solid overall. I don't see any real weakness, a minor presentation nit: the use of “—” is inconsistent in lines 268–269 and 278–279. Please see some clarification questions in question section below.\n\nThank you for your advice. We will revise the paper accordingly in the next version.\n\n> Q1: The paper examines scaling of CPT, SFT, and RL individually. While this may be outside the current scope, it would be helpful to hear the authors’ thoughts on budget allocation across PT, CPT, SFT, and RL when total compute is fixed.\n\nThank you for your thoughtful suggestion. Allocating a fixed compute budget requires a sequential, trade-off-aware approach. First, we should allocate enough pre-training compute to reach the point of diminishing returns—around 80x to 160x the model's size in tokens—but no more, to conserve resources. Next, a critical portion of the budget must go to CPT to adapt the model to a specific domain, incorporating a small amount of data replay (e.g., 5%) to mitigate catastrophic forgetting. Finally, the remaining post-training budget should be split between SFT and RL according to this goal: allocate more to SFT for peak in-domain performance, or more to RL to enhance out-of-domain generalization. While we agree this is an important and practical direction for future work, a comprehensive study of optimal budget allocation is beyond the scope of the current paper.\n\n> Q2: In Section 3.3 (Figures 5 and 6), why does scaling the dataset size appear to have more impact than scaling the number of epochs? Any insights into this effect?\n\nScaling the number of epochs involves repeatedly showing the model the same fixed dataset, which leads to over-specialization; the model memorizes the specific training examples, causing in-domain performance to saturate and OOD performance to degrade as it fails to learn robust, transferable principles.\nIn contrast, scaling the dataset size exposes the model to a wider variety of unique problems, forcing it to learn more general, underlying logic rather than superficial statistical patterns. This leads to more durable improvements for in-domain tasks and better, more stable performance on OOD tasks because the model is genuinely learning problem-solving skills, not just recognizing solutions.\n\n> Q3: In Section 4.1 (lines 264–273), clarification is needed: does it mean the difference of using intermediate checkpoint is no learning rate decay? Do the authors use EMA checkpoints, and if so, how do they influence performance? Also, any insights on \"continue pretraining\" to include more tokens (different from CPT)?\n\nYes, the primary difference is that intermediate checkpoints are extracted before full training convergence and learning rate decay has occurred.\n\nWe did not use EMA checkpoints in our study. All evaluations are conducted on final checkpoints obtained after full training with complete learning rate schedules.\n\nWe did not \"continue pretraining\" to include more tokens. In our study, we simulate the real-world scenario where practitioners get pretrained checkpoints and start CPT on top of those checkpoints. Moreover, our CPT strategy is data- and objective-specific: we add domain-specific data while interleaving a small amount of replayed general-domain pretraining data. This is different from simply continuing generic pretraining to include more tokens, which could lead to catastrophic forgetting or degraded generalization as shown in Table 2 and Figure 3 (page 5) that effective CPT requires careful data balancing.\n\n> Q4: Are the filtering strategies equally effective across all model scales? Some trends (e.g., impact of data entropy) could vary depending on model size, would be good to confirm.\n\nWhile we do not explicitly evaluate filtering methods in this study, we designed our experiments to reveal how data scale interact with model scale. For instance, Section 3.1 show that smaller models (e.g., 1B) can outperform larger models (e.g., 4B) under constrained pretraining budgets, suggesting that data efficiency—an implicit effect of filtering—is especially crucial for smaller models. Additionally, in our CPT experiments, the effectiveness of modest replay appears consistent across model sizes, indicating that filtering-like strategies may transfer well across scales. That said, a more systematic study of filtering efficacy at different scales would be a valuable future direction.\n\n> Q5: Do filtering strategies benefit certain task types more than others (e.g., reasoning vs. factual recall)?\n\nAlthough we do not apply explicit filtering strategies, our results suggest that data curation strategies, e.g., replay or fine-tuning dataset quality control, affect tasks differently. For example, domain-specific CPT yields large improvements on reasoning-heavy tasks like math, whereas excessive pretraining or fine-tuning can sometimes degrade performance on OOD tasks that rely more on general knowledge, e.g., StrategyQA. This points to a possibility that filtering strategies may indeed have differential benefits, e.g. potentially favoring structured reasoning tasks more than open-domain factual recall. We view this as another promising direction for deeper follow-up work."}, {"review_id": "BiipOAVB7D", "reviewer": "Reviewer_mn1m", "summary": "This paper presents a highly systematic and transparent empirical study of large language model (LLM) training dynamics across all major phases: pre-training, continued pre-training, supervised fine-tuning (SFT), and reinforcement learning (RL). The authors train over 100 decoder-only LLaMA-2 models with 1B and 4B parameters from scratch, controlling for variables such as model size, token budgets, and data mixture. The study evaluates both upstream (language modeling) and downstream (reasoning/problem-solving) performance on a suite of in-domain and out-of-domain tasks. Key findings include: (1) diminishing returns and even potential performance degradation with excessive pre-training, (2) the critical role of domain-specific continued pre-training with replay to mitigate catastrophic forgetting, (3) the interplay and trade-offs between SFT and RL for in-domain and OOD generalization, and (4) practical recipes and heuristics for effective training at each stage. All models, datasets, and code are released for reproducibility and further research.", "questions": "1. **Architectural Generalization:**  \n   *To what extent do you expect your key empirical findings (such as the pre-training saturation points, replay ratios, and RL/SFT trade-offs) to hold for other architectures (e.g., Mistral, GPT-2/3, Mixture-of-Experts, or even non-transformer LMs)? Could the authors comment on whether they have tried, or plan to try, these recipes with alternative backbones? This is a crucial point for the impact and universality of the work.*\n\n2. **Deeper Analysis of Negative Transfer:**  \n   *The observed degradation of downstream performance with excessive pre-training is both surprising and important. Can the authors provide further analysis, ablation, or speculative hypotheses on the underlying mechanism? Is it optimization (e.g., overfitting, sharpness), distributional mismatch, or something else? Any additional insight here would elevate the paper's significance.*\n\n3. **Broader Task Diversity:**  \n   *How might the best practices and trade-offs identified here change for other core LLM tasks—especially dialogue, tool use, multi-turn instruction, code synthesis, or safety-critical objectives? Have the authors considered expanding the evaluation suite or at least discussing potential limitations in generalizing these findings?*\n\n4. **Computational Cost and Reproducibility at Scale:**  \n   *While the open release is commendable, running such a suite remains computationally expensive. Are there any practical recommendations or heuristics for researchers with more limited resources, or plans to provide smaller-scale ablations for the community?*\n\nScore could be increased if the authors provide empirical evidence or at least in-depth analysis that the key findings are robust across architectures, or offer new insight into the mechanisms underlying negative transfer from over-pretraining.", "limitations": "Yes, the authors have a dedicated discussion of limitations and are appropriately upfront about the primary weaknesses of their study. One additional limitation, however, is the architectural specificity: all findings are from LLaMA-2-based models, and this may restrict the universality of their \"best practices\" until further replication.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 4, "originality": 2, "strengths_and_weaknesses": "## Strengths and Weaknesses\n\n### Strengths\n\n- **Extremely systematic and transparent:** The study is one of the most comprehensive empirical investigations of LLM training dynamics to date. The careful control of experimental variables, end-to-end training from scratch, and consistent evaluation protocol enable clear, actionable insights.\n- **Open science:** Release of 100+ model checkpoints, all training data, and full code pipeline. This sets a strong standard for reproducibility and enables the community to build on this work.\n- **Practical best practices:** The paper distills an array of \"recipes\" and trade-offs for LLM practitioners, offering guidance on how much to scale each stage and how to balance domain adaptation, generalization, and forgetting.\n- **Novel empirical insights:** The finding that excessive pre-training can degrade downstream performance after SFT/RL is notable, as is the systematic exploration of replay ratios for catastrophic forgetting.\n- **Clear writing and presentation:** The structure, figures, and explanations are clear and facilitate understanding, even for complex, multi-stage training dynamics.\n\n### Weaknesses\n\n- **Architectural specificity:** All experiments are conducted on the LLaMA-2 architecture. This is a major limitation, as some conclusions and observed trends (e.g., the precise saturation regimes, the degree of negative transfer, RL/SFT trade-offs) may not generalize to other architectures (e.g., Mistral, Mamba, Mixture-of-Experts, or non-transformer architectures). Without replication on other backbones, it's unclear how universal the best practices are.\n- **Interpretability of negative transfer:** While the paper robustly documents the phenomenon of diminishing returns and even performance degradation from over-pretraining, it stops short of providing a deep analysis or theoretical explanation. Is this due to optimization issues, overfitting, pre-training/fine-tuning distribution mismatch, or something more subtle? Some ablation or hypothesis testing in this direction would increase the work's significance.\n- **Empirical, not theoretical:** The paper is almost entirely empirical. While this is not a flaw per se, the lack of any attempt to connect to theoretical frameworks (e.g., statistical learning theory, information theory, optimization landscapes) somewhat limits the generalizability and scientific depth of the conclusions. However, this is a minor issue compared to the main contributions.", "was_revised": true, "final_justification": "The authors' response mostly addressed my questions and comments, but did not provide sufficient new information for me to raise my score. I'll maintain my score of 5 and vote for accept.", "rebuttal": "# Response to Reviewer mn1m\n\nThank you for your thoughtful and constructive feedback. Below, we address your comments and suggestions point by point:\n\n## About architectural specificity\n> W1: Architectural specificity: All experiments are conducted on the LLaMA-2 architecture. This is a major limitation, as some conclusions and observed trends (e.g., the precise saturation regimes, the degree of negative transfer, RL/SFT trade-offs) may not generalize to other architectures (e.g., Mistral, Mamba, Mixture-of-Experts, or non-transformer architectures). Without replication on other backbones, it's unclear how universal the best practices are.\n> \n> Q1: Architectural Generalization: To what extent do you expect your key empirical findings (such as the pre-training saturation points, replay ratios, and RL/SFT trade-offs) to hold for other architectures (e.g., Mistral, GPT-2/3, Mixture-of-Experts, or even non-transformer LMs)? Could the authors comment on whether they have tried, or plan to try, these recipes with alternative backbones? This is a crucial point for the impact and universality of the work.\n\nThe qualitative trends—such as pre-training saturation, the need for data replay in CPT to combat forgetting, and the SFT/RL trade-off between in-domain specialization and out-of-domain generalization—are expected to hold for other dense, decoder-only transformers like Mistral or GPT, as evidence by many related works in the literature [1, 2]. These are fundamental dynamics of learning from data, not just architectural quirks. However, the exact quantitative numbers, like the 80x-160x saturation point or the 5% replay ratio, would almost certainly need recalibrating for different models. For radically different architectures like Mixture-of-Experts, these recipes may require significant adaptation, and confirming this generalizability is a promising direction for future work.\n\n[1] D-CPT Law: Domain-specific Continual Pre-Training Scaling Law for Large Language Models\n\n[2] Overtrained Language Models Are Harder to Fine-Tune\n\n## About negative transfer\n> W2: Interpretability of negative transfer: While the paper robustly documents the phenomenon of diminishing returns and even performance degradation from over-pretraining, it stops short of providing a deep analysis or theoretical explanation. Is this due to optimization issues, overfitting, pre-training/fine-tuning distribution mismatch, or something more subtle? Some ablation or hypothesis testing in this direction would increase the work's significance.\n> \n> Q2: Deeper Analysis of Negative Transfer: The observed degradation of downstream performance with excessive pre-training is both surprising and important. Can the authors provide further analysis, ablation, or speculative hypotheses on the underlying mechanism? Is it optimization (e.g., overfitting, sharpness), distributional mismatch, or something else? Any additional insight here would elevate the paper's significance.\n\nThank you for highlighting the importance of analyzing the mechanism behind the degradation from excessive pretraining. While our current work focuses only on empirical study of this effect across stages of the training pipeline, we believe our results are highly consistent with—and complemented by—recent theoretical insights. Springer et al. [1] provide a compelling explanation via the concept of progressive sensitivity, which increases with the amount of pretraining. Their analysis shows that as pretraining progresses, model parameters become more fragile to post-training updates. This increased sensitivity causes greater forgetting or distortion of acquired knowledge during post-training, leading to what they call catastrophic overtraining. They demonstrate that this degradation is inevitable when pretraining is extended without constraints and is exacerbated by distribution mismatch between pretraining and post-training tasks.\n\nIn our work, we extended the study to both upstream and downstream tasks and in-domain and out-of-domain tasks, and observe similar phenomenon. Models trained beyond a certain token threshold begin to show degraded performance on downstream reasoning tasks, even though upstream perplexity continues to improve. This aligns with their theoretical finding that the inflection point—the point beyond which sensitivity dominates the benefits of additional pretraining—marks the onset of degradation. We view a more formal exploration of the sensitivity–plasticity trade-off as an exciting future direction, and we appreciate the reviewer’s encouragement to deepen this analysis.\n\n[1] Overtrained Language Models Are Harder to Fine-Tune\n\n## Other questions\n> W3: Empirical, not theoretical: The paper is almost entirely empirical. While this is not a flaw per se, the lack of any attempt to connect to theoretical frameworks (e.g., statistical learning theory, information theory, optimization landscapes) somewhat limits the generalizability and scientific depth of the conclusions. However, this is a minor issue compared to the main contributions.\n\nWe acknowledge that our study is primarily empirical by design, as our goal was to provide a rigorous, controlled, and transparent investigation into training dynamics across the full lifecycle of language models. That said, we agree that connecting our findings to broader theoretical frameworks could enrich the scientific depth and interpretability of our results. We view this as a valuable direction for future work and hope that our released models and pipeline can serve as a foundation for such theoretical explorations by the community.\n\n> Q3: Broader Task Diversity: How might the best practices and trade-offs identified here change for other core LLM tasks—especially dialogue, tool use, multi-turn instruction, code synthesis, or safety-critical objectives? Have the authors considered expanding the evaluation suite or at least discussing potential limitations in generalizing these findings?\n\nWhile it is true that our empirical results of are focused on reasoning tasks, this was a deliberate methodological choice necessary to establish a clear, reproducible baseline. As we acknowledge in limitations, the study's focus leaves dynamics for tasks like tool use and safety unexplored. However, the work's primary contribution is not a universal recipe but a foundational framework for analysis. We argue that the core principles identified—such as pre-training saturation, catastrophic forgetting during CPT, and the trade-off between specialization (SFT) and generalization (RL) are fundamental learning dynamics that will likely persist across different objectives. The key change will be in the nature of the trade-offs: for dialogue, the SFT/RL balance might shift from logical correctness to one between stylistic alignment and helpfulness; for tool use, RL could be rewarded on successful API execution; and for safety, the entire paradigm might be augmented with additional constraint-based methods. Therefore, this study does not claim to have the final answer for all tasks, but rather provides the essential blueprint and a set of validated hypotheses required to begin investigating these more complex and subjective domains systematically.\n\n> Q4: Computational Cost and Reproducibility at Scale: While the open release is commendable, running such a suite remains computationally expensive. Are there any practical recommendations or heuristics for researchers with more limited resources, or plans to provide smaller-scale ablations for the community?\n\nWhile the computational cost of this suite is indeed significant, its primary purpose is to serve as a foundational, pre-computed roadmap precisely because such large-scale studies are beyond the reach of most researchers. Rather than requiring a full reproduction, the key takeaways from our study are intended to be the heuristics themselves. For instance, our findings on pre-training saturation provide a clear guideline on when to stop costly training runs (Takeaway 2), and the strong correlation between ORM scores and task performance (Takeaway 13) offers a much cheaper proxy for model evaluation than running extensive downstream tests. Therefore, the value for the resource-constrained researcher lies not in re-running our experiments, but in using these validated principles and proxy metrics to make more informed, cost-effective decisions in their own work."}, {"review_id": "dePPmoe8k8", "reviewer": "Reviewer_3Qki", "summary": "This paper releases a suite of over 100 language models with varying decisions across the full training pipeline: from pretraining (#tokens, #params in {0.5B, 1B, 4B}) to continued pretraining (#tokens new mix, #tokens original mix), SFT (#examples, #epochs), RL (#examples, #epochs). Data is fixed for pretraining (FineWeb-EDU), continued pretraining (FineMath), SFT (Various math data), and RL (same sources as SFT). Alignment algorithm (PPO) is also fixed, and a fixed set of hyperparams are used for each model size. They provide analysis of how changes in the varied decisions impact evaluations upstream (Cloze Form HellaSwag, Winogrande, PIQA, OBQA, ARC) and downstream (generative GSM8K + MATH, CruxEval, BGQA, TabMWP, StrategyWA). These analyses are used to provide 13 practical takeaways for model developers, such as not over-pretraining and the need to replay original pretraining data during the continued pretraining.", "questions": "- What can the authors say about how easy it will be to build on their suite? Whats it like to run post training steps on top of one of your pretrained models? If you want to run a new pretraining experiment but with a smaller configuration how easy is that? How about availability of intermediate checkpoints? Optimizer states? How easy is it to run different kinds of evaluations?\n    \n\n- Would love to see an analysis of the relationship of upstream to downstream performance (and how this may be altered by various training decisions)\n    \n\n- Font in figure in figures is frequently too small to read.\n    \n- line103 define ORM score before use\n    \n- how do I interpret ORM avg@16 in figure 2. why are the scores negative? Is higher better? Maybe include a hyperlink in the caption to section 2.2 for metric definitions.\n    \n- table 1 is difficult to read, could be better as a figure. at least tell us the take away in the caption.\n    \n- Table 2 doesn’t say what param size and pretraining tokens trained are used in these experiments.\n    \n- maybe put the value into the x axis label not just the caption, e.g. figure 5 should have x axis “SFT Epochs”\n    \n- what are the x marks in the fig 7 b? also it would be interesting to see this subplot but with more than 1 epoch since it seems that epoching multiple times is very important based on figure 7 a.\n    \n- I like the use of color in fig 8! Maybe also just quickly say SFT allocation increases further to the right.\n    \n- It would be great to have a table or figure telling us what are all the models released.\n    \n- table 3 is hard to parse but I’m not sure how to explain it better since it’s subtle, could this be a line plot instead with the 20BT x position highlighted?\n    \n- Figure 9 is waaay too small, maybe just say the R^2 values in a table or make it big enough to see the plots.\n    \n- 32K vocab size seems like a bit of an uncommon choice (I think of >64K being more common). You could argue this is well motivated for helping you explore smaller, less expressive models but regardless might be worth pointing out somewhere outside of the appendix.\n    \n- What is the LR schedule for pretraining? Cosine?\n    \n- How were the hyperparams selected?\n    \n- A possibility (but totally fine if out of scope): the analysis in fig 3 could also compare using a moving weight space averaging of checkpoints or some other form of regularization instead of data replay. Also a lack of annealing of the LR of the base model before CPT may play a role in disruption caused by the data distribution shift (but I think you drop the LR pretty low right?) But maybe your LR is too high during CPT or your warm up is too short?\n    \n- line 205: Does overly large epochs mean too many epochs or that each epoch is too large (i.e., too many examples)?\n    \n- figure 7: don’t use shades of blue to indicate different metrics as its hard to distinguish them. Use differences in hue and or changes in marker.\n    \n- is the correlation in 4.2 between ORM and Downstream performance both on fully post trained models or is the ORM score on just the mid trained model before it’s post-trained?", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "strengths:\n\nTo my knowledge this is the first work that releases a suite sweeping over token/param ratios in pretraining and also conducts post training over may configurations. The most expensive to reproduce result open sourced by this suite is confirming the effect of over pretraining an reducing SFT and SFT + RL performance. This was previously observed by Springer et al (2025), as cited in this paper, but they only examined intermediate checkpoints rather than doing runs with different scheduled endings. Most of the rest of the analyses in the paper just build off the 1B-160BT model and thus could have just been conducted on top of an off the shelf base model. But future work can use their artifacts to explore the effect of token/param ratio on numerous other aspects of the post-training pipeline.\n\nBeyond the compute expense of trying multiple pretraining configurations, this suite of artifacts will also save developer time simply by having saved checkpoints for lots of different post training configurations all in one place. This is not as expensive to reproduce, but if this suite is well organized it could speed up research built on top of it. In particular I see a great role for this suite in pedagogy. It would be great tool for students learning about the full language model training pipeline.\n\nThis work also examines various subtleties of post-training: the relative importance of CPT and SFT (possibly hinting at CPT cannibalizing SFT?), The effect of RL on confidence, trade offs between SFT and RL for ID and OOD performance.\n\nweakness:\n\nWhile still an interesting avenue of inquiry, this suite has a domain specific mid and post training focus on math, contrary to the more common practice of post training models for generalist performance. The evaluation does includes OOD eval but even these are very targeted tasks and do not include general evaluations of preference alignment eg MT-bench, alpaca-eval.\n\nThe current paper and appendix don’t clearly list everything that will be available in the release. One can assume that hopefully all the models mentioned analyses will be released, but if these are all that will be released it’s worth noting that then the suite sweeps much more sparsely over some options than others (for example the 4B models participate in far fewer of the analyses than the 1B models). It’s understandable that a full grid search over all the complex configurations would be very challenging to put together, but given the expense that goes into pretraining I hope the cheaper to sweep stages of training are more densely sampled.\n\nMuch of the practical utility of this study consists of finding specific good settings for things like how much data to replay in CPT, but it’s not discussed much how one should go about generalizing these findings to other configurations (e.g. bigger models). The answer may just be that without sampling more than one point over model size for example, we can’t do better than hoping that the ratio of replay to new CPT data is constant, which is fine. But some discussion of what we can know and how we might know more would be important to have.", "was_revised": true, "final_justification": "I think this suite of models will be quite impactful, as I am not aware of previous sweeps over both pretraining and post training configurations in a single suite. I'm increasing my score since I can see in the table of experiments provided in the author response that the experiment sweeps are denser than I realized. I also think the findings in the paper are helpful bring together evidence for a wide set of best practices in pre and post training all in one set of experiments. My other concerns are mostly just reasonable limitations that the authors embrace as well motivated scoping decisions (e.g. focusing on math evals since they're more verifiable). I recommend this paper be accepted.", "rebuttal": "# Response to Reviewer 3Qki\n\n## General issues:\n\n> W1: While still an interesting avenue of inquiry, ...\n\nIt is correct that we intentionally focuses on the domain of mathematics, but we consider this to be a deliberate methodological choice rather than a limitation. The common practice of training for generalist performance is often evaluated using subjective, preference-based benchmarks like MT-Bench, which are noisy, difficult to reproduce, and can obscure the true effects of specific training decisions. By anchoring the entire study in a domain with verifiable, objective correctness, we can isolate and measure the fundamental dynamics of training—such as saturation points and the trade-offs between SFT and RL with a scientific rigor that would be impossible in a purely subjective setting.\n\n> W2: The current paper and appendix don’t clearly list everything that will be available in the release...\n\nDue to anonymization policies, we cannot release assets at this stage. However, all models and code and data discussed in the paper will be released post-acceptance, along with full training and evaluation documentations. While the 4B sweep is sparser due to resource constraints, we prioritized high-resolution sweeps at the 1B scale and plan to expand post-training variations for 4B after acceptance.\n\n> W3: Much of the practical utility of this study consists of finding specific good settings for things like how much data to replay in CPT, ...\n\nThank you for raising a crucial point regarding the generalizability of specific hyperparameters, but the primary contribution of our paper is not to prescribe universal constants, but rather to identify and validate the fundamental dynamics at play, thereby providing the very methodology for \"how we might know more.\" For example, by demonstrating that a modest replay budget of around 5% effectively mitigates catastrophic forgetting (Takeaway 4), the study provides an invaluable, empirically-grounded starting point that drastically reduces the search space for researchers working with different model sizes or domains. \n\nOur paper's value lies in establishing what to look for and why it matters, which equips researchers with the knowledge that they must consider a data replay strategy and should begin their search in a low-single-digit percentage range, rather than having to discover this principle from scratch. Thus, the work offers not a static answer, but a validated, efficient template for inquiry.\n\n> Q1: What can the authors say about how easy it will be to build on their suite? ...\n\nWe expect that the practical usability of the model suite is important, and the project was designed with this exact concern in mind. The release accompanying our paper goes beyond merely providing model weights; it includes the entire open-source training, post-training, and evaluation pipeline used in the study. \n\nThis was a deliberate design decision we made to ensure the suite is a modular and extensible toolkit, not a static artifact. Consequently, running post-training steps on our models is as simple as loading a chosen pre-trained checkpoint into the provided pipeline; conducting a new, smaller-scale pre-training experiment involves modifying the transparently documented configurations (Appendix C); and running different evaluations means plugging a new dataset into our open-source evaluation harness. \n\nBy providing not just the models but the complete, end-to-end code, we have deliberately lowered the barrier to entry for precisely the kinds of follow-on research and custom experiments you describe. For example, we expect our checkpoints are useful for downstream performance scaling law fits and other reliability studies like data attribution.\n\n> Q2: Would love to see an analysis of the relationship of upstream to downstream performance ...\n\nThank you for the suggestion. We agree that understanding the relationship between upstream and downstream performance is interesting, and in fact, our study has incorporated investigations into this aspect across training stages. Specifically:\n\n1. Section 3.1 shows that while more extensive pretraining consistently improves upstream performance (e.g., lower perplexity), downstream performance saturates and can even degrade—highlighting a clear decoupling between upstream and downstream gains in the overtraining regime.\n2. Section 3.2 further demonstrates that continued pretraining (CPT) on domain-specific data leads to degraded upstream performance due to catastrophic forgetting, yet this step is critical for improving downstream task performance, especially for in-domain reasoning tasks.\n\nThese results reveal that training decisions such as scaling pretraining or incorporating replay in CPT can alter the alignment—or misalignment—between upstream and downstream behavior. We will clarify this connection more explicitly in the next version to highlight that this relationship is a core focus of our analysis.\n\n## Clarification issues:\n\n> how do I interpret ORM avg@16 in figure 2...\n\nThank you for pointing this out. ORM avg@16 refers to the average ORM score over 16 sampled generations per input prompt. As explained in Section 2.2, this scalar score—produced by an external reward model—serves as a proxy for solution quality, where **higher scores indicate better responses**. The scores can be negative because the ORM model outputs unnormalized scores centered around zero, depending on the alignment and quality of the response relative to the input. To avoid confusion, in our next draft we will (1) explicitly state in the figure caption that higher is better, (2) clarify that negative values are possible due to the ORM’s scoring convention, and (3) include a hyperlink to Section 2.2 for detailed metric definitions.\n\n> Table 2 doesn’t say what param size and pretraining tokens trained are used in these experiments.\n\nThank you for pointing this out. In line 155, we have pointed out that we use 1B model pretrained with 160 billion tokens. We will add this information to the caption of the table.\n\n> what are the x marks in the fig 7 b? ...\n\nWe apologize for the confusion. The x-marks in Figure 7b denote models that frequently generate overly long responses exceeding the context window, which leads to truncated outputs and degraded performance—an issue we refer to as model collapse. These failures typically arise when training with very large RL datasets (e.g., ≥350K examples) for one epoch, where the model over-optimizes toward verbosity rather than correctness. We agree that comparing against multi-epoch RL runs—as in Figure 7a—would be insightful, since multiple epochs appear to yield more stable and robust improvements without triggering collapse. We will consider including such a comparison in the revision or supplement to illustrate how scaling depth (epochs) versus breadth (examples) influences model behavior under RL.\n\n> 32K vocab size seems like a bit of an uncommon choice (I think of >64K being more common)...\n\nThank you for the thoughtful observation. You're right—the 32K vocabulary size is smaller than what is commonly used in larger-scale models (often >64K). We intentionally chose 32K to align with our focus on studying smaller models (1B and 4B) under controlled compute budgets, where a more compact vocabulary helps reduce embedding size and training overhead. We agree this design choice deserves more visibility and will add a brief explanation in the main text to clarify its motivation beyond what's currently noted in the appendix.\n\n\n> What is the LR schedule for pretraining? Cosine?\n\nYes, we use cosine LR scheduler. \n\n\n> How were the hyperparams selected?\n\nThank you for the question. We used a consistent and carefully tuned set of hyperparameters across all experiments to isolate the effects of scaling compute, model size, and training stages. For pretraining, we adopted a cosine learning rate schedule with warmup, and tuned the base learning rate and batch size separately for 1B and 4B models to ensure stable convergence and optimal upstream task performance. For post-training stages (SFT, RL), we selected hyperparameters based on preliminary grid searches, guided by prior work on hyperparameter scaling and training stability in LLMs, including Wortsman et al. [1] and Zhang et al. [2]. To improve transparency, we will add a table of all key hyperparameters to the Appendix in the newer version, including learning rates, batch sizes, optimizers, and scheduling details for each training phase.\n\n[1] Small-scale proxies for large-scale transformer training instabilities\n\n[2] How does critical batch size scale in pre-training?\n\n> A possibility (but totally fine if out of scope)...\n\nThank you for the thoughtful suggestions. You're absolutely right that techniques such as weight space averaging or additional forms of regularization during CPT could help mitigate forgetting, and we agree that comparing those approaches to data replay would be valuable. Similarly, adjusting the LR schedule could influence the stability of adaptation. However, we consider these directions beyond the scope of our current study, which focuses on data-centric strategies like replay for mitigating distribution shift. We do use a decayed learning rate at the end of pretraining and adopt conservative learning rate settings for CPT, but a deeper sweep over schedule configurations remains an interesting avenue for future work.\n\n> line 205: Does overly large epochs mean too many epochs or that each epoch is too large (i.e., too many examples)?\n\nIt means \"too many epochs\".\n\n> is the correlation in 4.2 between ORM and Downstream performance both on fully post trained models or is the ORM score on just the mid trained model before it’s post-trained?\n\nThey are fully post-trained models.\n\n## Formatting issues:\n\nPer formatting issues you have mentioned, we will modify each of them in our next version. Thank you very much for pointing them out!"}]

Author rebuttal:

Meta-review: Reviewers are aligned that this is a careful, end-to-end empirical study of LM training dynamics across PT --> CPT --> SFT --> RL, with unusually systematic sweeps (100+ models at 0.5B/1B/4B) with clear takeaways on pretraining saturation, the need for small replay during CPT to mitigate forgetting and SFT/RL trade-offs for ID vs. OOD. The rebuttal helpfully clarified hyperparameter choices and schedules, CPT warm-up, and provided a denser inventory of planned artifacts. Remaining concerns are scope rather than flaws. Overall, the work meets the bar on correctness, transparency and community value.
Decision: Accept (oral)