Front-Loading Reasoning: The Synergy between Pretraining and Post-Training Data
Authors: Syeda Nahida Akter
Shrimai Prabhumoye
Eric Nyberg
Mostofa Patwary
Mohammad Shoeybi
Yejin Choi
Bryan Catanzaro
Venue: ICLR 2026 Poster (2026)
Keywords: Pretraining
Supervised Finetuning
Reasoning
LLM
Primary area: foundation or frontier models, including LLMs
TL;DR: Dissecting the Synergy of Reasoning Data in Pretraining and SFT

Abstract: The prevailing paradigm for enhancing the reasoning abilities of Large Language Models (LLMs) revolves around post-training on high-quality, reasoning-intensive data. While emerging literature suggests that reasoning data is increasingly incorporated also during the mid-training stage---a practice that is relatively more proprietary and less openly characterized---the role of such data in pretraining remains unclear. In particular, due to the opaqueness of pretraining corpora in most frontier models, the effect of reasoning data introduced at different phases of pre- and/or post-training is relatively less reported in the scientific literature. This raises several important but unsettled questions: Is adding reasoning data earlier during pre-training any better than introducing it during post-training, when the token counts are controlled? Could earlier inclusion risk overfitting and harm generalization, or instead establish durable foundations that later fine-tuning cannot recover?  To address these questions, we conduct the first systematic study of how reasoning data—varying in scale, diversity, and quality—affects LLM performance when introduced at different stages of training. Our findings reveal that front-loading reasoning data into pretraining is critical (19% average gain), establishing foundational capabilities that cannot be fully replicated by later-stage SFT, even with more data. We uncover an asymmetric principle for optimal data allocation: pretraining benefits most from broad diversity in reasoning patterns (11% average gain), while SFT is more sensitive to data quality (15% average gain with high quality data). Furthermore, we show that high-quality pretraining data has latent effects, activated only after SFT, and that naively scaling SFT data can be detrimental, washing away the benefits of early reasoning injection. Collectively, our results challenge the conventional separation of language modeling and reasoning, providing a principled guide for strategically allocating data across the entire training pipeline to build more capable models.

Published as a conference paper at ICLR 2026
FRONT-LOADING REASONING: THE SYNERGY BE- TWEEN PRETRAINING AND POST-TRAINING DATA
Syeda Nahida Akter?: Shrimai Prabhumoye 1.3 Eric Nyberg?, Mostofa Patwary', Mohammad Shoeybi', Yejin Choi Bryan Catanzaro'
NVIDIA', Carnegie Mellon University?, Boston University Stanford University
sakter@andrew cmu edu, sprabhumoye@nvidde com
ABSTRACT
The prevailing paradigm for enhancing the reasoning abilities of Large Lan- guage Models (LLMs) revolves around post-training on high-quality, reasoning- intensive data. While emerging literature suggests that reasoning data is increas- ingly incorporated also during the mid-training stage-a practice that is relatively more proprietary and less openly characterized the role of such data in pretrain- ing remains unclear. In particular, due to the opaqueness of pretraining corpora in most frontier models, the effect of reasoning data introduced at different phases of pre- and/or post-training is relatively less reported in the scientific literature. This raises several important but unsettled questions: Is adding reasoning data ear- lier during pre-training any better than introducing during post-training, when the token counts are controlled? Could earlier inclusion risk overfitting and harm generalization, or instead establish durable foundations that later fine-funing can- not recover? To address these questions, we conduct the first systematic study of how reasoning data-varying in scale, diversity, and quality- affects LLM per- formance when introduced different stages of training. Our findings reveal that front-loading reasoning data into pretraining is critical (19% average gain), es- tablishing foundational capabilities that cannot be fully replicated by later-stage SFT, even with more data. We uncover asymmetric principle for optimal data pretraining benefits most from broad diversity in reasoning patterns (11% average gain), while SFT is more sensitive to data quality (15% average gain with high quality data). Furthermore, we show that high-quality pretraining data has latent effects, activated only after SFT, and that naively scaling SFT data can be detrimental, washing away the benefits of early reasoning injection. Col- lectively, our results challenge the conventional separation of language modeling and reasoning, providing principled guide for strategically allocating data across the entire training pipeline to build more capable models.
allocation:
INTRODUCTION
The reasoning abilities of Large Language Models (LLMs) have advanced considerably, with post- training reasoning data driving significant breakthroughs in reasoning tasks, such math compe- titions (Hendrycks et al., 2021b), PhD-level scientific QA (Rein et al., 2024; Phan et al., 2025), and software engineering (Jimenez et al., 2024). This progress has been largely driven by mid- post- ceasoning-intensvve training LLMs on high-quality, reasoning- intensive datasets -otten featuring long chain-of-thought
(CoT) examples (Guha al., 2025; Moshkov al., 2025; Zhou et al., 2025; Gandhi et al., 2025; Wang et al., 2025). While this approach has proven effective, it treats reasoning as specialized skill be layered onto generalist base. addition, the impact of incorporating reasoning data during pretraining-ann and the potential synergistic effects on subsequent post-training- s remains critical yet less explored frontier. This research gap persists due to the prohibitive computational cost of end-to-end pretraining experiments and the opacity surrounding proprietary training recipes, which has concentrated community efforts on the more accessible post-training phase.
Work done during internship NVIDIA

---

Published as a conference paper at ICLR 2026
Pretraining
SFT
RL
Accuracy (%)
SPO
D Doss
Diversin
Quality
Pretraining Corpora
Reasoning Data (Dres)
Reasoning Tasks
Figure I: We systematically inject reasoning-style data (Dres) at different phases of train- ing-prrtainn versus SFT-while varying its diversity, quantity and quality. Our results show an asymmetric principle: diversity and scale matter most during pretraining, whereas quality dominates in SFT. This allocation strategy compounds through reinforcement learning (RL), yielding sustained gains across complex reasoning benchmarks.
The synergy between post-tue mest-training phases has been widely explored (Liu et al., 2025; Chen et al., 2025b; Chu et al., 2025). yet conclusions vary with training data and scale, and their applicability to pretraining remains vague in the current literature. In this work, we investigate not just which reasoning data, but when train with such reasoning data by studying the synergy between pre- training and post-training. Our central goal is to determine the ideal balance of such reasoning data across the two phases order to maximize downstream accuracies after reinforcement learning. This motivates the following research questions:
a reasoning-rich pretraining essential, or can model "catch up"? We investigate whether model pretrained without reasoning data match the performance of its reasoning- -wware coun- terparts by simply undergoing a more intensive SFT phase.
Does inclusion of reasoning data make the base LLM overfitted and less generalizable to sus- tain gains in subsequent training phases? While recent literature highlights overspecialization of reasoning during post-training gations of this effect pretraining remain limited.
be detrimental (Gupta et al., 2025; Luo et al., 2025b), investi-
Does data diversity in pretraining impact stability and specialization during SFT? Specifically, does using the same reasoning data in both pretraining and SFT lead to robust skill mastery, or does narrow pretraining focus risk catastrophic forgetting when the model is later fine-tuned?
Does the complexity and quality of reasoning data matter when incorporated during pretraining of the base model? Current literature explores this mostly from SFT stage (Zhou et al., 2023; Guha et al., 2025), making obscure whether difficulty or noisiness in the early phase of training directly impacts reasoning development or not.
This work provides systematic analysis of the interplay between reasoning data and the distinct phases of LLM training. Our primary findings are summarized as:
Front-loading reasoning data into pretraining creates durable, compounding advantage. Injecting reasoning data during pretraining establishes a superior foundation that widens at every stage of post-training, culminating in a +19% lead on expert-level benchmarks. This refutes the catch-up and overfitting hypotheses, proving that SFT cannot compensate for a weak foundation and that pretraining choices dictate the final performance ceiling.
The optimal data strategy asymmetric: prioritize diversity pretraining and quality in SFT. Our results reveal clear, phase-dependent principle. Pretraining benefits most from diversity and scale (a +11+1% % gain with diverse corpus), while SFT is dominated by data quality +15% % gain with high-quality reasoning data). This provides an actionable heuristic for data allocation that is more nuanced than simplistic "more is better" approaches.
Naive scaling of SFT data is ineffective and harmful. Blindly scaling SFT with mixed-quality data yields no average improvement and actively harmed mathematical reasoning by -5% on av- erage, while marginal (0.4%) addition of high-quality data consistently improved performance.
2

---

Published as a conference paper at ICLR 2026
High-quality pretraining data have latent effect unlocked by SFT. We found that high- quality data added to diverse pretraining mix showed minimal immediate benefit but "unlocked" additional +4% gain over model pretrained with diverse, mixed quality data after SFT revealing a deeper synergy where pretraining can instill a latent potential in the model that is only activated during the alignment phase.
METHODOLOGY
Our methodology is designed systematically determine the optimal strategy for allocating rea- soning data between the pretraining and supervised fine-tuning stages of LLM development. We frame this as an optimization problem where the goal maximize the final model's downstream accuracies, P. This is a function of the reasoning data introduced during pretraining, Dres and the data used for supervised fine-tuning, DSFT, Our objective is to find the optimal data configurations, (DPT*, res ), that solves the following:
res
(DPT. DSFT*) arg
max P(Otina)) DET DEFT
where final represents the parameters of the final model trained on data recipes defined by the choice of reasoning data at both stages.
Let D base denote the general pretraining corpus and we define a model _M(I) with parameters trained in two stages:
Pretraining:
OpT arg min ((.w))) UDFT (IM(Jo(s),v),
9
SFT:
6SFT = arg min B(z.))~DEPT (ser(fo(z).0)
Evaluation Objective.
The central research question can be expressed as analyzing the function:
P(DPT DEFT) E [Ace (t))] res
where T is of downstream reasoning tasks (math, science, code, general reasoning) and the expectation Er defines single fine-tuned model that is evaluated across T.
Our study can be summarized as optimizing the allocation of Dres between pretraining and SFT:
P(DPT DSFT res
IDPT
+ |DSETT,
DPT DEFT
Max
s.t.
res
where is the total budget of reasoning data available. This captures the trade-off early, scale/di- versity late, quality/complexit: DPT
DSFT
2.1 MODEL ARCHITECTURE AND BASELINE
We select a hybrid transformer with mixture of Mamba 2 (Dao & Gu, 2024), self-attention and FFN layers (NVIDIA, 2025a) with an 8B parameter for our base model, M and pretrain from scratch for I trillion tokens. This size strikes balance between computational feasibility and the capacity to learn complex reasoning patterns.
2.2 DATA PIPELINE
Our experimental design relies a careful distinction between categories of data: (1) large- scale, general- purpose pretraining corpus, and (2) a reasoning- focused,. instruction patructioonttning (SFT- style) datasets of varying quality and scale, This separation allows us to precisely control the injec- tion of reasoning data at different stages of training.
General Pretraining Corpus (Dbase). For the base training corpus, we adopt the dataset intro- duced in NVIDIA (2025b), which contains 6.2T tokens drawn from high-quality Common Crawl, mathematics, and code sources. This corpus provides broad coverage of languages and technical domains, serving the backbone of all pretraining experiments.
3

---

Published as a conference paper at ICLR 2026
Reasoning Datasets (Dres). To investigate the impact of data quality, diversity, and complexity, we curate four distinct reasoning- focused datasets in the question-anse answer format:
Large-Scale, Diverse Data (Di.DO). To simulate a "quantityover--uulayy" strategy, we em- Setiro--Peerrining-SFFAE dataset (NVIDIA, 2025b). This massive 268M samples of
ploy the dataset offers extensive domain coverage, with a composition of approximately 56% math, 17% code, and 27% science and general-purpose reasoning. The dataset covers tasks ranging from sim- ple Q&A to multi-turn dialogues, but with heterogeneous quality and reasoning depth, reflecting large-scale real-world availability.
Small-Scale, High-Que Saahh-Quaityy Data (DshQ). To capture the effect of long chain-of-thought traces from strong teacher models, we include the dataset of Guha al. (2025), comprising 1.2M care- fully curated examples (71% math, 21% code, 8% science). Compared to DLMQ. this corpus is smaller, less diverse, but significantly higher quality, emphasizing detailed reasoning paths.
Large-Scale, Mixed-Quality Data (DLMQ). To balance diversity with quality, we construct combined dataset that is a straightforward union of the two datasets above: DLMQ = Di.D + DsHQ. i.e., DLMQ contains 269.2M samples in total. This mix preserves large-scale coverage while injecting a fraction of curated, high-quality reasoning traces.
Answer-Length Filtered Data (DaL.). ). To investigate the feature of data quality, we create sub- set (7.1M samples) of PLDQ by retaining examples where the answer length exceeds 4096 tokens, based on the principle that longer responses often correspond to more complex CoT reasoning. This dataset allows us isolate the impact of reasoning complexity in different training phases.
2.3 SYNERGY BETWEEN PRETRAINING AND SFT
In this work, we aim to disentangle the contribution of reasoning data when incorporated at different points in the training pipeline. We structure the study into three stages: (i) large-scale Pretraining, where reasoning data may or may not be injected alongside the base corpus, (ii) Supervised Fine- tuning (SFT), where pretrained models are further adapted on reasoning data of varying quality and diversity, and (iii) Reinforcement Learning (RLVR) to determine the sustainability of early reasoning gain in the final model.
Phase 1: Pretraining. Prior work has primarily explored reasoning supervision either on top of fully pretrained LLMS (Wang et al., 2025) or by introducing small amounts of long chain-of th-tought (CoT) data into intermediate checkpoints (AI et al., 2025). These approaches leave open two ques- tions: how to inject reasoning data at scale during end-to-end pretraining, and whether the benefits persist when combined with high-quality base corpora. To address these questions, pretrain all models from scratch for IT tokens using mix of Drase and different types of Dres. Across all models, we keep the token ratio between Dhase and Dres fixed during pretraining. When reasoning dataset is small, it is repeated so that the model still observes the same total volume of reasoning tokens. To correctly state the data distribution, we pretrain all models for 600B tokens using D Dbase followed by 400B tokens on a mixture of 80% Drase and 20% Dres. This results in a constant budget of 80B reasoning tokens across all experiments. Dres can be any one source of data among the three reasoning datasets: DsHQ- DLDQ DLMQ. This token ratio has been maintained across all three pretraining runs with reasoning data.
Based on the reasoning data introduced, we train four distinct models:
Mbase: This model serves as our baseline and is pretrained without any reasoning data.
MI.QQ: Pre-trained with large-scale, diverse DLDQ reasoning dataset along with Drase-
MsHq: Pre-trained with DEHQ and Drase allowing us to isolate the effect of data quality versus the quantity and diversity of MLDQQ
MIMQ: Finally, this model is exposed to our combined reasoning DLMQ dataset.
In the subsequent analysis, we use Mies to denote the aggregate performance of the models pre- trained with reasoning data, representing the average score across ManQq MLDQQ and MLMQQ
Phase 2: Supervised Finetuning, Following pretraining, each of the four model variants (Mhases
MI.DQ Msnq. MLMQ) is adapted through supervised finetuning (SFT). This second phase is
4

---

Published as a conference paper at ICLR 2026
crucial for understanding the synergies, redundancies, and trade-offs between the data introduced during pretraining versus the SFT stage. To this end, we design a controlled set of SFT experiments, where each pretrained model is finetuned on different reasoning corpora introduced in Section 2.2 to address the following rearch questions:
The "Catch-Up" Hypothesis: Can intensive SFT on high-quality reasoning data allow the base- line model, Musse match or exceed the accuracy of models that were exposed reasoning data during pretraining? This directly tests the criticality of early data injection versus late-stage specialization.
Impact of Pretraining Data Scale and Diversity: We investigate how the scale and diversity of reasoning data used during pretraining influence the final model's capacity to absorb high-quality instruction data. Specifically, we ask: Does scaling up diverse reasoning data in pretraining provide lasting benefits even after all models finetuned on the same high-quality SFT corpus? By fine-tuning both the model pretrained on large, diverse data (MLDQ) and on smaller, less diverse data (MSHq) the same high-quality SFT set, deep initial exposure to reasoning yields a better foundation for downstream specialization.
determine whether broad
Impact of SFT Data Quality and Complexity: By fine-tuning all four base models on datasets of varying quality (DLDQ DSHQ) and complexity (DALF), we measure the marginal utility of data quality at the SFT stage as function of the model's initial pretraining condition.
This design enables us to address three critical dimensions: (1) the synergy between pretraining and SFT data, (2) the gains of increasing SFT data scale when reasoning was already introduced in pretraining, and (3) the role of data complexity and diversity in determining whether reasoning supervision should be injected early, late, or across both stages. Together with the pretraining ex- periments, these SFT studies form fully crossed setup, providing the first systematic assessment of how reasoning-style SFT data interacts with pretraining to shape the reasoning abilities of LLMs.
Phase Reinforcement Learning. To further observe the impact of reasoning centric pretraining and heavy supervised finetuning, we deploy RL. using Group Relative Policy Optimization (GRPO) (Shao et al., 2024) with verifiable rewards on top of the base models. Here we use NEMOTRON- CROSSTIIIKK (Akter et al., 2025) which has shown to be effective for multi-domain reasoning.
EXPERIMENTAL SETUP
3.1 TRAINING
Pretraining. To prepare base models, we pretrain a 8B LLM on our pretraining data blend till IT tokens using 512 H100 80GB SXM5 GPUs. During training, we use the AdamW optimizer (Loshchilov & Hutter, 2019) with 91 = 0.9, B2 = 0.95 and weight decay of 0.1. We use 8-way tensor and pipeline parallelism to train the model. We set the maximum value of learning rate to 3e minimum to 3e and use batch size of 6M tokens with a 8192 context length.
Post- Training. After pretraining, each 8B LLM is finetuned on 4.8M reasoming samples from Dres- SFT uses Adam with (8),8)) = (0.9, 0.95), weight decay 0.01, warmup ratio 0.05, learning rate 10- batch size 512, and context length 32k. We then apply GRPO via the veRI. framework for one epoch on NEMOTRON-CROSSTIE
data with constant LR 1x10-6 batch size 128, PPO mini-batch 128, and context length 8192. Each step samples 128 prompts with 8 rollouts (temperature= 1.0, top-p- = 1.0), and a KL penalty coefficient of 0.001.
3.2 EVALUATION METRICS
We report average accuracies of all tasks under each of the following categories.
Base Model Evaluations. We conduct thorough benchmark assessment to evaluate the general- izability of the base models, using a series of datasets using LM Eval Harness (Gao et al., 2024).
hhaps//gitthh.ccm//occnnnneeeeee
5

---

Published as a conference paper at ICLR 2026
General Purpose Reasoning (GPRpT AVG). We consider four standard commonsense and logical reasoning tasks in O-shot: ARC challenge (Clark et al., 2018), HellaSwag (Zellers et al., 2019), WinoGrande (Sakaguchi et al., 2021), and reading comprehension task: RACE (Lai et al., 2017).
Math Reasoning (MATHpr AVG). We evaluate the math reasoning ability with two benchmarks- they encompass math challenges from elementary to college level complexity demanding qualita- tive reasoning (8-shot GSM8K (Cobbe et al., 2021), 4-shot MATH-500 (Hendrycks et al., 2021b))-
Science Reasoning (SCIENCEpr AVG). We evaluate on 5-shot MMLU (Hendrycks et al., 2021a) and MMLU-Pro (Wang al., 2024) that spans multiple domains, from professional academic, testing the model on specialized subjects.
Code Reasoning (CODEPT AVG), For code tasks (HumanEval (Chen et al., 2021), MBPP (Odena al., 2021)) evaluate the EvalPlus variants along with the sanitization of generations (Liu et al., 2023), in a O-shot setup. We estimate avg @3, pass @ from 32 generations per prompt.
SFT Model Evaluations. To evaluate the reasoning ability of different SFT models, focus reasoning centric benchmarks unlike in base model evaluations, where mostly focus on the general- izability of the LLM. We conduct evaluations using MeMolSkills".
Math Reasoning (MATHsFT AVG). In addition to the GSM8K and MATH-500, we evaluate the models on two more complex math tasks AIMEE4 and AIME25 (Vecraboina, 2023).
Science Reasoning (SCIENCESFT AVG). On top of MMLU and MMLU-Pro, we evaluate on graduate level QA task: GPQA- Diamond (Rein et al., 2024).
Code Reasoning (CODESFT AVG). We choose LiveCodeBench (Jain et al., 2025) to test complex code reasoning ability.
Instruction Following (INSsFT AVG). For broader evaluation on diverse capabilities, we use IFE- val (Zeng al., 2024).
We report Pass @ 1 average of 16 runs for AIME-2024, AIME-2025 and average of 4 runs for MATH- 500, GSM8K, MMLU, MMLU-Pro, GPQA-Diamond, Bive and IFEval.
RL Model Evaluations. In this phase, we evaluate the models on complex reasoning tasks such as AIME24,25, MATH-500, GSM8K, MMLU, MMLU-Pro, GPQA-Diamond, LiveCodeBench fol- lowing the evaluation metric in SFT phase.
EXPERIMENTS AND RESULTS
Immediate Foundational Gains from Reasoning Data in Pretraining. Table shows the erage accuracies of our four model variants immediately after the IT token pretraining phase (see Table for individual benchmarks). The results provide clear evidence that integrating reasoning- style corpora from the start builds significantly more capable foundation. Every model exposed to
Model
Average
MATHpr AVG
SCIENCEet AVG
CODE AVG
GPRpr AVG
Mbase
52.70
47.17
47.13
40.89
75.63
Mssq
54.98
52.60
46.90
44.32
76.09
MUDQ
64.09
75.56
54.38
49.94
76.48
MLMQ
64.07
72.37
54.49
52.60
76.83
Miss
61.05
66.84
51.92
48.95
76.46
Table 1: Average Accuracies of base models trained without or with varying Dres. Pretraining with diverse reasoning data yields immediate gains, with scale and diversity driving math and code improvements, more than quality. Mres represents the average of MMHQ MLDQ. and MLMQQ
reasoning data surpasses Mbase- Mres average of the three reasoning-augmented variants trained under the same 1'T token budget, improves over Mbase by +8.35% on overall average aceuracy. The largest improvements come from models trained on large-scale, diverse data; MLDQ achieves

---

Published as a conference paper at ICLR 2026
highest average. driven by +28.4% gain in mathematics and a +9% gain in code over the baseline. Interestingly, the smaller, less diverse, high-quality dataset (MsHo) provides a modest lift, suggest- ing that at this early stage, the scale and diversity of the reasoning data are more critical than its curated quality for establishing a broad and robust reasoning foundation. Our experiments with a 1.2B Transformer (see Table 14) demonstrate that this front-loading strategy yields consistent, scal- able performance confirming the robustness of our approach across varying architectures.
Pretraining Advantage is Maintained and Amplified Post-SFT. We evaluate whether strong SET phase can close the accuracy gap established during pretraining with diverse reasoning data Dres- At the same time, we examine whether the inclusion of such data causes the model to overfit and reduce generalization, thereby diminishing subsequent post-training gains. We finetune each pretrained model on three reasoning datasets (DsHQ. PLDQ PLMQ), producing 12 models in total. We report the average results in Table 2 and include the full breakdown in Table 13.
Model
Average
MATHlaF AVG
SCIENCESP1 AVG
CODE AVG
INSaFT
M Mase + SFT
26.62
34.48
20.92
7.09
43.98
Mres + SFT
35.92
40.61
34.77
16.75
51.52
Table 2: Average Accuracies of SFT models pretrained with varying Dres. SFT amplifies the pretraining coaanntge--mddee with reasoning-rich pretraining significantly outperform baseline.
The results Table 2 indicate that the advantage gained during the pre-training phase not only persists but is amplified. The group of models pretrained with reasoning data (Myss + SFTT) out- performs the baseline group (MMase + SFTT by a significant 9.3% on average. This result strongly refutes the "cctch-up" hypothesis, showing that SFT is not substitute for strong reasoning foun- dation built during pretraining. While recent works have found reasoning-centrr post-training to be most effective math domains, the improvement science is minimal (Prabhakar et al., 2025; Luo et al., 2025a; Huan et al., 2025). However, the accuracy disparity our findings is most prominent often overlooked in reasoning focused post-training work. This suggests
in science domains, that pretraining with reasoning data does more than teach facts; it helps the model develop effective internal representations for abstract and logical structures to enhance problem solving ability across domains. It does not overfit the model rather infuses the critical thinking ability that comes into full potential after post-training (Appendix B). Consequently, the model's capacity absorb and leverage the SFT data is fundamentally enhanced, leading to greater learning efficiency and a higher performance ceiling. SFT acts as powerful enhancer, but its ultimate effectiveness constrained by the quality of the foundation established during pretraining.
Math Reasoning
Science & Code Reasoning
Model
Avg.
MATH-500
GSM8K
AIME24
AIME25
GPQA
MMLU
MMLU-PRO
LCB
Mbasc + SFT RL
37.92
72.05
83.83
12.29
16.04
28.16
41.10
36.69
13.16
MIMa
STTs RL
56.66
87,13
93.07
45.21
33.96
31.69
72.91
56,91
32.43
Table 3: Average accuracies of RL models pretrained and fine-tuned with varying Dres. Intro- ducing reasoning data early provides significant reasoning boost after post-training.
Pretraining Strategy Dictates Final Accuracy on Expert-Level Tasks. The final RL phase re- veals the definitive impact of our pretraining interventions, particularly on expert-level reasoning benchmarks. We select MLMQ + SFT SHQ and Mbase + SFT sHQ finetuned using DsHq as our two extreme pretraining backbones. As shown in Table 3, the accuracy gap between the two models continues to diverge, with the fully-aligned MLMQ models achieving 18.57% lead over the Mbase model on average. The most striking results appear on the highly challenging AIME competition math problems, where the reasoning- pretriined models deliver 39.32% improvement over the base- line. This provides conclusive evidence that carly investment in reasoning data yields compounding returns, becoming the decisive factor in achieving frontier accuracies the most demanding tasks.
5 ABLATIONS
Does the scale and diversity of the reasoning data matter in Pretraining? As detailed in Ta- ble 1, plainly increasing size and diversity of Dres in pretraining has significant improvement on
7

---

Published as a conference paper at ICLR 2026
Model
Average
MATH AVG
SCIENCEErT AVG
CODEgFT AVG
INSgr7 AVG
Mbuse
SFTsHQ
29.92
42.79
35.83
10,48
30.59
Mose
SPTsho(2Xepochs)
34.01
48.05
40.69
14.60
32.70
Msuq + SFTsHQ
37.33
50.52
40.00
24.76
34.06
MLDQ + SFIsHQ
46.70
60.79
50.67
28.57
46.79
MLMQ SHO
50.95
64.67
53.74
35.55
49.82
Table 4: Impact of diverse pretraining Dres on SFT phase. Doubling SFT for the baseline fails to "catch up" to reasoning-pretraiinn models, while the latent advantage of the mixed-quality pretrain- ing (MLMq) emerges, making it the top performer.
the base model. The model pretrained on large, diverse data (MLDQ) achieves an absolute +9.09% average gain over the model trained on the smaller, less diverse corpus (Mshq), with the largest gains observed in math, science, and code domains that explicitly demand structured reasoning. GPRpr AVG shows limited sensitivity to diversity due to the nature of tasks that require commonsense and general knowledge. In contrast, scaling DLDQ with DsHQ (high-quality but less diverse) as in MLMQ provides minimal further benefit on the reasoning tasks undersordrso that broad exposure to diverse reasoning patterns during pretraining is impactful for building a strong foundation.
The Pretraining Advantage Persists and Resists "Catch-Up" Attempts via SFT. A central ques- tion is whether a model without a reasoning-rich pretraining (Mbase) can compensate for this deficit by undergoing more intensive SFT phase. We test this "catch-up" hypothesis by fine-tuning Mbase with two times more epochs using the same SFT data (DshQ). The results in Table 4 prove this hy- pothesis false. While doubling the SFT epochs improves the baseline's 's average score by 4.09%, this enhanced baseline still fails to match the performance of even our weakest reasoning pretrained model, MąHQ SFTSHQ (+3.32%). This provides strong evidence that pretraining instills a foun- dational reasoning capability that cannot be fully replicated by simply scaling the SFT phase.
POs high-quality data reveals latent value. The downstream consequences of these pretrain- ing choices become more nuanced after SFT. To isolate and test whether these effects persist into post-training, we finetune all base models with the same high-quality SFT recipe (DshQ). Results diverse corpora continue substantially outperform less diverse counterparts even after SFT, confirming that a diverse pretraining foundation enhances a model's capacity to benefit from SFT. More surprisingly, while the immediate gains of scaling with high-quality but narrow data (MLMQ) were muted at the pretraining stage, SFT reveals a latent ad- vantage: MLMQ achieves an additional +4.25%% gain over MLDQ POSt-SFT. This reveals a critical finding that high-quality but less diverse data may act as a complementary amplifier, whose benefits emerge after alignment- underlining the latent impact of quality of data during the pretraining.
in Table 4 confirm that models pretrained
Model
Average
MATHET AVG
SCIENCESFT AVG
CODESET AVG
INSgr7 AVG
Mbuse + SFTsHQ
29.92
42.79
35.83
10.48
30.59
Mns + SFTLMQ
31.21
30.91
27.73
9,79
56.41
M + SFT.DQ
31.54
32.28
28.43
10.85
54.61
Mas + SFTsHQ
44.99
58.66
48.14
29.63
43.56
Table 5: Impact of diverse SFT Dres SFT phase. Fine-tuning the small, high-quality corpus
(DsHQ) is highly effective, while using large, diverse corpora (DLDQ) degrades reasoning.
SFT dominated by data quality, not diversity. We finetune all reasoning-pretrainee models (M..)) on each of our distinct reasoning datasets, and report the averaged results in Table 5. The findings reveal a striking contrast: while diversity beneficial in pretraining, blindly scaling diverse reasoning data during SFT degrades performance. Models trained with DLDQ DIMQ during SFT underperform relative to those finetuned on the smaller, high-quality, long-Cor CoT dataset, DsHQ despite having been exposed to reasoning data during pretraining. In fact, the use of large-scale, mixed-quality data at the SFT stage not only erodes the benefits of reasoning- pretraining but even lead to worse outcomes than the baseline Mbase finetuned with DsHo in math, code, and
8

---

Published as a conference paper at ICLR 2026
Model
Daase:
Drss
Overall
MATH AVG
SCIENCEst AVG
CODEPT AVG
GPRpr AVG
MLMQ
80 20
64.07
72.37
54.49
52.60
76.83
90 / 10
63.97
75.24
53.21
50.19
77.23
60/ / 40
67.28
79.63
55.73
56.47
77.31
Table 6: Effect of varying reasoning data ratio during pretraining. Increasing the reasoning proportion improves reasoning-focuue focused benchmarks while preserving general-domain performance.
science tasks which benefit from reasoning. This result confirms the widely held view that data quality and long reasoning data is critical for effective SFT (Zhou al., 2023; Zhao et al., 2024; Prabhakar et al., 2025). Our findings, however, extend this understanding by showing that simply applying high-quality data at every stage is not optimal. Instead, the most effective strategy is asym- metric: pretraining benefits most from broad and diverse reasoning data to establish generalizable priors, whereas SFT requires high-quality, reasoning-heayy data for targeted refinement.
Reasoning Ratio Sensitivity and Its Interaction with Alignment. As outlined in Section 2.3, the ratio between reasoning data and general pretraining data is fixed at 20% on 400B tokens in our main experiments to ensure fair comparison across DSHQ PLDQ, and DLMQ- While this provides controlled setting, the optimal ratio is inherently empirical and may vary across domains and datasets. To study the sensitivity of this hyperparameter, we conduct additional experiments by varying the reasoning proportion using Dres = DLMQ-
As detailed in Table 6, increasing the reasoning ratio from 20% to 40% improves overall pretraining accuracy and substantially boosts math, science, and code henchmarks. Reducing it to 10% produces a mild drop in overall performance. These results indicate that the reasoning signal scales positively with its proportion during pretraining.
Model
Overall
MATHEFT AVG
SCIENCEgrt AVG
CODE AVG
INSgp7 AVG
MLMQ SFTsHQ
50.95
64.67
53.74
35.55
49.82
MiMQ SFTSHS sHQ [60/40]
52.63
67.71
54.19
43.81
44.81
Table Effect of pretraining reasoning ratio on downstream SFT performance. Higher reason- ing proportion improves reasoning benchmarks but slightly reduces fnslluction-folloiingg metrics.
We further examine how these ratios affect downstream performance after SFT. Using DsH for fine-tuning under the same SFT recipe, we compare the 80/20 and 60/40 pretrained models in Table 7. The 60/40 model continues to improve reasoning benchmarks after SFT, particularly in math and code, but shows decline in fostructionfoollowing performance. This aligns with the breadth aelinthel trade-off discussed in Table 17. Increasing reasoning proportion strengthens structured reasoning capabilities, while general pretraining data contributes broader stylistic and formatting exposure that supports instruction-follwvii indicate that the reasoning ratio is a meaningful control knob. Higher proportions consistently strengthen reasoning performance both before and after SFT, but may modestly reduce alignment- sensitive metrics. The optimal balance may therefore depend on the target deployment domain, and systematic exploration across datasets remains important direction for future work.
flexibility. In summary, these experiments
How should we expand reasoning data during SFT? We next ablate the effect of scaling reason- ing data during the SFT phase by contrasting two strategies: (i) scaling with data of similar quality and diversity, and (ii) scaling with data of higher quality and reasoning depth. As shown in Ta- ble 8, simply doubling the amount of diverse but mixed-quality data yields negligible improvement in average accuracy with a 4.92% drop math noisy or shallow reasoning data may dilute the useful signal and actively harm reasoning sppccifcc domains. The small gains in science and code do not offset this regression, highlighting the limits of ouantity-driven scaling in SFT. In contrast, when scaling DALF with high-quality DsHQ (DALE)) the average accuracy improves further, with math and instruction- followwng tasks benefiting most. Im- portantly, this qualitative expansion is achieved with only a marginal increase in dataset size (0.4% more samples). These contrasting outcomes provide clear evidence that SFT is phase of targeted
cauuracy-uuggeeee
that increasing the volume of

---

Published as a conference paper at ICLR 2026
Model
Average
MATHarT AVG
SCIENCESpT AVG
CODEaFT AVG
INSapr AVG
MLDQ + SFTLDQ
32.84
28,38
35.22
10.16
57.61
MLDQ + SFT2xLDQ
32.99
23.46
39.65
11.75
57.10
MI.DQ + AFT
42.66
60.95
47.29
22.54
39.87
MLDQ + SFT AFFALF
43.04
61.61
45.78
22.53
42.23
Table 8: Impact of scaling reasoning data in SFT phase. Naively doubling mixed-quality data is detrimental to math reasoning, whereas targeted scaling of high-quality data yields consistent gains.
refinement, not broad data absorption; the most effective scaling strategy is strategically enhance the training corpus with high-quality, reasoning-intensive examples.
RELATED WORK
Reasoning in Pretraining and Midtraining, Cheng et al. (2024) study instruction pretraining by converting raw text into short QA pairs and report gains on general-purpose reasoning tasks that require minimal reasoning. While effective for broad linguistic alignment, their setup does not explicitly target reasoning-inten intensive domains such as mathematics, graduate level science, or code. Morcover, their pipeline of self-distilled instruction generation demonstrates that Instruct- PT outperforms vanilla PT after instruction tuning, but it does not assess whether these marginal pretraining gains persist once models undergo reasoning-heavy SFT and reinforcement learning- In contrast, our work systematically varies the complexity, quantity, and diversity of reasoning- style SFT data-ccntaining intermediate thoughts and answers-across both pretraining and SFT, allowing us to probe whether early exposure yields durable downstream advantages.
More recent efforts have begun explore the interplay between pretraining and instruction tun- ing. Liang et al. (2025) augment the instruction-tuning pool to better align with the distribution of pretraining data, reinforcing consistency between the two stages. While complementary in spirit, their method is applied only during SFT and does not address whether reasoning specific supervi- sion at the pretraining stage provides sustained benefits. Similarly, Wang et al. (2025); Al et al. (2025) introduce mid-training phase, continuing pretraining on small but high-quality reasoning dataset before SFT and RLVR. They report substantial downstream gains, particularly in mathe- matics benchmarks, highlighting the promise of mid-training interventions. However, because their corpus is heavily math-centric, it is difficult to disentangle whether the improvements stem from scale, complexity, domain diversity, and the generalizability to science code remains unclear.
spaccfing-ppecffic
complementary direction is pursued by Gandhi et al. (2025), who inject algorithmically generated "cognitive behavioral" reasoning traces during mid-training. demonstrating improvements after inforcement learning. This underscores the potential of early reasoning supervision but remains limited in scope: the interventions are restricted to small datasets and narrow tasks, leaving open questions about scalability, diversity, and phase-specific allocation of reasoning data. Our work builds on these insights by conducting the first systematic, large-scale analysis of reasoning data across both pretraining and SFT, providing principled framework for understanding when and how reasoning supervision should be applied.
7 CONCLUSION
Our study provides the first systematic investigation of how reasoning data, varying in scale, diver- sity, and quality, influences LLMS across the entire training pipeline. We show that reasoning must be introduced early: front-loading into pretraining creates durable foundations that post-training alone cannot recover. Crucially, we uncover an asymmetric allocation prineiple-diversitty drives pre- training effectiveness, while quality governs SFT prov-dividing a clear, actionable blueprint for data strategy. Further, we demonstrate that high-quality pretraining data yield latent benefits activated only during SFT, and that naive SFT scaling with noisy data can be actively harmful. Collectively, these findings challenge the conventional division between pretraining and reasoning, positioning reasoning-awee aware pretraining as a critical ingredient in building more capable, generalizable, and compute-efficient language models.
10

---

wttp:///c.a..oooooooo///02...0 Published as a conference paper at ICLR 2026
REFERENCES
Essential AI, :, Darsh J Shah, Peter Rushton, Somanshu Singla, Mohit Parmar, Kurt Smith, Yash Vanjani, Ashish Vaswani, Adarsh Chaluvaraju, Andrew Hojel, Andrew Ma, Anil Thomas, An- thony Polloreno, Ashish Tanwer, Burhan Drak Sibai, Divya S Mansingka, Divya Shivaprasad, Ishaan Shah, Karl Stratos, Khoi Nguyen, Michael Callahan, Michael Pust, Mrinal Iyer, Philip Monk, Platon Mazarakis, Ritvik Kapila, Saurabh Srivastava, and Tim Romanski. Rethinking reflection in pre-training, 2025. URL https:/arri..orr/as55550.4444
Syeda Nahida Akter, Shrimai Prabhumoye, Matvei Novikov, Seungju Han, Ying Lin, Evelina Bakhturina, Eric Nyberg, Yejin Choi, Mostofa Patwary, Mohammad Shoeybi, and Bryan Catan- zaro. Nemotron--crosthinn: Scaling self-learning beyond math reasoning. 2025. URL https: //axiwworg/abs/2504.1333
Hardy Chen, Haoqin Tu, Fali Wang, Hui Liu, Xianfeng Tang, Xinya Du, Yuyin Zhou, and Cihang Xic. Sft or ri? an early investigation into training rl ri-like reasoning large vision-language models, 2025a. URL http:///aaxiv.or/ab///5555/
1468.
Jierun Chen, Tiezheng Yu, Haoli Bai, Lewei Yao. Jiannan Wu, Kaican Li, Fei Mi, Chaofan Tao, Lei Zhu, Manyi Zhang, Xiaohui Li, Hou, Lifeng Shang, and Qun Liu. The synergy dilemma of long-cot sft and rl: Investigating post-training techniques for reasoning vims, 2025b. URL https://arxiv.orr/ab//2555 077.066......
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fo- tios Chantzis, Elizabeth Barnes, Ariel Hosbert-Voss, William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Josh Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob Mc- Grew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojeiech Zaremba. Evaluating large language models trained on code, 2021. URL https:/arkiv..or/ab///000.0000
Daixuan Cheng, Yuxian Gu, Shaohan Huang, Junyu Bi, Minlie Huang, and Furu Wei. Instruc- tion pre-training: Language models are supervised multitask learners. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Proceedings of the 2024 Conference Methods in Natural Language Processing, pp. 2529-2550, 2024. Association for Computational Linguisties. doi: Miami, Florida, USA, November main.
Empirical
main. 148. URL
Tianzhe Chu, Yuexiang Zhai, Jihan Yang. Shengbang Tong, Saining Xie, Dale Schuurmans, Quoc V. Le, Sergey Levine, and Yi Ma. Sft memorizes, generalizes: A comparative study of foundation model post-training, 2025. URL https///axxix..og///555000.166
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge, 2018. URL https:/aarivvorg/ab///833
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems, 2021. URL https://arxiv.org/ab//111 10. 14168.
Tri Dao and Albert Gu. Transformers are SSMs: Generalized models and efficient algorithms through structured state space duality. In Ruslan Salakhutdinov, Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berkenkamp (eds.), Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pp. 10041-10071. PMLR, 21-27 Jul 2024. URL https:/proocedding..mll pressiv33/dao22aahhn!
11

---

Published as a conference paper at ICLR 2026
Kanishk Gandhi, Ayush K Chakravarthy, Anikait Singh, Nathan Lile, and Noah Goodman. Cog- nitive behaviors that enable self- improving reasoners, or, four habits of highly effective STars. In Second Conference on Language Modeling, 2025. URL http:://oemeevii..e//fonmniie QGJ9IXI.Ty.
Leo Gao. Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Fos- ter; Laurence Golding, Jeffrey Hsu, Alain Le Noac'h, Haonan Li, Kyle McDonell, Niklas Muen- nighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang, Anish Thite, Ben Wang. Kevin Wang, and Andy Zou. A framework for few-shot language model evaluation, 07 2024. URL
Etash Guha, Ryan Marten, Sedrick Keh, Negin Raoof, Georgios Smymis, Hritik Bansal, Marianna Nezhurina, Jean Mercat, Trung Vu, Zayne Sprague, Ashima Suvarna, Benjamin Feuer, Liangyu Chen, Zaid Khan, Eric Frankel, Sachin Grover, Caroline Choi, Niklas Muennighoff, Shiye Su, Wanjia Zhao, John Yang, Shreyas Pimpalgaonkar, Kartik Sharma, Charlic Cheng-Jie ji Ji, Yichuan Deng, Sarah Pratt, Vivek Ramanujan, Jon Saad-Falcon, Jeffrey Li, Achal Dave, Alon Albalak, Kushal Arora, Blake Wulfe, Chinmay Hegde. Greg Durrett, Sewoong Oh, Mohit Bansal, Saadia Gabriel, Aditya Grover, Kai-Wei Chang, Vaishaal Shankar, Aaron Gokaslan, Mike A. Merrill, Tatsunori Hashimoto, Yejin Choi, Jenia Jitsev, Reinhard Heckel, Maheswaran Sathiamoorthy, Alexandros G. Dimakis, and Ludwig Schmidt. Openthoughts: Data recipes for reasoning models, 2025. URL
Sonam Gupta, Yatin Nandwani, Asaf Yehudai, Dinesh Khandelwal, Dinesh Raghu, and Sachindra Joshi. Selective self-to-supervised fine-tuning for generalization in large language models, 2025. URL https//arkii..or/bbb/22500... 130.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. Proceedings of the Interna- tional Conference on Learning Representations (ICLR), 2021a.
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. NeurIPS, 2021b.
Maggie Huan, Yuetai Li, Tuney Zheng, Xiaoyu Xu, Seungone Kim, Minxin Du Radha Pooven- dran, Graham Neubig, and Xiang Yue. Does math reasoning improve general Ilm capabilities? understanding transferability of Iim reasoning, 2025. URL https://axxiv.or/abs//5500 http./oor//abs/550000222222 .00432.
Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang. Armando Solar-Lezama, Koushik Sen, and Ion Stoica. Livecodebench: Holistic and contamination free evaluation of large language models for code. In The Thirteenth International Conference on Learning Representations, 2025. URL http://ppeeeeeeee.eteett/o//llld
Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik R Narasimhan. SWE-bench: Can language models resolve weal-world github issues? In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview. net/forum "iVTormmmmddVVTTNNNNMMMMM
Guokun Lai, Qizhe Xie, Hanxiao Liu, Yiming Yang. and Eduard Hovy. RACE: Large-scale ReAd- ing comprehension dataset from examinations. Martha Palmer, Rebecca Hwa, and Sebastian Riedel (eds.). Proceedings of the 2017 Conference Empirical Methods in Natural Language Processing, pp. 785-794, Copenhagen, Denmark, September 2017. Association for Computa- tional Linguistics. doi: 10.1665//1/DD1--1
URL http:://ccaanhhollg.ooo//1111111
Yiming Liang, Tianyu Zheng, Xinrun Du, Ge Zhang, Jiaheng Liu, Xingwei Qu, Wenqiang Zu, Xingrun Xing, Chujie Zheng, Lei Ma, Guoyin Wang, Zhaoxiang Zhang. Wenhao Huang. Xiang Yue, and Jiajun Zhang. Aligning instruction tuning with pre-training. 2025. URL https://arxiv. org/abs/2501.0995
Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and LINGMING ZHANG. Is your code gen- crated by chatgpt really correct? rigorous evaluation of large language models for code generation. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine
12

---

Published as a conference paper at ICLR 2026
(eds.), Advances in Neural Information Processing Systems, volume 36, pp. 21558 21572. Curran Associates, Inc., 2023. URL
Zihan Liu, Zhuolin Yang, Yang Chen, Chankyu Lee, Mohammad Shoeybi, Bryan Catanzaro, and Wei Ping. Accreason-nemootron 1.1: Advancing math and code reasoning through sft and rl synergy, 2025. URL https://axxiv.orr/aas/55555
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Confer- ence on Learning Representations, 2019. URL http:///pereeooooo..//o/oo/dd/ad
Michael Luo, Sijun Tan, Justin Wong, Xiaoxiang Shi, William Y. Tang, Manan Roongta, Colin Cai, Jeffrey Luo, Erran Li, Raluca Ada Popa, and lon Stoica. Deepscaler: Sur-
passing ol ol-preview with 1.5b model by scaling rl. https://pretty-radie rraito--rr---bbbee rwdir-aadd.anaeee 075.notion.site/ 1-ath------5--Modeell
Sarppasinng-O-PPeee 2025a. Notion Blog.
Yun Luo, Zhen Yang, Fandong Meng, Yafu Li, Jie Zhou, and Yue Zhang. An empirical study of catastrophic forgetting in large language models during continual fine-tuning, 2025b. URL hutps//arxivvorr/abb/3333 08747.
Ivan Moshkov, Darragh Hanley, Ivan Sorokin, Shubham Toshniwal, Christof Henkel, Benedikt Schifferer, Wei Du, and Igor Gitman. Aimo-2 winning solution: Building state-of-the-art mathe- matical reasoning models with openmathreasoning dataset, 2025. URL https://aniii.orr/aa//25544
16891.
NVIDIA. Nemotron-h: A family of accurate and efficient hybrid trabb-transformer models, 2025a. URL http:::://aaxiiorrr////22004//33
NVIDIA. Nvidia nemotron nano 2: An accurate and efficient hybrid mamba-transformer reasoning model, 2025b. URL https://aniv.orr/abb/5500.. 14444.
Augustus Odena, Charles Sutton, David Martin Dohan, Ellen Jiang. Henryk Michalewski, Jacob Austin, Maarten Paul Bosma, Maxwell Nye, Michael Terry, and Quoc V. Le. Program synthesis with large language models. In n/a, pp. n/a, n/a, 2021. n/a.
Long Phan et al. Humanity's 's last exam, 2025. URL https//arri..or//a//23111.41
Vignesh Prabhakar, Md Amirul Islam, Adam Atanas, Yao-Ting Wang, Joah Han, Aastha Jhun- jhunwala, Rucha Apte, Robert Clark, Kang Xu, Zihan Wang, and Kai Liu. Omniscience: A domain-speccialzed Ilm for scientific reasoning and discovery, 2025. URL https://arxiw.or/abbs 2503.17604.
David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R. Bowman. GPQA: A graduate-level google-proof q&a benchmark. In First Conference Language Modeling, 2024. URL https://opemneeiww.neel 71775544--6655844998.
Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adver- sarial winograd schema challenge at scale. Communications of the ACM, 64(9):99-106, 2021.
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y. K. Li, Y. Wu, and Daya Guo. Deepseekmath: Pushing the limits of mathe- matical reasoning in open language models, 2024. URL https///axii.oog///2222222200000
Hemish Veeraboina. Aime problem set 1983-2024, 2023. URL http://www.kggll..ommaasssss hemissveeraaboinaaaime
htwprrrbbiiiiaaiiem-rolbmmmeeese problem-set- 1983-2024.
Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang. Tianle Li. Max Ku, Kai Wang, Alex Zhuang, Rongqi Fan, Xiang Yue, and Wenhu Chen. Mmlu-pro: more ro- bust and challenging multi-task language understanding benchmark. In A. Globerson,
13

---

Published as a conference paper at ICLR 2026
L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomezak, and C. Zhang (eds.), Ad- vances in Neural Information Processing Systems, volume 37. pp. 95266-952999 Cur- Associates, Inc., 2024. URL https://procceednns.. www.pss..suiic.ccpaappee files/paper/222/fiile 122 Paper-Datasees and Benchmarks Track.pdf.
Zengzhi Wang, Fan Zhou, Xuefeng Li, and Pengfei Liu. Octothinker: Mid-training incentivizes
reinforcement learning scaling. arXiv preprint arXiv:2506.20512 2025. URL https://arxiv.org/ abs/2506.20512.
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can ma- chine really finish your sentence? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, 2019.
Zhiyuan Zeng, Jiatong Yu, Tianyu Gao, Yu Meng, Tanya Goyal, and Danqi Chen. Evaluating large language models at evaluating instruction following. In International Conference on Learning Representations (ICLR), 2024.
Hao Zhao, Maksym Andriushchenko, Francesco Croce, and Nicolas Flammarion. Long is more for aligmment: A simple but tough-to-beat baseline for instruction fine-tuning. arXiv preprint 240.222400.04833, 2024.
Chunting Zhou, Pengfei Liu, Puxin Xu, Srinivasan Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, Lili Yu, et al. Lima: Less is more for alignment. Advances in Neural Information Processing Systems, 36:55006-55021, 2023.
Fan Zhou, Zengzhi Wang. Nikhil Ranjan, Zhoujun Cheng, Liping Tang, Guowei He, Zhengzhong Liu, and Eric P. Xing. Megamath: Pushing the limits of open math corpora, 2025. URL https: //aaxiv.org/abs/250400220
A EXPERIMENTS AND RESULTS
Breakdown of Pretraining Results. Table 9 provides a fine-grained view of the pretraining ef- fects across individual benchmarks, complementing the domain-level averages reported in Table 1. The improvements are not confined to a small subset of tasks but are broadly distributed, with the largest gains concentrated in mathematically intensive and program synthesis benchmarks. For ex- ample, on GSM8K and MATHI-50,, models trained with large-scale reasoning data exhibit dramatic jumps over Mbase, with MLDQ and MLMQ more than doubling performance on MATH-500. Sim- ilar trends appear in code generation tasks such as HUMANEVAL, JUUANEVALPLUS and MBPP, models consistently outperform the baseline, indicating stronger pro- cedural and compositional capabilities.
where masoning-auugmenee
In contrast, gains on general-purpose reasoning benchmarks such as ARC-C, RACE, WINO- GRANDE, and MELLASSAG are more modest, suggesting that these tasks are less sensitive to ex- plicit reasoning supervision during pretraining. Nevertheless, even in these cases, performance re- mains stable or slightly improved, indicating that incorporating reasoning data does not harm broad language understanding. Overall, the benchmark- results reinforce the conclusion that early exposure to diverse reasoning corpora yields systematic improvements, with the most pronounced benefits emerging on tasks that require multi-step deduction, symbolic manipulation, or structured program synthesis.
Detailed SFT Benchmark Performance. While the Table 2 presents aggregate performance met- rics across different reasoning domains, Table 10, 11, and 12 provide a granular breakdown of how each reasoning model performs across individual benchmarks when fine-tuned on DSHQ DLDQ. and DLMQ respectively.
These detailed results confirm that the "reasoning ddvaatagee"estaaliiii during the IT pretraining phase is not limited to aggregate scores but persists across all distinct evaluation categories, includ- ing AIME, GSM8K, MMLU, and LIVECODEBENCH Models pretrained with reasoning-sper specific corpora (MDdQ and MLMq) consistently outperform Mbase and MssQ across these benchmarks, regardless of the SFT dataset used. Specifically, MLMQ + SFT frequently achieves the highest
14

---

Published as a conference paper at ICLR 2026
Benchmark
Mbase
Msnq
MLDQ
MLMQ
Mres
ARC-C
80.89
80.46
81.40
81.83
81.15
RACE
73.59
75.41
78.28
79.43
76.68
WINOGRANDE
70.61
71.43
69.53
69.38
70.25
HELLASWAG
77.38
77.06
76.69
76.67
76.95
GSM8K
59.74
65.20
82.71
85.14
73.20
MATH-500
34.60
40.00
68.40
59.60
50.65
MMLU
61.67
61.45
65.87
65.42
63.60
MMLU-PR
32.59
32.34
42.89
43.56
37.85
HUMANEVAL
37.44
41.04
48.63
51.68
44.70
HUMANEVALPLUS
32.59
35.03
42.74
46.28
39.16
MBPP
41.64
47.47
48.85
51.47
47.36
MBPP[SANITIZED]
51.87
53.74
59.53
60.97
56.53
МATHºт AVG
47.17
52.60
75.56
72.37
61.92
SCIENCE,t
17.13
46.90
54.38
54.49
50.72
CODEşT AVG
40.89
44.32
49.94
52.60
46.94
GPR AVG
75.63
76.09
76.48
76.83
76.25
Overall
52.70
54.98
64.09
64.07
61.05
Table 9; Breakdown of base model accuracies across benchmarks. With increasing diversity and quality, the difference between Mbase and models pretrained with reasoning data increases.
SFT Dataset: DSHQ
Benchmark
Mbuse + SFT
MsHQ + SFT
MLDq + SFT
MLMq + SFT Mree + SFT
IFEVAL
30.59
34.06
46.79
49.82
43.56
AIME-24
8.12
18.33
35.21
41.88
31.81
AIMB-25
11.88
18.12
29.38
33.12
26.87
GSMSK
81.24
86.58
91.05
92.84
90.16
MATH-500
69.90
79.05
87.50
90.85
85.80
MMLU
52.14
62.90
71.15
73.49
69.18
MMLU-PRO
39.45
48.63
53.45
55.54
52.54
GPQA-DAN DDAA-DD
15.91
8.46
27.40
32.20
22.69
LIVECODEBENCH
10.48
24.76
28.57
35.55
29.63
MATHay AVG
42.79
50.52
60.79
64.67
58.66
SCIENCEs1 AVG
35.83
40.00
50.67
53.74
48.14
CODEgFT AVG
10.48
24.76
28.57
35.55
29.63
INS AVG
30.59
34.06
46.79
49.82
43.56
Overall
35.52
42.32
52.28
56.14
50.25
Table 10: Breakdown of model accuracies across benchmarks after training SFT phase on the DsHQ-
peak performance in complex reasoning tasks like AIME-24/25 and MATH-500, suggesting that high-quality, dense reasoning pretraining provides a more robust initialization that SFT can more effectively refine.
To facilitate direct comparison of how different SFT strategies interact with varying pretraining foundations, Table 13 aggregates the results across model-SFT combinations, The data under- scores clear synergy: the best overall performance is achieved when models are pretrained on diverse, high-quality reasoning data and subsequently fine-tuned on a compatible SFT dataset. This highlights that while SFT is crucial for aligning model behavior, the "foundation" built during pre- training serves as the primary determinant for the model's upper bound on complex reasoning ca- pabilities. Even when given the same SFT budget, models lacking reasoning-speciffe pretraining
15

---

Published as a conference paper at ICLR 2026
SFT Dataset: PLDQ
Benchmark
Mbase + SFT
MMHQ + SFT
MMnq + SFT MLMQ +
SFT Mns. + SFT
IFEVAL
50.86
47.01
57.61
59.21
54.61
AIMB-24
1.15
2.50
6.37
4.90
4.59
AIME-25
0.83
3.12
7.71
9.38
6.74
GSM8K
73.56
75.11
59.81
77.62
70.84
MATH-500
46.70
44.98
39.63
56.28
46.96
MMLU
15.25
9.95
49.15
56.81
38.64
MMLU-PRO
16.26
14.24
30.50
33.51
26.08
UPQA-DIAMONT
8.97
7.39
26.01
28.35
20.58
LIVECODERENCH
6.04
10.48
10.16
11.91
10.85
MATHaF AVG
30.56
31.43
28.38
37.04
32.28
SCIENCEEeT AVG
13.49
10.52
35.22
39.55
28.43
CODE AVG
6.04
10.48
10.16
11.91
10.85
INSaFT AVG
50.86
47.01
57.61
59.21
54.61
Overall
25.24
24.86
32.84
36.93
31.54
Table 11: Breakdown of model accuracies across benchmarks after training SFT phase on the DLDQ-
SFT Dataset: DLMQ
Benchmark
Mbsse + SFT MaHq + SFT MuDQ + SFT
MLMq SFT
Mis SFT
IFEVAL
50.50
52.65
57.78
58.79
56.41
AIME-24
1.25
3.13
8.23
1.69
5.35
AIMB-25
0.84
2.92
6.98
7.09
5.66
GSM8K
72.93
74.03
57.70
76.27
69.33
MATH-500
45.33
42.18
36.93
50.75
43.28
MMLU
15.75
6.90
50.90
55.15
37.65
MMLU-PRO
15.57
13.18
32.09
33.37
26.21
GPQA-DIAMOND
8.97
4.87
23.17
29.99
19.34
LIVECODEBENCH
4.76
9.37
10.95
9.05
9.79
MATHe²T AVG
30.09
30.56
27.46
34.70
30.91
SCIENCEST AVG
13.43
8.31
35.39
39.50
27.73
CODE AVG
4.76
9.37
10.95
9.05
9.79
INS577 AVG
50.50
52.65
57.78
58.79
56.41
Overall
24.69
25.22
32.89
35.51
31.21
Table 12: Breakdown of model accuracies across benchmarks after training SFT phase the DLMQQ
struggle to close the performance gap, particularly in challenging domains like code generation and advanced mathematics.
B ADDITIONAL ABLATIONS
Generalization and Robustness Across Model Scales and Architectures. To ensure trans- parency and reproducibility. all pretraining, SFT, and RL dataaetts-includdng Common Crawl, Arxiv, Wikipedia, StackExchange, GitHub, Webneet and OpenWebMath- fully open corpora. To validate the external validity of our data strategy and the effects of front- loading reasoning data across varying scales and architectures, we extended our experiments to Transformer baasd model M with 1.2B parameters trained 125B tokens. We compared the base- line Muuse against My which integrates early reasoning exposure from DiMO while maintaining constant token ratio between D and DLMO-
derived from
16

---

Published as a conference paper at ICLR 2026
Model
gT AVG
SCIENCEeFT AVG
CODE AVG
INSaFT AVG
Musse SFIsHQ
42.79
35.83
10.48
30.59
Mbose
LFD
30.56
13.49
6.04
50.86
Musse
LFT
30.09
13.43
4.76
50.50
Mbsse SFT
34.88
20.92
7.09
43.98
Maa + SFTTsHQ
50.52
40.00
24.76
34.06
MLDQ + SFTsHQ
60.79
50.67
28.57
46.79
MIMQ SHT
64.667
53.74
35.55
49.82
MsHq + SFTTnQ
31.43
10.52
10.48
47.01
MLDQ + LFQ
28.38
35.22
10.16
57.61
MLMQ LFD
37
39.55
11.91
59.21
MsnQ + SF'TLMQ
30.56
8.31
9.37
52.65
MLDQ + SFTTMQ
27.46
35.39
10.95
57.78
MLMQ SFTLMQ
34.70
39.50
9.05
58.79
Mas + SFT
40.62
34.77
16.75
51.52
Table 13: Results of all SFT models with varying pretraining and SFT data. Model pretrained with reasoning data obtains the highest gain after SFT phase of training.
Model
MATHE AVG
SCIENCEPT AVG
CODEşt AVG
GPRrr AVG
Overall Avg
Mbsse 6,92
18.91
12.82
40.92
19.89
MIMQ
16.39
21.50
23.44
41.69
25.75
Table 14: Effect of early reasoning exposure on a 1.2B Transformer model. The reasoning- augmented model improves substantially on math, science, and code benchmarks while maintaining general-domain performance.
As summarized in Table 14, the integration of reasoning data consistently enhances task-specific performance across different architectures. We observed substantial gains, specifically +9.47% in Math, +2.60% in Science, and +10.62% Code benchmarks. Critically, these enhancements do not degrade generalized capabilities; performance on general benchmarks (GPRp7 AVG) remains stable, indicating that early exposure to reasoning data does not perturb the model's fundamental language modeling objectives. These findings provide strong empirical evidence that our data strategy is robust and scalable, demonstrating that front-loading reasoning data is an effective mechanism for augmenting complex deduction capabilities regardless of the underlying model scale architecture.
Anatomy of high-quality reasoning data in SFT. Our previous results establish that SFT bene- fits immensely from high-quality data, but what precisely constitutes "quality" remains unclear. In this ablation, we investigate defining characteristic of such data: the depth and complexity of its reasoning traces. Specifically, we compare datasets that differ both in reasoning length and con- struction method. The high-quality corpus DsHQ consists of answers generated by strong teacher models, characterized by long Shain-of-thought with average length exceeding 10k tokens. In contrast, DLDQ provides reasoning data from diverse domains but with much shorter and noisier reasoning traces (average ~550 tokens). This distinction highlights potential mechanism underly- ing quality: longer reasoning chains may serve as richer supervisory signals, encouraging models to internalize structured multi-step inference rather than surface-leved heuristics.
To test this hypothesis, we extract from DLLO only the longest reasoning traces, creating a new dataset DLLF- Although it represents only ~2% of the original DLLQ corpus, DALF is highly skewed toward domains with inherently deeper reasoning (75% math, with the remainder in science, code, and general reasoning). We then conduct SFT on top of the Milg model using both DLLQ (quantity and diversity) and DALP (length-filtered complexity).
As shown in Table 15, emphasizing depth in reasoning traces has a significant impact on downstream reasoning tasks. While finetuning with DELo yields only modest improvements, switching to the
17

---

Published as a conference paper at ICLR 2026
Model
Average
MATHsFT AVG
SCIENCESFT AVG
CODEaFT AVG
INSaFT AVG
MLDQ + SFTLDQ
32.84
28.38
35.22
10.16 22.54
57.61 39.87
MLDQ + ALF
42.71
60.95
47.50
Table 15: Impact of depth reasoning traces in data on SFT phase. Model trained on longer CoT reasoning data outperforms the one trained diverse reasoning traces.
50 times smaller, filtered by reasoning depth via answer length DALF boosts the overall score to 9.87%, with particularly strong gains in math, science and code. Interestingly, this comes the cost of slightly reduced accuracy on cntructtio-followiiin and reasoning-specifte depth. These results provide strong evidence that longer chain-of-thought supervision is a critical marker of quality in SFT data. Even when drawn from noisy, large-scale corpus, selecting for reasoning depth alone can yield outsized improvements, making length-filtering simple yet cost-effective heuristic for constructing impactful reasoning datasets for SFT phase.
tasks, reflecting a trade-off between breadth
Data Redundancy Reinforces Foundational Skills, Not Overfitting. A critical consideration in our two-phase approach is whether using the same reasoning data in both pretraining and SFT leads catastrophic forgetting or brittle overfitting, known concern in sequential fine-tuning (Luo al., 2025b; Chen et al., 2025a).
Our results, shown in Figure suggest this concern is unfounded and that the opposite is true: for reasoning, strategic redundancy is highly beneficial. The baseline model, Mbase, ex- posed the high-quality DSHQ data only during SFT. is the lowest performer across all cate- gories. In contrast, MsHQ which sees this same data in both phases, demonstrates a significant performance uplift, indicating that the second exposure reinforces rather than overwrites learn- ing. We hypothesize this occurs because the two training phases serve different learning func- tions. During pretraining, the reasoning data is integrated slowly into the model's core represen- tations alongside vast, diverse knowledge, forcing internalization of abstract logical patterns. The SFT phase then acts not as a new learning task, but as powerful reinforcement signal on an already- prepared foundation. This benefit amplified by a diverse pretraining context: the top-performing MLMQ model leverages its broad exposure to various reasoning styles to most fectively capitalize on the repeated, high-quality signal from DsHQ. This suggests that data redundancy be- tween pretraining and SFT should be viewed as a powerful mechanism for skill consolidation, where a diverse pretraining builds the capacity for reasoning and redundant SFT sharp- it.
Mo. AFN)
Masg SFT)),ng
80
Mexaq PFT!
Misru amol
40
MATH AVI
SCIENCE AVG
CODE AVG
INST AVO
Figure 2: The model that saw the same high-quality data in both pretraining and SFT (MsHq) handily beats the base- line (Mosse) that only saw the data once.
Prefrained Foundations vs. Scaling SFT Data. In Table 4, the notation SFT 2 refers to extend- ing the SFT phase for twice the number of epochs over the existing dataset, rather than expanding the training set with unique samples. To further investigate whether reasoning performance is an artifact of data quantity-ant and to address potential concerns regarding overfitting with repeated data-we conducted a controlled experiment comparing a no-reason base model against our reason-aware baseline. We examined whether the no-reason base model, Mhasee could recover the performance of the beason-award model, MLDQ. by utilizing significantly larger volume of unique SFT tokens. We fine-tuned Mbase on combined dataset of DLDQ and DALE (totaling 268M unique samples) and compared it against Manqo which was fine-tuned solely on D ALF (7.1M unique samples). Both models evaluated under the identical SFT evaluation setup.
18

---

Published as a conference paper at ICLR 2026
Model
MA AVG
SCIENCEgFT AVG
CODEgFT AVG
INSgFr AVG
Overall
Misse + SF.TLDQ+ALE MUDQ SF ALF
33.66
29.15
3.49
56.86
30.79
60.95
47.29
22.54
39.87
42.66
Table 16: Comparison between SFT with large amounts of unique data and reasoning- augmented pretraining. The reasoning- -basdd model substantially outperforms the no-reason base- line despite using significantly fewer unique SFT samples.
Instruction Type
MInq SFTALF
Munq + SFTTDD
Diff
punctuation:no comma
27
length. constraints:number words
change caserenglish lowercase
19
28
frequency
13
21
8
saannge.css:eenlisll... capital
16
language:response language
detectable. format:numbr..bull
21
combination:two sespoossssssssssss
11
16
5
wwwoordsforbidden..o words
12
17
5
detectable format:titl
32
36
startend-quotation
change case:capital word frequency
12
length_conste con..oostrainttsnmmeerrrrr htterrainnsnuubmbbrraarrrrra paragraphs
12
3
length_com weeethhconsttriittsnthhh paragttpa.ppphh..... wird word
length_consttenee constraints:numberrrro somta -eettencssss
29
detectable format:json format
startend:end checker
11
13
detectable format:number highlighted sections
40
41
detectable detectable.fommatteoo formatt:constriineddd...... formaatconstaaineerrsssssss response
10
detectable numbert
wwwpammeeeeeeapllderrrrrr. placeholders
combination.repeatt prompt
detectable_formen wtt.ffotmmmmillleeecctttoooooooo
12
12
detectable.comtentttostssrr
23
22
keywords:existence keywords.frequeecy
31
Table 17: Instruction-wise accuracy comparison IFEval between models trained with less diverse (Dalf) and more diverse (DI.Dq) corpora.
As demonstrated in Table 16, the reason-aware base model achieves 39% relative improvement in the Overall score and consistently outperforms the no-reason baseline across all domains, despite the latter receiving substantially more unique training data. This result highlights that strong reasoning foundations established during pretraining cannot be trivially recovered through additional SFT, even when the model is provided with significantly larger scale of unique data.
Breadth vs. Alignment: The Role of Data Diversity in Instruction Following. Table 5 reveals consistent trade-off: length-filtered long-CoT SFT DALF improves reasoning performance while weakening instruction- followiin ability. This pattern suggests that reduced data diversity affects alignment-sensitive behaviors. We hypothesize that this degradation stems from distributional skew and stylistic rigidity, as DALF is heavily concentrated in math and code domains, similar in structure DsHQ- While such data reinforces precise token-level reasoning, exposes the model nar- rower range of linguistic forms and formatting variations. To investigate this phenomenon, we com- pare instruction-level accuracy on the IFEval benchmark between two models: MLDQ + SFTTLF (less diverse SFT) and MLDQ SFTLDQ (diverse SFT).
As shown in Table 17, the diverse DLDQ dataset yields substantial improvements in linguistic ma- nipulation tasks, including punctuation control, case transformation, word-count constraints, and formatting instructions. These categories require flexibility in natural language generation, sensitiv- ity to stylistic variation, and the ability to adapt output structure. Exposure to diverse reasoning data
19

---

Published as a conference paper at ICLR 2026
Model
Overall Avg
MATHa? AVG
SCIENCEs7 AVG
CODE
GPRşr AVG
Mnnq 64.09
75.56
54.38
76.48 49.94
MALF
52.59
46.73
46.03
41.66
75.96
Table 18:
Effect of reducing scale and diversity of reasoning data during pretraining. MALF trained on a length-filted filtered subset of PLDQ that smaller and less diverse.
introduces broader linguistic cues, Markdown structures, and formatting styles, enabling stronger control over output form.
In contrast, the math- and code-heavy DALF dataset reinforces strict token precision and pattern adherence. This benefits tasks involving keyword presence, token counting, or rigid structural con- straints but provides limited stylistic variability. As result, models trained on less diverse data excel at "hard" lexical constraints while underperforming on "soft" linguistic transformations that require adaptability. These findings clarify that data diversity plays critical role in preserving instruction- following capabilities. Reasoning gains from length-filtered long-Cor -ooT data come with a narrowing of stylistic exposure, and broader linguistic coverage helps maintain alignment-related flexibility without sacrificing reasoning performance.
Controlling for Scale and Diversity in Reasoning Pretraining. To isolate the role of scale and diversity from other potential confounders such as dataset source or content differences, we conduct a controlled pretraining experiment using downsampled subset of DI.D Specifically, we con- struct Ale by retaining only examples whose answer length exceeds 4096 tokens. This filtering procedure selects long-chain-of-theee of-thought samples that are typically math- and code-centric, resulting in data that is arguably high quality in reasoning depth but substantially smaller in scale and less di- verse in topical coverage than the full DLDQ corpus. During pretraining of .MALP, we maintain the same token ratio between Dase and DALF in the original setup ensure controlled comparison.
As detailed in Table 18, the model trained on the downsampled subset exhibits an absolute 11.5 % drop in overall pretraining accuracy compared MLDQQ The degradation is particularly pro- nounced in math and code benchmarks, while general-doman performance remains relatively stable. These results indicate that although long-chain-of-thouge samples capture high-quality reasoning traces, reducing dataset scale and diversity substantially weakens the overall pretraining signal.
This controlled comparison supports the claim that reasoning quality alone insufficient to explain performance gains. Instead, diversity and scale play a dominant role during pretraining, enabling broader generalization across reasoning- iasoning-intensive domains.
20


Reviews:
[{"review_id": "tF2tstfEsW", "reviewer": "Reviewer_6sC9", "summary": "The paper studies when to introduce reasoning-style data in the LLM pipeline. Using an 8B hybrid Mamba-Transformer trained from scratch for 1T tokens, the authors compare injecting reasoning data during pretraining vs. during SFT (and then RL). Core findings: (1) adding diverse reasoning data early (during pretraining) creates durable advantages that post-training cannot “catch up” to (reported ~19% average gain on expert benchmarks); (2) an asymmetric allocation principle: diversity & scale matter most in pretraining, while quality (long CoT) dominates SFT; (3) naïvely scaling mixed-quality SFT data can hurt reasoning; (4) some high-quality pretraining effects are “latent” and only unlock after SFT.", "questions": "see weakness", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Comprehensive empirical study. Thorough, well-controlled experiments quantify how reasoning data affects both pretraining (PT) and post-training (SFT/RL).\n\nActionable principle. Clear rule of thumb: prioritize diversity and scale of reasoning signals during PT; prioritize quality (e.g., long, high-fidelity CoT) during SFT—immediately useful for practitioners.\n\nCross-stage validation. Systematic swaps across PT/SFT/RL (controlling data type/quality/scale) repeatedly confirm the “front-loading” thesis, with consistent gains (Tables 1–6).\n\nInformative negative result. Simply doubling mixed-quality SFT data provides no benefit and can degrade math performance, underscoring that quantity cannot replace quality.\n\nLatent PT benefit. A small amount of high-quality reasoning introduced in PT yields improvements that only manifest after SFT, clarifying stage interactions.", "weaknesses": "External validity & dataset specificity. The pretraining relies on NVIDIA-curated corpora (~6.2T tokens) and SFT sets such as OpenThoughts and Nemotron-Pretraining-SFT-v1. It’s unclear how well the conclusions transfer to other pretraining bases or fully open corpora.\n\nClarity on PT/SFT data mixing and table notation. For Tables 2–3, please spell out the exact SFT composition used during post-training for each pretraining variant and standardize the notation. In Table 4, the meaning of SFT×2 is ambiguous: is the dataset literally duplicated (two passes over identical examples) or expanded with additional unique samples? If duplicated, did the model simply see the same data twice (risking overfitting), and was the token budget controlled elsewhere?\n\nBreadth vs. alignment trade-off. Length-filtered long-CoT SFT (DALF) appears to improve reasoning while weakening instruction-following. Any concrete hypothesis (e.g., length effects, stylistic drift, or distribution shift) plus diagnostics on this?\n\n\nPretraining mixture definition (80% D_base / 20% D_res). The paper states: “we pretrain from scratch for 1T tokens using 80% D_base and 20% D_res.”  how is that 20% constructed on different reasoning dataset settings?  If any sampling strategy is used? Please clarify this.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4my3DXYPXu", "reviewer": "Reviewer_2v5m", "summary": "This paper investigates the benefits of adding reasoning data earlier during pretraining. The authors vary the scale, diversity and quality of the reasoning data and show that front-loading reasoning data into pretraining stage not only builds better pretrained models but also synergies with subsequent post-training. They also show that diversity of the reasoning data is crucial for pretraining, whereas quality is important for SFT stage.", "questions": "1. Is D_{res} always 20% of the data (mentioned at Line 189)? If yes, why there are different scales of reasoning datasets (D_{SHQ} and D_{LDQ})? Can you clarify?\n\n2. How is the ratio of reasoning data decided? Did you try other ratio (higher or lower than 20%) and does that affect your conclusions? \n\n3. What is D_{LLQ}? at Line 171-172? It seems not defined.\n\n4. Line 349-350, could you explain why D_{SHQ} is mentioned, should it be D_{LMQ}?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper systematically studies the effects of front-loading different types of reasoning data (in terms of quality and diversity) into pretraining, which is novel and timely.\n\n- The experiments are conducted at a large-scale and evaluated on diverse downstream tasks, which makes the conclusions more practical.\n\n- The finding that high-quality pretraining data shows benefits only after SFT is interesting.", "weaknesses": "1. Major: 'Scale and diversity of the reasoning data are important in pretraining' is not well supported. D_{LDQ} and D_{SHQ} are from different datasets, and scale and diversity is not the only difference. The content of the dataset, as well as the quality of the dataset, are likely to be confounders. I think a more controllable comparison is to reduce the size and diversity of D_{LDQ} (-->D'_{LDQ}) and compare M_{LDQ} and M'_{LDQ}.\n\n2. Minor: Some observations remain to be better understood, for example why exactly pretraining benefits more from diversity and SFT betnefits more from quality.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "947bRWjGb7", "reviewer": "Reviewer_WLmB", "summary": "This paper investigates the stage to introduce reasoning data into the training process of Large Language Models (LLMs). The authors challenge the conventional approach of adding reasoning skills during the post-training stage and  argue for incorporating it early on, during pretraining.", "questions": "1. The objective in Equation 1 requires clarification. The notation $E_{t \\sim \\mathcal{T}}$ is ambiguous as to whether a single fine-tuned model is evaluated across all tasks in $\\mathcal{T}$, or if a unique model is fine-tuned specifically for each task $t$.\n2. Why choosing a hybrid model, is the conclusion the same for Decoder-only transformer?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "1. This paper provides a detailed comparison of pretraining with $\\mathcal{M_{LDQ}}$, $\\mathcal{M_{SHQ}}$, and a mixture of both, thoroughly evaluating their impact on the model's performance across different reasoning tasks after pretrain and SFT. \n2. This paper shows that introducing reasoning data in pretrain stage can improve model performance in both pretrain stage and SFT stage \n3. This paper demonstrates that the SFT stage should leverage higher-quality data with greater reasoning depth to significantly enhance the model's final performance.", "weaknesses": "1. All the performances are evaluated in reasoning tasks, the performance on the memry task are not evaluated.\n2. The paper does not clearly specify how the $\\mathcal{M_{LDQ}}$ and $\\mathcal{M_{SHQ}}$ datasets were mixed to create $\\mathcal{M_{LMQ}}$ (especially ratio). It would be important to investigate whether different mixing ratios significantly impact the model's performance, and specifically, whether an optimal ratio exists that would allow $\\mathcal{M_{LMQ}}$ to outperform $\\mathcal{M_{LDQ}}$.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "d4zlv3Gu4o", "reviewer": "Reviewer_hBYq", "summary": "This paper challenges the conventional practice of adding reasoning data only during fine-tuning. Through controlled experiments, it demonstrates that front-loading diverse reasoning data into pretraining creates an advantage that cannot be matched by later stages.\n\nTheir key findings are:\n\n1. Pretraining benefits most from scale and diversity of reasoning data, building broad and robust foundational capabilities.\n2. SFT is dominated by data quality, where small amounts of high-quality, long chain-of-thought data are vastly more effective than large, diverse but lower-quality mixtures. Naively scaling SFT data with mixed quality can even be detrimental.\n3. High-quality pretraining data has a latent effect, where its full benefit is \"unlocked\" only after the subsequent SFT phase.\n\nThe study concludes that a strategic, asymmetric data strategy—prioritizing diversity in pretraining and quality in SFT—is essential for building more capable and efficient reasoning models.", "questions": "See weakness.", "limitations": "", "rating": 4, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. The authors present a systematic and thorough experimental setup, including baseline comparisons, multiple data regimes (scale, quality, diversity), and evaluation across a broad suite of reasoning benchmarks and domains (math, science, code, general reasoning). This breadth of empirical evidence is a clear strength.\n2. The central finding—that diversity in reasoning data benefits pretraining most, while data quality dominates SFT effectiveness—is supported by clear quantitative results (e.g., Tables 1, 2, and 3), providing a practical guideline for future LLM training recipes.\n3. The paper performs extensive ablation studies (Section 5), specifically investigating scaling strategies, catch-up hypotheses, and the latent effects of high-quality data. For example, Table 4 rigorously compares the effects of doubling SFT data versus diverse pretraining.", "weaknesses": "1. The description of data volume is highly unclear. In Section 2.2, $D_{LDQ}$ is described as containing 336B tokens, while $D_{SHQ}$ is described as containing 1.2M samples. It is also unclear exactly how much base data and reasoning data each model used during the pretraining phase, especially for the $M_{SHQ}$ model. \n2. The results in Tables 2, 3, and 5 only report the average scores of the three models pretrained with reasoning data. However, Table 1 has already demonstrated significant performance differences between the base models of LDQ and SHQ. The authors should provide the complete results for each individual model group across all tables.\n3. The experimental results in this work lack validation across different model scales and on the more widely adopted Transformer architecture.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)