Front-Loading Reasoning: The Synergy between Pretraining and
  Post-Training Data
Authors: name: Syeda Nahida Akter, name: Shrimai Prabhumoye, name: Eric Nyberg, name: Mostofa Patwary, name: Mohammad Shoeybi, name: Yejin Choi, name: Bryan Catanzaro
Organization: NVIDIA

The prevailing paradigm for enhancing the reasoning abilities of LLMs
revolves around post-training on high-quality, reasoning-intensive data. While
emerging literature suggests that reasoning data is increasingly incorporated
also during the mid-training stage-a practice that is relatively more
proprietary and less openly characterized-the role of such data in pretraining
remains unclear. In particular, due to the opaqueness of pretraining corpora in
most frontier models, the effect of reasoning data introduced at different
phases of pre- and/or post-training is relatively less reported in the
scientific literature. This raises several important questions: Is adding
reasoning data earlier during pretraining any better than introducing it during
post-training? Could earlier inclusion risk overfitting and harm
generalization, or instead establish durable foundations that later fine-tuning
cannot recover? We conduct the first systematic study of how reasoning
data-varying in scale, diversity, and quality-affects LLM performance when
introduced at different stages of training. We find that front-loading
reasoning data into pretraining is critical (19% avg gain), establishing
foundational capabilities that cannot be fully replicated by later-stage SFT,
even with more data. We uncover an asymmetric principle for optimal data
allocation: pretraining benefits most from broad diversity in reasoning
patterns (11% avg gain), while SFT is more sensitive to data quality (15% avg
gain). We show that high-quality pretraining data has latent effects, activated
only after SFT, and that naively scaling SFT data can be detrimental, washing
away the benefits of early reasoning injection. Our results challenge the
conventional separation of language modeling and reasoning, providing a
principled guide for strategically allocating data across the entire training
pipeline to build more capable models.