Echo Chamber: RL Post-training Amplifies Behaviors Learned in Pretraining
Authors: Rosie Zhao
Alexandru Meterez
Sham M. Kakade
Cengiz Pehlevan
Samy Jelassi
Eran Malach
Venue: COLM 2025 (2025)
Keywords: reinforcement learning
language models
post-training
ppo
pretraining
reasoning
TL;DR: We conduct a systematic end-to-end study of RL fine-tuning from scratch for mathematical reasoning, uncovering how RL shapes model behavior across scales and data mixtures.

Abstract: Reinforcement learning (RL)-based fine-tuning has become a crucial step in post-training language models for advanced mathematical reasoning and coding. Following the success of frontier reasoning models, recent work has demonstrated that RL fine-tuning consistently improves performance, even in smaller-scale models; however, the underlying mechanisms driving these improvements are not well-understood. Understanding the effects of RL fine-tuning requires disentangling its interaction with pretraining data composition, hyperparameters, and model scale, but such problems are exacerbated by the lack of transparency regarding the training data used in many existing models. In this work, we present a systematic end-to-end study of RL fine-tuning for mathematical reasoning by training models entirely from scratch on different mixtures of fully open datasets. We investigate the effects of various RL fine-tuning algorithms (PPO, GRPO, and Expert Iteration) across models of different scales. Our study reveals that RL algorithms consistently converge towards a dominant output distribution, amplifying patterns in the pretraining data. We also find that models of different scales trained on the same data mixture will converge to distinct output distributions, suggesting that there are scale-dependent biases in model generalization. Moreover, we find that RL post-training on simpler questions can lead to performance gains on harder ones, indicating that certain reasoning capabilities generalize across tasks. Our findings show that small-scale proxies in controlled settings can elicit interesting insights regarding the role of RL in shaping language model behavior.

Published as a conference paper at COLM 2025
Echo Chamber: RL Post-training Amplifies Behaviors Learned in Pretraining
Rosie Zhao'
Alexandru Meterez* Harvard University
Sham Kakade
Harvard University
Harvard University
Kempner Institute
Kempner Institute
Kempner Institute
Cengiz Pehlevan Harvard University
Samy Jelassi* Harvard University
Eran Malach
Harvard University
Kempner Institute
Kempner Institute
Abstract
Reinforcement learning (RL)-based fine-tuning has become crucial step in post-training language models for advanced mathematical reasoning and coding. Following the success of frontier reasoning models, recent work has demonstrated that RL fine-tuning consistently improves performance, even in smaller-scale models; however, the underlying mechanisms driving these improvements are not well-understood. Understanding the effects of RL fine-tuning requires disentangling its interaction with pretraining data composition, hyperparameters, and model scale, but such problems are exacerbated by the lack of transparency regarding the training data used in many existing models. In this work, we present a systematic end-to-end study of RL fine-tuning for mathematical reasoning by training models entirely from scratch different mixtures of fully open datasets. We inves- tigate the effects of various RL fine-tuning algorithms (PPO, GRPO, and Expert Iteration) across models of different scales. Our study reveals that RL algorithms consistently converge towards a dominant output distribu- tion, amplifying patterns in the pretraining data. We also find that models of different scales trained on the same data mixture will converge to distinct output distributions, suggesting that there are scale-dependent biases in model generalization. Moreover, we find that RL questions can lead to performance gains on harder ones, indicating that certain reasoning capabilities generalize across tasks. Our findings show that small-scale proxies in controlled settings can elicit interesting insights regarding the role of RL in shaping language model behavior."
on simpler
1 Introduction
Reinforcement learning-based fine-tuning has emerged as a crucial step in the post-training process for enhancing language models' capabilities in advanced mathematical reasoning and coding (Jaech et al., 2024; Guo et al., 2025; Shao et al., 2024; Team et al., 2025). Open- source efforts to reproduce the fine-tuning strategies used in state-of-the-art reasoning models have further demonstrated that reinforcement learning consistently boosts perfor- mance in these domains (Lambert et al., 2024; Havrilla et al., 2024; Luo et al., 2025; Zeng et al., 2025), even when applied to smaller-scale pretrained models or synthetic environments (Pan et al., 2025).
'Equal contribution. Correspondence Rosie Zhao rrosieehaogg.harvarrr.coo harvard.edu) and Alexandru Meterez (ame ter.eeee. hww.ard.aadddd.eaoooo.. edu).
*Fqual contribution.
!Our code is available at httpp:
All pretrained base
models can be found here, and intermediate checkpoints from RI. fine-tuning for two 1B pretrained models can be found at the following links: TinyGSM OM11 + OM12 and TinyGSM OMI2.

---

Published as a conference paper at COLM 2025
Pre-training
Post-training
Base Model
Finetuned Model
SSL
Eval
Figure 1: We conduct systematic end-to-end study of RL fine-tuning for mathematical reasoning by training models entirely from scratch using different mixtures of datasets. The instruction datasets included in our pretraining mixes contain distinct formats which we can track in the model's generations after pretraining and RL post-training; we find that after rost-training, the model consistently converges to dominant output distribution coinciding with a significant increase in performance.
While RL post-training has demonstrated empirical success, the underlying mechanisms driving these improvements are being actively studied. Several hypotheses have been proposed explain the effectiveness of RL, including its potential encourage longer chains of thought (Wei et al., 2022; Yeo et al., 2025), facilitate backtracking behaviors (Guo et al., 2025), generalize to unseen task variants (Chu et al., 2025), and improve overall reasoning accuracy. However, a limitation of these studies is their lack of control over the pretraining data increasingly recognized factor in providing the proper model initialization needed for effective fine-tuning (Abdin et al., 2024; Allal et al., 2025; Petty et al., 2024; Penedo et al., 2024). This gap especially salient given that most existing reproductions and analyses begin from base models whose pretraining datasets proprietary or insufficiently documented. A prominent example is the Qwen family of models (Yang et al., 2024), which is commonly used in RL post-training studies but the synthetic math and code data used for pretraining remains undisclosed. Prior work has shown that some models demonstrate substantial improvements while others stagnate when applying these post-training techniques (Gandhi et al., 2025), highlighting the critical influence of pretraining data despite it being the most opaque part of the training pipeline for reasoning models. Consequently, it is difficult to isolate the role of RL in shaping model behavior, as its effects are entangled with unknown factors in the pretraining data.
either
In this work, we seek to clarify the relationship between pretraining data and RL-based post-training. Specifically, we ask the following: how does the composition of pretraining data affect the efficacy of RL fine-tuning? And how does this interaction depend on the choice of RL algorithm, the choice of hyperparameters, and model scale? To answer these questions, we construct controlled experimental setting that allows us to systematically examine these factors, providing clearer picture of how pretraining and RL jointly shape model behavior.
To isolate the effects of RL fine-tuning, we pretrain language models from scratch on curated mixtures of open-source datasets, including both document-style corpora and synthetic instruction datasets with diverse characteristics. This setup gives us full control over what the model is exposed to during pretraining and allows to track the influence of specific instruction datasets. We then fine-tune these models using reinforcement learning on mathematical question-answering tasks. This controlled setting enables quantitative and qualitative shifts in the model's generations across different stages of training, offering clearer view into the mechanisms by which RL fine-tniig interacts with pretraining data.
monitor both
2

---

Published as a conference paper at COLM 2025
Our primary contributions as follows:
We conduct principled investigation of RL fine-tuning starting from models of various scales that we have pretrained from scratch on mixtures of fully open datasets (Section 2).
We find that RL fine-tuning consistently drives models to converge on generating outputs in the format of single pretraining distribution (Section 3.1), often yielding improved pass@1 accuracy but reduced diversity. Despite occasional failure cases (Section 3.2), the preferred distribution is typically the most performant one as measured on the base model's accuracy restricted to the specific distribution. Quali- tative properties within the preferred distribution are also further refined during RL fine-tuning (Section 3.3).
The preferred distribution reveals a scale-dependent bias: smaller models favor simpler, code-like formats, while larger models shift toward natural language outputs (Section 3.4).
We provide evidence of positive transfer from RL fine-tuning, showing that models improve on evaluation datasets not seen during post-training (Section 4).
Experimental Setup
2.1 Pretraining
Architecture: We train decoder-only language models using the OLMo codebase (Groen- eveld al., 2024; OLMo et al., 2024) of two sizes: 150M and 1B parameters. The models have widths of 768 and 2048, and depths of 12 and 16 layers respectively. The MLP hidden dimension is 8x of the width, and we use SwiGLU activations (Shazeer, 2020) and RoPE positional encodings (Su et al., 2024).
Datasets: We train on a mixture of datasets related to mathematics; for all models, unless otherwise specified we train on FineMath-3++ (Allal et al., 2025) and the Algebraic-Stack subset of the Proof-Pile-2 (Azerbayev al., 2023). Aside from these datasets consisting of documents with mathematical content, we also train on instruction datasets such as TinyGSM (Liu et al., 2023), OpenMathInstruetl (Toshniwal et al., 2025b), and Open struct2 (Toshniwal et al., 2025a). We repeat these question-answer datasets in various ratios in our mixtures, sometimes with multiple passes over the same dataset we denote this using the x symbol throughout the manuscript (eg. 4x TinyGSM refers to four passes over the TinyGSM dataset). We pretrain on the question-answer datasets by concatenating the prompt and the answer and adding them to the general corpus, without any chat template or special formatting.
TinyGSM synthetic dataset of 12.3M problem-solution pairs generated from the GSM8K and GSM-IC (Shi et al., 2023) training subsets, with code solutions generated by GPT-3.5. OpenMathinstrruttl consists of 1.8M problem-solution pairs generated from the GSM8K and MATH training subsets, with code solutions generated by Mixtral-8x7B (Jiang et al., 2024). Finally, OpenMathlnstruct22 consists of 14M problem-solution pairs also generated from the GSM8K and MATH training subsets, with natural language solutions generated by Llama3.1 as
405B-Inst We focus on these datasets because each has distinct characteristics- tags and specific formatting- thht we can search within the model's generations, enabling us to monitor the presence of each dataset throughout training. We provide more details and representative examples from each dataset in Appendix B.
Pretraining Hyperparameters: For all models we use the AdamW optimizer (Kingma & Ba, 2014; Loshchilov & Hutter, 2017) with learning rate of 0,001 and weight decay of 0.1. We use a linear warmup 5000 steps and a cosine decay scheduler to 10% of the peak learning rate.
3

---

Published as a conference paper at COLM 2025
2.2 Reinforcement Learning Fine-tuning
We perform fine-tuning using various RL algorithms directly on the models that we have pretrained from scratch. We use the OpenRLHF (Hu et al., 2024) implementation of Pol- icy Optimization (PPO) (Schulman et al., 2017) and Group Relative Policy Optimization (GRPO) (Shao et al., 2024). We train using verifiable rewards (Lambert et al., 2024), where the reward function for RL fine-tuning is 1 if the model's 's answer matches the ground truth, and otherwise.
We additionally fine-tune our models with Expert Iteration (EI) (Anthony et al., 2017). Starting from our pretrained models, we generate 64 generations for each problem in the train set of GSM8K, and create de de-duplicated dataset of the generations which lead to correct answer. We use this dataset to then perform supervised fine-tuning on the pretrained model. This procedure can be done in iterations, where the fine-tuned model from the previous iteration is used to generate the de-duplicated dataset of correct generations, and supervised fine-tuning is done the base model.
For the results presented in Section 3 we fine-tune using questions from the train split of GSM8K and study the performance and format of the generations of the models on the test split of GSM8K, both during and after fine-tuning. In Section 4 we take the models fine-tuned using questions from GSMSK and evaluate on the test set of MATH-500 and AIME 1983-2024. In Appendix we also perform PPO on questions from the train split of MATH. For more details about the hyperparameters used, refer to Appendix C.
3 RL on Models Pretrained from Scratch with Different Mixtures
In this section, we present summary of our results after applying reinforcement learn- ing fine-tuning using problems from GSM8K on our models which were pretrained from scratch. With the exception of a few results in Section 3.3, we always include FineMath3+ and Algebraic-Stack in our pretraining mixtures, and vary quantities of TinyGSM, Open- Mathlnstructl, and MathnMathlnstruuc. Furthermore, unless otherwise specified, figures in this section correspond to our runs with PPO on models with 150M parameters; we conduct further analysis on models with 1B parameters in Section 3.4 and Appendix E, and comparisons with other RL algorithms and Expert Iteration are provided in Section 3.5 and Appendix F, Finally, we provide brief theoretical justification of our results in Section 3.6.
3.1 RL converges to favour one distribution in the mixture
We begin by highlighting a striking pattern consistently observed during RL fine-tuning across all pretraining data mixtures: the model rapidly converges to producing outputs that follow the format of a single data distribution seen during pretraining, supressing the other ones. In Figure 2, we illustrate both the percentage of generations corresponding to each dataset and their respective accuracies when fine-tuning model pretrained on TinyGSM, OpenMathlnstructl, and Open.MathInstruct. For more details on dataset examples, how we evaluate the correctness of model generations, and the metrics that we report, see Appendix B. The model quickly shifts toward generating answers in the format of one Distribtiinn-TinySSS in this case within the first epoch (note the log-scaled x-axis). This transition coincides with the largest gain in overall pass@1 accuracy.
We also observe that while majority@64 accuracy improves by approximately 5% due fine- tuning, pass@64 accuracy declines towards the end of training, in line with prior findings on reduced generation diversity following RLHF/ et al., 2025).
/ RL. fine-tuning (Kirk et al., 2024; Dang
Additionally, we find that increasing the coefficient for the KL penalty during fine-tuning preserves some outputs in formats from other distributions besides the preferred one. As shown in Figure 3, fine-tuning with a higher KL coefficient for the same pretrained model from Figure still results in preference for TinyGS--style outputs, but subset of generations in natural language / OpenMathInstrucc2 format still remains. This leads to comparable pass@1 accuracy relative to the lower KI. setting, while pass@64 accuracy
4

---

Published as a conference paper at COLM 2025
remains stable. In Appendix D, we demonstrate that this tendency to favor single data distribution is consistent across all pretraining mixtures evaluated, and we also show that removing the KL penalty altogether yields similar performance.
Finally, although we focus on accuracy and percentage metrics for our analysis here and henceforth in this section, show that similar phenomena manifest when tracking confidence-based metrics- such as the average probability of the TinyGSM and OpentMahlnnsrruct-ss/// initial token formats- in Appendix G.
(3MIl-syee)
0.8
(TixyB5M
0.8
Teet
Porcentage
0.5
Accuracy
0.6
peuo)
Rggrity004
peud044
0.4
0.2
0.2
0.0
Samples
Samples
Figure 2: Starting from a 150M model pretrained with TinyGSM, OpenMathInstructt,, and OpenMathInstrucch. we track the following throughout PPO training: (Left) Percentage of generations on GSM8K test which adhere to the formats TinyGSM, OMI1, and Text (referring to the formats of TinyGSM, OpenMathInstructt, and OpenMathInstruut2// n/atural language respectively) and (Right) GSM8K test accuracy restricted to the generations in each dataset format as well as overall pass@1, pass@64, and majority@64 accuracy. The generations quickly converge to outputting exclusively in the format of TinyGSM within the first epoch training, which coincides with the greatest increase overall accuracy. While majority@66 @64 experiences slight increase after fine-tuning, pass@64 performance decreases slightly at the end of training.
(SMII-9AM))
0.8
0.8
Tei
Percentage
(900)
Accuracy
0.6-
0.4
0.2
0.2
0.0
0.0
Samples
Samples
Figure 3: GSM8K test accuracy across epochs over the data during PPO when starting from the same 150M model as Figure 2 but with higher KI. coefficient (0.01 as opposed to 0.001). The model still retains some generations using the format from OpenMathInstruct2, but reaches a similar final pass@1 accuracy as in Figure
3.2 RL doesn't always favor the most performant, nor the most common distribution
In the previous section, observed that RL fine-tuning amplifies generations coming from one distribution, while downweighting the others. This raises a natural question: does the model consistently favor the distribution that yields the best performance, the distribution with the highest proportion of generations at initialization?
We find that the answer is nuanced and can depend on the pretraining data mixture. We provide two representative examples: in Figure 4, we present the evolution of the percentage of generations for each distribution and their accuracies during fine-tuning for models

---

Published as a conference paper at COLM 2025
pretrained TinyGSM combined with varying amounts of OpenMattInstruutt.. In Figure (a), although the model initially produces more OpenMathinstruttllstte compared to TinyGSM-style solutions (28%), it ultimately converges to generating TinyGSM- style outputs within the first epoch. In contrast, Figure 4 (b) shows that when the number of 1tenstthInstruct1 samples is doubled during pretraining, the model instead converges to OpenMathhnntruutt-sstt generations. This occurs despite the initial generation distribution being similar Figure 4 (a) and despite TinyGSM generations achieving higher accuracy than Mathnssttlttttuutt generations at initialization. However, in (b), the model achieves lower performance after fine-tuning compared to (a) and eventually degrades further near the end of training. We consider this a failure mode of RL fine-tuning, Nonetheless, in most of our experiments, the model tends select the distribution with the highest performance a after pretraining -Tnnnnn,, in the case of the 150M models-ace across the majority of fine- tuning runs.
solutions (62%)
Cok (OMilstty)) (TRyG5M-
0.8
Percentage
0.6
Accuracy
mejelye04
0.2
0.2
10'
10
Samples
Samples
(a) PPO initialized from a model trained on TinyGSM and 4 OpenMathIInstrutti.
(OMIi--te
0.8
(TeyCSMaayk)
0.8
Percentage
0.6
Accuracy
majoiiy044
(owsB)
0.4
0.2
0.2
0.0
0.0
Samples
Samples
(b) PPO initialized from model trained on TinyGSM and 8 OpenMathInstruct..
Figure 4: Proportion of generations by data format (left) and corresponding accuracies (right) during PPO fine-tuning with pretraining 150M models on TinyGSM and varying amounts of OpenMathInstruetl. In (a), where the pretraining set includes 4x OpenMathInstruutll, the model rapidly shifts within the first epoch to predominantly generating TinGGSM-style outputs, despite their lower frequency at initialization. In (b), increasing the amount of OpenMathInstructll in pretraining further results in the base model retaining similar initial generation distribution. However, during fine-tuning, the model transitions to almost exclusively producing OpenMathinstructl-se generations, which coincides with a drop in overall accuracy.
3.3
How does performance within one distribution improve during RL?
In the preceding sections, we examined models pretrained on varying proportions of the TinyGSM, OpenMathInstructl, and OpenMathInstruct) datasets (as reminder, we always include FineMath3+ and Algebraic-Stack as well unless otherwise specified). We observed that, in most instances, the largest gains in pass@1 accuracy were associated with the model conforming to the format of a single distribution most cases, TinyGSM. This naturally raises the question of whether model generations exhibit meaningful progress within a given distribution, and whether performance improvements are achievable when pretraining is done on single dataset.

---

Published as a conference paper at COLM 2025
Figure 5 (left) demonstrates that increasing the amount of TinyGSM data (specifically, repeat TinyGSM 1, 2, 4, and times in the pretraining mix) in the pretraining of 150M- parameter models leads to improved performance acToss pass@l, pass@64, and majority accuracy after fine-tuning. Figure 5 (right) further illustrates the progression of pass@1 accuracy across training epochs, where we observe that models pretrained with the highest proportion of TinyGSM not only achieve the best final performance but also exhibit the largest performance gain from fine-tuning. We track the progression of pass@64 and majority@66 accuracy in Figure 10 in the Appendix. These findings suggest that selectively repeating subsets of pretraining data, rather than incorporating additional diverse datasets, may yield more substantial improvements due to RL-based fine-tuning.
1.0
0.8
ThyGSM
ThyySM
ThyGSS
Accuracy
0.8
Accuracy
0.6
TiyySSM
0.6
0.1
pos001
0.4
powe2611
majonity64
0.2
2
0
200000
400000
600000
Amount of TinyGSM
Samples
Figure 5: (Left): Top pass@1, pass@64, and majority@64 accuracy on GSM8K test across epochs after training with PPO on 150M models pretrained with different amounts of TinyGSM. (Right): GSM8K pass@1 test accuracy across PPO training for models trained on different amounts of TinyGSM.
Finally, we pretrain a 150M parameter model from scratch using only TinyGSM, excluding FineMath3+ and Algebraic-Stack. Our goal was to answer two questions: does RL fine- tuning still yield performance gains in the absence of additional datasets, and so, what underlies these improvements?
As shown in Figure 6 (left), performance continues to improve after applying PPO to this model. To better understand how the model's generations evolve during fine-tuning, track characteristic features of TinyGSM solutions - such as including a docstring that replicates the original question and having a lack of additional comments. In Figure 6 (right), we plot the proportion of model outputs that follow these conventions. We observe that, over training, the model increasingly conforms to the TinyGSM style, including settling on a consistent docstring format (e.g. shifting from mixed usage of single and double apostrophes consistently using apostrophes). This supports the view that fine-tuning not only steers the model toward a preferred distribution but also refines outputs within that distribution. We further explore how fine-tuning improves generation quality beyond distributional preference in Section 4, where we discuss positive transfer effects to external evaluation datasets.
3.4 The effect of scale: larger models prefer different distributions
In this section, we examine how the trends identified above change with model scale. We pretrain 1B parameter models on various dataset mixtures to compare their behavior after fine-tuning with that of the corresponding 150M parameter model pretrained on the same mixture. We find that while models at both scales maintain preference for single distribution's format, the specific favored distribution changes with scale. Notably, 150M models tend to predominantly output TinyCaatformma generations, whereas the 1B models tend to prefer OpenMathinsstuuc2-syy by IpenMathInstruutt-sttee not the preferred choice for the 1B models, and their final accuracy surpasses that of the smaller model pretrained on the same mixture. This points to scale-dependent bias in
natural language responses, followed code. As shown in Figure 7 and Appendix E, TinyGSM is
7

---

Published as a conference paper at COLM 2025
10
Carcle
Monhel derming
Apctoobl
Acouracy
Percentae
Des spottighe
0.6
0.0
0.2
10a
1MP
11P
11f
Samples
Samples
Figure (Left): pass@l, pass@64, and majority@64 accuracies on the GSM8K test set during fine-tuning of 150M model pretrained solely with 4x TinyGSM (no Algebraic-Stack or FineMath3+). As with other pretraining mixtures, we continue final performance. (Right): Monitoring qualitative properties of the model's generations throughout fine-tuning, such as whether the docstring copies the question, the inclusion of comments, and the choice between single or double apostrophes for docstrings. The model progressively refines its outputs during training and increasingly aligns with the TinyGSM format, which coincides with improved accuracy.
observe gains in
behavior, likely tied to the larger model's greater capacity to answer questions correctly in natural language. In contrast, the 150M model may rely more heavily on the simpler, more deterministic TinyGSM-style code to produce accurate answers.
(DMiI-ut s)
Cook (TMyCoM)
lect
Percetaae
posiit
0.6
Accuracy
0.6
majohby044
0.4
0.0
0.0
10*
Samples
Samples
Figure Percentage of generations (left) and respective accuracies (right) during PPO train- ing for 1B model pretrained on TinyGSM, IpenMathInstruutt) and Oathnaatnnstruud2. Although a 150M model pretrained on the exact same data converges on outputting only LinyGS--formatter generations (see Figure 2), here guage solutions, even though natural language has the lowest percentage across generations and TinyGSM is the more performant distribution at initialization.
the model amplify natural lan-
3.5 The effect of the RL algorithm
In Appendix F we report analogous results from the previous sections with GRPO and Expert Iteration. For GRPO in Appendix F.1 we observe the same trend in the percentage of generations where the model converges to favoring the format one distribution, but the training of GRPO is generally less stable and often experiences brief collapse in performance before recovering by the end training. Additional results from multiple rounds of Expert Iteration are presented in Appendix F.2. In our setup, this approach consistently underperforms PPO and exhibits only a mild shift toward favoring a single dataset format. We believe this is likely due to repeatedly fine-tuning from the original base model. The nuanced differences we observe across RL algorithms highlight the need for further investigation into how specific algorithmic choices influence model behavior.
8

---

Published as a conference paper at COLM 2025
3.6 Supporting theory
We now provide some theoretical explanation for the results detailed above. We emphasize that the focus of this paper is not theoretical analysis of reinforcement learning, and we simply reiterate known results that explain the findings of this work. Let X' be the space of inputs and be the space of responses. Letr X x y + {0,1} be reward function, and let re be our reference policy (before RL). Assume that our reference policy is in fact a mixture of k different policies T,,..., 7 s.t. [
Σ; a, for a1, ai, E [0, satisfying Lia; = 1. For example, each π.; can be a different solution format for math questions (code, text, etc.). We can frame the problem of reinforcement learning solved by egg. PPO as maximizing the expected reward under KL-regularization"..
arg max Ey-n(r(y,xx) KL (et
Then, the maximizer would correspond to:
π (y|x) a (x)((y)x) exp (r(y.x)/B) L (xpp(((x.))//)m)m))
Namely, reweight the original mixture of policies corresponding to the rewards from each policy in the original mixture. This consistent with our experiments, which show that RL mostly converges to the strategy which maximizes the reward.
Transfer to other evaluation datasets
In Section 3.3, we observed that RL fine-tuning improve the structure of model outputs in ways that align with the format of the favored training distribution. While the qualitative attributes highlighted in Figure may contribute to the model generating more accurate answers, our goal in this section is to gather stronger evidence that RL fine-tuning produces changes that directly enhance performance general capabilities like arithmetic. To this end, we focus on evaluating our models on datasets that were not used during fine-tuning, aiming to assess whether the models demonstrate positive transfer to more challenging tasks. For our 1B models, we evaluate on MATH-500 after performing PPO with the train questions from GSM8K and provide pass@1 and majority@64 performance before ('Base') and after ('FT') fine-tuning in Table 1. We observe consistent performance gains following fine-tuning, with some models improving by much as 10%. Although MATH-500 is considered out-of-distribution relative to the fine-tuning data, models pretrained on mixtures that include either Mathnntiinttruct datasets have already encountered synthetic problems resembling those in MATH. These models show the largest improvements on MATH-500 after fine-tuning, highlighting the benefit of pretraining on data that structurally similar to the downstream task.
such as reducing error rates or improving
In Appendix H.1, we analyze these improvements from a qualitative lens by prompting GPT-4.5 Preview classify the types of errors made by the base model for incorrect gener- ations and later corrected following fine fine-tuning. In Appendix H.2 we present evaluation results on AIME for the same models and find little to no improvement on pass@1 and majority@64 performance for the AIME 2022-2024 benchmark across all pretrained models, but improvements are observed for pass@64 performance. In Appendix H.3 we provide ex- amples of model generations MATH-500 and AIME 2022-2024 before and after doing RL fine-tuning on GSM8K, where the base model was previously incorrect and the fine-tuned model provides correct answer.
Discussion and Conclusion
In this work, explored the effect of the pretraining data on the post-training stage in end-to-end manner. Through pretraining models across different scales (150M and 1B) on
We note that our experimental results hold even without adding the Kl-reuularization term. We leave analysis of this setting to future work.

---

Published as a conference paper at COLM 2025
Pretraining Data Mixture
Pass@1 Base
Pass@1 FT
Maj@64 Base
Maj@64 FT
TinyGMM + 4xOMI1
8.60%
12.60%
22.60%
26.00%
TinyGSM + OMT2
33.40%
43.60%
46.20%
52.80%
OMI2 + MMQA
34.60%
44.40%
51.20%
55.00%
'inyGGM
4.80%
9.60%
7.80%
12.20%
TinyGSM + OMI1 + OMI2
33.40%
43.80%
48.60%
54.60%
Table 1: Pass@1 and majority@64 performance of 1B models on the MATH-500 benchmark before and after RL fine-tuning with PPO GSM8K train questions. Each row corresponds to a different pretraining data mixture. Results show consistent improvements after fine- tuning, suggesting that RL not only improves output formatting but also enhances general mathematical capabilities.
data mixtures containing general mathematics corpus and various ratios of question-answer datasets, our study has shown the following:
RL fine-tuning amplifies specific mode from the pretraining mixture while col- lapsing the others.
The mode that gets amplified depends on the scale of the model, and the degree of amplification depends the hyperparameters namely, the coefficient for the KL penalty.
RL post-training on simpler datasets such as GSM8K gives a performance boost on harder mathematical datasets such MATH, and to lesser extent on AIME.
Small-scale proxies offer valuable insights into the scientific aspects of RL fine-tuning in LLMs.
Our work opens up several exciting research directions towards understanding RL post- training and extracting more performance from these models. One potential question how our results extend to more complicated data mixtures, such as including multilingual data in the mix. Moreover, there notion of an optimal pretraining mixture that would lead to the best reasoning performance downstream, and how does this mixture differ across model scales?
Crucially, believe that one major confounder in the existing literature is the reliance on pretrained models. While several open-source reasoning models are openly available, the pretraining datasets are not public, which a critical aspect of the performance of the base models on reasoning tasks (Yang et al., 2024; Grattafiori et al., 2024). Naturally, this discrepancy gets amplified downstream fine-tuning and evaluation, leading to spurious conclusions about the abilities and behaviors of these models. We believe that studying
LLM fine-tuning in controlled settings starting from scratch is a necessary and underexplored avenue for research, amenable for exploring in academic settings using the small scale proxies introduced in this manuscript.
6 Acknowledgements
SK, RZ, AM, and SJ acknowledge support from the Office of Naval Research under award N00014-22-1-2377 and the National Science Foundation Grant under award #TS 2229881. This work has been made possible in part by gift from the Chan Zuckerberg Initiative Foundation to establish the Kempner Institute for the Study of Natural and Artificial Intelligence. RZ is supported by Simons Investigator Fellowship, NSF grant DMS-2134157, DARPA grant W911NF2010021. and DOE grant DE-SC0022199, CP is supported by NSF grant DMS-2134157, NSF CAREER Award IIS-2239780, DARPA grant DIAL-FP-033 a Sloan Research Fellowship, and The William F. Milton Fund from Harvard University. RZ and AM are supported by Kempner Institute Graduate Research Fellowships
10

---

Published as a conference paper at COLM 2025
References
Marah Abdin, Jyoti Aneja, Harkirat Behl, Sébastien Bubeck, Ronen Eldan, Suriya Gunasekar, Michael Harrison, Russell J Hewett, Mojan Javaheripi, Piero Kauffmann, et al. Phi-4 technical report. Xiv preprint ar a/Xt::2412.08905, 2024.
Arash Ahmadian, Chris Cremer, Matthias Gallé, Marzich Fadace, Julia Kreutzer, Olivier Pietquin, Ahmet Üstün, and Sara Hooker. Back to basics: Revisiting reinforce style optimization for learning from human feedback in llms, arxio preprint arXiv:2402.147700 2024.
Loubna Ben Allal, Anton Lozhkov, Elie Bakouch, Gabriel Martín Blázquez, Guilherme Penedo, Lewis Tunstall, Andrés Marafioti, Hynek Kydlfček, Agustín Piqueres Lajarín, Vaibhav Srivastav, Joshua Lochner, Caleb Fahlgren, Xuan-Son Nguyen, Clémentine Four- rier, Ben Burtenshaw, Hugo Larcher, Haojun Zhao, Cyril Zakka, Mathieu Morlon, Colin Raffel, Leandro von Werra, and Thomas Wolf. Smollm2: When smol goes big data-centric training of small language model, 2025. URL https://arxiv: org/ /502. 02737.
Thomas Anthony, Zheng Tian, and David Barber. Thinking fast and slow with deep learning and tree search. Advances in neural information processing systems, 30, 2017.
Zhangir Azerbayev, Hailey Schoelkopf, Keiran Paster, Marco Dos Santos, Stephen McAleer, Albert Q. Jiang, Jia Deng, Stella Biderman, and Sean Welleck. Llemma: An open language model for mathematics, 2023.
Maciej Besta, Nils Blach, Ales Kubicek, Robert Gerstenberger, Michal Podstawski, Lukas Gianinazzi, Joanna Gajda, Tomasz Lehmann, Hubert Niewiadomski, Piotr Nyczyk, et al. Graph of thoughts: Solving elaborate problems with large language models. In Proceedings of the AAAT Conference on Artificial Infelligence, volume 38, pp- 17682-17690, 2024.
Tianzhe Chu, Yuexiang Zhai, Jihan Yang, Shengbang Tong, Saining Xie, Dale Schuurmans, Quoc V Le, Sergey Levine, and Yi Ma. Sit memorizes, rl generalizes: A comparative study of foundation model post-training arXiv preprint ar arXiv:2501.17100 2025.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. xrv preprint ar arXiv:2110.141 2021.
Ganqu Cui, Lifan Yuan, Zefan Wang, Hanbin Wang, Wendi Li, Bingxiang He, Yuchen Fan, Tianyu Yu, Qixin Xu, Weize Chen, et al. Process reinforcement through implicit rewards. arxiv preprint ar Xxxx:2502..01556 2025.
Xingyu Dang, Christina Baek, J Zico Kolter, and Aditi Raghunathan. Assessing diversity col- lapse in reasoning. In Scaling Self- Improving Foundation Models without Human Superoision, 2025.
Hanze Dong, Wei Xiong, Deepanshu Goyal, Yihan Zhang, Winnie Chow, Rui Pan, Shizhe Diao, Jipeng Zhang, Kashun Shum, and Tong Zhang. Raft: Reward ranked finetuning for generative foundation model alignment. ar preprint ar Xrx::2304.06767, 2023.
Kanishk Gandhi, Ayush Chakravarthy, Anikait Singh, Nathan Lile, and Noah Good- man. Cognitive behaviors that enable self-improving reasoners, or, four habits of highly effective stars. Xiv preprint arXio:2503.01307, 2025.
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models. Xrviv preprint ar arXiv:2407.21758 2024.
Dirk Groeneveld, Iz Beltagy, Evan Walsh, Akshita Bhagia, Rodney Kinney, Oyvind Tafjord, Ananya Jha, Hamish Ivison, lan Magnusson, Yizhong Wang, et al. Olmo: Accelerating the science of language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp- 13789-15809 2024.
11

---

Published as a conference paper at COLM 2025
Xinyu Guan, Li Lyna Zhang, Yifei Liu, Ning Shang, Youran Sun, Yi Zhu, Fan Yang, and Mao Yang. rstar-math: Small iims can master math reasoning with self-evolved deep thinking. arXiv preprint ar 11X002250.00551, 2025.
Caglar Gulcehre, Tom Le Paine, Srivatsan Srinivasan, Ksenia Konyushkova, Lotte Weerts, Abhishek Sharma, Aditya Siddhant, Alex Ahern, Miaosen Wang, Chenjie Gu, et al. Reinforced self-training (rest) for language modeling. arXiv preprint ar XrXi::2308.08998, 2023.
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in Ilms via reinforcement learning. arXiv preprint ar arXiv:2501.12 110425 2025.
Shibo Hao, Sainbayar Sukhbaatar, Dijia Su, Xian Li, Zhiting Hu, Jason Weston, and Yuan- dong Tian. Training large language models to reason in a continuous latent space. arv preprint ar Xir::2412.677 06669, 2024.
Alex Havrilla, Yuqing Du, Sharath Chandra Raparthy, Christoforos Nalmpantis, Jane Dwivedi-Yu, Maksym Zhuravinskyi, Eric Hambro, Sainbayar Sukhbaatar, and Roberta Raileanu. Teaching large language models to reason with reinforcement learning. Xrv preprint 7Xix:2403.04642 2024.
Joy He-Yueya, Gabriel Poesia, Rose Wang, and Noah D Goodman. Solving math word problems by combining language models with symbolic solvers. arve preprint ar 0tt:23044091022 2023.
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. Xiv preprint ar arXiv:2103.032: 038874, 2021.
Jian Hu. Reinforce++: A simple and efficient approach for aligning large language models. Xiv preprint ar xrxi:25011032666 2025.
Jian Hu, Xibin Wu, Zilin Zhu, Xianyu, Weixun Wang, Dehao Zhang, and Yu Cao. Open- rlhf: An easy-to-use, scalable and high-performance rlhf framework. arXiv preprint ar Xiv:2405.11143, 2024.
Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed KI-Kishky, Aiden Low, al. Openai of system card.
Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, et al. arxiv preprint arXit:2412.1672)) 2024.
Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Ar- mando Solar-Lezama, Koushik Sen, and Ion Stoica. Livecodebench: Holistic and contami- nation free evaluation of large language models for code. arXiv preprint 4xXiv:2403.07974, 2024.
Albert Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, et al. Mixtral of experts. arXiv preprint ar Xriv:2401.04088, 2024.
Amirhossein Kazemnejad, Milad Aghajohari, Eva Portelance, Alessandro Sordoni, Siva P
Reddy, Aaron Courville, and Nicolas Le Roux. Vineppo: Unlocking rl potential for Ilm reasoning through refined credit assignment. arXiv preprint arXio:2410.01679, 2024.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint ar 4rXiv:1412.6988 2014.
Robert Kirk, Ishita Mediratta, Christoforos Nalmpantis, Jelena Luketina, Eric Hambro, Edward Grefenstette, and Roberta Raileanu. Understanding the effects of rlhf on Ilm gen- eralisation and diversity. In The Twelfth International Conference on Learning Representations, 2024.
Hynek Kydlíček. Math Verify: Math Verification Library, 2025. URL Btts:tts:// //gi weithuu..com com/ nuggingfacee/aat--eerrt
12

---

Published as a conference paper at COLM 2025
Nathan Lambert, Jacob Morrison, Valentina Pyatkin, Shengyi Huang, Hamish Ivison, Faeze Brahman, Lester James V Miranda, Alisa Liu, Nouha Dziri, Shane Lyu, et al. T, ulu Pushing frontiers in open language model post-training. arx preprint ar drXio:2411.15124, 2024.
ShengyiHuangg
Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let's verify step by step. In The Twelfth International Conference on Learning Representations, 2023.
Bingbin Liu, Sebastien Bubeck, Ronen Eldan, Janardhan Kulkarni, Yuanzhi Li, Anh Nguyen, Rachel Ward, and Yi Zhang. Tinygsm: achieving? 80% on gsmök with small language models. Xiv preprint ar arXiv:2312.09241, 2023.
Zichen Liu, Changyu Chen, Wenjun Li, Penghui Qi, Tianyu Pang, Chao Du, Wee Sun Lee, and Min Lin. Understanding r1-zero-like training: A critical perspective. arXiv preprint ar arXiv:2503.20783, 2025.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arv preprint arXir:1711.05101, 2017.
Michael Luo, Sijun Tan, Justin Wong, Xiaoxiang Shi, William Y. Tang, Manan Roongta, Colin Cai, Jeffrey Luo, Tianjun Zhang, Li Erran Li, Raluca Ada Popa, and Ion Stoica. Deepscaler: Surpassing ol-preview with 1.5b model by scaling rl. http:://pretty-rraii-b75.... site/ 1LLL11966611111110000beeeee00300 3a4e2,
2025. Notion Blog.
Niklas Muennighoff, Zitong Yang, Weijia Shi, Xiang Lisa Li, Li Fei-Fei, Hannaneh Hajishirzi, Luke Zettlemoyer, Percy Liang, Emmanuel Candès, and Tatsunori Hashimoto. sl: Simple test-time scaling. arviv preprint ar axXiv:2501.19393 2025.
Team OLMo, Pete Walsh, Luca Soldaini, Dirk Groeneveld, Kyle Lo, Shane Arora, Akshita Bhagia, Yuling Gu, Shengyi Huang, Matt Jordan, al. 2 olmo 2 furious. arXiv preprint ar arXie:2501.00556, 2024.
Jiayi Pan, Junjie Zhang, Xingyao Wang, Lifan Yuan, Hao Peng, and Alane Suhr. Tinyzero. 2025. Accessed: 2025-01-24.
https:/ httpss//ggithbb.coo com
Guilherme Penedo, Hynek Kydlíček, Anton Lozhkov, Margaret Mitchell, Colin A Raffel, Leandro Von Werra, Thomas Wolf, et al. The fineweb datasets: Decanting the web for the finest text data at scale. Advances in Neural Information Processing Systems, 37:30811-30849 2024.
Jackson Petty, Sjoerd van Steenkiste, and Tal Linzen. How does code pretraining affect language model task performance? ar Xiv preprint arXiv:2409.04556, 2024.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly reward model. Advances in Neural Information Processing Systems, 36:53728-5377 2023,
David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R Bowman. Gpqa: A graduate-level google- proof q&a benchmark. In First Conference on Language Modeling, 2024.
Timo Schick, Jane Dwivedi-YY, Roberto Dessi, Roberta Raileanu, Maria Lomeli, Eric Hambro, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. Toolformer: Language models can teach themselves to use tools. Advances in Neural Information Processing Systems, 36: 68539-68551, 2023.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. Xiv preprint ar xixx:1707.00347 2017.
13

---

Published as a conference paper at COLM 2025
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arviv preprint arXiio2402.03300, 2024.
Noam Shazeer. Glu variants improve transformer. arxiv preprint arXi::2002.05202, 2020.
Freda Shi, Xinyun Chen, Kanishka Misra, Nathan Scales, David Dohan, Ed H Chi, Nathanael Schärli, and Denny Zhou. Large language models can be easily distracted by irrelevant context. In International Conference on Machine Learning, pp. 31210-31227. PMLR, 2023.
Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding. Neurocomputing, 568:127063, 2024.
Kimi Team, Angang Du, Bofei Gao, Bowei Xing, Changjiu Jiang, Cheng Chen, Cheng Li, Chenjun Xiao, Chenzhuang Du, Chonghua Liao, et al. Kimi k1. 5: Scaling reinforcement learning with llms. ar Xiv preprint ar arXiv:2501.1259) 2025.
Shubham Toshniwal, Wei Du, Ivan Moshkov, Branislav Kisacanin, Alexan Ayrapetyan, and Igor Gitman. Openmathinstructt2: Accelerating ai for math with massive open-source instruction data. In The Thirteenth International Conference on Learning Representations, 2025a.
Shubham Toshniwal, Ivan Moshkov, Sean Narenthiran, Daria Gitman, Fei Jia, and Igor Gitman. Openmathinstruct-l:: A 1.8 million math instruction tuning dataset. Advances in Neural Information Processing Systems, 37:34737-34774, 2025b.
Jonathan Uesato, Nate Kushman, Ramana Kumar, Francis Song, Noah Siegel, Lisa Wang, Antonia Creswell, Geoffrey Irving, and Irina Higgins. Solving math word problems with process-and outcome-based feedback. arXiv preprint ar xxX::22211.14275 2022.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoe Le, Denny Zhou, et al. Chain-of-though prompting elicits reasoning in large language models. Advances in neural information processing systems, 01224424-28831 2022.
Fang Wu, Weihao Xuan, Ximing Lu, Zaid Harchaoui, and Yejin Choi. The invisible leash: Why rlvr may not escape its origin. arXiv preprint X104:2507..1443, 2025.
Haotian Xu, Xing Wu, Weinong Wang, Zhongzhi Li, Da Zheng, Boyuan Chen, Yi Shijia Kang, Jiaming Ji, Yingying Zhang, al. Redstar: Does scaling long-cot data unlock better slow-reasoning systems? ar Xiv preprint ar Xiv::25001 11284, 2025.
An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2. technical report. arxie preprint arXiv:241.115115 2024.
Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large language models. Advances in neural information processing systems, 36:11809-11822, 2023a.
Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and
Yuan Cao. React: Synergizing reasoning and acting in language models. In International Conference on Learning Representations (ICLR), 2023b.
Edward Yeo, Yuxuan Tong, Morry Niu, Graham Neubig, and Xiang Yue. Demystifying long thain-of-thought reasoning in llms. Xiv preprint ar xxXiv:2502..3373, 2025.
Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Tiantian Fan, Gaohong Liu, Lingjun Liu, Xin Liu, et al. Dapo: An open-source 1lm reinforcement learning system at scale. Xrv preprint ar arXiv:2503.14477 2025.
Lifan Yuan, Wendi Li, Huayu Chen, Ganqu Cui, Ning Ding, Kaiyan Zhang, Bowen Zhou, Zhiyuan Liu, and Hao Peng. Free process rewards without process labels. aiv preprint ar ar11::2412.01981 2024.
14

---

Published as a conference paper at COLM 2025
Eric Zelikman, Yuhuai Wu, Jesse Mu, and Noah Goodman. Star: Bootstrapping reasoning with reasoning. Advances in Neural Information Processing Systems, 35:15476-15488, 2022.
Eric Zelikman, Georges Raif Harik, Yijia Shao, Varuna Jayasiri, Nick Haber, and Noah Goodman. Quiet-star: Language models can teach themselves to think before speaking. In First Conference on Language Modeling, 2024.
Weihao Zeng, Yuzhen Huang, Wei Liu, Keqing He, Qian Liu, Zejun Ma, and Junxian He. 7b model and 8k examples: Emerging reasoning with reinforcement learning is both effective and efficient.
witssstttsssieeee wtteeeimmlerrrreeeeeeeeeeeeeeeee 2025. Notion Blog,
15

---

Published as a conference paper at COLM 2025
A Related Works
There is an extensive and rapidly expanding body of literature covering the understanding of post-training on the performance of LLMs in reasoning domains.
Reasoning in Large Language Models: Following the introduction of chain of thought (CoT) (Wei et al., 2022), LLMs have improved drastically in their reasoning capabilities. Frontier language models (Jaech et al., 2024; Grattafiori et al., 2024) have achieved impressive performance on hard mathematical and coding benchmarks (Hendrycks et al., 2021; Jain et al., 2024; Rein al., 2024; Cobbe et al., 2021). Further lines of work expand upon the CoT concept towards more complex structures such as trees and graphs (Yao et al., 2023a; Besta et al., 2024). Another approach to improve performance on reasoning tasks is by combining CoT approaches with tools (Schick et al., 2023; He- Yueya et al., 2023; Yao et al., 2023b), or by teaching the model to produce formal representations such as code, alongside the natural language generations (Guan et al., 2025). More recently, there have been several works proposing reasoning in latent thoughts, using different amounts of thinking tokens at training time and inference time (Hao et al., 2024; Zelikman et al., 2024).
Reinforcement Learning Fine-tuning: The post-training stage has been shown to be a crucial step towards improving LLM reasoning. Broadly, these can be split in supervised fine-tuning approaches (SFT) which involve fine-tuning on a dataset, or distilling from a teacher model (Muennighoff et al., 2025; Xu et al., 2025), 2025)_,xxpert Iteration (EI) approaches- usually involving training on multiple rounds on correct samples generated by the policy itself (Anthony et al., 2017; Dong et al., 2023; Gulcehre et al., 2023; Zelikman et al., 2022)-, and RL approaches- bassed on using policy optimization algorithm (Schulman et al., 2017; Guo al., 2025; Yu et al., 2025; Liu et al., 2025; Hu, 2025; Ahmadian et al., 2024; Kazemnejad et al., 2024). Recently, reinforcement learning with verifiable rewards (RLVR) (Lambert et al., 2024) has become the de facto standard for improving reasoning in LLMs, especially in mathematics and coding domains. In the case of reinforcement learning from human feedback (RLHF) for aligning models to human preferences, a reward model (Uesato et al., 2022; Lightman et al., 2023; Rafailov et al., 2023) is employed in order to rank the answers of the model to a prompt either at the end of the generation termed outcome reward models (ORMs) (Cobbe et al. 2021), or at each intermediate step termed process reward models (PRMs) (Cui et al., 2025; Yuan et al., 2024).
Despite the large literature covering RL post-training, there is still lack of understanding for the connection between the pretraining data and the effect it has on RL post-training optimization. To the best of our knowledge, we are the first to perform an extensive end-to- end study of the effect of pretraining data mixtures for mathematical reasoning in LLMs of different scales, and explore the difference between the common policy optimization algo- rithms. A theoretical explanation for the diversity collapse brought by RLVR is presented in Wu et al. (2025), who argue that RLVR is inherently limited to the support of the base model. Havrilla et al. (2024) is the closest work to our own, studying the performance of PPO across scales both on base models and fine-tuned models. Pan et al. (2025) also explores the emergence of the "Aha" moment in base LLMs, trained for solving countdown and multiplication tasks. Finally, Gandhi et al. (2025) leverage continued pretraining on Llama models towards bringing their performance closer to the Qwen models, and show that this improvement correlates with the reasoning abilities of the initial model.
B Dataset and Evaluation Details
As mentioned in Section 2.1, we include TinyGSM, OpenMathInstructll and OpenMathIn- struct2 instruction datasets in the pretraining mixture. Each of these datasets have distinct characteristics that can be searched for in the model's generations. We provide more details for each dataset here.
16

---

Published as a conference paper at COLM 2025
B.1 TinyGSM
In TinyGSM, answers are formatted as Python code enclosed within a function named math.problem(). This function consistently ends with return result, where result represents the final numerical solution to the grade-school math problem. To identify model generations that follow the TinyGSM format our experimental results, we search for the function signature def simple math problem(): To evaluate for correctness, we run the code within htapllemat__ppooooee replicates the problem statement. We track these characteristics in our experimental analysis, as discussed Section 3.3. Below, we provide a representative example of a question and its corresponding solution.
Additionally, these solutions include a docstring that
Representative Question TinyGSM
Benjamin picked some oranges at the fruit stand that cost $0.75 each. When Benjamin reached the cash register, he realized he was short the total price, so his friend Mason funded the rest. If Benjamin had $18 him, how many oranges did he buy?
Representative Answer in TinyGSM
def simple mapppee.math.pooeeem () -> int:
Benjamin picked some oranges at the fruit stand that cost $0.75 each. When Benjamin reached the cash register, realized he was $9 short of the lotal price, s0 his friend Mason funded the rest. If Benjamin had $18 on him, how many oranges did he buy? ...
cost-per-oranc amount-short 9
0.75
benjamin money 18
total_cost benjamin.money amount.short number_of_orangee = total_cost / cost.per-orange
result number of oranges return result
17

---

Published as a conference paper at COLM 2025
B.2 OpenMathInstructl
In OpenMathlnstructl,, answers are structured with code wrapped within <l1n=code></mmm-codd>>>> tags. Additionally, the parsed numerical result is enclosed in tags, followed by a final boxed answer. For GSM8K evaluations, we execute the model-generated code within the <l1m-cooee</11mmcoom tags to assess correctness. In the case of MATH, since models may post-process the code output, we evaluate correctness based on either the executed code and the final boxed result. To identify model generations in our experimental results that adhere to the OpenMathInstructi format, we search for the presence <l1m-code><IImm-oooee tags. A representative question and answer is given below.
Representative Question from OpenMaahhnntruutll
Martha has 18 crayons. She lost half of them, so she bought new set of 20 crayons. How many crayons in total does Martha have after the purchase?
Representative Answer from OpenMathInssruuttl
Let's solve this problem using Python code. <llm-code>
oooount.o..oss..rryooooo 18
ofoount.o..ne.crraee 20
totult of + amount.of.mew.crryom
total amount
</1lm-code>
<llm-code-outpul
Output: 29.0
</1mm-code-output>>
Thus, Martha has
29
crayons total.
18

---

Published as a conference paper at COLM 2025
B.3 OpenMathInstrucc2
In OpenMathInstructz, responses are provided in natural language with a final boxed answer. Since this is the only dataset we consider that exclusively generates answers in natural language, if no Python code is detected in the specified TinyGSM or OpenMath- Instruct1 format, we classify the model's 's output as 'Text' and assume it likely follows the OpenMathInstruct22 format.
Representative Question from OpenMatthnnstrutt2
bag contains red marbles and an unknown number of blue marbles. A player randomly selects marble from the bag. If a red marble is selected, the player wins 3 dollars, but if a blue marble is selected, the player loses 3 dollars. the expected amount won for playing the game is dollar, then how many blue marbles in the bag?
Representative Answer from OpenMathInstrrut22
Let be the number of blue marbles in the bag. The total number of marbles in the bag is + b. The probability of selecting a red marble is 8-1.
The probability of selecting blue marble is 8
b
The expected amount won is calculated as the sum of the products of the amount won and the probability of each outcome:
8
b
Expected amount won (3)
+
(-3)
8 8
8 b
We are given that the expected amount won is dollar:
8
b
(3)
+ (-3)
8
8 + b
Simplify the equation:
3b
8 + b
8 +++ b
Combine the fractions:
24 3b 1
b
Multiply both sides by + b:
24 3b 8 b
Solve for b:
24 8 3b + - 16
4b
+
+
4
B.4 Evaluation
To evaluate model generations, we apply different procedures depending on the output format. If the model produces code-such as in the TinyGSM or OpenMathInstrucct formats-we execute the code and extract the result: for TinyGSM, this is the value of the result variable, and for OpenMathinstructl, tags. Whether the model outputs code natural
the value of the variable on the last
line within the 11lm=code></1mm-ooo language, the final answer is parsed using the Math- Math-Verify library (Kydlíček, 2025) from HuggingFace to determine whether the prediction matches the correct answer.
19

---

Published as a conference paper at COLM 2025
We report three overall accuracy metrics: pass@1, pass@64, and majority@64. Pass@1 measures the percentage of questions correctly answered with a single generation using greedy decoding. Pass@64 reflects the percentage of problems for which at least one out of 64 sampled generations using temperature 0.7 produces a correct answer. Majority@64 measures the percentage of questions for which the most frequent final answer across 64 generations using temperature 0.7 matches the correct solution.
C Additional Experimental Details
We use the OpenRLHF (Hu et al., 2024) implementation of PPO and GRPO. The default hyperparameter configurations we for these algorithms in Table 2. We also vary KL coefficient to be 0 or 0.01. Other hyperparameters are set as default from OpenRLHI, for instance, for PPO we use the token-level KL penalty which added the reward, and for GRPO we incorporate the KL penalty in the loss and use the non-negative 'k3' estimator. We also use the hyperparameters Appendix F.2, where k = 64 is the number of samples we generate per problem before checking for correctness and filtering. We swept over peak learning rate values in [5 x 10 10-6, x 10-5, x 10 10-#, 000001] and observed very marginal gains (1-2%) for other learning
Table 3 for Expert Iteration (EI) results in
rates the first iteration of ET aside from x 10
Parameter
Value
Training Batch Size
64
Epochs
10
Prompt Max Length
1024
Generate Max Length
1024
Actor Learning Rate
1 10-6
Critic Learning Rate
7 10-6
Temperature
0.7
KL Coefficient
x 10-3
Rollout Batch Size
64
Samples per Prompt
8
Reward Normalization
True
A
0.95
Clip
0.2
Warmup
0.03
Adam Betas
(0.9, 0.95)
Table 2: Configuration for PPO and GRPO runs.
Parameter
Value
k
64
Training Batch Size
256
Epochs
2
Prompt Max Length
1024
Generate Max Length
1024
Learning Rate
x 10-4
Adam Betas
(0.9, 0.95)
Table 3: Hyper-Paramete Configuration for El runs.
D Additional Mixtures 150M Models
D.1 Mixtures with OpenMathInstruct and OpenMathInstruetz2
We provide additional results analogous to Figure and Figure for two other pretraining mixtures 150M models: TinyGSM and OpenMathInstruct11 (Figure 8) and TinyGSM
20

---

Published as a conference paper at COLM 2025
and OpenMatthnstruct22 (Figure 9). As before, we also include FineMath3+ and Algebraic- Stack in the pretraining mixture. Across both mixtures we see the model converges to outputting IinyGSM-format code, with the exception of high KL coefficient; note in particular that for all of our mixtures, KL coefficient yielded similarly performant results to the default setting 0.001, in line with prior work proposing to remove the KL. penalty for fine-tuning reasoning models (Yu et al., 2025).
siy
0.8
(Tey syle)
Tec:
Pecentage
poeedt
0.6
Accuracy
mujoriye4
0.4
0.0
0.0
10
Samples
Samples
(a) PPO model trained on TinyGSM and 1x OpenMathInstrrucl with KL coefficient 0.001.
1.0
(DM53-oyy)
Code (TayCBMuyy))
Ted:
Percentage
Accuracy
0.6
0.4
0.2
0.0
10
10
10'
10P
Samples
Samples
(b) PPO on a model trained on TinyGSM and 1x OpenMathlnstrueel with KL coefficient 0.01.
Codk (OMIl
0.8
0.8
Percenaag
Accuracy
0.6
majeiv804 (owelll)
0.4
0.2
0.2
10'
10
Samples
Samples
(c) PPO on model trained on TinyGSM and 1x OpenMathinstructt with KL coefficient 0.
Figure 8: Tracking percentage of generations and accuracy for PPO runs with varying KL coefficients, starting from a 150M model pretrained on TinyGSM and Open We observe that TinyGMM the consistently preferred distribution, and using KL coefficient 0 behaves similarly to KL coefficient 0.001.
D.2 TinyGSM Varying Fractions (1x, 2x, 4x, 8x)
In Figure 10 we show how pass@64 and majority@64 performance progresses throughout PPO training starting from models pretrained on various amounts of TinyGSM (along with FineMath3+ and Algebraic-Stack). While majority@64 yields 5-10% improvement across training, we note that pass@64 performance increases with the amount of TinyGSM shown in training but does not improve from model initialization during fine-tuning.
21

---

Published as a conference paper at COLM 2025
0.8
0.8
Percentage
0.6
Accuracy
nojobyfed
par
0.4
0.2
0.2
0.0
0.0
Samples
Samples
(a) PPO model trained on TinyGSM and OpenMathInstruct22 with KL coefficient 0.001.
1.0
[TbySSM eyle)
poesilt
Perentage
0.6
mejoiiyt4
0.6
Accuracy
044
0.4
0.0
0.0
10'
10
10
Samples
Samples
(b) PPO model trained TinyGSM and OpenMathInstrrud22 with KL coefficient 0.01.
1.0
Inct Code (TMyGSM--yk)
(owdd)
Percentage
Accuracy
0.6
0.4
0.2
0.2
10'
10
Samples
Samples
(c) PPO on a model trained on TinyGSM OpenMaathnnstruct22 with coefficient 0.
Figure 9: Tracking percentage of generations and accuracy for PPO runs with varying KL coefficients, starting from a 150M model pretrained on TinyGSM and OpenMathInstruct.. We observe that TinyGSM is the consistently preferred distribution, and using KL coefficient 0 behaves similarly to KL coefficient 0.001.
E
Additional Mixtures 1B Models
Below we provide additional figures showing the percentage of generations and respective accuracies starting from 1B parameter models pretrained on different mixes of TinyGSM, OpenMathInstructl, and OpenMMathnstruutt. For all of our 1B models, we include the FineMath3+ and Algebraic-Stack datasets. In Figure 11 we perform PPO on 1B model pretrained on TinyGSM and 4x OpenMathInstrucct (corresponding 150M model shown in Figure 4(a)) and in Figure 12 we perform PPO on a 1B model pretrained on TinyGSM and OpenMathInstruct2 (corresponding 150M model shown in Figure 9a)). We find that at this model scale, the model converges to outputting natural language rather than TinyGSM- or OpenMaathInstructl-sttt
code. We also verify that mixing TinyGSM and Open Instruct2 yielded the highest performing model after fine-tuning, instead of having only TinyGSM only OpenMathInstrucee and MMQA in the pretraining mix (see Figure 13).
22

---

Published as a conference paper at COLM 2025
Pass064
Majority@64
1.0
1 ThyGSM
2x Thy63M
0.8
0.8
Thy
Accuracy
Accuracy
x ThyGSM
0.6
0.6
1x Thy ThyGSM
TnyGSW
ThyGSM
0.2
0.2
200000
400000
600000
Samples
Samples
Figure 10: Pass@64 and majority@ 064 performance across epochs for the corresponding runs shown in Figure 5. While pass@k performance does not significantly improve after RL training, there 5-10% improvement in majority@k performance.
(TeyCSMaakk)
0.8
0.8
Text
Percentage
0.6
Accuracy
0.6
nojobtytt
pandt
0.2
0.0
0.0
Samples
Samples
Figure 11: Percentage of generations (left) and respective accuracies (right) during PPO training for 1B model pretrained on TinyGSM and 4x OpenMathInstructt.. This is the same pretraining data used for the 150M model in Figure 4 (a), but here we see the 1B model amplify the OpenMatlInstructl code format and obtaining a better final accuracy compared to the 150M model.
Other RL Algorithms: GRPO, EI
E1
GRPO
We also perform RL fine-tuning using GRPO (Shao et al., 2024) using the same hyperpa- rameters for PPO. In Figure 14 we present analogous results for GRPO as Figure 2 and Figure were for PPO. Across different data mixtures, we generally observed GRPO to exhibit the same phenomenon of preferring one distribution; however, it was less stable than PPO and often experienced brief collapse in performance before recovering again by the end of training. In Figure 14, we see that the model switches its preference from natural language generations to TinyGSM, coinciding with this drop in performance. GRPO with higher KL coefficient still exhibits the convergence to the TinyGSM format in contrast to PPO.
In Figure 15 we present analogous results as Figure 4 for GRPO. We see similar evolutions of the percentage of generations as in PPO, and the accuracy shows similar collapse (in the case of training with 8x OpenMathInstructf, this model does not recover from this collapse).
Finally in Figure we present analogous results as Figure 6 where we perform GRPO on model trained on 4x TinyGSM only (without Algebraic-Stack and FineMath3+) and in Figure where we do GRPO on models trained on varying amounts of TinyGSM (with Algebraic-Stack and FineMath3+ included). We that performance is very similar to PPO, with GRPO performing slightly worse for increasing amounts of TinyGSM the pretraining data.
23

---

Published as a conference paper at COLM 2025
0.8
0.8
pamiit
Percentage
0.6
Accuracy
0.6
nojobyytt
pas
0.2
0.2
0.0
0.0
Samples
Samples
Figure 12: Percentage of generations (left) and respective accuracies (right) during PPO training for 1B model pretrained on TinyGSM and OpenMathInstruct2. Although our 150M pretrained models most frequently converged on only outputting only TinyGSM- formatted generations, here we see the model amplify natural language solutions, even though TinyGSM is the more performant distribution at initialization.
1.3
Coke (T ayGoM up
Tes
0.5
0.5
Accuracy
Accuraos
0.5
0.5
0:4-::-----
0.4
0.2
0.2
0.0
Samples
Samples
Figure 13: Accuracy during PPO training for a 1B model pretrained on TinyGSM (left) and OpenMathInstrucch and MMQA (right). For the 1B model on the left, its final accuracy is higher than the corresponding 150M model pretrained on the same amount of data (See Figure 5). However, both models trained on these subsets alone do not reach the same final accuracy as the model pretrained with the two datasets mixed (see Figure 12).
F2 Expert Iteration
We also ran Expert Tteration on subset of our 150M pretrained models. As outlined in Section 2, we began by generating k = 64 candidate solutions per problem from the GSM8K training set using the pretrained model. From these, we constructed de-duplicated dataset consisting only of generations that yield the correct final answer. This dataset was then used for supervised fine-tuning of the pretrained model. We repeated this process over multiple iterations: each time, the fine-tuned model was used to regenerate correct samples, while the training continued from the original base model. Our main goals were to assess whether one data format tends to dominate over others in the mixture and to compare performance against our PPO results, following similar questions posed in Havrilla et al. (2024). To ensure comparable x-axis with our PPO results, track the percentage and accuracy of generations as a function of the cumulative number of training samples. Specifically, for each iteration, increment the total sample count by multiplying the number of training epochs with the size of the de-duplicated dataset.
In Figure 17, we present results from three iterations of Expert Iteration starting from the same 150M base model used in Figure 2, pretrained on a mixture of TinyGSM, OpenMathIn- structl, and OpenMathInstruct2.. Despite seeing comparable number of training samples, final performance lags behind that of PPO, and the model's 's generations do not show strong preference for any particular dataset format. Nonetheless, there a modest trend toward increased preference for TinyGSM over time, though this shift is slower and less pronounced; see Figure 18 and Figure 19 for similar experiments using base models pre- trained on TinyGSM OpenMaahlnsttuutll and TinyGSM + OpenMathlnstruct2, respectively. Overall, we find that Expert Iteration consistently underperforms PPO- even in settings without dataset mixtures. For example, in Figure 20, starting from base model pretrained
24

---

Published as a conference paper at COLM 2025
0.8
(TeyCSMaakk)
0.8
Percentage
0.6
Accuracy
majaiiy044
(owsi)
0.4
0.2
0.2
0.0
0.0
Samples
Samples
(a) GRPO with KL coefficient 0.001.
1.0
(DMB3-dy a) Cook (3MyCkk)
Tect
Pereetage
0.6
posi8t
0.6
Accuracy
majohty04
044
0.4
0.0
0.0
10'
10
10'
112
Samples
Samples
(b) GRPO with KL coefficient 0.01.
Figure 14: The analogous results using GRPO starting from the same model pretrained with TinyGSM, Ophhnaatlnstruuttt, and OpenMathInstructt. with low KL ((a), analogous to Figure 2) and high KL coefficient ((b), analogous to Figure 3). GRPO exhibits less stable dynamics compared to PPO, where it appears that one distribution is about to be preferred but suddenly switches its preferences, corresponding with a drop in overall accuracy. Once the model has converged on one distribution, the accuracy begins recovering again. We also note that GRPO is more robust to high KL, likely due to the presence of the KL penalty in the loss as opposed to the reward (see Appendix C).
on 8x TinyGSM (which achieves 60% GSMSK test accuracy after PPO), accuracy after three El iterations remains below 45%.
We also two iterations of El on three of our pretrained 1B models. In Figure 21 observe similar trends where accuracy marginally improves and there is modest trend towards an increased preference for OpenMathInstrruct, cnatturucctnauualll natural language-style answers.
We hypothesize that the slower shift toward a dominant format is due to the repeated fine-tuning from the fixed base model, in contrast PPO or GRPO's more online nature. This may suggest that more offline update steps in RL fine-tuning help maintain the original distribution, which could be beneficial for preserving generation diversity. We leave further exploration of RL algorithms and their associated design choices to future work.
G Co Metrics
Our results in Section 3 highlight how different pretraining data mixtures influence both the stylistic distribution and accuracy of model outputs. We now show that these preferences also manifest in confidence-based metrics.
During RL fine-tuning, we track the average probability outputs beginning with def simple_ma wttee_aa__ppoooe (rirhhprrobeem)) and Let's solve this problem using Python code. <1lm-code> on the GSMSK test set. As detailed in Appendix B, these token prefixes are characteristic of TinyGSM and OMII generations, respectively. (We exclude OM12 from this analysis due to the lack consistent initial token pattern.) As shown in Figure 22, the average probabilities closely follow the trends in output proportions presented in Figures 2, 4(a),
25

---

Published as a conference paper at COLM 2025
0.8
(TeyCSMaakk)
0.8
Percentage
0.6
Accuracy
majariy044
0.4
0.2
0.2
0.0
0.0
Samples
Samples
(a) GRPO initialized from model trained on TinyGSM and 4 x OpenMathInstructl..
1.0
(DMB3-dt a)
Cook (TMyCSMa
Text
Perentage
0.6
passit
0.6
Accuracy
majorby854
044
0.4
0.0
0.0
10'
10
10
Samples
Samples
(b) GRPO initialized from model trained TinyGSM and 8 OpenMathInstruuttl.
Figure 15: Analogous figure as Figure 4 when using GRPO instead of PPO. We see the same conclusion that TinyGSM preferred in (a) and OpenMathInstruct!! is preferred in (b) which results in collapse in performance. We observe the same initial increase and collapse later in training as mentioned in Figure 14.
1.0
pasett
majornyy54
ponabos
Accuracy
Accuracy
0.8
pass2]
majoriy/itd
0.6
0.1
10'
Amount TinyGSM
Samples
Figure 16: Analogous figures Figure 5 (Left and Figure 6 (Right) when using GRPO instead of PPO. We see near-identical trends as in PPO, with the exception of pass@1 accuracy being slightly worse when increasing quantities of TinyGSM compared to PPO.
and 4(b), albeit with smoother trajectory. Additionally, the narrowing error bars over the course of training suggest further stability.
Overall, we found that the average generation probabilities increase throughout train- ing-even even after the output format has largely stabilized-indias indicating that the model's confi- dence continues to grow within the dominant output distribution.
H Further Transfer Learning Investigations
H.1 Qualitative Analysis on MATH-500 Generations
In Section 4, we demonstrated that 1B models fine-tuned on GSM8K questions showed improved performance on MATH-500. To further analyze these gains, for each of our models
26

---

Published as a conference paper at COLM 2025
0.8
0.8
Tect
Percentage
0.6
Accuracy
mojottyte
0.4
0.2
0.2
0.0
0.0
200000
400000
600000
200000
400000
600000
Samples
Samples
Figure 17: Percentage of generations (Left) and respective accuracies (Right) function of cumulative number of training samples for the same 150M model pretrained on TinyGSM, Open and OpenMMathInstruut2-aa We note lower increase in overall performance for roughly a similar number of examples for PPO, and the percentage of generations show only a slight preference for TinyGSM- format generations.
Figure 2-across three iterations of EI.
1.0
Code (OMi)
Codl (Tby eyle)
Percenaage
0.6
0.6
Accuracy
rojoityt4
0.4
0.4
0.0
0.0
0
100000 200000 300000 800000
50000
200000
400000
Samples
Samples
Figure 18: Percentage of generations (Left) and respective accuracies (Right) as a function of cumulative number of training samples for 150M model pretrained TinyGSM and Open across three iterations of El. Here we see the final accuracy is lower than that of PPO (see Figure 8 (a)) and an increasing preference for TinyGSM.
identified the subset of questions where the model's answer initially incorrect after pretraining but became correct following fine-tuning. For each of these cases, we prompted GPT-4.5 Preview to explain why the base model's response was incorrect, why the fine- tuned model's response was correct, and to indicate which type of error was corrected between the two generations, from the following predefined set:
Arithmetic error Mistakes in calculation, sign, order of operations, rounding, or undefined operations.
Formula/application mistake - Using the wrong formula, incorrect substitutions, or misapplying rules (e.g., differentiation, integration, exponentiation, trigonometry).
extra terms, or flawed Algebraic/logic flaw Incorrect manipulation, missing/ eextra reasoning in problem-solving.
Nissinerpreaaiionniiseeddng Incorrect understanding of the problem, assump- tions, or misusing given information.
Notation/repessenation issue Errors in variables, indexing, units, graphing, or coordinate representation.
Incomplete answer Incorrect solution was incomplete or collapsed (started repeat- ing, included irrelevant content, etc.)
Figure 24 presents breakdown of error types made by each pretrained model, sorted in descending order from left to right. Across most models, the dominant sources of error stem from misinterpreting the question making flawed algebraic logical deductions.
27

---

Published as a conference paper at COLM 2025
0.8
0.8
Percentage
0.6
Accuracy
nojobyyte
pas
0.4
0.2
0.2
0.0
0.0
100000 200000 30000 400000 600000
GO0O00
400000
0O0000
Samples
Samples
Figure 19: Percentage of generations (Left) and respective accuracies (Right) as function of cumulative number of training samples for a 150M model pretrained on TinyGSM and OpenMath Instrruct222 across three iterations of EI. Here we see the final accuracy is lower than that of PPO (see Figure 9 (a)) with performance plateauing by the third iteration. We do see a similar trend as in Figure (a) where TinyGSM-for formmft code starting to occupy larger percentage of generations compared to natural language, but the effect is much slower compared to PPO.
Cok: (TMyGSM-ay))
Text
0.8
Percentage
0.6
Accuracy
0.2
0.2
200000
400000
600000
S00000
200000
400000 600000 800000
Samples
Samples
Figure 20: Percentage of generations (Left) and respective accuracies (Right) function of cumulative number of training samples for a 150M model pretrained on 8x TinyGSM across three iterations of EI. After three iterations of EI, the model performance is below 45%, whereas after PPO the accuracy reaches almost 60%.
This suggests that the gains from fine-tuning are not driven by improvements just in better arithmetic accuracy. Instead, they appear to enhance the model's ability to comprehend the problem and reason through its solution, along with the format-level refinements discussed in Section 3.3.
H.2 AIME Evaluation
In Section 4, we showed that evaluation on MATH-500 improved after applying PPO on GSM8K training questions. Here, we present additional evaluation results on AIME. As shown in Table 4, performance on AIME 2022-2024 questions exhibits minimal to no improvement in pass@1 and majority@64 metrics following PPO.
In contrast, Table 5, which includes broader evaluation set spanning AIME 1983-2024, shows more substantial gains both metrics. However, we do observe improvement in pass@64 performance for the two AIME subsets in Table 6. Notably, models pretrained on mixtures incorporating OpentMathInstruct datasets (which include synthetic problems derived from MATH) achieved the largest improvements after post-training. The observed pattern suggests that data similarity between pretraining and evaluation distributions is crucial for transfer. In particular, AIME questions prior to 2022 are known to have potential data contamination with MATH.
In Figure 23, we perform the same qualitative analysis on the generations for the AIME pass@64 evaluation as in Section H.1.
28

---

Published as a conference paper at COLM 2025
1.0
Cok (0Missty
(TRy
0.8
0.8
Percentage
0.6
Accuracy
majortycc
0.4
0.2
0.2
0.0
0.0
200000
400000
000000
500000
200000
400000
600000
800000
Samples
Samples
(a) EI on 1B model trained on TinyGSM, OpenMathlnstruutll, and OpenMathInstruutt..
1.0
(OMii-ate
(TeyGSMayy))
0.8
Perentage
0.6
Acturacy
0.6
mojontytt
0.4
0.0
0.0
0.0
0.2
0.4
0.5
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
Samples
x10*
Samples
x1I
(b) EI on 1B model trained TinyGSM and 4x Optnnaatlnstruuttl.
1.0
1.0
Code (SirCC5))
Percentag
Accuracy
0.6
mejoiiy644
pamillsi
powdi
0.4
0.4
0.2
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.4
0.6
1.0
Samples
Samples
x1P
(c) ET on a 1B model trained on TinyGSM, OpenMattInstruutt2, and MMQA.
Figure 21: We perform two iterations of El for starting from three 1B pretrained models. We see only a slight increase in overall performance, and a trend towards preferring natural language answers (consistent with findings regarding the preferred distribution chang- ing with scale in Section 3.4).
TinyGSM OMI OM2
TinySSM 4x0MI1
TinyGSM BxOMI1
1.0
1.0
0.8
Probabitt
Probabiity
Probuhiity
0.2
OVB Teet9
1.2
0.2
Tay
18
109
Samples
Samples
Samples
Figure 22: Average probability of def simple_math hompleemath.problend and Let's solve this problem using Python code. <ilm-code> occurring after each problem in the GSMSK test for models pretrained from TinyGSM, OpenMathInstructt and OpenMath[nstructt (left), TinyGSM and 4x Mathssutintt (middle), and TinyGSM and 8x OpenMath- Instruct1 (right). The average probability corresponding to generations from the preferred dataset in the percentage plots (from left to right, Figure 2, Figure 4(a), and Figure 4(b)) is similarly amplified over the course RL fine-tuning.
29

---

Published as a conference paper at COLM 2025
Error Type
TinyGSM + OM12
Algebraic; logic flow
Miinter praati////n/n/
TinyGSM OMI1 OM12
Formila /'pplication mistake
Incomplete
TinyGSM + 4 OMI1
Arilhmelic ence
Nocation//rp frabisnntnnrrnnnaattion
TinyGSM
OMI2 MMQA
10%
20%
30%
40%
60%
70%
90%
100%
Figure 23: Distribution of error types on AIME for each 1B pretrained model before fine- tuning GSM8K.
Error Type
TinyGSM OMI2
Algebraic, logic flow
Misinter
TinyGSM OMI1 OM12
hormuia /application mistake
Incomplete
TinyGSM 4 OMI1
Av Arthhmetic encrr
Netetion//ngrreentaati
TinyGSM
OMI2 MMQA
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
Figure 24: Distribution of error types on MATH-500 for each 1B pretrained model before fine-tuning on GSM8K.
30

---

Published as a conference paper at COLM 2025
Pretraining Data Mixture
Pass@1 Base
Pass@1 FT
Maj@64 Base
Maj@64 FT
TinyGMM + 4xOMI1
0.00%
0.00%
0.00%
0.00%
TinyGSM + OMT2
0.00%
1.11%
0.00%
2.22%
OMI2 + MMQA
1.11%
2.22%
1.11%
3.33%
TinyGSM
0.00%
0.00%
0.00%
1.11%
TinyGSM + OMI1 + OMI2
0.00%
2.22%
1.11%
2.22%
Table 4: Pass@1 and majority@64 performance of different pretraining data mixtures on the AIME 2022-2024 benchmark both before and after doing PPO on GSM8K.
31

---

Published as a conference paper at COLM 2025
Pretraining Data Mixture
Pass@1 Base
Pass@1 FT
Maj@64 Base
Maj@64 FT
TinyGMM + 4xOMI1
0.00%
0.00%
0.00%
0.00%
TinyGSM + OMT2
2.47%
6.54%
6.43%
13.93%
OMI2 + MMQA
2.89%
7.93%
7.40%
14.36%
TinyGSM
0.00%
0.21%
0.21%
0.75%
TinyGSM + OMI1 + OMI2
2.47%
7.18%
6.54%
13.50%
Table 5: Pass@1 and majority@64 performance of different pretraining data mixtures on the AIME 1983-2024 benchmark both before and after doing PPO on GSM8K.
Pretraining Data Mixture
1983-2024 Pass@64 Base
1983-2024 Pass@64 FT
TinyGSM + 4xOMI1
0.00%
0.00%
TinyGSM OM12
26.37%
37.41%
OM12 MMQA
26.58%
36.55%
TinyGSM
4.72%
7.07%
TinyGSM + OM11 + OMI2
25.08%
38.16%
Pretraining Data Mixture
2022-2024 Pass@64 Base
2022-2024 Pass@64
GSM + 4xOM11
0.00%
4.44%
TinyGSM OM12
0.00%
18.89%
OMI2 MMQA
8.89%
21.11%
TinyGSM
1.11%
8,89%
TinyGSM OMI1 OM12
11.11%
15.56%
Table 6: Pass@64 results on the AIME 1983-2024 and 2022-2024 benchmarks after PPO on GSM8K.
H.3 Example Generations on MATH and AIME 2022-2024
Below we provide an example problem, an incorrect solution generated by the pretrained 1B model, and a subsequently correct solution generated by the fine-tuned 1B model for both MATH-500 and AIME 2024.
Example Problem from MATH-500
Problem: The two-digit number "B6," where B is the tens digit, is the square of positive integer. How many distinct possible values are there for B?
Example Problem from AIME 2024
Alice chooses set A of positive integers. Then Bob lists all finite nonempty sets B of positive integers with the property that the maximum element of B belongs to A. Bob's list has 2024 sets. Find the sum of the elements of A.
Problem:
32

---

Published as a conference paper at COLM 2025
Example Incorrect Generation from MATH-500
Let B be a two-digit number, so B 10a + b, where a is the tens digit and b is the units digit. Since B is the square of positive integer, we can write:
x2
where x is positive integer. We can rewrite the equation as:
10a + xx
Since and b are digits, know that 1 s 9 and b 9. We can try different values of x to find the possible values of For x have:
10a + b = 1 =
10a
0 0 10a < 9 + 0 10a LA
8
=
0 a 0.8
This a contradiction, so + 1.
For x = 2, we have:
10a b 2 4 b= 10a
0 0 4 10a < 9 0 < 10a <4=05a55 0.4
This is contradiction, so + 2.
For x = 3, we have:
10a b 3 9 b b= 10a
0 10a < 9 = 0 < 10a < 9 = 0<
This is a contradiction, so + 3.
For x have:
10a + b = 42 = 16 b = 16 10a
< 16 10a < 9 = 0 10a = 0 0.
This a contradiction, so x + 4. For x we have:
10a + = 52 = 25 b 25 10a
10a < 9 = 0 < 10a < 1 0
This a contradiction, so + 5.
For x = 6, we have:
10a + 62 36 = b 36 10a
0 36 10a < 9 0 < 10a < 0
This is contradiction, so + 6.
For x = 7, we have:
10a + b 7 49 = b = 49 - 10a
0 49 10a < 9 = 0 < 10a 40 = 0
This is a contradiction, so x + 7.
For x 8, we have:
10a + = 82 = - b b= 64 10a
0 64 10a < 9 = 0 < 10a < = 0 5.5
This a contradiction, so x + For x we have:
10a + 92 81 + 81 10a
81 10a < 9 0 S 10a < 72 = 7.2
This contradiction, so 9. Therefore, there are no possible values for B. The answer is:
33

---

Published as a conference paper at COLM 2025
Example Correct Generation from MATH-500
We are given that the two-digit number B6 is the square of a positive integer. Let's denote this integer as x. Then we have:
B6 = x²
Since B is the tens digit, it can range from to 9. We can write the equation as:
10B 6 =
We need find the possible values of B that satisfy this equation.
Notice that x² must be a two-digit number, so 10 S x² 100. This means < 10. We can try different values of x and see if we get valid B:
For x we have 10B
16, so 10B 10, which gives B =.
For = 5, we have 10B + 6 25, so 10B 19, which does not give integer value
for B.
For
have 10B 6 36, so 10B 30, which gives B = 3.
For x = 7, we have 10B + 6 = 49, so 10B 43, which does not give an integer value
for B.
For x = 8, we have 10B + 6 = 64, so 10B = 58, which does not give an integer value
for B.
For x = 9, we have 10B + 6 = 81, so 10B 75, which does not give an integer value
for B.
We found two possible values for B: and 3.
The answer is:
2
34

---

Published as a conference paper at COLM 2025
Example Incorrect Generation from AIME 2024
Let's denote the elements of A as a1, 42,02... "2,,.,..,, We want to find the sum of the elements of A.
First, we notice that for each from 1 to there are at most 2024 distinct sets B, with the property that the maximum element of B; belongs to A,-
Since there are 2024 sets in total, we have:
11 b 2024
Σ BEB,
Now, let's consider the sum of the elements A. We can rewrite it as:
Σ
Σ b
1bCB
1
Since A; is a subset of B,, we have:
" EI
Σ a ElbbA;
Notice that for each i, the sum of the elements of A; is equal to the sum of the elements of B; minus the maximum element of B;. Since the maximum element of B; belongs to A;, we have:
H Σ
Σ
ΣΣ
E
aj Σ
1
bE
BEA:
Simplifying, we get:
Σ 2
Σ a; = 2024
1=1 beΛ;
Since each a; appears at most 2024 times in the sum, we have:
Σ
Σ bEA
ai S 2024 2 4048
We also know:
n
"
Σ
Σ S 2024 bEB;
Σ
Σ S 2024.00400
1-1 -bEA;
Thus:
"
Σ
Σ bEA;
4; 2
Then:
aj 2024
2024 2024 4096
35

---

Published as a conference paper at COLM 2025
Example Correct Generation from AIME 2024
Let the elements of A be 0j, 01,02....,,4 An. A set B is allowed if its maximum element is in A.
Every subset B must be nonempty set of positive integers such that max B) E A. Now, fix an element a E A. How many subsets B have maximum a? Every such is nonempty subset of {1,2, ..., ) with maximum a, which means the rest of the elements in B must be from (1,2, 1}. There are 2 such subsets (each subset of {1,..,, a - 1} plus a).
So the total number of such sets is:
Σ 2 1 2024
REA
Now we want to find set A of positive integers such that Loc 20 2024. Note that 2024 in binary is:
2024 10 1023 is too small, 2 2048 too big
Try breaking 2024 into powers 2:
2024 1024 + 512 256 128 + 64 + 32 + 4 11, 10,1,,,,,,,7,64
Then the sum of elements of A is:
11 + 10 9 +
55
I RL Fine-tuning on MATH
The results presented in Sections 3 and 4 focus on performing PPO fine-tuning with problems from the GSM8K train split; we also ran PPO with problems from the MATH train split for three of our 1B models pretrained with difference mixtures. Due to computational resources we keep the same hyperparameters as detailed in Appendix C.
We show the change in performance on MATH-500 in Table as well as performance AIME 1983-2024 and AIME 2022-2024 in Table 9, Table 8, and Table 10. Compared to fine-tuning on GSMSK train questions, we observe less improvements in performance on MATH-500 and similar results when evaluating on AIME, where only pass@64 performance yields significant improvements.
36

---

Published as a conference paper at COLM 2025
Pretraining Data Mixture
MATH Pass@1 Base
MATH Pass@1 FT
TinyGSM OMI2
33.40%
39.80%
OM12 + MMQA
34.60%
42.80%
TinyGSM OMIL OMI2
33.40%
39.20%
Pretraining Data Mixture
MATH Maj@64 Base
MATH Maj@64 FT
TinyGSM OM12
46.20%
49.20%
OM12 + MMQA
51.20%
50.00%
TinyGSM OMI1 OMI2
48.60%
49.40%
Pretraining Data Mixture
MATH Pass@64 Base
MATH Pass@64 FT
TinyGSM + OMI2
80.40%
83.00%
OMI2 + MMQA
80.60%
83.80%
TinyGSM + OMI1 OMI2
83.40%
82.40%
Table Pass@1, majority@64, and pass@64 performance of different pretraining data mixtures on the MATH-500 benchmark both before and after doing PPO on MATH.
Pretraining Data Mixture
Pass@1 Base
Pass@1 FT
Maj@64 Base
Maj@64 FT
TinyGSM + OMI2
1.11%
3.33%
1.11%
3.33%
OM12 + MMQA
0.00%
1.11%
0.00%
2.22%
TinyGSM + OMI1 + OMI2
0.00%
2.22%
1.11%
3.33%
Table 8: Pass@1 and majority@64 performance of different pretraining data mixtures on the AIME 2022-2024 benchmark both before and after doing PPO on MATH.
Pretraining Data Mixture
Pass@1 Base
Pass@1 FT
Maj@64 Base
Maj@64 FT
TinyGSM + OMI2
2.47%
6.65%
6.43%
11.79%
OM12 + MMQA
2.89%
7.72%
7.40%
13.40%
TinyGSM + OMII + OM12
2.47%
7.82%
6.54%
14.36%
Table 9; Pass@1 and majority@64 performance of different pretraining data mixtures on the AIME 1983-2024 benchmark both before and after doing PPO on MATH.
Pretraining Data Mixture
1983-2024 Pass@64 Base
1983-2024 Pass@64 FT
TinyGSM + OM12
26.37%
34.51%
OM12 MMQA
26.58%
34.41%
TinyGSM + OMT1 OMI2
25.08%
35.58%
Pretraining Data Mixture
2022-2024 Pass@64 Base
2022-2024 Pass@64
TinyGSM + OM12
10.00%
18.89%
OM12 MMQA
0.00%
15.56%
TinyGSM + OMI1 OMI2
10.00%
18.89%
Table 10: Pass@64 results on the AIME 1983-2024 and 2022-2024 benchmarks after PPO MATH.
37


Reviews:
[{"review_id": "8ChmAjhbog", "reviewer": "Reviewer_cgzq", "summary": "The paper seeks to explore the mechanism(s) underlying the ability of reinforcement learning fine-tuning to improve performance on mathematical reasoning and coding tasks.  The authors pre-train language models of different scale from _scratch_ on curated mixtures of open datasets.  These datasets have distinct format that function as _fingerprints_ which allow the authors to track the output distributions of the language model before and after reinforcement learning fine-tuning.  They find that language models fine-tuned for mathematical reasoning and coding tasks often converge to the distribution of a single component of the training corpus, and that this specific distribution influenced by size.", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "*\tThe paper systematically establishes a link between pre-training data and a model’s output distribution after RL fine-tuning across three RL algorithms. \n*\tThe experimental setup concise and crystal-clear, with sound rationale for the selected architecture, datasets and reinforcement learning algorithms.\n*\tPre-training from _scratch_ on datasets with _fingerprints_ is insightful as it provides a mechanism to track otherwise intractable output distributions.\n*\tResults obtained from experimenting with the KL divergence coefficient are interesting, and illustrate the coefficient is indeed effective at guiding the output distribution.\n*\tThe observation that pre-training on a single dataset (line 195 -197) is potentially impactful.\n*\tThe finding that smaller models seem to prefer code-style pre-training, while a larger model’s output distribution converges to a natural language distribution is a strong foundation for future research.\n*\tResults are thoroughly ablated with respect to dataset mixtures and reinforcement earning algorithms.\n*\tThe paradigm of moving towards pre-training from _scratch_ instead of relying on existing pre-trained models is excellent.", "reasons_to_reject": "*\tWhile the empirical findings are highly valuable, more insight on the underlying _mechanism_ might be helpful.\n*\tExploring other families of tasks or model architectures could make the findings more durable.\n*\t(minor) Presenting results in tabular format would help with comparisons among settings.\n*\t(minor) Providing examples of the _distinct formats_ which facilitate distribution tracking would make this important consideration less abstract.", "questions_to_authors": "None", "extra_scores": {}}, {"review_id": "p2SLwrdS7N", "reviewer": "Reviewer_va6E", "summary": "The paper presents a systematic end-to-end investigation into how reinforcement learning (RL) fine-tuning shapes the behavior of language models trained for mathematical reasoning. The authors pretrain models from scratch using fully open datasets with varying characteristics and then apply several RL algorithms (PPO, GRPO, Expert Iteration). They find that RL fine-tuning causes models to converge towards a single dominant output distribution, typically aligning with the most performant pretraining dataset format. This effect is shown to be scale-dependent, with smaller models favoring code-like outputs and larger models preferring natural language. The study also demonstrates that fine-tuning on simpler tasks (like GSM8K) can yield improvements on harder ones (like MATH), suggesting generalization of reasoning capabilities.", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Controlled Experimental Design: The authors pretrain models from scratch using open datasets, allowing for full transparency and control over training data. Most of the existing work does not ablate the pre-training data factor.\n\n- Findings on Distributional Collapse: The discovery that RL fine-tuning amplifies a dominant pretraining format (i.e., \"echo chamber\" effect) provides a novel perspective on how RL interacts with model representations, offering valuable implications for interpretability and diversity.\n\n- Scale-Specific Analysis: By comparing 150M and 1B models, the paper reveals scale-dependent biases in format preference and performance, enriching our understanding of model capacity and generalization.", "reasons_to_reject": "- Format vs. Semantics: One of the core findings is that RL fine-tuning pushes the model toward a dominant output format, but it’s unclear whether this reflects true improvements in reasoning or just better adherence to a surface-level template. Are these models actually reasoning better, or just getting better at mimicking the structure of their most familiar training examples? \n\n- Narrow Dataset Scope: The authors do well to control the pretraining pipeline with open datasets, but the scope is quite limited: centered on mathematical instruction. This is understandable given the focus of the study, but it raises questions about generality. Would similar dynamics emerge in other reasoning and non-reasoning tasks? Could this kind of format collapse still occur if the model were exposed to less structured or more heterogeneous instruction types?\n\n- Heavy Reliance on PPO: PPO is the main algorithm used, and while GRPO and Expert Iteration are mentioned, they’re treated more as side notes. It would be good to understand how much of the observed behavior is an artifact of PPO itself. For instance, would methods with less aggressive policy updates show more format diversity? Or could different RL objectives (like reward shaping with process-level feedback) lead to less homogenization?", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "WvUoL7OfPY", "reviewer": "Reviewer_9LBw", "summary": "This paper investigates how RL fine-tuning interacts with the pretraining data of language models in mathematical reasoning tasks. The authors perform a comprehensive, end-to-end analysis using fully open pretraining datasets and various RL algorithms (PPO, GRPO, Expert Iteration) across several model scales. They find that RL post-training amplifies a single distribution from the pretraining mixture, reducing diversity. The favored format is often the most performant one, but it varies with model scale: smaller models favor simpler, code-like formats while larger models shift towards natural language outputs. The paper is well written and easy to follow.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Novelty and Relevance: The paper fills a significant gap in understanding how RL interacts with pretraining, especially in controlled settings. This is a good contribution given the opacity in the current state-of-the-art LLMs.\n- Clear Empirical Findings: The “echo chamber” effect (convergence to one pretraining format) is clearly and convincingly demonstrated.\n- Well-designed experiments: I like the comparison between pass 1 and pass 64 design, which provides us a good overview of how the model’s distribution on these responses looks like (peakier distribution yields higher pass 1 while higher pass 64 means a larger and better support).", "reasons_to_reject": "- Some of the interesting findings could be further analyzed and explained, e.g., the role of the KL term.\n- Only tracking the accuracy and percentage might not explain the phenomenon here. I suggest tracking some confidence-related metrics, e.g., entropy of the distribution, log-probabilities of them, etc., would make the paper stronger.\n- There’s no theoretical or mechanistic explanation for why the model collapses to one format. Could this be tied to reward sparsity, easier credit assignment, squeezing effect (mentioned later), or inductive bias?", "questions_to_authors": "- I find the phenomenon demonstrated in this paper, i.e., RL tuning lets the model consistently converge to a dominant output distribution coinciding with a significant increase in performance, is quite similar to those reported in [1] on off-policy DPO. Would the theoretical framework provided in [1] explain the findings in this paper?\n- The results show that the converged type or distribution is not directly correlated to the initial ratio of the pretrained data. Would the average initial confidence in different types of responses be a good measurement?\n- Some small typos (or inconsistencies), e.g., “150m model” in the caption of Figure 3, should it be “150M”?\n\n[1] Ren, Yi, and Danica J. Sutherland. \"Learning dynamics of llm finetuning.\" ICLR 2025", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper proposes a controlled study where they analyze the interaction between the pretraining mixture and RL finetuning finding that different mixtures lead to very different output distributions after RL finetuning. This is an interesting finding that all reviewers appreciated. Reviewers were impressed by the effort and clean experimental methodology and findings.

As a sidenote, this finding is perhaps not surprising given that the solution to the problem of maximizing E(R) - KL(\pi || \pi_ref) is the original distribution tilted by an exponent of the reward  (\pi^*(y | x) \propto \pi_ref(y | x) exp(R(x,y / \beta).
Decision: Accept