Smaller, Weaker, Yet Better: Training LLM Reasoners via Compute-Optimal Sampling
Authors: Hritik Bansal
Arian Hosseini
Rishabh Agarwal
Vinh Q. Tran
Mehran Kazemi
Venue: ICLR 2025 Poster (2025)
Keywords: large and small language models
reasoning
math
compute-optimal
sampling
supervised finetuning
Primary area: foundation or frontier models, including LLMs

Abstract: Training on high-quality synthetic data from strong language models (LMs) is a common strategy to improve the reasoning performance of LMs. In this work, we revisit whether this strategy is compute-optimal under a fixed inference budget (e.g., FLOPs). To do so, we investigate the trade-offs between generating synthetic data using a stronger but more expensive (SE) model versus a weaker but cheaper (WC) model. We evaluate the generated data across three key metrics: coverage, diversity, and false positive rate, and show that the data from WC models may have higher coverage and diversity, but also exhibit higher false positive rates. We then finetune LMs on data from SE and WC models in different settings: knowledge distillation, self-improvement, and a novel weak-to-strong improvement setup where a weaker LM teaches reasoning to a stronger LM. Our findings reveal that models finetuned on WC-generated data consistently outperform those trained on SE-generated data across multiple benchmarks and multiple choices of WC and SE models. These results challenge the prevailing practice of relying on SE models for synthetic data generation, suggesting that WC may be the compute-optimal approach for training advanced LM reasoners.

Published as a conference paper at ICLR 2025
SMALLER, WEAKER, YET BETTER: TRAINING LLM REASONERS VIA COMPUTE-OPTIMAL SAMPLING
Hritik Bansal 1,2 Arian Hosseini Rishabh Agarwal Vinh Q. Tran', Mehran Kazemi
1 Google DeepMind, 2 UCLA, Mila
Correspondence: hbansallg. ucla. edu and mehrankazemi eeceeeemm@@@ogggoo @google.....ooooooooooogooooe... com
ABSTRACT
Training on high-quality synthetic data from strong language models (LMs) is a common strategy to improve the reasoning performance of LMs. In this work, we revisit whether this strategy is compute-optimal under a fixed inference bud- get (e.g., FLOPs). To do so, investigate the trade-offs between generating synthetic data using a stronger but more expensive (SE) model versus a weaker but cheaper (WC) model. We evaluate the generated data across three key met- rics: coverage, diversity, and false positive rate, and show that the data from WC models may have higher coverage and diversity, but also exhibit higher false pos- itive rates. We then finetune LMS on data from SE and WC models in different settings: knowledge distillation, self-improvement, and a novel weak-to-strong improvement setup where a weaker LM teaches reasoning findings reveal that models finetuned on WC-generated data consistently outper- form those trained on SE- SE-generated data across multiple benchmarks and multiple choices of WC and SE models. These results challenge the prevailing practice relying SE models for synthetic data generation, suggesting that WC may be the compute-optimal approach for training advanced LM reasoners.
stronger LM. Our
Knowledge distillation with Gemini-1 5 data MATH
Ground
matched)
MATH Dataset
#lsh data tprioermatchedi
(Knowledge distillation)
Finetuning
(Self inprovement)
Finetuning
(Weak-tn-strip improvement)
accuray
46
Fass@1 Accuracy (86)
45
28
+6.0%
38
34
26
30
24
39
Anetured Models
(a) Finetuning LMs with Gemma2 data.
(b) Finetuning LMs with Gemini 1.5 data.
Figure 1: Summary the results. (a) We finetune Gemma-7B Gemma2-9R, and Gemma2-27B on the synthetic data collected from stronger but more expensive LM (Gemma2-27B) and weaker but cheaper LM (Gemma2-9B) a compute-matched setup for the MATH dataset. We find that training with Gemma2-9 9B data is more compute-optimal across diverse finetuning paradigms knowledge distillation, self-improvement, and weak-to-strong improvement (i.e. using a weaker model to improve stronger model). (b) We finetune Gemma models (7B/9B/27B) on synthetic data generated by Gemini-1.5-Pro and Gemini-1.5-Flaat a price-matched setup. We find that finetuning with Flash-generated data consistently outperforms Pro-generated data.
*I "almma experiments in this paper were conducted only parties outside of Google. Authors affi aifild with Google responsible for the Gemini and Gemma experiments.

---

Published as a conference paper at ICLR 2025
INTRODUCTION
Language models (LMs) have demonstrated impressive reasoning capabilities, but their success heavily reliess being trained on vast amounts of (problem, solution) pairs. Collecting this data from humans is costly and time-consuming. Recent studies have demonstrated the feasibility of synthetically generating this data using LMs themselves, offering a more scalable and efficient ap- proach to training data acquisition. One widely-adopted approach is to sample multiple candidate solutions for problem from an LM, filters them for final answer correctness, and finetune models on the correct solutions (Zelikman et al., 2022). Several works show that LMs trained with such synthetic solutions outperform those trained with human-written solutions (Yuan et al., 2023; Yu et al., 2023; Yue et al., 2023; Singh et al., 2023; Pang et al., 2024). Practitioners often sample solu- tions from strong LMs to ensure high quality (Teknium, 2023; Roziere et al., 2023; Mukherjee al., 2023; Xu et al., 2023). However, sampling from strong LMs is expensive and resource-intensive,, and limits the number of solutions that can be generated for practical sampling budgets.
In this paper, we explore an alternative sampling approach. Given a fixed compute budget, we in- vestigate sampling from weaker but cheaper (WC) model as opposed to the commonly-used approach of sampling from stronger but more expensive (SE) model. We start by comparing data from WC vs SE across three axes that play crucial roles in the utility of such synthetic data: 1- covenage, the number of unique problems that are solved, 2- diversity, the average number of unique solutions we obtain per problem, and false positive rate (FPR), the percentage of prob- lems that arrive at the correct final answer but with a wrong reasoning. We find that since we can generate more samples from the WC model compared to the SE model under fixed budget, the data from WC may exhibit higher coverage and diversity. However, due the lower quality of the WC model, it may also have a higher FPR. As a particular example for the Gemma2 family (Team et al., 2024a;b) on the MATH dataset (Hendrycks et al., 2021), Gemma2-9B achieves 11% higher coverage and 86% higher diversity, but also with 7% higher FPR compared to Gemma2-27B.
aamppin
LM Expensive
(Fze)
eemples
Wooker and Cheap LM (Pec params)
N Pee
Fincuundd LM FFel
Aoourasy Fie: >
Illustration of the approach. Given a fixed sampling budget, can either generate fewer samples from stronger but more expensive (SE) model or more samples from weaker but cheaper (WC) model. The latter may lead to solving wider range of problems and also more correct solutions per question. We compare the utility of these two syn- thetically generated datasets for training LM reason- ers in various supervised fineturing setups and show that training with the data from WC consistently out- performs training data from SE.
Figure 2:
We then fine-tune models on data from SE and WC (see Figure 2) across diverse setups correspond- ing to three paradigms: 1) knowledge distillation, where student LM learns from teacher LM (Hinton et al., 2015); 2) ielfooimpove where an LM learns from self- eeff-genrrated data (Huang et al., 2022); and 3) a new paradigm we introduce called Weak-to-Strong Improvement, where a strong student LM improves using synthetic data from a weaker teacher L.M. Using two (WC, SE) model pairs, one from the Gemma2 family and another from the Gemini 1.5 family (Reid et al., 2024), we show on multiple benchmarks that training WC-generated data consistently outper- forms training on SE-generated data under the three setups, with relative gains of up to 31.6% per- cent (see Figure for summary of the results). Our results indicate that it is more compute-optimal to sample from a WC model as opposed to the common-practice of sampling from SE model. With the performance gap between small and large LMs getting narrower over time (especially at larger scales - see Appendix K for details), our results establish a solid foundation for training the next generation of LM reasoners.
PRELIMINARIES
Let D = fgr, aikir be training dataset of size n with reasoning questions qi and final answers (aka labels) o, A successful approach to leverage such data to improve models for reasoning is as follows. We sample multiple solutions for each qi at non-zero temperature and create the synthetic data DG = (g:: [frr dzy }}] where k is the number of samples, f.z is the j-th reasoning chain (i.e. solution) generated by the model for qi, and āi is the model's final answer for qi in the j-th
2

---

Published as a conference paper at ICLR 2025
sample. Then, we filter the incorrect solutions by comparing dzy to a, and removing the solutions whose final answer do not match that of the gold answer'. Finally, supervise finetuned model on the remaining data De to maximize (() ( [log(po(r, al alg))], i.e. the probability of generating the reasoning and final answer given the question q. This approach was first proposed in (Zelikman et al., 2022) and was then extended in multiple works including (Zelikman et al., 2024; Singh et al., 2023).
For a dataset Do. we compute coverageak (aka passūk) (Chen et al., 2021) as EDa 1- (M-M)) where the number of solutions, out of M, with correct answers and ED denotes the expectation over the problems and solutions in the generated dataset. Conceptu- ally, coverage@k measures the fraction of unique questions that have at least one correct solution, assuming that we sample & solutions per question from the model. We also define diversity@k as the average number of unique correct solutions we obtain per question when we sample k solutions per question. Finally, we define false positive rate (FPR) as the percentage of solutions in Da where the reasoning incorrect, despite the final answer being correct.
Different choices of the LM to sample solutions from and the LM to finetune lead to different setups. Knowledge Distillation (Hinton al., 2015) corresponds training student LM on the synthetic data sampled from a stronger and larger LM. Self- Improvemnn (Huang et al., 2022) corresponds to training an LM on samples generated from itself.
3 COMPUTE-MATCHER SAMPLING AND TRAINING
To generate dataset DG with synthetic solutions from D, one can leverage different models for generating solutions. Specifically, at a fixed sampling budget (FLOPs), one can generate more samples from a weaker but cheaper (WC) model or fewer samples from a stronger but more ex- pensive (SE) model. Given a WC model with PwC parameters and SE with PSæ parameters, we compute the sampling ratio at fix budget for the two models, focusing on decoder-only trans- former models (Vaswani, 2017). Following (Kaplan et al., 2020), we note that the FLOPs per inference token is 2P, for model with P parameters. As result, the FLOPs for T inference tokens is 2PT. Further, we assume that generating each solution requires an average of W infer- ence tokens for both models². Let Swe and Syg represent the number of samples we generate per question for the two models. The total cost of generating samples for the dataset D will then be Costwe x Swe x W (2Pwc) and COstsE expensive models, respectively. At fixed sampling budget, we have:
SSE x W (2PSE) for the cheap and
Swc = PSE SSF Pwc
n x Swe X W x (2Pwc)= X SSF X W (2PSR)
(1)
Equation I indicates that fixed sampling budget, for each question we can generate PSE/ Pwo more samples from WC; the ratio scales linearly with the model parameters ratio". Sampling more solutions from WC may increase the likelihood of correctly solving larger subset of the problems (high coverage) and obtaining more correct solutions per question (high diversity).
Given fixed budget, a WC model, and then finetune models for a fixed number of steps on the data from each these models to measure and compare the utility of the data from each model. Specifically, we generate PSE/ Pwo more samples from the WC model compared to the SE model. We consider three fine- tuning setups that consists of diverse finetuning paradigms. The paradigms include the widely used knowledge distillation, the emerging framework of self-improvement, and a novel weak-to-strong improvement paradigm we introduce in this work. We define weak to-strong improvement (W2S-I) as enhancing the reasoning capabilities of a strong model using samples generated from weaker
either generate fewer samples from SE model more samples from
While it is possible to use other approaches for filtering (c.g., pracess-based or outcome-hased reward model (Uesato et al., 2022)), we mainly focus final answer correctness as it has shown quite strong. "hii is mostly reasonable as solutions are expected to he model-agnostic, but note that one model may solve a question using more optimal way compared to the other model thus producing smaller solution. Note that this may also depend the available hardware, which we ignore in this work.
3

---

Published as a conference paper at ICLR 2025
Data (L) Finctuning setup (-)
Student-LM
WC-LM
SE-LM
WC (Compute-matched) SE
Knowledge distillation
Self-improvement
Weak-to-strong improvement
Knowledge distillation
Knowledge distillation
Self-improvement
Table 1: Summary the supervised finetuning setups. We finetumed the language models under three
setups: (a) Student LM,, (b) Weak-Cheap (WC) LM, and (c) Strong-Expenniiee (SE) L.M. For each setup, we employed different finetuning paradigms based the source of the synthetic data. For example, training separate student LM with data from both WC and SE models falls under the knowledge distillation paradigm. In contrast, training a WC model with its own samples new paradigm, weak-to-strong improvement, where the samples from the WC model is used to improve the reasoning capabilities of the SE model at the fixed compute budget.
relf-improvement. Finally, we also introduce a
model. The three setups are as follows (a summary of the three setups and the finetuning paradigms that each case corresponds to can be found in Table 1).
Student-L.M finetuning: Conventionally, the supervised finetuning data for training student LM acquired from SE models to ensure high-quality (Teknium, 2023). However, we aim to understand whether WC models can replace SE models for distillation at the fixed sampling budget. To do so, finetune student LM separate from the WC and SE models the WC and SE data, which corresponds to distillation in both the cases.
WC-LM finetuning: Prior work (Singh et al., 2023) has shown that finetuning WC model through self- seff-generated data lags behind distillation from SE data. However, their setup spends a higher sampling budget collecting data from SE than WC. In this work, revisit this finetuning setup under the fixed sampling budget and finetune the WC model on the WC and SE data at a fixed budget for both. Note that training the WC model on its own data corresponds self-improvement whereas training WC on the data from SE corresponds to distillation. Hence, this setup compares self-improvement on WC data with distillation from SE data.
SE-LM finetuning: It is commonly believed that to improve a SE model, we either need synthetic data from the SE model itself or from an even stronger (and perhaps more expensive) model. Here, we test an alternative approach to understand whether the synthetic data from the WC model can improve the SE model. To this end, we finetune the SE model on the WC and SE data. Training SE on data from WC corresponds to W2S-1 and training SE on data from SE corresponds to self- improvement. Overall, this setup compares W2S-I by WC data with reff-improvement by SE data.
4 EXPERIMENTAL SETUP
Datasets:
We mainly experiment with MATH (Hendrycks et al., 2021) and GSM-8K (Cobbe et al., 2021) datasets, which are widely adopted in the literature. We generate the solutions for the prob- lems in the MATH using 4-shot prompt and for GSM-8K using an 8-shot prompt. We generated the candidate solutions in the synthetic dataset using TopK (K= 3) strategy with temperature 0.7.
Data Generation: We use Gemma2 models for synthetic data generation, with pretrained Gemma2- 9B and Gemma2-27B acting as the WC and SE models respectively. Since the 9B model roughly 3 times smaller than the 27B model, at a fixed sampling compute budget we can sample 3x more sample solutions per problem for Gemma2-9B. For our experiments, we consider two sampling budgets: a low budget, where we generate and 3 candidate solutions per problem from Gemma2- 27B and Gemma2-9B, respectively, and high budget, where we generate 10 and 30 candidate solutions per problem. Further, study the transfer of the reasoning capabilities for the models trained on MATH at the high sampling budget on the Functional MATH dataset.
Model Finetuning: We summarize the details for finetuning setups in the Table 1. In the Student-LM finetuning setup, we finetune the Gemma-7B model (Team et al., 2024a), for WC-LM finetume Gemma2-9B, and for SE-LM we finetune Gemma2-27B. Further, across different setups with the human- solutions as a ground- baseline. We finetuned the Gemma2-9B and Gemma2-27B models with batch size of 32 for 600 and 6000 steps under
train the LMs
4

---

Published as a conference paper at ICLR 2025
False Positive (MATH)
Coverage (MATH)
Diversity (MATH)
27
73
quesoon
12
278
278
25
99 (compute-macched)
(ompaale-malcced)
Icampptteematcheed
coveragni (86
6.5
23
57
8.21
correct solutions
49
19
41
15
13
Human
Gemini-1.5
Low
High
Low
High
Annotator
Sampling budget
Sampling budget
(a) Coverage MATH.
(b) Diversity on MATH.
False
Positive
Rate
on
MATH.
Figure 3: Synthetic data analysis for MATH dataset. The (a) coverage, (b) diversity, and (c) false
positive rates for Gemma2-278 and Gemma2-98 on the MATH dataset, at two sampling budgets.
the low and high sampling budget, respectively. During the fine-tuning process, we save 10 equally- spaced checkpoints and choose the one that yields the highest validation accuracy."
Synthetic Data Evaluation: To assess the quality of the synthetic data from the SE and WC models, we measure the coverage, diversity and for at a fixed cost. From Equation 1, we know that sampling one solution from SE takes the same FLOPs as sampling Psæ/ Pwe solutions from WC. Therefore, we compare coverage@k for SE to coverage Pak k) for WC to allow a similar budget to both models. Specifically, we compare coverage@k and coverage@3k for our SE and WC models. Similarly we compare diversity@kk and diversity(i3k for our SE and WC models. Since FPR cannot be computed automatically, subset of the data, where solutions from each model were selected randomly and rated for reasoning correctness by the authors, and 2- automatic evaluation where we sampled 500 solutions and prompted Gemini-B Priii-ro--..5 (Reid et al., 2024) to rate the correctness of the reasoning paths. To sample solutions, for the MATH dataset selected uniformly from each diversity level. In experiments, we find that the FPR estimates are close to each other for the human and automatic evaluation. We provide few qualitative examples for the false positive instances in Appendix F.
compute it using two proxies: 1- human evaluation on
We use pass@1 accuracy to evaluate the performance the fine- tuned LMs. Specifically, generate single solution for the problem (zero-shot) from the test split, using a sampling temperature of 0.0 (greedy decoding) for the fine-tuned LM and measure the percentage of problems that where the final answer matches the golden final answer. We also report maj@k @k (k- = 1,4.8 8, 16) for part of our experiments, where we generate & solutions per problem at a sampling temperature of 0.7 and select the final answer that appears most among the l samples.
Evaluating Finetuned Models:
5 EXPERIMENTS AND RESULTS
We compare data from WC and SE models along several axes. First, analyze the data along various quality metrics ($5.1). Subsequently, we present the supervised finetuning results for the different setups ($5.2). Finally, perform ablation studies to study the impact of dataset size, sampling strategy, and the role of quality dimensions the model performance ($F.
5.1 SYNTHETIC DATA ANALYSIS
We compare WC and SE data across three key quality metrics (coverage, diversity, and FPR) at fixed sampling budget. We present the results for MATH at the low and high sampling budgets in Figure and for GSM-8K in the Appendix Figure 20.
Coverage: We find that the data from Gemma2-9B (WC) outperforms Gemma2-278 2-2B (SE) by 11% and 6% (absolute) the low and high sampling budgets, respectively, for the MATH dataset, and
We provide details in Appendix
5

---

Published as a conference paper at ICLR 2025
Student-LM Finetuning
WC-LM
Finetuning
42
34
Ground truth
Ground truth
Ground trush
Pass@r Accuracy (96)
32
278
(oomgute-matched)
Pass@1 Accuracy 1%6
41
278
Icompute-matthed
278
(com pute
47
40
39
28
38
passe1
-3
26
37
24
Low
High
36
Low
High
39
Low
High
Sampling Budget
Sampling Budget
Sampling Budget
(a) Finetuming Gemma-7
(b) Finetuning Gemma2-99
Finetuning Gemma2-27B
Figure 4: guppeviieddfinetuniie results (MATH). The results for finetuning various LMs on the MATH synthetic data from the WC (Gemma2-9B) and SE (Gemma2-27B) models, fixed sam- pling budget. We observe that training with the samples from the WC model consistently outper- forms training with SE data.
8% and 1% (absolute) for GSM-8K. This highlights that the higher number of samples for the WC model aids in solving more unique problems for both the reasoning datasets. We provide the cov- erage trends for diverse sampling budgets in Appendix G. In addition, we observe that the coverage of the WC model increases across various difficulty levels the MATH dataset for the high sam- pling budget (see Appendix Figure 21). This highlights that synthetic data from the WC model can solve more unique questions at various difficulty levels compare to the SE model, at a fixed sampling budget (Tong al., 2024). Further, provide qualitative example that gets solved by repeated sampling from Gemma2-9B but remains unsolved by Gemma2-27B at the fixed high sampling budget (Table 6).
Diversity:
The diversity for the data from Gemma2-99 higher than Gemma2-27B by 86% and 125% (relative) the low and high sampling budgets for the MATH dataset, and 134% and 158% (relative) at for the GSM-8K dataset. This implies that many unique reasoning chains in the synthetic data from the WC model lead the correct solutions. We also observe that the absolute diversity scores are lower for MATH compared to GSM-8K at high sampling budget, indicating that models generate fewer correct solutions for the more challenging datasets when using repeated sampling.
Since we utilize the final answer correctness for filtering the synthetic data, it does not remove the solutions with incorrect intermediate reasoning steps. Our human evaluations suggest that the FPR for the WC-generated solutions is 7% and 2% (absolute) higher than SE-generated solutions on the MATH and GSM-8K, respectively. The trends from the automatic evaluation are similar to that of human evaluation. Due to the differences in the difficulty of the problems, absolute FPRs are much lower for GSM-SK compared to MATH. We also note that the development of high-quality verifiers will be essential to filter bad chain-of-thoughts from the synthetic data (Lightman et al., 2023).
FPR:
note that the
Given the mixed signals of high coverage and diversity coupled with high FPR, it remains unclear whether it is compute-on optimal to sample from the WC model or the SE model for training strong reasoners. We study this in the next section.
5.2 COMPUTE-OPTIMALLE RESULTS FOR TRAINING
We compare the utility of the synthetic data generated from the Gemma2-9B (WC) and Gemma2- 27B (SE) model for the MATH and GSM-SK dataset across the diverse finetuning paradigms in Figure 4 and Figure 5, respectively. In addition, present the results for training with human- written -hatu-hs- from the original training sets as a baseline.
Student-LM Finetuning. The Gemma-7B finetuned with the synthetic data from WC consistently outperforms the one finetuned on data from SC with relative gain of 6% and 5,8% at the low and high sampling budgets, respectively, for the MATH dataset and 4 2% and 1. 1.3% for GSM-8K.

---

Published as a conference paper at ICLR 2025
Rinetuning
WC-LM (GSM-8K)
SE-LM Rinetuning
Ground truth
Ground truth
Ground trush
Pass Accuracy 196)
78
278
Pass@1 Accuracy (%)
278
lcomputemmtthhed
88
278
(computee pute
Pass@ Accuracy
76
86
61
79
82
70
68
Low
High
77
Low
High
80
Low
High
Sampling Budget
Sampling Budget
Sampling Budget
Finetuming Gemma-
(b) Finetuning Gemma2-99
Finetuning Gemma2-21
Figure 5: pnpervised-finetunie results (GSM-8K). The results for finetuning various L.Ss on the GSM-8K synthetic data from the WC (Gemma2-96 and SE (Gemma2-27B) models, fixed sampling budget. We observe that training with samples from the WC model leads to stronger reasoners than training with SE data.
Student-LM Finstuning (Functional MATH)
WC-LM M Fineunning (Functional MATH!
SE-1M Rinetuning (Functional MATH)
272
4B jromp me-marhhed)
Issmpote
tearr puteeratceed
40
Majgk199
37
47
Makkk 063
34
31
28
16
(a) Gemma-7B evaluation.
(b) Gemma2-9B evaluation.
(c) Gemma2-278 evaluation.
Figure 6: Generalization Results (Functional MATH). The performance the models trained with the synthetic data from the MATH data high sampling budget the Functional MATH dataset. The results suggest that training with WC data enhances the generalization capabilities over the SE data, at fixed sampling budget.
Contrary to the common belief of stronger models being better for knowledge distillation, our results indicate that finetuning on data from WC is more compute-optimal than data from SE.
WC-LM Finetuning. We compare the performance of Gemma2-9B finetuned with the WC data (i.e. self-generated data) and SE data (i.e. data from Gemma2-27B). The results for MATH and GSM-8K reported in Figures 4b and 5b. We observe that the self-generated data (WC data) improves over knowledge distillation from a strong model (SE data), achieving relative gains of 3.8% and 2% at the low and high sampling budgets, respectively, for the MATH dataset, and 1.5% at the low sampling budget for the GSM-8K dataset. However, we find that the WC model finetuned with WC data matches the SE data for the GSM-8K dataset at high sampling budget. This is mainly due to the lower difficulty of the GSM-8k dataset, where it becomes saturated at higher sampling budgets (see Figure 20a). Interestingly, our empirical findings suggest that training WC model on synthetic data from its own is more compute- optimall than distillation from a stronger model.
SE-LM finetuning. We present the results for finetuning Gemma2-22 with the Gemma2-92 generated data and self-generate data. The results for MATH and GSM-8K are reported in Fig- ure 4c and 5c. Surprisingly, we observe that the model finetuned with the WC data outperforms the SE data, achieving relative gains of 5.8% and 4.3% at the low and high sampling budget, respec- tively, for the MATH dataset and 1.2% and 1.5% for the GSM-8K dataset. This result is even more surprising given that the Gemma2-27B data is expected to be more in-distribution than the Gemma2- 9B data. Contrary to the common belief self self-generated data or data from a stronger model being better, empirical findings show that training model in W2S-I setup from a WC data may be more compute-ope optimal than training it in self-improvemment setup on its own data. This result also establishes new paradigm for improving frontier models compute-efficient way, by generating

---

Published as a conference paper at ICLR 2025
FP81
solutions from finetuned models (MATH) FPR of solutions from finetuned (MATH)
Faise Positive Rate (%)
278
False Positive Rate (%)
Pro
(compute matched)
Icost: 1x Pro)
14
13
Student-LM
SE-LM
Gemma-78
Gemma-98 Gemma-278
Finetuning setups
Finetuning setups
Figure 7: False positive rates (FPR) of finetuned models. The FPR of finetuned models on MATH assessed by Gemini-1.5-Prro for models finetuned with (Left) Gemma2-27B and Gemma2-9B data (compute-matcher) and (right) Gemini-Pro and Gemini-Flas data pprice-matched..
synthetic data from much smaller models. We also perform the experiments on the Ilama models in Appendix D. In this case too, we observe that WC data outperforms the SE data across Student-LMM, WC-LM, and SE-LM finetuning, highlighting at the robustness of our conclusions.
FPR of Finetuned Models: We showed that models finetuned on WC data achieve higher final answer accuracy. However, since WC data had a higher FPR compared SE data, a question that may arise is whether the WC finetuned models mainly learn to arrive at the correct final answer but with wrong reasoning chains. To study this, similar to the experiment in Figure 3c, we use Cemini--15-Pae
judge to estimate the FPR of the finetuned models. To reduce noise, do this three times and average the results. We report the results for finetuned models with (Gemma-27B, Gemma-9B) and (Gemini-Pro, Gemini-Flash as the (SE, WC) data in Figure 7. Despite the larger FPR of the WC data, we observe that the FPR of the WC finetuned models is as good as the FPR of the SE finetuned models across different finetuning setups and choices of SE/WC data.
Generalization. Here, we aim to study the transfer capabilities of the models trained with the WC and SE data. Specifically, we evaluate the models finetuned with the synthetic solutions for the MATH datasets at the high sampling budget on the Functional MATH dataset. The results in Figure show that the Gemma-7B finetuned with the WC data consistently outperforms the SE data, where the relative gains range from 5.8% 6.5% at different values of k. In addition, we observe that the Gemma2-9B finetuned with the self- gelf-generated data outperforms knowledge distillation with the Gemma2-27B data achieving relative gains ranging from 2.5% 4.5% at different values of k. Moreover, finetuning Gemma2-27B with WC data matches closely with the SE data, except for k 8 where the gap is relative gain of 2%. Our results highlight that finetuning the LMs with the WC data enhances the generalization capabilities over the SE data at the fixed sampling budget.
Ablations studies: In Appendix E.I. show that results hold for train sets with smaller sizes and Appendix E.2 we show that the higher coverage and diversity both play positive roles in the superior performance of the WC data. While introduced the notion of compute-matchin sampling in this work, in the literature, comparisons between WC and SE data have been mostly done in nuther-match setup, where one generates equal number of samples from both models. In Appendix E.3, we show that SE data indeed outperforms WC data in this setup. We conjecture this be the main reason why SE data has been previously favored. In Appendix C, we extend our results to coding where we observe that the benefits from the WC can be context-dependent.
Overall, our findings challenge the conventional wisdom that advocates training on samples from the SE model, by showing that training samples from the WC model may be more compute-optimal across various tasks and setups.
Takeaway:
SCALING TO STATE-OF-THE. ART LANGUAGE MODELS
In the prior experiments, we focused on the synthetic data acquisition from open LMs. Here, we aim to show that data from the weaker SoTA LM can train better reasoners than stronger SoTA LM at fixed sampling budget. To this end, we scale our method to sampling data from Cemini--.5-PPe and Gemini-1 1.5--15shl As the model sizes not publicly available, utilize the ratio between their
8

---

Published as a conference paper at ICLR 2025
pricing per output token a proxy to perform compute matuche sampling. As of August 2024, note that the price per million output tokens is $10.5 and $0.3 for Gemini-1.5-PPe and Gemini- 1.5- Flash, respectively. Hence, sample 1 and 35 solutions per problem from 1.5-Pro and 1.5-Flash, respectively. We conduct our experiments on the MATH dataset.
We perform knowledge distillation Gemma-7B, Gemma2-9B, and Gemma2-27B LMs with the synthetic data from Pro (SE) and
the
Knowledge distillation with Gemini-1. 1.5 dala for MATH
Pro data
50
Flash
+5.7%
Flash (WC). We present the results in Figure 8. Interestingly, we find that finetuning with the
Flash data (cost: 1x of Pro)
46
Pass81 Accuracy
WC data outperforms the SE data, achieving relative gains of 31.6% 16,, 14.4%, and 10.9% for Gemma-7B, Gemma2-9B, and Gemma2-27B, respectively. This can be attributed the differ- ence in the coverage of the models at the fixed sampling budget, which 61.1% and 81% for 1.5-Pro and 1--Fla,h, respectively.
42
38
+31.0%
+29.1%
30
26
Gemmaa778
Gemmaz-98
GemmaZ-278
Finetuned Models
Reducing the cost of data sampling, Further, investigate training the LMs with the WC data that less expensive than collecting so- lution per problem from the SE model. Specif- ically, we create dataset by sampling 5 solu- tions per problem from the Flash (WC) model, which is 7x more economical than generating 1 solution from the Pro (SE) model, terms of the price ($). Upon training the LMs on the 0.15x cost data regime (Figure 8), we find that training on this data can also outperform training with SC data, achieving relative gains of 19.1%, 9.8%, and 5.7% for finetuning Gemma-7B. Gemma2-9B. and Gemma2-27B, respectively. This can be attributed to higher coverage of the weaker model (69%), even in the more economical scenario, in comparison to the stronger model (61.1%).
Figure 8: We finetune Gemma models (7B/9B/27B) on synthetic data generated by the state-of-the-art LMs Gemini-1.5-Pro and Gemini-
We find that finetuning with Flash geash-generate data consistently outperforms Pro-generated data not only the same sampling monetary cost as Gemini-1.5-Pro, but also at 0.15x of the cost.
Takeaway: We demonstrate that price-matched sampling from weaker SoTA LMs produces supe- rior reasoners compared to finetuning with data from stronger SoTA models.
EXTENDING RESULTS TO SCENARIOS LACKING GROUND-TRUTH LABELS
We have far assumed access to final gold answers which allows us to filter the synthetically generated solutions through final answer correctness. Here, we extend our approach to two scenarios where ground-truth labels unavailable: 1- the MATH dataset while assuming do not have the ground-truth labels (details in Appendix B. and 2- single-turn chat (instruction-folloiiing data which lacks the concept of ground-truth labels (details in Appendix B.2).
Performance on Reasoning, We study the impact of two settings on the performance of the fine- tuned models using SE and WC data at fixed sampling budget. In the first setting, we perform no verification of the candidate solutions; that is, we include all the synthetic solutions in the finetuning mix. In the second setting, we perform verification for the candidate solutions using model-based verifier. We present the results for finetuning LMs on the Gemma-9 9B (WC) and Gemma-27B (SE) data with no verification and LM as a judge Figure 11. Overall, the trends suggest that whether WC data is superior to SE data or not in the case of lacking ground truth data depends on the quality of the overall models and the finetuning setup.
Performance on Instruction-follower from WC and SE data at fixed sampling budget, for training Instruction-following LMs. We present the results in Appendix Figure 9. Interestingly, we observe that finetuned models with WC data significantly outperform the SE data across different model sizes. In particular, the instruction- level accuracy of Gemma-9B trained with Flash data outperforms Pro data by achieving relative gain of 12.8%. summary, our results highlight the usefulness of WC data over SE data for training capable dnstuction-folloviin models fixed sampling budget.
Task. Here, we study the usefulness of synthetic responses

---

Published as a conference paper at ICLR 2025
8 RELATED WORK
Gemini-1.5 (Performance on
Pro data
LMs for reasoning. The ability to solve reason- ing tasks has been a long standing goal of artifi- cial intelligence (Reid et al., 2024; Achiam et al., 2023; Dubey et al., 2024; Team, 2024; Anthropic, 2024: AI, 2024). In this regard, LMs trained on the intemet-scale data have achieved great success for math, code, and other reasoning tasks (Lewkowycz et al., 2022; Azerbayev et al., 2023; Kazemi et al., 2024). There have been several works that aim enhance the reasoning capabilities of the LMs ei- ther via prompting (Kojima al., 2022; Wang et al., 2022: Zheng et al., 2023a; Kazemi et al., 2022) or finetuning (Yue al., 2023; Yu et al., 2023). In this work, we focus on finetuning the LMs with task- specific datasets to build strong reasoners. Specif- ically, our method closely aligns with the widely adopted STaR (Zelikman et al., 2022) where the syn- thetic data from the LMs are used to elicit strong rea- soning capabilities.
Instructio--lvee Accuracy (%6
Flash data (cost: 1x of Pro)
73
69
65
57
Gemma-778
Gemmaa-998 Finetuned Models
Gemmaz-278
Figure Performance of finetuned mod- els on IFEval. The results present the instruction-level accuracy (%) on IFEval of the models finetuned with Gemini-Pro and Gemini-Flas (price-matched) data.
Finetuning LMs. Within the finetuning paradigm, there have been several works that improve reasoning with synthetic data. Broadly, these works focus on knowledge distillation from a strong but expensive LM (Wu et al., 2024; Yue al., 2023) seff-improvement (Gulcehre et al., 2023; Singh et al., 2023). While is common to filter the synthetic data for the final answer correctness (akin Zelikman et al. (2022)), there are several works that aim build task-specific verifiers to train strong reasoners (Lightman et al., 2023; Wu et al., 2024; Hosseini et al., 2024; Yuan et al., 2024). In this work, we explore the utility of the synthetic data from the weak but cheap LMs for training strong reasoners. We do not explore using model-based verifiers with the synthetic data for enhanced reasoning, and leave as future work. Our weak-to- tt-son improvement paradigm, where a strong model is trained with the generations from the weak model, is related to several prior work (Bowman et al., 2022; Burns et al., 2023; Yang et al., 2024b) which study the ability of a strong LM to Iearn from the data generated by weaker LM. However, the aim of these works is to recover the full capabilities of the strong model from weaker data, whereas we aim to enhance the strong model capabilities further. Our work also studies compute-optimal sampling from weak and strong models, which is absent in previous work.
Large and small LMs. While training large LMs has led to significant advancements across various tasks, there has recently been a growing interest in developing capable small LMs (HF, 2024b; Javaheripi et al., 2023). Specifically, capable small LM is faster to run, and easier serve to millions of users on the edge devices (Gunter et al., 2024). As a result, several recent works aim understand the utility of the weak but cheaper LMs in comparison to the strong but expensive LMs for reasoning. Specifically, Brown et al. (2024); Song et al. (2024); Snell et al. (2024) show that the solve rate of the small LMs can increase significantly with repeated sampling. In addition, Hassid et al. (2024) demonstrate that repeated generations from smaller LMs can outperform the data generated by larger LMs at a fixed sampling computational budget during inference for coding tasks. In this work, we go beyond these works and show the utility of the synthetic data from the small LMs for training strong reasoners across a diverse set of supervised finetuning setups.
9 CONCLUSION
In this work, we provide a framework for compute-optimal sampling from weak but cheap LM for reasoning tasks. Specifically, we show that at a fixed sampling compute budget, repeated sampling from smaller model can achieve higher coverage and diversity than from strong but more ex- pensive model. Furthermore, our empirical findings highlight that fine-tuning LMs with data from the small LM can consistently outperform data from the large LM under the same compute budget. Our results can serve as a foundation for training LM reasoners, especially as the performance gap between small and large LMS continues to narrow time (Appendix K).
10

---

Published as a conference paper at ICLR 2025
REPRODUCIBILITY STATEMENT
In this paper, we generated synthetic data either using open-weight language models (Gemma2 family and Llama), or models that are publicly available through API calls (Gemini 1.5 family). We also used publicly available datasets, MATH and GSM-8K. The data generation process is detailed SK. Additionally, we focus our finetuning experiments to open-weight Gemma models (7B, 9B, and 27B) only, with the finetuning details provided in Appendix J. Finally, the evaluation details are also covered $4.
REFERENCES
J. Achiam. S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt, S. Altman, Anadkat, al. Gpt-4 technical report. arXiv preprint 4rX110000000 2023.
M. AI. Au Large- mistral.ai
Anthropic. Claude 3.5 sonnet model card addendum. 2024. URL https://wwwwweeeeee anthropic. com/ 9 93a1 Caro_Claude, 3___aadd_____ddddmmmm pdf.
J. Austin, A. Odena, M. Nye, M. Bosma, H. Michalewski, D. Dohan, E. Jiang, C. Cai, M. Terry, Q. Le, et al. Program synthesis with large language models. arXiv preprint arXiv:2108.0772 2021.
Azerbayev, Schoelkopf, Paster, D. Santos, McAleer, A. Q. Jiang, Deng, S. Bi- derman, and S. Welleck. Llemma: An open language model for mathematics. arXiv preprint arXiv:2310. 10..613,,, 2023.
S. R. Bowman, J. Hyun, E. Perez, E. Chen, C. Pettit, S. Heiner, K. Lukošiūté, A. Askell, A. Jones, A. Chen, et al. Measuring progress on scalable oversight for large language models. arXiv preprint arXiv:2211.00
2022.
B. Brown, J. Juravsky, Ehrlich, R. Clark, Q. V. Le, C. Ré, and A. Mirhoseini. Large language monkeys: Scaling inference compute with repeated sampling, arXiv preprint arXiv:2407.21787. 2024.
C. Burns, P. Izmailov, J. H. Kirchner, B. Baker, L. Gao, L. Aschenbrenner, Y. Chen, A. Ecoffet, M. Joglekar, J. Leike, et al. Weak-to-strong generalization: Eliciting strong capabilities with weak supervision. arXiv preprint arXiv:2312.09390 2023.
M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. D. O. Pinto, J. Kaplan, H. Edwards, Y. Burda, N. Joseph, G. Brockman, et al. Evaluating large language models trained code. arXiv preprint arXiv:2107.03374. 2021.
K. Cobbe, V. Kosaraju, M. Bavarian, M. Chen, H. Jun, L. Kaiser, M. Plappert, J. Tworek, Hilton, R. Nakano, et al. Training verifiers to solve math word problems. arXiv preprint 1rXhv:2110.14166 2021.
A. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. Al-Dahle, A. Letman, A. Mathur, A. Schelten, A. Yang, A. Fan, et al. The llama 3 herd of models. arXiv preprint arXi::2407.21788, 2024.
C. Guluceeree T. Paine, S. Srinivasan, K. Konyushkova, L. Weerts, A. Sharma, A. Siddhant, A. Ah- em,, M. Wang, C. Gu, et al. Reinforced self- self-training (rest) for language modeling, arXiv preprint arXiv:2308.0890 2023.
T. Gunter, Z. Wang, C. Wang, R. Pang, A. Narayanan, A. Zhang, B. Zhang, C. Chen, C.-C. Chiu, D. Qiu, et al. Apple intelligence foundation language models. arXiv preprint 2407 2024.
M. Hassid, T. Remez, J. Gehring. R. Schwartz, and Y. Adi. The larger the better? improved Ilm code-generation via budget reallocation. arXiv preprint 40Xiv:2404.00725, 2024.
11

---

Published as a conference paper at ICLR 2025
D. Hendrycks, C. Burns, S. Kadavath, A. Arora, S. Basart, E. Tang, D. Song, and J. Steinhardt. Mea- suring mathematical problem solving with the math dataset. arXiv preprint arXi:22103.38874 2021.
HF. Open LLM Leaderboard 2 a Hugging Face Space by open-Ilm-leaderboard
huggingface.co. https: / /huggi ngwaaccae....o. acee cooccoaaaooooo /so/spaces// peaccee//pen-eeee open- 1m- rboard/ 2024a.
open_1 1 wellm_eaaeebbr
HE SmolLM blazingly fast and remarkably powerful huggingface.co. https:// huggingface co/blog/ hwlog/mmolnn....oooooooomoomoomm 2024b.
G. Hinton, O. Vinyals, and Dean. Distilling the knowledge in neural network. arXiv preprint arXiv:1503.02531, 2015.
A. Hosseini, X. Yuan, N. Malkin, Courville, A. Sordoni, and R. Agarwal. V-star: Training verifiers for self-taught reasoners. arXiv preprint arXiv:2402.06457, 2024.
Huang, Gu, L. Hou, Y. Wu, X. Wang. H. Yu, and J. Han. Large language models can self-improve. arXiv preprint arxiv:22 1661/ 2022.
M. Javaheripi, Bubeck, M. Abdin, J. Aneja, S. Bubeck, C. T. Mendes, Chen, A. Del Giorno, R. Eldan, S. Gopi, et al. Phi-2: The surprising power of small language models. Microsoft Research Blog, 2023.
A. Q. Jiang, A. Sablayrolles, A. Roux, A. Mensch, B. Savary, C. Bamford, D. S. Chaplot, D. d. I. Casas, E. Hanna, F. Bressand, et al. Mixtral of experts. arXiv preprint arXiv:2401.04088 2024.
Kaplan, S. McCandlish, Henighan, T. B. Brown, B. Chess, R. Child, S. Gray, A. Radford, J. Wu, and D. Amodei. Scaling laws for neural language models. arXiv preprint arXi::2001.08361 2020.
M. Kazemi, N. Kim, D. Bhatia, X. Xu, and D. Ramachandran. Lambada: Backward chaining for automated reasoning in natural language. arXiv preprint arXiv:2212.138994 2022.
M. Kazemi, N. Dikkala, A. Anand, P. Devic, I. Dasgupta, F. Liu, B. Fatemi, P. Awasthi, D. Guo, S. Gollapudi, et al. Remi: A dataset for reasoning with multiple images. arXiv preprint arXiv:2406.099 09155 2024.
T. Kojima, S. S. Gu, M. Reid, Y. Matsuo, and Y. Iwasawa. Large language models are zero-shot reasoners. Advances in neural information processing systems, 35:22199-22213, 2022.
A. Köpf, Y. Kilcher, D. von Rütte, S. Anagnostidis, Z. R. Tam, K. Stevens, A. Barhoum, D. Nguyen, O. Stanley, R. Nagyfi, al. Openassistant conversations-demee democratizing large language model alignment. Advances in Neural Information Processing Systems, 36, 2024.
A. Lewkowyez, A. Andreassen, D. Dohan, Dyer, Michalewski, Ramasesh, A. Slone, C. Anil, I. Schlag, T. Gutman-Solo, et al. Solving quantitative reasoning problems with language models. Advances in Neural Information Processing Systems, 35:3843-3857, 2022.
H. Lightman, V. Kosaraju, Y. Burda, H. Edwards, B. Baker, T. Lee, J. Leike, J. Schulman, Sutskever, and K. Cobbe. Let's verify step by step. arXiv preprint wrXiv:2305.20050. 2023.
S. Mukherjee, A. Mitra, G. Jawahar, S. Agarwal, H. Palangi, and A. Awadallah. Orca: Progressive learning from complex explanation traces of gpt-4. arXiv preprint arXiv:2306.022 2023.
S. Muralidharan, S. T. Sreenivas, R. Joshi, M. Chochowski, M. Patwary, M. Shoeybi, B. Catanzaro, Kautz, and P. Molchanov, Compact language models via pruning and knowledge distillation. arXiv preprint 240. 2024.
R. Y. Pang, W. Yuan. K. Cho, H. He, S. Sukhbaatar, and J. Weston. Iterative reasoning preference optimization. arXiv preprint 2444:3240.117333, 2024.
12

---

Published as a conference paper at ICLR 2025
M. Reid, N. Savinov, D. Teplyashin, D. Lepikhin, T. Lillierap, J.-b. Alayrac, R. Soricut, A. Lazari- dou, O. Firat, Schrittwieser, et al. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv preprint arXiv:2403.055300 2024.
B. Roziere, J. Gehring, F. Gloeckle, S. Sootla, I. Gat, X. E. Tan, Y. Adi, J. Liu, T. Remez, J. Rapin, et al. Code Ilama: Open foundation models for code. arXiv preprint arXiv:2308, Stti::2308..29900 12950, 2023.
Z. Shao, D. Dai, D. Guo, B. Liu, and Z. Wang. Deepseek-v2: A strong, economical, and efficient mixuure-of-experts language model. ArXiv, abs/2405.044 .44334, 2024. URL https: //api. www.eteesaanotteeeeeee
www.ooo///oooooo wow.oooppoo.DDD....... 26:96613826
A. Singh, J. D. Co-Reyes, R. Agarwal, A. Anand, P. Patil, P. J. Liu, J. Harrison, J. Lee, K. Xu,
A. Parisi, al. Beyond human data: Scaling self-training for problem-solving with language models. arXiv preprint arXiv:2312.06585 2023.
C. Cnell,, J. Lee, K. Xu, and A. Kumar. Scaling Ilm test-time compute optimally can be more effective than scaling model parameters. arXiv preprint arXiv:2408.03314 2024.
Y. Song, G. Wang, Li, and B. Y. Lin. The good, the bad, and the greedy: Evaluation of Ilms should not ignore non-determinism. arXiv preprint (tfi:24077.105577 2024.
R. Taori, I. Gulrajani, T. Zhang. Y. Dubois, X. Li, C. Guestrin, P. Liang, and T. B. Hashimoto. Stan- ford alpaca: An instruction-folloiing Ilama model. attpss:/ggttnei com/tatsu-lal stanford_alpane
2023.
G. Team, T. Mesnard, C. Hardin, R. Dadashi, S. Bhupatiraju, S. Pathak. L. Sifrre, M. Rivière, M. S. Kale, J. Love, et al. Gemma: Open models based on gemini research and technology. arXiv preprint ar 08X2 2024a.
G. Team, M. Riviere, S. Pathak, P. G. Sessa, C. Hardin, S. Bhupatiraju, L. Hussenot, T. Mesnard, B. Shahriari, A. Ramé, et al. Gemma 2: Improving open language models at practical size. arXiv preprint arXiv:2408.000 18, 2024b.
Q Team. Introducing Qwen1.5 -wenlm.giituubb......... https::/qwenlm: http://///eeen...ttt....o..o//// wtt.hu.....bbllg/// qwen1. 5/, 2024.
Teknium. Openhermes 2.5: An open dataset of synthetic data for generalist Ilm assistants, 2023. htto/daaaets/////oooo wtteeets///iiiuieeeeeeeeeeeittee http:/tteiiiiiigeeeeee htt///Opneeeeeeeee 2.5.
URL https::/huggiiggfeeeeeeeeee
Y. Tong, X. Zhang, R. Wang, Wu, and J. Hc. Dart-math: Difficulty-aware rejection tuning for mathematical problem-solving, arXiv preprint arXiv:2407. 13690, 2024.
J. Uesato, N. Kushman, R. Kumar, F. Song, N. Siegel, L. Wang, A. Creswell, G. Irving, and I. Hig- gins. Solving math word problems with process-and outcome-based feedback. arXiv preprint 1.Xiv:2211114275: 2022.
A. Vaswani. Attention is all you need. arXiv preprint
2017.
X. Wang, J. Wei, D. Schurrans, Q. Le, E. Chi, S. Narang, A. Chowdhery, and D. Zhou. Self-consistency improves chain of thought reasoning language models. arXiv preprint arXiv:2203.1117. 2022.
T. Wu, W. Yuan, O. Golovneva, J. Xu, Y. Tian, J. Jiao, J. Weston, and S. Sukhbaatar. Meta- rewarding language models: Self-improving alignment with llm-as-a metaa--e arXiv preprint 1rXiv:2407.19594 2024.
XAI. Grok-1 1 Model Card x.ai. https: attps://x.ate htt.ptttsss//kkaa////////o/to/tt
2024.
C. Xu, Q. Sun, K. Zheng, X. Geng, P. Zhao, J. Feng, C. Tao, and D. Jiang. WizardIm: Empowering large language models to follow complex instructions. arXiv preprint arXiv:2304.12244 2023.
A. Yang, B. Yang, B. Hui, B. Zheng, B. Yu, C. Zhou, C. Li, C. Li, D. Liu, F. Huang, et al. Qwen2 technical report. arXiv preprint arXiv:2407. 2024a.
Y. Yang, Y. Ma, and P. Liu. Weak-to-strong reasoning. arXiv preprint 24l.:2207.134477 2024b.
13

---

Published as a conference paper at ICLR 2025
A. Young, B. Chen, C. Li, C. Huang, G. Zhang, G. Zhang, H. Li, J. Zhu, J. Chen, J. Chang, et al. YE: Open foundation models by 01. ai. arXiv preprint arXiv:2403.04652 2024.
L. Yu, W. Jiang, H. Shi, J. Yu, Z. Liu, Y. Zhang, J. T. Kwok, Z. Li, A. Weller, and W. Liu. Meta- math: Bootstrap your own mathematical questions for large language models. arXiv preprint arXiv:2309. 12309.112880 12284, 2023.
W. Yuan, R. Y. Pang. K. Cho,, S. Sukhbaatar, J. Xu, and J. Weston. Self-rewarding language models. arXiv preprint arXiv:2401.10020, 2024.
Z. Yuan, H. Yuan, C. Li. G. Dong, K. Lu, C. Tan, C. Zhou, and Zhou. Scaling relationship on learning mathematical reasoning with large language models. arXiv preprint arXiv:2308.01822 2023.
X. Yue, X. Qu, G. Zhang, Y. Fu, W. Huang, H. Sun, Y. Su, and W. Chen. Mammoth: Building math generalist models through hybrid instruction tuning. arXiv preprint arXiv:2309.0566 2023.
E. Zelikman, Wu, J. Mu, and Goodman, Star: Bootstrapping reasoning with reasoning. Ad- vances in Neural Information Processing Systems, 35:15476-15488 2022.
E. Zelikman, G. Harik, Y. Shao, V. Jayasiri, N. Haber, and N. D. Goodman. Quiet-star: Language models can teach themselves to think before speaking. arXiv preprint arXiv:2403.0962) 2024.
H. S. Zheng, S. Mishra, X. Chen, H.-T. Cheng, E. H. Chi, Q. V. Le, and D. Zhou. Take a step back: Evoking reasoning via abstraction in large language models. arXiv preprint arXiv:2310.06117, 2023a.
L. Zheng, W.-L. Chiang, Y. Sheng. S. Zhuang, Z. Wu, Y. Zhuang, Z. Lin, Z. Li, D. Li, E. Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 36:46595-46623, 2023b.
J. Zhou, T. Lu, S. Mishra, S. Brahma, S. Basu, Y. Luan, D. Zhou, and L. Hou. Instruction-follooiing evaluation for large language models. arXiv preprint 1.7991331..07911, 2023.
14

---

Published as a conference paper at ICLR 2025
pata pollution with no ground-truth labels
Data pollution with no ground-truth labels
92
venfication
48
verification
*e of incorrect solutions
84
LM as judge verifier
of incorrect solutions
as judge verifier
76
44
68
40
60
36
52
32
44
28
36
28
g 24
20
27B
9B (matched)
20
Pro
Flash(matched)
(a) Analyzing Gemma2-99 and 27B data.
(b) Analyzing Gemini- and Flash data.
Figure 10: Analyzing the percentage of bad solutions in the synthetic data. The results present the amount of bad solutions, that lead to incorrect final answer, if we do not have access to oracle verifier (final answer correctness) for MATH dataset. Specifically, we consider two strategies: no filtering and using language model as a judge. Gemma-27B and Gemma-9B compute-matchedd (b) We analyze the amount of data pollution in Gemini-Pro and Gemini-Flash (price-matched).
(a) We analyze the amount of data pollution in
A DISCUSSION
In this work, introduce compute-matches sampling in the context of data generation from weak and cheap (WC) model and a strong and expensive (SE) model. We demonstrate that WC data can train stronger language models (LM) for reasoning tasks than SE data when constrained by fixed compute budget. A relevant area for future work, and current limitation of this study, is to explore the conditions under which WC data consistently outperforms SE data in model finetuning (c.g., based on relative gains/losses in terms of coverage, diversity, and false positive rate). Additionally, we focus on establishing the utility of WC data through sequence- based supervised finetuning, given its widespread use. However, it would also be valuable to examine the behaviors of WC and SE data iterative finetuning (Singh et al., 2023), as well as supervised finetuning through logit matching. In addition, it will be interesting study the implications of findings for pretraining where the experimental designs are non-trivial. In particular, pretraining of language models requires a more complicated infrastructure due the scale of tokens (trillions) and diversity of data domains (natural language, math, coding, multilingual data) involved in it. Finally, an essential aspect of training reasoning models involves verification (Cobbe al., 2021), and it would be appropriate to investigate the impact of WC and SE data on training LM verifiers for reasoning tasks.
B ADDITIONAL DETAILS: SCENARIOS LACKING GROUND-TRUTH LABELS
In the prior experiments, we assumed having access to final gold answers which allowed us to filter the synthetically generated solutions through final answer correctness, following the STaR framework. Here, we extend our approach to scenarios where ground-truth labels are unavailable. In particular, we consider two scenarios: 1- the MATH dataset while assuming we do not have the ground-trui truth labels ()B.1), and 2- single-turn chat (instruction-folllownng data which lacks the concept of ground-trui truth labels ($B.2).
B.1 PERFORMANCE ON REASONING
We study the impact of two settings on the performance of the finetuned models using SE and WC data at a fixed sampling budget. In the first setting, we perform verification of the candidate solutions; that is, we include all the synthetic solutions in the finetuning mix. In the second setting, perform verification for the candidate solutions using model-based verifier. Specifically, use an language model (LM) as a judge (Zheng et al., 2023b) setting for verification where, akin to prior work (Yuan et al., 2024), an LM is prompted verify if solution is correct not. Note, however, that in practice one can use any other type of verifier, including a verifier that has been previously trained to judge the quality of the solutions. Due the lack of ground-truth data, LM as
15

---

Published as a conference paper at ICLR 2025
(compute matchedi
(compute masched)
mencermance w/ ground-truth (No verilication)
Performance no ground-truth (LM Judge Verifier)
Studeen-LM Revetuning
WC:UM Resturing
SE-UM Finetuning
Stabent-LM Reaturing
WC-UM Resturing
SE-LM Finetuning
Acourcy
26
34
39
12
35
16
26
18.
31
(a) Finetuning w/ Gemma data without filtering.
(b) Finetuning w/ Gemma data using as a judge.
Finetuning with Gemma data without access to ground-truth labels. The results present the accuracy of the finetuned models with Gemma-27B and Gemma-9B ccompute-matchien data without access to the ground-truth labels. (a) We do not perform any filtering the synthetic data. (b) We perform filtering using language model as judge.
Figure 11:
KD with Gemini-1.5 for MATH (No Verification)
KD with Gemini-1.5 for MATH (LM as Judge Verifier)
52
50
Pro data
Pro data
48
Flash data (cost: 0.15x of Pro)
Flash data (cost: 0.15x of Pro)
Flash data (cost: 1x Pro]
46
Flash data (cost: 1x Pro]
Pass@1 Accuracy (%)
Pass@1 Accuracy (%)
44
40
38
36
34
32
28
30
24
26
Gemma2-277
Gemma2-278
Finetuned Models
Finetuned Models
(a) Finetuning w/ Gemini data without filtering.
(b) Finetuning w/ Gemini data with L.M as judge.
Figure 12: Finetuning with Gemini data without access to ground-truth labels. The results present the accuracy of the finetuned models with Gemini-Pro and Gemini-Flash (price-matched) data without access to the ground-truth labels. (a) We do not perform any filtering on the synthetic data. (b) We perform filtering using language model judge.
judge is expected to be better than no verification but worse than oracle verifier in filtering incorrect solutions from the data.
Setup We experiment with the same (WC, SE) model pairs as in the previous experiments, ie. (Gemma-9B, Gemma-27B) and (Gemini- 1.n.i-1.5-FFP Flash,,hh Gemini-1.5-Par Following the compute- matched setup, we generate and solutions per problem from Gemma-27B and Gemma-9B. following the price-matched setup, we generate 1 and 35 solutions problem from Pro and Flash. We also consider a cheaper version where we collect solutions per problem from Flash, as done in the previous experiments. Post-generation. Gemma-9p 9B and Flash data, and the Pro model to verify the final answers for Gemma-27B and Pro data. This is ensure that do not spend more compute (or cost) for the WC setup. Subsequently. we perform supervised finetuning of Gemma-7B/9BB277 with the (fnteriltered synthetic data.
the Flash model to verify the final answers for the
Data Analysis We start by analyzing the data in the no-verification and LM judge setups and present the percentage synthetic data that leads to incorrect final answer for the two strategies in Figure 10. We find that the majority of the synthetic solutions from Gemma-9B and Gemma-27B, 65%+.. lead to incorrect final answer without any verification. However, we observe that LM as a judge verification significantly reduces the amount of bad solutions from Gemma-99 and Gemma- 27B (down to 25%). On the other hand, we observe that the percentage of bad solutions is between 40% 48% for Gemini-Pon and Gemini-Flash without any verification. Similar Gemma
16

---

Published as a conference paper at ICLR 2025
models, the amount of bad data reduces to 23% after LM judge verification. Now, we will study the impact of finetuning LMs on this data.
Results We present the results for finetuning LMs on the Ge (WC) and Gemma-27B (SE) data with no verification and LM as a judge in Figure 11. We observe that finetuning models with the SE data slightly outperforms WC data across the two strategies (Figure 1 la and 11b). This indicates that the finetuned models are more sensitive to the incorrect solutions from Gemma-9B data in comparison to the Gemma-27B data at the fixed sampling budget. Further, we present the results for finetuning LMs on the Gemini-Flashe (WC) and Gemini-Pro (SE) data in Figure 12, indicating that the finetuned models with the WC data consistently outperform the SE data across the two strategies (Figure 12a and 12b). Interestingly, we observe that cheaper Flash data (e.g., 5 solutions per problem) outperforms price-matched version of Flash data (e.g., 35 solutions per problem) for training Gemma-7B and Gemma-9B without any verification (Figure 12a). This can be attributed to the presence a larger number of bad solutions among 35 solutions in comparison to 5 solutions in the finetuning mix. Overall, the trends suggest that whether WC data is superior to SE data or not in the case of lacking ground truth data depends on the quality of the overall models and the finetuning setup.
B.2 PERFORMANCE ON CRORUCTIIO-FOOLOOWIII TASK
Apart from the reasoning tasks, the synthetic data from the SE models is also used for instilling corructionfoolowinng (chat) capabilities (Taori et al., 2023; Teknium, 2023). Due to the subjectivity of the chat data, the notion of final answer correctness may be ill-defined. For instance, there is no ground-truth for the instruction 'poem on strawberries and beaches'. Here, we study the usefulness of synthetic responses from WC and SE data at a fixed sampling budget, for training instruction- following LMs.
Setup: We use Gemini-1.5-Pre and 1.5-Fnn- the SE and WC models, respectively, as they have the capability to follow user instructions. In particular, we prompt the generators with 5000 random instructions from the OpenAssistant] dataset (Köpf et al., 2024). We generate and 35 responses per instruction for Pro and Flash respectively, following a price-matched setup. Subsequently, we perform supervised finetuning of for Gemma-7B, 9B and 27B with the synthetic instruction-followee data. Finally, we evaluate the finetuned models on the IFEval data (Zhou et al., 2023) and report the instruction-level accuracy.
Results: We present the results in Figure 9. Interestingly, we observe that finetuned models with WC data significantly outperform the SE data across different model sizes. In particular, the instruction- level accuracy of Gemma-9B trained with Flash data outperforms Pro data by achieving a relative gain of 12. 2%. In summary, our results highlight the usefulness of WC data over SE data for training capable cnstruction-follovii models at a fixed sampling budget.
C EXTENDING OUR RESULTS TO CODING TASKS
Here, we aim to understand the utility of the synthetic data from the Gemma2-9B (WC) and Gemma2-27B (SE) model coding tasks. To this end, generate candidate solutions for the MBPP (Austin et al., 2021) dataset from WC and SE models at the low and high sampling budgets and finetune models in three setups on these data. We use the santizied version of MBPP contain- ing 427 problems overall; we used 3 problems for fewshot prompting (used for sampling from the models), 324 problems for synthetic training data generation, and 100 problems for validation. The candidate solutions are filtered by the unit tests that accompany each instance of the dataset. After finetuning. we evaluate the LMs on 164 problems from the HumanEval dataset (Chen al., 2021).
We compare the coverage and diversity of the synthetic datasets in Figure 13 and observe that the coverage of the WC model is higher than SE at low data regime while it is similar SE in the high sampling budget regime. In addition, we find that the diversity of the WC model is more than that of the SE model for the low and high sampling budgets. Subsequently, we finetune Gemma-7B, Gemma2-9B, and Gemma2-27B models with the ground-truth and synthetic datasets and evaluate on
Shtpps /ttp://hhgggggoe wggagenngaace..oooooooooo co/dataseeee googl e research- wwwerrrchdattstttteee /ttpttattteettapppppppppt viewer sanitized
17

---

Published as a conference paper at ICLR 2025
Coverage (MBPP)
Diversity (MBPP)
correct salutions per question
18
91
27B
27B
87
9B (compute-matched)
15
9B (compute-matched]
83
12
75
9
67
59
Low
High
Low
High
Sampling budget
Sampling budget
(a) Coverage on MBPP.
(b) Diversity on MBPP.
Figure 13: Synthetic data analysis for MBPP dataset. We present the (a) coverage. and (b) di- versity for subset of the santized MBPP dataset for Gemma2-278 and Gemma2-9B at two fixed sampling budgets.
Student-LM LL Finetuning
WC Finetuning (MBPP==HumanEval)
[MBBP->HumanEva]
SE-LM Finetuning MMBPP->HumaEEWw
Pass@11 Accuracy
61
278
Ground-truth
matched)
Pass@1 Accuracy (%)
62
64
2
Crouns-bruth
matchedi
Accura
71
(computeemeithe))
60
52
58
65
49
62
56
43
54
59
40
LoW
Sampling Budget
High
52
Low
Sampling Budget
High
56
Low
Sampling Budget
High
(c) Finetuning Gemma2-27B
(a) Finetuning Gemma-7B.
(b) Finetuning Gemma2-9B.
Figure 14: supervised-linetuning with MBPP and evaluation on HumanEval. We report the results for finetuning diverse language models on the MBPP synthetic data from the SE model (Gemma2-99) and WC model (Gemma2-27B) at the fixed sampling budgets.
HumanEyal (Figure 14). Our empirical findings indicate that finetuning with WC data outperforms SE data for the student-L.M and WC-LM finetuning setups, while the performances similar for SE-LM finetuning setup at the low sampling budget. At the high sampling budget, where the models have similar coverage, we find that training with the SE data better for student-LM finetuning while WC-data is better for WC-LM finetuning. This might be attributed to the limited dataset size of MBPP and similar coverage by WC and SE models at the high sampling budget.
D EXPERIMENTS ON LLAMA MODELS
Here, extend our results on another set of open language models from the Llama series Dubey et al. (2024). Specifically, we consider Llama-3.22 3.2-BB...B-BBBB-- In-trust and Llama-3. 3. 1-8B-instruct as the pair of WC and SE models, respectively. Subsequently, we sample solution per problem and solutions problem from the WC and SE model, in accordance with the matchee sampling ratio for the problems in the MATH train dataset. addition, we filter the solutions that lead to the incorrect final answer. We finetune Llama-3. 2- In-Irrcttt (student-LMD), Llama-3.2-3 3.2-3B 2-3B-Instrruct (WC-LM), and Llama-3, --8--tssruttt (SE-LM) on the WC and SE data. Finally, these models are evaluated on the problems from the MATH500 test set. We present the results in Table 2.
Consistent with our results on Gemma models, we find that training with the WC data is more compute-optimal than SE data across diverse finetuning setups including knowledge distillation, rell-improveemett and weak-to-stron. improvement. These benefits can explained by the high coverage and diversity of WC data in comparison SE data. Specifically, observe that the WC
18

---

Published as a conference paper at ICLR 2025
Data
Student-LMM F.T. WC-LMET. SE-LMET.
Llama-88
5.6
31.6
36.4
Llama-3B (compute-matched)
7.2
33.2
38.2
Table 2: Results on LJama models. We find that WC data more compute-optimal than SE data
across diverse finetuning setups for the Llama models well. We abbreviate finetuning F.T.
Student-LM Finetuning
WC-LM Finetuning
SE-LM Finetuning
49
98 (compute matched)
41
Icompute-maaie maathttttttccee
98 ccompute-matcirt
Pass$1 Accuracy (%)
Accuracy
47
39
45
28
43
26
37
24
41
22
500
7500
35
7500 problems in the dataset
500
39
500
7500
problems in the dataset
problems in the dataset
(a) Finetuning Gemma-7B.
(b) Finetuning Gemma2-99
(c) Finetuning Gemma2-278
Figure 15: Impact of the dataset size. The performance of finetuned LMs on the synthetic data from WC and SE models, different sizes of the training set. Training with the WC data leads to better models than training with the SE data at both dataset sizes.
model has coverage of 67% and a diversity of 2.2, whereas the SE model has coverage of 49% and diversity 1.
E ABLATION STUDIES
We perform several ablation studies better understand the merit WC data.
E. IMPACT OF DATASET SIZE
We study whether the benefits of the synthetic data from the WC model hold at different dataset sizes. We repeat our experiments for the MATH dataset at the high budget, but when only having access to 500 training data (selected randomly from the training set). We present the results for the finetuned models in Figure 15. We observe that models trained with the WC data outperform those trained with the SE data, achieving relative gains of 12.93%, 11.4%, and 5.1% for the three paradigms, respectively. This highlights the utility of generating more data from the WC model instead of the SE model in the low-problem regimes the fixed sampling budget.
E.2 COVERAGE AND DIVERSITY
We aim to understand the role of coverage and diversity in enhancing the performance of models trained with WC-generated synthetic data. To this end, for the MATH dataset, consider the original high-sampling (30 solutions per problem) WC dataset as a (high coverage, high diversity) dataset. We then construct (high coverage, low diversity) version by only selecting one correct solution per question from our samples. This reduces the diversity of the original WC dataset from 11 to 1, while maintaining the coverage. We also create a (low coverage, low diversity) dataset where we generate just one solution per problem from the WC model and filter it for the correctness of the final answer. The coverage this dataset (27%) is lower than that of the WC dataset with 30 solutions per problem (43%). We train models across the three finetuning setups on these sets and present the results in Figure 16. Our results indicate that across all setups, the high coverage and high diversity data is better than high coverage and low diversity, and high coverage and low diversity is better than low coverage and low diversity. This reveals that both the coverage and diversity play a critical role in training strong reasoners from the smaller LMs.
19

---

Published as a conference paper at ICLR 2025
Ablation: Role coverage and diversity
49
coverage, diversity
45
coverage, diversity
Pass.1 Accuras CX4 41
high coverage, high diversity
37
33
29
21
Student-LM
WC-LM
SE-LM
Finetuning setups
Figure 16: Understanding role coverage and diversity for training strong reasoners WC model. We compare the performance of training the LMs with synthetic data acquired by collecting (a) solution per problem (low diversity, low coverage), (b) 30 solutions per problem (high diversity, high coverage), and (c) 30 solutions per problem but keeping just correct solution (high coverage, low diversity). We find that both high diversity and coverage are helpful for training strong reasoners.
Ablation: number vs campute-matched (MATH)
Ablation: number compute-matched (GSM-8K)
98 (numberematched)
278
Passia1 Accuracy 189
98 (compute matched)
3B (compute matched)
Pass@g Accuracy
36
76
32
73
70
20
Student-LM
WC-LM
SE-LM
Student-UM
WC-LM
SE-LM
Finetuning setups
Finetuning Setups
(a) Finetuning LMs on MATH data.
(b) Finetuning LMs on GSM-8K data.
Figure 17: Comparison between number-matchen sampling and compute-matched sampling from the WC model. We report the results for finetuning diverse LMs with synthetic data from WC and SE model the low sampling budget. Conventionally, practitioners would compare performance of the models trained with WC data and SE data at the fixed number of samples from both models. However, we observe larger gains using the samples from WC model that acquired at the fixed sampling budget as that of SE model.
E.3 DEFAULT VS COMPUTE-OPTIMAL SAMPLING FROM CHEAP LMS
We anticipate that the reason why data from SE models has been previously preferred over data from WC is because they have been tested in a setup where an equal number of samples have been generated from the two models (e.g., see (Singh et al., 2023)), as opposed setup. To verify this, we generated solution per problem number-matched) from the WC model for the MATH and GSM-8K datasets and trained the models under the three fine-tuning setups on this generated data, after filtering for final answer correctness. We then compare the performance of the models trained with synthetic data, where we generate solutions per problem from the WC model, matched in sampling compute the SE model. We present the results Figure 17. We see that the models trained with the number-matcher WC data are sub-optimal in comparison to the models trained with the compute- compute-matched WC data, and lead to worse models compared to training with the SE data. This highlights that the future comparisons between synthetic data from weak and strong models should be made in the sampling compute-matcher regime.
compute-mate matched
E.4 MIXING STRONG AND WEAK- MATCHED DATA
Here, aim study the impact of distributing from both the SE and WC models. To do so, we sample solutions per problem from the Gemma- 27B (SE) and 15 solutions per problem from the Gemma-9B (WC) data. We compare this data with
fixed budget sampling candidate solutions
20

---

Published as a conference paper at ICLR 2025
Mixing SE WC data (MATH)
48
(105)
9B (305)
Pass@1 Accuracy (%)
44
Mixed (278:55 and 98:15S)
40
36
32
24
20
Student-LM
WC-LM
SE-LM
Finetuning setups
Finetuning models with mixing strong and weak (compute-matched) data. The results present the performance of the models finetuned with mixing the data from Gemma2-27B (SF) with the data from Gemma2-9B (WC) for the fixed sampling budget. Specifically, we mix solutions (5S) per problem from SE model with the 15 solutions (15S) problem from WC model.
Figure 18:
two non-mixture settings: 1- 10 solutions per problem from SE model and no solutions from the WC model, and 2- 30 solutions per problem from WC model and no solutions from the SE model. We observe the mixed data has coverage of 68.8% in comparison to the 70.7% from WC data. This indicates that the compute-matched sampling from WC model solves more unique problems than mixing SE and WC data at the same sampling budget. We then finetune models on the mixed data and present the results for Student-LMM, WC-LM, and SE-LM finetuning in Figure 18. We observe that in the student-LM and SE-LM setups, mixed data underperforms whereas in the WC- LM setup it slightly outperforms the non-mixed setups. This could be due to the fact that mixing two datasets results in two data distributions that might be harder for models to learn Overall, our results highlight that the usefulness of data mixing might be context-dependent. We leave a rigorous study of SE and WC data mixing for optimal performance future work.
F QUALITATIVE EXAMPLES
We present few qualitative examples for model-generater solutions that lead the correct final answer with incorrect (or correct) reasoning for the MATH dataset. We provide two bad reasoning examples in Table 3 and Table The existence of such examples contributes to the false positive rates the synthetic dataset. In addition, we provide a good reasoning example in Table 5.
Problem:
Wher three standard diice arw tossed, the numbers a, by are obtained. Pind the probabil lity that ubc.
180.
Ground-truth:
He first Factox 280 into these posi.iow integera from Lhe sut. (1,2,3,4,5,6). Since 180 5 125, at Teast one of integers most 6. Since 180 >
150, least two integers must equal Indead, 160 = 5-6-6 is the only such way to factor 180. Theretoze, (a,b.c) = (5,6, (6, 5,6), (6, are probability (3) zia probabiliey
only possibilities a, Each
abc 180
Gonerated solution (Incorvect reasoning):
There are
216 possible out cones, The numbere 1,3, are the only ppesible values for b,e. = 1, thea = and 6, za the probability of this outcane is ZIG if = 3, then b =
and e = 6, sa the probabi of this ourcone is also te 1f a 6, then 3 and 1, so the
probabi lity of this outcone is slso A Therefore, the probabi 1 ity that abe 180 14141
210
Pinal Answer: The final answer is +. hoge it cocrect.
Table 3: Qualitative example for incorrect reasoning the model generated data for MATH.
21

---

Published as a conference paper at ICLR 2025
Problem:
prossbility that point cleser to thar 07 Express your answer as docinal the nearest zench. sayylanncseeaa0.2. 21nch)) drav(10, (5, draw -0.))-000,0-----0--000000 dCaN( (2,-0_1))= (2,111))22)))2) 0.1117 (1, 10-000011----------0-- (3,0.11)7 (6, (1,-1)1---0--.00111111100 0.11)7 draxi -5,111))-- (1,0011 label ("rggs, 00,-0,11, S1; poolii"1"" 20111 label "oel:""g"", (2,-0.2) 1sbel ("1gg, (8,-11,,, label
point solected zandon
portion
sumber line
deaw (11, (1,1--10,001111
2111), label (nren, (3,-0.21,9) [/aav].
Gruud-truth:
Me ting the
which 15 equidietant is closer Lo Uhan D. So, Uhe probability is equal the Tength of this zegion
and 4, Clearly, this
all 2,
-G
Gevrrttd solution (Incorrcct ressoning::
total length From to 4 is Final answer is U.S. hopa it is correct.
number line is
The lengeh the portion number
3. Therefore, the probability la 0.6 0.80
0.6
Final Answers The
Table 4: Qualitative example for incorrect reasoning the model generated data for MATH.
Problem:
cistingutchanie balls into 2 indistinguishabie boxes?
Ground-truth:
In this problen ve don't t pare which box is which, we only care which be is are toget her and which aswnf b. For wach ball, there are chalows of which bax to place it in. Since this chal number
balls, multiply independent waye to place dist i balls into 2 dixt ingutishal ishable baxes.
tegether.
there
2 runber of ways Lu actange the boxes. There ase
then divide by the
2 ways to acrange the 2 boxwa, au theze are
8 ways to arrange dist ingul shable belly inco 2. ind ltt indiatingg ishable boxes. Note: This nethed does not quneralize there more than boxas. Generated solution (Correct reasoning):
It
there 2 ways pat balls
boxes.
However,
since
indistingutehabb.c xe divide by 21, ansves The final ansver is 8. hope it correct.
Anewer:
Table 5: Qualitative example for correct reasoning the model generated data for MATH.
G COVERAGE TRENDS
We present the coverage trends for diverse sampling budgets on the MATH and GSM-8K dataset in Figure 19a and Figure 19b, respectively.
Covorage
sampling budgets
Coverage GSM-8K different sampling budgets
0.7
0.95
Pass (95
Pess value 1%))
0.85
0.80
0.4
0.75
Pass@K Gemmaa-270 SBK Gemma2-98
Pass@K Germia
2-98
0.3
Pas Gemma2-98
Pass@3KK Gemma2-96
0.65
9 10
(a) MATH dataset.
(b) GSM-SK dataset.
Figure 19: Coverage (Pass@K) trends for synthetic data acquisition from Gemma2-9B and Gemma2-27B on the (a) MATH and (b) GSM-8K datasets. For a compute-matched comparison, Pass 3K for Gemma2-9B should be compared against Pass@K K for Gemma2-27B.
H DATA ANALYSIS: GSM-8K
We presented the coverage, diversity, and false positive rate of the synthetic data from Gemma2-27B and Gemma2-98 on the MATH dataset in the main text. In Figure 20, we present these metrics for the GSM-8K dataset.
22

---

Published as a conference paper at ICLR 2025
Rate
Coverage (GSM-8K)
Diversity (GSM-8K)
24
278
98
21
278
(comp.deemattchs))
94
18
solutions
15
Percenta
82
78
6
Human
Gemini-1.5 1.5
Low
High
Low
High
Annotator
Sampling budget
Sampling budget
False Positive Rate on GSM- 8K.
(a) Coverage GSM-8K.
(b) Diversity GSM-8K.
Figure 20: Synthetic data analysis for GSM-8K. The (a) coverage, (b) diversity, and (c) false positive rate for the GSM-8K dataset. The results are provided for synthetic data generation from Gemma2-27B and Gemma2-9B two sampling budgets.
Percent of Solved Problems (%)
Solving levels from MATH-Train dataset
100
278
90
98 (compute-matched)
80
70
60
50
40
30
20
2
Level
Figure 21:
Coverage across different levels of the MATH dataset. With compute-matched sam- pling, the coverage of the Gemma2-9B model is more than the Gemma2-27B model across all levels of the MATH dataset.
I SOLVING PROBLEMS ACROSS LEVELS FOR MATH
We present the effect of repeated sampling from the weak but cheaper LM and stronger but expensive LM on solving the problems across different levels for the MATH dataset in Figure 21.
So for
UB Ue-generated solutio::
Town
have
2(n 2)
2
2((2 m))
2 2
Thorefore,
3-m WFF
3, so 2
3n and 4n -1 and
Firal Ansvezs The final answez 1s
hoge st 1e cocrect.
Table 6: Qualitative example from Level 5 of MATH dataset that gets solved by repeated sam- pling from Gemma2-9B (30 solutions) but remains unsolved by Gemma2-27B (10 solutions) at fixed sampling budget.
23

---

Published as a conference paper at ICLR 2025
J EXPERIMENTAL SETUP DETAILS
As mentioned in the main text, we mainly experimented with MATH (Hendrycks et al., 2021) and GSM-8K (Cobbe et al., 2021) datasets, which are widely adopted for evaluating reasoning and mathematical problem solving. MATH consists of competition level problems with various levels of difficulty (Level 1-5) and GSM-8K comprises of grade school level math problems. Each dataset contains 7500 math problems in their training split. We evaluate the models on 500 problems from the MATH test split (Lightman et al., 2023) and 1319 problems from the GSM-8K test split. Further, use 500 problems from the MATH test split and 500 problems from GSM-8K the validation dataset.
We generate the solutions for the problems in the MATH using a 4-shot prompt and for GSM-8K using an 8-shot prompt. We generated the candidate solutions in the synthetic dataset using TopK (K= 3) strategy with temperature of 0.7. The MATH prompts are identical to the Minerva prompts (Appendix D.2 in (Lewkowycz et al., 2022)). In addition, the GSM-8K prompts look similar to the ones found in open-source evaluation frameworks.
In addition, we train the Gemmal-7B model with a batch size of for 2400 and 24000 step under the low and high sampling budget, respectively. We perform hyperparameter search for the learning rates {le- - 7, 5e 7, le - 6} based on the model performance on the validation datasets.
K A FUTURE PERSPECTIVE
Variation in reasoning capabilities over time for open language models LLaM3 (708)
70
Small LM (1B-9B)
Large LM (20B-80B)
MATH Performance (%)
Owen2 (728)
50
DeepSeeki22 (21B Owen2
Mixtral (228)
Gemmaz
(348)
40
owen1.5 (728)
30
Grok-1 (788)
Gemman a1 (7B)
Qwan2 (1B)
Owen1.5 (7b)
20
Nov-2023
Feb-2024
Mar-2024
April-2024
Jun-2024
Jul-2024
Aug-2024
Figure 22: Variation in the performance of open LMs on the MATH dataset over time. The fitted trendlines suggest that the quality of smaller LMs is improving more rapidly than that of larger LMs over time. This highlights that our findings utilizing smaller LMs for training strong reasoners will become increasingly relevant in the future.
We showed that for the current WC and SE models, training reasoners through sampling from WC models may be more compute-optimal Here, we aim to discuss the relevance of these results for the future set of WC and SE models. To do so, we surveyed 17 LMs that pass the following criteria: 1- the model size is known and falls within [IB, 9B] or [20B, 80B] range, 2- the model is released in the past one year, 2- the technical report of the model reports results on the MATH dataset and the model is capable on it (> 20%), 4- ranks high on the OpenI_LM leaderboard under the pretrained models category (HF, 2024a). This resulted in models from seven families including Gemma-2 (Team al.,
httpps: http://////hththt//////e/ thub. com/E1 /B1 eut heral /1mmmvvaa i on hwr-aaanees//eeeee /bl ob/mai in 1m_eval/ basks/gsmäte hasmmmkk/gssmkooooooo 11 ama. yam1
24

---

Published as a conference paper at ICLR 2025
2024b), LLaMA-3 (Dubey al., 2024), Mixtral (Jiang al., 2024), Qwen (Team, 2024; Yang al., 2024a), Grok-1 (xAI, 2024), DeepSeek-v2 v2 (Shao et al., 2024), and Yi (Young et al., 2024). We grouped these models into small LM (IB to 9B) and large LMs (20B to 80B). We then plotted in Figure 22 the model performances on the MATH dataset against their date the publication release arxiv and fitted trendlines the data points representing the small and large LMs using the least squares method'.
Our analysis reveals that, despite the variance, the trendline for the smaller LMs is steeper than that of the larger LMs. This indicates that the reasoning performance of the small LMs may be improving more rapidly over time compared the larger LMs. The rapid rise in the performance of the small LMs can be attributed to factors such as the enhanced quality and scale of the pretraining data (e.g., LLLMM-33 employs 15T tokens), pruning and knowledge distillation (Muralidharan al., 2024). With the performance gap between small and large LMs narrowing over time, we anticipate that our results will become even more relevant the future.
WWe consider the number of active model parameters for mixture-of-experts LMs.
25


Reviews:
[{"review_id": "owFvo6zTlt", "reviewer": "Reviewer_6hvc", "summary": "This paper challenges the common practice of using strong but expensive (SE) language models to generate synthetic training data, proposing instead that using weaker but cheaper (WC) models may be more compute-optimal. The authors introduce a \"compute-matched sampling\" framework that enables fair comparison between WC and SE models by accounting for their relative compute costs. At a fixed compute budget, this framework shows that one can generate P_SE/P_WC more samples from a WC model than an SE model. The authors evaluate this approach across multiple model pairs (Gemma2 9B/27B and Gemini Flash/Pro), tasks (primarily mathematical reasoning), and training paradigms (knowledge distillation, self-improvement, and a novel \"weak-to-strong improvement\"). They assess the generated data along three key dimensions: coverage (problems solved), diversity (unique solutions per problem), and false positive rate (correct answers with incorrect reasoning). The results consistently show that training with WC-generated data outperforms SE-generated data when properly compute-matched.", "questions": "I will try and cluster my questions in sensible groups.\n\n1. _Theoretical Understanding_:\n   - Can you provide theoretical insights into when WC sampling should outperform SE sampling?\n   - How does the optimal sampling ratio change with model size and task complexity?\n   - What are the key factors that determine the success of weak-to-strong improvement?\n\n2. _Methodology_:\n   - How would the results change with more sophisticated filtering strategies?\n   - Could you provide more details about the specific prompting strategies used?\n   - How sensitive are the results to the choice of temperature and sampling parameters?\n\n3. _Generalisation_:\n   - What characteristics of a task make it more/less suitable for WC sampling?\n   - How would the results scale to even larger model sizes?\n   - What is the relationship between FPR and final model performance?\n\n4. _Practical Implementation_:\n   - How would you recommend implementing this in scenarios without ground truth?\n   - What modifications would be needed for different domains or tasks?\n   - Could you provide more detailed guidance on optimal sampling strategies for different scenarios?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. _Originality_:\n   - Introduces a novel compute-matched sampling framework with clear mathematical foundations\n   - Proposes a new \"weak-to-strong improvement\" training paradigm that challenges conventional wisdom\n   - Provides a fresh perspective on the compute-quality trade-off in synthetic data generation\n\n2. _Experimental Rigour_:\n   - Comprehensive evaluation across multiple dimensions:\n     * Multiple model pairs (both open and closed models)\n     * Various compute budgets and training paradigms\n     * Different dataset sizes and difficulty levels\n   - Thorough ablation studies that isolate the impact of coverage and diversity\n   - Both human and automated evaluation of false positive rates\n   - Clear validation of results through transfer learning (Functional MATH)\n\n3. _Practical Impact_:\n   - Demonstrates significant cost savings potential (0.15x cost for comparable or better performance)\n   - Shows consistent improvements across model sizes (7B to 27B)\n   - Provides actionable insights for practitioners\n   - Results particularly relevant given the trend of improving smaller models\n\n4. _Technical Depth_:\n   - Rigorous mathematical formulation of compute-matching\n   - Analysis of traed-offs between coverage, diversity, and error rates\n   - Ablation studies support main claims\n   - Clear empirical validation of theoretical framework", "weaknesses": "1. _Theoretical Foundation_:\n   - Lacks formal analysis of when WC sampling should outperform SE sampling\n   - No theoretical bounds on the optimal sampling ratio\n   - Missing analysis of the relationship between model size and optimal sampling strategy\n   - Limited exploration of failure modes and their characteristics\n\n2. _Methodology Limitations_:\n   - Heavy reliance on ground truth for filtering solutions\n   - Limited exploration of alternative filtering strategies\n   - FPR evaluation methodology could be more robust (50 human samples probably insufficient)\n   - Some key implementation details relegated to appendices\n\n3. _Generalisation Concerns_:\n   - Primary focus on mathematical reasoning tasks\n   - Limited exploration of other domains (coding results show context-dependency)\n   - Unclear scalability to larger model sizes\n   - Performance on more complex reasoning tasks not fully explored\n\n4. _Practical Considerations_:\n   - Deployment challenges in scenarios without ground truth not fully addressed\n   - Resource optimisation strategies could be explored more\n   - Limited discussion of integration with existing training pipelines\n   - Cost-benefit analysis could be more comprehensive across different scenarios", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "IbhPg9J38o", "reviewer": "Reviewer_rLZh", "summary": "This paper revisits the trade-offs between generating synthetic data using a stronger but more expensive (SE) model versus a weaker but cheaper (WC) model, and finds that at a fixed sampling compute budget, finetuning LMs with data from a WC model can consistently outperform data from a SE model in multiple settings.", "questions": "1. Although both low and high budgets are studied, could you please provide the results of an extremely high budget where the cost is not an important factor? This should be indicative of diverse data scales.\n2. Despite the train-test splits of MBPP, this paper only trains models on MBPP and tests them on HumanEval. The testing results on MBPP are expected to be provided for a more comprehensive understanding.\n3. Writing：\n- 1. All the ref links are invalid.\n  2. l101, l104: \"i.e.\" should be \"i.e.\".\n  3. l103: grammar error for \"we supervise finetune\".\n  4. l109: use \\citet{} for \"(Zelikman et al., 2024;Singh et al., 2023).\".", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The research question is significant, focusing on performance comparison of data sampled from WC and SE models, respectively.\n2. The findings are impressive, challenging the traditional belief that data from a strong model is better for finetuning models.\n3. The evaluation settings are diverse, demonstrating the effectiveness and robustness of this method, despite only the Gemma series models.", "weaknesses": "1. This paper centers exclusively on the Gemma series, and it is essential to extend the analysis to the Llama series to demonstrate the robustness of the conclusions.\n2. While the paper aims to highlight the lower computational cost of the WC model for data synthesis (particularly important for large-scale data generation), all the experiments are conducted on relatively small datasets. This discrepancy undermines the overall contribution of the paper.\n3. Compared to the SE model, the WC model can be regarded as a more diverse yet lower-quality variant. Therefore, it is crucial to compare it with techniques designed to enhance output diversity. Specifically, if adjusting the sampling temperature of the SE model consistently results in performance degradation relative to the WC model, this suggests that the WC model provides a superior quality-diversity trade-off compared to merely increasing the sampling temperature.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Ngg10RaBoy", "reviewer": "Reviewer_qzpX", "summary": "This paper presents the novel observation that generating synthetic data using a weaker but cheaper (WC) model is more effective than using a stronger but more expensive (SE) model. The authors demonstrate that, under the same budget, data generated by WC models tend to have higher coverage and diversity, though with a corresponding increase in false positive rates. Additionally, they show that models fine-tuned on WC-generated data consistently outperform those trained on SE-generated data.", "questions": "It seems that the difference in data quality between the WC and SE models becomes larger at lower budgets. Is it possible that the WC and SE models generate data of similar quality when the budget is very high?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The paper is well-structured and clearly written, making the methodology and results easy to follow.\n\nThe experiments are well-executed and provide convincing evidence of the benefits of the proposed approach.\n\nIt addresses a critical issue in synthetic data generation, offering a valuable contribution to this area of research.", "weaknesses": "The conclusion may not hold when using models from different companies. Based on my experience, under the same budget, data generated by a larger model like Qwen2.5 7B could outperform that of a smaller one like Gemma2 2B.\n\nThe paper could benefit from experimenting with more complex reasoning tasks, such as tree search algorithms, and using a reward model to evaluate the quality of the generated data.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "BaOPicAjUI", "reviewer": "Reviewer_gC6v", "summary": "The paper investigates whether it is better to (self-)distill from a Gemma-27B LLM or to distill three times more finetuning data from a three times smaller Gemma-9B model. It finds that the three times more data of the smaller model, despite including more errors, leads to a higher performance of the finetuned student model.", "questions": "* Your distillation setup is limited to finetuning. One setup where it would be more realistic to not have enough budget is pretraining. Do you have any results on this? I of course do not expect to pretrain a network until convergence during the rebuttal, but it would already be helpful if you could show the first couple of iterations just to make sure the worse data (higher FPR) does not seem to converge to a much worse model.\n* I'd be interested in sample-matched figures. The figures where I'd be most interested in a sample-matched comparison are Figures 4c and 5c. This would allow finding out if a small model can successfully improve a larger model, which would challenge beliefs in the field.\n* Just to go sure: In the self-improvement setups, you keep training a model iteratively on its own generations from the current parameters? Or do you mean that you finetune a \"fresh\" 7B model using an already converged 7B model?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 2, "strengths": "* The paper investigates not only knowledge distillation, but also a self-improvement setup\n* In Figures 4 and 5, it is interesting that training on model-generated solution paths (one per question; at least in the \"27B low compute\" setup) gives a better performance than training on human-provided solution paths (also one per question)\n* Figure 7 carries an interesting finding: Despite training on the data generated by the small model which has more errors, the ultimate trained model does not have more errors in its reasoning. This implies that the additional data mitigates its lower quality, which might add evidence to the discourse beyond the setup studied in this paper.\n* The writing and flow of experiments is mostly clear", "weaknesses": "* In 21 of the 22 Figures, the paper hinges on \"matching the compute\", i.e., being allowed to generate 3 times more data when using the 3 times smaller LM. This confounds two factors of variation, making it hard to interpret the findings. This is the main weakness of the paper. One idea to improve on this weakness would be to test out distilling 1, 3, and 9 samples from the large LLM and 3, 9, and 27 samples from the small LLM (instead of the current 1+10 vs 3+30), so that there are both overlapping settings with a matched number of samples and with a matched compute.\n* In the only figure where the small LLM is compared to the large LLM without this advantage (Figure 20 in the appendix), the large LLM produces better training data. It can be expected that if we use the large LLM to generate enough data until the student model converges, it will make a better distilled model. Thus, the only real application of the proposed method is when we do not have enough budget to produce enough data to converge. For the finetuning setup of the paper, that would amount to not being able to generate data for 8k-12.5k questions. This is a setup with limited applicability in practice. It would increase the contribution (score) of the paper to investigate problems where budget limits are hit more frequently in practice, like pretraining, see also my question below.\n* Relative and absolute increases are reported inconsistently. E.g., in Figure 3b the fact that the proposed small model finds 11 instead of 5 solution paths per question (when it is allowed to generate 3 times more paths in total) is reported as a 125% increase (line 268), whereas the fact that 24% of its solutions paths are wrong compared to 17% of the large model is reported as a 7% increase (line 310). This inconsistency becomes problematic when reporting the increase on percentage numbers (e.g., line 258), where it is unclear whether this is a relative or absolute increase. Keeping the reporting consistent would increase both the presentation and the scientific soundness scores.\n* The paper only evaluates Gemma (/Gemini) models. It would help judge the generalization of the claims (and increase the contribution score) to test it out on at least one other LLM, like a Llama model.\n* The datasets are very limited to two math datasets, limiting the contribution. As above, more datasets would help judge the range of applicability, especially whether it also works on non-math and non-reasoning datasets.\n* The paper does not compare to baselines, despite citing multiple closely related approaches\n* The method still requires annotated data, because the LLM-generated data needs to be filtered out if it does not match the GT. It would increase the applicability of the score (and thus the contribution score) if there would be an ablation without filtering, i.e., answering whether the unfiltered erroneous data from the smaller model can still train a better model.\n\nSmall notes that did not influence my score and don't need to be rebuttled, I just note them to make the camera-ready better:\n* The first paragraph of Section 3 could be shortened; it's message (in Equation 1) is just \"if a model has x times more parameters, it takes x times longer to generate\".\n* typo in line 54, \"filters\"\n* typo in line 103 \"we supervise finetune\"\n* typo in line 151, \"consists\"\n* typo in line 157, \"for training student LM\"\n* typo in line 241, \"that where\"\n* The references exclusively list arxiv versions of the papers, not their actual published versions\n* The reference .bib file should best use double brackets for \"{{OpenAI}}\", \"{{The Llama Team}}\", to prevent the ill formatting in line 483 (\"Team, 2024; Anthropic, 2024; AI, 2024\")", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)