Mind the Gap: Examining the Self-Improvement Capabilities of Large Language Models
Authors: Yuda Song
Hanlin Zhang
Carson Eisenach
Sham M. Kakade
Dean Foster
Udaya Ghai
Venue: ICLR 2025 Oral (2025)
Keywords: LLM
self-improvement
synthetic data
post-training
test-time optimization
Primary area: foundation or frontier models, including LLMs
TL;DR: We conduct a comprehensive examination on LLM self-improvement capability via the generation-verification gap.

Abstract: Self-improvement is a mechanism in Large Language Model (LLM) pre-training, post-training and test-time inference. We explore a framework where the model verifies its own outputs, filters or reweights data based on this verification, and distills the filtered data.  Despite several empirical successes, a fundamental understanding is still lacking. In this work, we initiate a comprehensive, modular and controlled study on LLM self-improvement. We provide a mathematical formulation for self-improvement, which is largely governed by a quantity which we formalize as the **generation-verification gap**. Through experiments with various model families and tasks, we discover a scaling phenomenon of self-improvement -- a variant of the generation-verification gap scales monotonically with the model pre-training flops. We also examine when self-improvement is possible, an iterative self-improvement procedure, and ways to improve its performance. Our findings not only advance understanding of LLM self-improvement with practical implications, but also open numerous avenues for future research into its capabilities and boundaries.

Published as a conference paper at ICLR 2025
MIND THE GAP: EXAMINING THE SELF-IMPROVEMENT CAPABILITIES OF LARGE LANGUAGE MODELS
Yuda Song
Hanlin Zhang
Carson Eisenach Amazon
CMU, Amazon
Harvard University
yudas@andrew.. cmu, edu
hanlinzhangeg. harvard. edu
cei ceiseneamazon.com
Sham M. Kakade
Dean Foster
Udaya Ghai Amazon ughai acchai@mmazon.ccom com
Harvard University, Amazon sham@seas, harvard.
Amazon Foster@amazon. com
ABSTRACT
Self-imppovement is a mechanism in Large Language Model (LLM) pre-training, post-training and test-time inference. We explore a framework where the model ver- ifies its own outputs, filters or reweights data based on this verification, and distills the filtered data. Despite several empirical successes, fundamental understanding is still lacking. In this work, we initiate a comprehensive, modular and controlled study on LLM self-improvement. We provide mathematical formulation for self- ceff-mmproveeent, which is largely governed by a quantity which we formalize as gap. Through experiments with various model families
the inneraation-eriffcaan and tasks, we discover scaling phenomenon of seff-improvement- a variant of the inreattion-vrificaation gap scales monotonically with the model pre-training flops. We also examine when self-improvement is possible, an iterative self-improve cef-iimprovemett procedure, and ways to improve its performance. Our findings not only advance understanding of LLM self- self-improvement with practical implications, but also open numerous avenues for future research into its capabilities and boundaries.
INTRODUCTION
Recent work increasingly explores the use of synthetic data in training large language models (LLMs), with applications in both pre-training and post-training (Bai et al., 2022; Meng et al., 2022; et al., 2023; Adler et al., 2024; Dubey et al., 2024; Yang et al., 2024; Hui et al., 2024: Li et al., 2024). While synthetic data, often generated by LLMs, offers a valuable complement to human generated data, human-generated
its misuse harm performance. Bertrand al. (2023) and Gerstgrasser et al. (2024) showed self- training on model-generated data leads to degradation. To mitigate this, incorporating a "reliable" ver- ifier label data has shown promise in preventing such performance collapse (Gillman et al., 2024).
straightforward verification mechanism to train a reward model on human-annotated data to assess the quality of synthetic data (Lightman al., 2023; Wang al., 2024a). However, this approach can be prohibitively expensive and may offer few signals in domains where models exhibit super-human performance. An alternative is use stronger model (Chang al., 2023; Havrilla et al., 2024) for annotation, but this becomes infeasible when the model is at the frontier of current capabilities. A promising solution is to use the model to label its own generations. Motivated by the intuition that "verification is easier than generation", one can hypothesize that the model may act as siieer-tha--rnndor verifier of its own outputs, enabling self-improvement (Zelikman et al., 2022).
Most previous self- infrroveeeee algorithms can be summarized as follows: 1) make multiple generations from the model, 2) use the same model to verify the generations, and 3) distill from the reranked/filtered generation (Zelikman et al., 2022; Huang et al., 2022; Wang al., 2022b; Yehudai et al., 2024; Madaan et al., 2024; Yuan et al., 2024; Xu et al., 2024; Liang et al., 2024). With this framework, self-impunn reff-improvement is also related improving inference quality (Wang al., 2022a; Welleck et al., 2024) if the model can verify its own generation, self self-improvementt can enhance test-time performance with additional computation towards more generations and updates.

---

Published as a conference paper at ICLR 2025
Qwen-1.5 1.5 (GSMBK)
Qwen-2 (GSMBK)
Liama-2 (GSMBK)
M11.5 (GSMBK)
Relative Gap (9)
30
20
20
20
5
10
10
5
-10
6
5.0
5.5
6.0
6.5
log, flops (1e21)
Owen-15 (MATH)
Qwen:2 (MATH)
Owen-2.5 (MATH)
Ulama-2 (MATH)
Relative Gap (%)
10
15
COT-
10
5
lag. flops (1e21)
Figure With proper verification method (e.g., CoT-S), the relative viterrattion-eeffiaaaton gap (Definition 2.2) scales monotonically with respect the pre-training flops. We conjecture that in this case, the relative gap is linear with respect to the log of the pre- vre-trannig flops. MC denotes Multiple Choice verification, CoT-S denotes Chain-of- Chain-of.Thhught-SS3 Thought-Score verification, and To denotes Tournament verification. The description of each verification can be found in Section 3.
Despite impressive empirical progress, fundamental understanding of LLM self- eeffimprrovemen remains limited. It is uncertain whether these results can be interpreted solely as an indication of the self-improvement capability of LLMs, given the potential for confounders various stages of the process. Moreover, much of the existing research focuses on just one model family or a single verification mechanism, limiting the broader applicability of the findings. In this work, we conduct a comprehensive study of the self-improvement capability of LLMS;; our contribution is as follows:
Self-Improvement Framework: Section 2 details the mathematical formulation of the self- improvement process, highlighting three critical desiderata. We propose the amneration-eeriiiaaati gap (GV-Gap) (Definition 2.1) as the central metric for evaluation. GV-Gap captures the "precision" of the model's 's verification over its own generations. Concretely, it is defined as the performance improvement obtained by re-weighting generations by the model's self-verification score (e.g.. 0 or 1). Our empirical findings indicate that GV-Gap is a more accurate metric for measuring self self-improvement versus the previous metric of the performance difference after a model update.
In Section 4, we measure the generation-verificatin gap across multiple model families, verification mechanisms, and tasks. i) Certain verification methods induce a scaling phenomenon for self improvement the relative Gap (Definition 2.2) increases monotonically with pre-training flops shown in Figure ii) We find that in coss-veriiiiation (i.e., using different models for generation and verification). the GV-gap increases with verifier capability and decreases with generator capability. iii) Finally, we observe that most models do not achieve relf-improvement in information retrieval or reasoning tasks that exceed their inherent reasoning or planning capabilities. By studying multiple types of verification, our results indicate general patterns beyond just prompt engineering.
Scaling Properties:
Iterative Self-Improvementt: In Section 5, we identify that i) GV-Gap saturates in handful rounds of iterative self- aafroveprvee:::: ii) the saturation rate is independent from the model capacity, iii) the effective diversity degrades during the iterative self-improvemmnt.
Verification Mechanisms: In Section 6, we consider methods to enhance self- seff-imrrovement through a fine-grae mne-grained study the verification methods. Key observations include: i) the same verification method induces consistent trends among different models; ii) different verification mechanisms have significant non-overlaps; iii) GV-Gap is not necessarily positively correlated with generation accuracy; iv) an ensemble of verification can enhance self-improvement.
A DISSECTION OF THE SELF-IMPROVEMENT FRAMEWORK
In this section, we introduce the setup of the self-improvement framework considered in the paper, which consists of the following three main components:
Due space constraint we defer related works Appendix B.
2

---

Published as a conference paper at ICLR 2025
Response Generation. Let A' be the prompt/context space, and y be the response space. Let F S (f : X + A())) be a class of generative models that maps a prompt to a distribution over responses. A task, task T (e.g., math, code, trivia, puzzles, etc.), defines distribution Mtask E (2)) over the prompt space X, and utility function Maakk X' x y + [Umin: Umax]: where Umin Uma denote bounds of the utility function. The goal is to find a generative model that maximizes the expected utility: fask == arg max JeF Jrask (f), where
Jrask = Bx- [By=f(-=)) [Meask (z, 3/]]
We will often drop the subscript task when is clear from the context.
Verify with Proxy Utility. Without the access to the ground truth utility function u, we rely on proxy utility function constructed by some model f, denoted as ū, X example, let Z = [10] == 2.... 10} be scores, the proxy utility (z, y) E0 [2] rates the quality of the response with a score from 1 to 10 with an instruction prompt prom. In Section 3, we provide more proxy utility functions used in our experiments.
[Uminn Umax] For
Update via Reweighting, Let t E [T] be the iteration index, ft be the model at iteration t, and y be the proxy utility defined by model f". The reweighted distribution ft f.(w((ū))) )] is defined such that
[ (y ()) x ft [ x) w(ū (z, y)), Vx, y E X y
Here w : R20 is a weight function that maps a utility from the verification procedure to a weight. The specific form of w is determined by the algorithm used for the model update step (we provide two examples below). The objective is to find a model se+1 E F such that
((fe+1 ((( )I) Ez= (e( x), ((((((()))))))) )- =))]
small (i.e., E-projecting fi [w [] onto F), for some distance measure d. Note that when f = ft. we have self-improvement though the framework can also be used for improving using utilities
produced by a different model as is studied in Section 4.2.
Example 2.1 (KL-regularized RL Update). One can treat the proxy utility û as reward, and perform RLHF style RL update with a reverse KL constraint (Christiano et al., 2017; Ouyang et al., 2022):
f(y x) fe xx)
f++1 arg max Eeggyroos (z..) 33 3logg fEF
In this case, we have w(s) exp(s/3). and d can be the KL divergence between fi+1(: 22) and fe (ū )-( x) (Nemirovskij & Yudin, 1983).
Example 2.2 (Rejection Sampling). In rejection sampling, we filter the generation by a threshold and then fie-tune the model on the filtered data:
fet1 = arg max Fzz,y=pof [log(S(y x)) 1 (x. 2 2T] JEF
In this case, we have w(()) 1 [ 2 r). and can be the total variation distance between fe+1 and fel fe(w(ū,)) (Zhang, 2006).
Finally, it is convenient to abuse the notation and allow w and to take batch input. For example, we allow to take list of score and then set the filtering threshold the quantile (n E [0, 1) of the score. We denote this as top-n or quantile-n filtering.
2.1 THREE KEY FACTORS OF SELF-IMPROVEMENT
For any meaningful self celf-improvement, iteration t, we would like find f(+1 such that J(f+11) > (ff.), where recall J(f) is the expected utility under the model f. We identify the three key conditions that may bottleneck improvement on model f:: 1. Improvable Generation. Our framework involves reshaping the generation distribution towards increased utility. In order for this to be useful, the utilities of generations must have variability, For example, if generation were done with greedy decoding, no improvement in this process would be possible. Fortunately, the improvable generation phenomenon has been well-observed in LLMs (Li et al., 2022; Brown et al., 2024) (see also Figure 4). 2. Informative Verification. Recall that weight function w(ū)) is defined by the proxy utility function Ty which is constructed by a verifier model g. If the verification capability is limited, the weighting may not provide useful signal for improvement. The following definition quantifies this intuition:
3

---

Published as a conference paper at ICLR 2025
Geternfion
Vertication
Double
Generalive
Gencrative
Ter's
Mic
Model
Model
Eree Agi
Cupt/.x) 06.% 80% 6.7%
Figure 2: A visualization of using rejection sampling as an example for the key definitions in the self-improvement framework. For simplicity, we assume a single prompt x.
Definition 2.1 (Generation- Verification Gap). For a generator f and verifier g, we define the generation-verfficttinn gap (GV-Gap) between and g as gap(f.g) == J[(([((ū))) J(f)
We include a visualization of Definition 2.1 in Figure 2. This is the core metric for our analysis throughout the paper. When the generator f and verifier g are the same, we denote the shorthand gap(f) == gap(f, which is the self-improvement generation-verificction gap. However, the GV-Gap is an absolute quantity, which does not fully capture the various qualities of the generation. Consider generator that already achieves 99% accuracy on task: first, the upper bound for gap(f) is only 0.01; second, incorrect responses are likely to be very subtle, and thus any improvement in the reweighted distribution might require a very strong verification model. This motivates the relative GV-Gap:
Definition 2.2 (Relative Generation Verification Gap). For a generator f and verifier g. we define the relative generation-verificattii gap between f and g as
(f.g) == Ezza Ey-s(m(a,)) (a,) (-rr))) [u(z y)1 Ey f(- a) m))
Uma Ey- [u(,, 3)]
That is, we weigh the gap of each prompt by its deficiency to the best possible utility. For simplicity, we will denote the self-GV-Gap as "gap" or gap and relative self-GV-Gap as "relative gap" or gap when the context is clear. domains where verification easier than generation, gap > likely holds, and indicates that there is additional signal that can be exploited. One can also check that, for all prompts x E X. if the weight function (x, :) and u(x,) -) is positively correlated? under the distribution of y ~ then we can always guarantee gap(f. .) > 0.
3. High-fidelity Model Update. The final condition is that the model fi41 mimics/distills the performance of the reweighted distribution fe ([w(ū,, i.e. [f(f+1)-- [w )) S Eupaate, with some small Supdate- For example, if through MLE we bound the TV-distance between the two by E', then by Holder's inequality have Eupdate S e' e'Umax- Combining it with the gap guarantee, have:
J(f++1) - J(h)) Z gap(Je, g)
Supdate-
With a sufficiently expressive LLM class, it is often observed that the distillation error éupdate small.
Note that sometimes we might observe J J(++1) J (fi[ ([[w((ū,)))) > O. For instance, a benchmark may require outputs in specific format; in such cases, the finetuned model fe+1 might outperform the reweighted distribution fo ([(((,,), simply by aligning outputs with the required format, even if the underlying answers remain unchanged. Recent works (Dubois al., 2024; Zhang al.. 2024c) highlight additional confounders, such as modifications to output length during finetuning, which may inflate perceived improvements without reflecting true model capabilities. Conversely, it is conceivable that intrinsic enhancements in the model's reasoning might occur; for example, mastering simpler tasks could indirectly boost performance on more complex problems requiring similar skills. However, in our experiment we only observe the former scenario. That said, both cases emphasize the need for caution when interpreting such improvements, and this further emphasizes the importance of our modular approach in dissecting the components of self- seff-improvement.
'Even under the case that w(ūg)) (x, :) and u(x,-) are negatively correlated, if we have small holdout (R,))) ))}, use the holdout set tune dataset with ground truth labels u(x,, y), we can define w() = exp(a w((ū,)
a, and use mn(a) reweight the distribution.
4

---

Published as a conference paper at ICLR 2025
3 EXPERIMENT SETUP
Our experiment is based on Im-evaluation-harnes (Gao et al., 2024). For all tasks we use the following setup: for generations and verification, use sampling parameters p = 0.9, 0.7, max length of 512 and 4-shot in-context samples. 9 For each model f. for each prompt æ, we sample 128 responses y ~ f(x)), and sample 1 verification for each response, which defines the proxy utility score ( v).* We mainly consider the rejection sampling setting (for completeness we investigate the RL setting in Section 5), and thus the weight function w the indicator function with either quantile or global threshold (c.r. Example 2.2). Then we calculate gap gaprel according to Definitions 2.1 and 2.2, which is the accuracy difference between the filtered generations and the original generations. Specifically, consider the following verification mechanisms (a formal description of each is provided in Appendix along with example prompts in Appendix G):
1. Multiple Choice (MC) (Zhao et al., 2023; Liu et al., 2023; Dong et al., 2024) asks the LM to label responses as "Correct" and "Incorrect" and uses the probability of "Correct" as continuous score.
2. Chain of Thought (CoT) asks the LM to score responses and to provide the justification (i.e. CoT (Wei et al., 2022)) and the score is parsed from the answer. Scores can be on a scale from 1 to 10 (CoT-Scoree) (Yuan et al., 2024; Liang et al., 2024) or binary (CoT-Binaryy
3. Tournament (To) involves sampling batch of generations and having verifier compare generation pairs in a single elimination tournament to produce a new generation distribution. We can repeat the process until there is only one response left from the batch.
We consider the following models families: Qwen-1.5 1.5 (Bai et al., 2023), Qwen-2 (Yang et al., 2024), Qwen-2.5 (Team, 2024), Llama-2 (Touvron al., 2023), Llama-3, Llama-3.1 (Dubey et al., 2024) and Yi-1.5 (Young et al., 2024). To avoid the confounding effect of the post-training, all experiments in this paper are performed on base models." Finally, all inference in this paper is performed with vLLM (Kwon et al., 2023).
SCALING PROPERTIES OF GENERATION-VERIFIE
GAP
In this section, we conduct a comprehensive study on measuring the scaling property of the generation- verification gap due to its valuable practical guidance in both pre-training and downstream tasks (Kaplan et al., 2020; Hernandez al., 2021; Isik et al., 2024; Ruan et al., 2024).
4.1 SCALING RESULTS
We start with the GSM8K benchmark (Cobbe et al., 2021), with 1320 questions on the test data split, and MATH benchmark (Hendrycks al., 2021), with 5000 questions the test data split. The ground truth utility u(z, y) = 1 the end of response is the correct answer to the question x, or u(x y) = 0 otherwise. We compute gap(f) f) for each model f and verification method, and we record the full results in Tables 3 and 4. In particular, we observe the following phenomena:
Small Models 1.5 0.5B, Qwen-2 0.5B and Llama-2 7B, gap(f) is non- con-positive for nearly all verification methods, even though the models have non-trivial generation accuracy. We also observe this phenomenon in Pythia (Biderman et al., 2023) and OPT (Zhang et al., 2022) model families. We believe this result indicates that eel--improeement requires a minimal level of instruction following and reasoning capa- bilities, which is not present in these small models. We will further illustrate this point in Section 4.4.
not Self-improve. For small (in terms of pre-training flops) models such Qwen-
CoT Verification is More Stable than MC. Some MC verification incurs non-positive gap even for medium-sized models such as Qwen-1.5 14/32B and Llama-3/3.1 8B models, while CoT verification
'In Appendix D.I. perform ablation different sampling temperatures, and observe that the same observations hold for range of reasonable hyperparameters.
"Note that an ideal verification should be sampling multiple verifications per generation. We only sample due computational constraints and leave multiple verifications along with understanding verification compute scaling for future work. As preliminary investigation this issue, in Appendix D.4, perform small-scale experiments on measuring the variance the verification self-improvement.
In Appendix D.2, we repeat subset of our experiments on the instruct models and we indeed observe that the results are more noisy.
5

---

Published as a conference paper at ICLR 2025
Lima 2 MC (0598X))
Laamai2 Cal'S (GMM5R)
Owen-22 MC (05
Owan 2 CaT5 (55M80
33.3033.3
0:54
3.08
500
5.16
5:27
33
3:30
2.70
3.79
15.30
35
6.81
3.18
9.17
0.25
0.97
4.96
30
5.29
1:30
1.55
5.30
728
+0.07
2.61
8.91
0:30
0.61
1.44
108
128
Varifer
Valtfer
Vurifirr
Verffer
Figure 3: GV-Gaps (%) in cross voss-improvement For each row (a fixed generator), gap increases as veri- fier capacity goes up. For each column (a fixed verifier), gap decreases as generator capacity goes up.
always has a positive gap for mediun/large-sized models. Our results align with recent studies showing that MC evaluation might be unreliable, especially for small models (Dominguez-OIin Olmedo et al., 2024). We perform more in-depth analysis on this point in Section 6.
ge (f) Scales with Pre-training Flops. We observe that with certain verification methods (such as CoT-Score) the relative gap grows monotonically with the pre-training flops, demonstrating a scaling property. We visualize the scaling results in Figure where we plot (f))a(()) with respect to the log- arithm of pre-training flops. Specifically, we hypothesize that in the case where the verification elicits the scaling property, gaprei(J") scales linearly with respect to the logarithm of the pre-training flops. However, note that we should not expect the slope for each model family to be the same. In Figure 12, we repeat the same plot for gap(f). f). but we do not observe a similar trend with the absolute gap.
4.2 CROSS VERIFICATION
In self-improvement, both generator and verifier change when transitioning between different models. To better understand the relationship between generation/verification ability and model capacity, we perform cross- vross-eeriiiattio study, where only alter either the generator or the verifier at time. We consider the Llama-2 and Qwen-2 model families, and the two most representative verification methods: MC with quantile threshold and CoT-Score. We present the results in Figure 3. We observe that the results are consistent with our intuition on the difficulty of verification: fix a generator model S. gap(J, g) increases as the model capacity (defined by pre-training flops) of the verifier model g increases. On the other hand, fix a verifier model gap(J,g) g) decreases as the model capacity of the generator model increases, as the error of the generator model becomes more difficult detect.
At first glance, the results seem to imply that selecting the largest model as the verifier, akin to teacher-student setup, is advantageous. However, considering the computational costs associated with larger verifier models, this approach might be suboptimal. Alternativey., a weak-to- veak-to-strong setup, where a smaller model verifies larger one, might be more cost-effective, but our findings indicate that a positive gap cannot always be assured. We believe an interesting future direction is to explore the compute-optimal configuration for cross-verification. This, however, might require a combinatorially large number of experiments to pinpoint the optimal verifier for each generator.
Takeaway on scaling of self self-improvement
LLMs demonstrate clear scaling trends in self- and cross- tross-improvement:
Self-Improvement:: With stable verification, the relative gap increases monotonically with pre-training flops.
Cross-Improvement: The gap scales directly with the verifier's flops and inversely with the generator's flops.
If the relative gap scales linearly with the logarithm of pre-training flops, this relationship could guide decisions synthetic data generation strategies in seff-improvement. Addition- ally, results from cross-verification suggest that a compute-optimal combination may exist to maximize efficiency in cross-improvement contexts.
4.3 UNIMPROVABLE TASKS
The primary objective of improventte is predicated on the assumption that "verification is easier than generation". As such, it is also worthwhile to consider tasks where such intuition would not

---

Published as a conference paper at ICLR 2025
Table 1: Gap (%) on Natural Question for Qwen- 2 models. While all models have non-trivial generation accuracy. all gaps are near O, indicat- ing that the task is unimprovable.
Table 2: Generation accuracy, gap and relative
gap (%) on Sudoku for Qwen-2 models. Only the 72B models can self-improve. For the 72B models, the improvement around 200%.
0.5B
1.5B
7B
72B
0.5B
1.5B
7B
72B
Generation Accuracy
6.51
13.87
29.09
41.45
Generation Accuracy
0.66
0.62
2.09
8.82
MC (top 0.8)
-0.06
0.04
0.79
0.28
gap
-0.09
0.04
-0.07
16.99
MC (r = 0.8)
-0.05
0.02
-0.05
-0.05
gaprel
-0.15
-0.61
-0.01
20.81
hold. One such scenario involves factual tasks that require generating a factually correct answer to a trivia question. We hypothesize that the capability generate a correct answer is contingent solely on whether the model has been trained with the relevant factual knowledge, and verification would provide little additional signal. To test this, measure gap(f) on the Natural Question dataset (Kwiatkowski et al., 2019), where u(x, y) = 1 if is one of the candidate answers to the question n, and u(x, y) 0 otherwise. Our analysis reveals that despite all models achieving non-trivial generation accuracy, the gap remains smaller than 1%, or is even negative, across all models. This suggests that certain tasks may not benefit from the current self- inffimppooveent framework. We include the full results in Table 5.
test subset of 3610 questions, presented in Table 1,
4.4 SUDOKU
Generalized sudoku is canonical example where the generation (NP-hard) is harder than the verification (P) (Haythorpe, 2016). We consider by 4 sudoku puzzles, each with unique solution, with 288 puzzles in total. We task the models to use CoT reasoning for both generation and verification. The results, presented in Table and detailed further in Appendix E.3, reveal a surprising pattern: only the largest models, such Qwen-1.577 72B and Llama 3.1 70B, exhibit non-trivial gaps. For these models, the improvement indeed more significant (50% 300% improvement in accuracy) than the improvement in the math task.
While the second observation aligns with common intuition, the first may be unexpected, as most models demonstrate the ability self eeff-impovee tasks where the gap between generation and verifi- cation appears even narrower, such as in GSM8K. We hypothesize that, despite sudoku verification being simpler than generation, it still necessitates certain level of reasoning and planning, even with explicit verification guidelines. This requirement is similar in mathematical tasks; however, is likely that most models have been exposed math verification during pre-training, unlike sudoku verification. Consequently, smaller models may lack the requisite reasoning capabilities to improve on sudoku tasks. Although our analysis primarily post-hoc, an interesting avenue for future research would be to develop a metric to predict a model's "self-imprrvaabiiit"" on specific tasks.
Takeaway on improvable tasks
LLMs do not universally self-improve across all tasks:
There is no significant generaaion-veriiicattonn gap, given the similarity in complexity between generation and verification.
Factual Tasks:
Sudoku:
Despite the exponential computational complexity separation between gener- ation and verification in generalized sudoku, most models fail to self-improve. When improvement occurs, it is notably significant.
These findings suggest that the model's inherent reasoning and planning capabilities developed during pre-training crucial for general self- oelf-improvement.
ITERATIVE SELF-IMPROVEMENT
Building on our understanding of single-round self saf--improveentt, a natural extension is to study iterative self-imm ceff-improvement. As additional information is introduced in the process, it is unrealistic to expect indefinite improvement. Thus in this section, we study the dynamics of the iterative self-improvement, and its relationship with model scales.
7

---

Published as a conference paper at ICLR 2025
Generation Diversity (Qwen-1.5-7.. GSMBK)
Iterative Self-imprveeeen (Qwen-1.5, GSM8K)
90
78
Paspiikk (%)
70 MC
Accuracy
70
G4p (%)
78 AL
Iteration
20
Iteration
60
Iteration Itoration
2 24 25 26 2?
Round
Number Generations
The generation accuracy and gap along the iterative seff-improvement process for Qwen- 1.5 models with CoT-Binary and MC verification. RL denotes using RL for model update. The horizontal line on the gap plot denotes 0.5 5%. Right: The change of effective generation diversity along the iterative cell-improveeett process for Qwen-1.5 1.5 7B model, measured by pass for different k.
Figure 4: Left:
In our experiment, we perform iterative self-improvement (Algorithm 1) on the Qwen- 1.5 model family with CoT-Binary verification on GSM8K. We also perform model update using RL in addition to rejection sampling. We present the results in Figure 4 and defer the finetuning hyperparameters to Appendix F. We observe that 1) the gap diminishes nearly to within two three rounds of self-improvemmett, consistent with the observation with previous works (Yuan et al., 2024; Liang et al., 2024). However, previous works failed disentangle the gap and the model update. 2) The rate saturation is similar across models with different capacities. 3) Notably, for the 7B and 14B models, the model accuracy at iteration one exceeds the sum of the generation accuracy and the gap at iteration 0, i.e., J (fi) > J [[ ()))) This increase is attributed to improved adherence to the required answer format post-finetuni
the discrepancy between "flexible match" and "exact match" (extract the answer from the required answer format) disappears after the first round. As we argued in the previous section, this additional accuracy gain is not due the self-improvement capability of the model, and thus our modular study reduces the confounding factors in understanding the self- eef--imprveement capability of the model.
To compare the dynamics between verification methods, in Figure 4 we also plot MC (quantile 0.7) verification for the 7B model. We observe that the gap immediately drops to near after the first round of self-improvement, and thus multi-round self-improvement with MC verification is unlikely, This rapid saturation is consistent across other thresholds for MC verification. We provide a more detailed study on the cause of this phenomenon in Section 6.1.
We also examine the "effective diversity" of generations throughout the iterative self-im celf-imprveement process using the metric pass0k. 6 We present the results in Figure 4. We observe when i: is small, pass@k increases with the number of rounds of self-improvement, validating the success of the self-improvement process. However, when & is large, pass&k decreases with the number of iterations, indicating that the diversity of the generations is reduced through the self-improvement process. This trend may result from the model's inability to verify rare, yet correct, answers, potentially leading to convergence on incorrect solutions during the self-improvement process.
For completeness, we also repeat the same experiments on the MATH benchmark. We defer the results to Appendix E.4. We observe the same phenomena of saturation limit and decrease in effective diversity in the MATH benchmark. In Appendix D.3, we also perform experiments on iterative improvement with more powerful verifier and ground-truth labels.
Takeaway on iterative seffvimprovement
LLMs can perform iterative self-improvement with an effective verification method:
Saturation Limit: Without new information, iterative self-improvement typically saturates
after two or three rounds, regardless of the model's capacity.
Cause of Saturation: diversity, caused by convergence on incorrect answers for certain questions.
A potential reason for this saturation is decrease in effective
Addressing the reduction in diversity could potentially extend the duration and effectiveness of the self-improvement process.
'Given a question, passūåk is if least of the generations of the model correct, or 0 otherwise.
8

---

Published as a conference paper at ICLR 2025
Owen-1.5 (GSMBK)
Qwen 1.5 Tournament (GSM-8K)
HC Quandic
CoT5
0
0.2
0.5
0.7
0.2
0.3
Threshold
Round
Figure 5: Change the gap as the threshold varies for each verification method. We only present the results for MC with quantile thresh- old and CoT-Score, because CoT-Binary 's gap stays constant as the threshold varies.
Figure 6: Change in the generation accuracy of the filtered dataset (left) and gap (right) with re- spect to the round of tournament. The right figure plots the gap with respect to the accuracy from the previous round instead of the base accuracy.
6 A FINE-GRAINED STUDY ON VERIFICATION
Among the three components of self-improvement, the verification step offers the most flexibility, whereas generation and update follow more fixed procedures. Therefore, this section presents detailed examination of the verification mechanisms. Through this particular study, we aim uncover practical ways to enhance the overall self- celf-improvement process.
6.1 GENERALIZATION OF VERIFICATIONS
In the rejection sampling framework, selecting an appropriate threshold for filtering generations based on verification is crucial practical concern. We explore verification methods adaptable to various thresholds, including MC, CoT-Score, and Tournament. Our analysis focuses how the gap changes with different thresholds for these methods. We present results for the Qwen-1.5 models in Figs. 5 and 6. For tournament verification, the threshold is defined as the number of rounds of tournament. We defer the results of other models to Appendix E.S. In Tournament, we note that the gap with respect to the accuracy in the previous iteration generally decreases monotonically; this trend occurs as the verification error more likely to be exploited by the remaining generations at later stages of the tournament.
We also conduct two additional sanity checks: in Figure 16, we plot the distribution of scores with CoT-Score Verification, and the mode of the score is 10 for all models. This self-bias behavior expected and is also observed in Xu et al. (2024). We also check the recency bias (Zhao et al., 2021) in Tournament verification: the average probability of preferring the first generation across all models and all rounds is 0.56 with a standard deviation of 0.12, indicating no critical recency bias.
We observe that the relationship between the gap and the threshold is consistent across most models when using any fixed verification method. For MC and Tournament, the gap follows a concave curve relative to the threshold, while for CoT-Score, increases monotonically. agree on the optimal thresholds for each verification across model families, and we use the optimal thresholds report our results for this paper. However, general, one should not expect the optimal threshold transfers between different tasks. That said, the consistency in threshold effects suggests practical approach: if determining the optimal threshold for large model is costly, one might first establish it for smaller model and then apply it to the larger model.
addition, most models
6.2 CORRELATION STUDIES BETWEEN VERIFICATION METHODS
As the verification methods are functionally similar, one might question the necessity study multiple verifications. To address this, We start by comparing the distribution of gaps induced by different veri- fication methods. We present the bar plot of the gap distribution of Qwen- 1.5 7B in Figure 7. Notably, there are significant discrepancies, especially between MC and CoT methods the variance in the gap considerably larger with MC. This aligns with our previous findings in Section where iterative self-improvemment with MC verification saturates more quickly than with COT. While CoT slightly improves the accuracy on most questions, MC will drive the accuracy to the extreme in one round of seff-improvemmnnt We observe that this pattern holds across all models, as detailed in Figure 17.

---

Published as a conference paper at ICLR 2025
Distribution of Gaps (Qwen-1.5-78, GSMBK)
Vericationn Coreiitinn 0.5.n
Frequency
Ai 6.074
0.158
1000
041
1,00
9.21
1M
2.00
0:77
2.04
677
1.43
35
3.55
4.00
500
2.24
1.00
0:20
0:6
0.33
100
244
3.00
8:5
0.35
04
100
4.13
-0.5
0.0
0.5
Gap
-0.5
0.0
0.5
1.39
0.35
1.00
3.10
1,02
MC-Q CoTB
Figure 7: Left: The empirical distribution of gaps of MC Quantile and CoT-Binary of Qwen-1.5- 7B on GSM8K. We cluster gaps in bins of intervals with width of 0.005. We label the mean (μ) and standard deviation (Ø) of each distribution. Right: the correlation plot of the output of each verification ū and the correlation plot of the gap from each verification and generation accuracy.
To further compare the verification methods, we calculate the Pearson correlation coefficient between the outputs of the proxy utility ū the gaps, shown in Figure 7. We Qwen-1.5 7B an example and defer full results to Figs. 18 and 19. We observe that the correlations between ū are generally low, suggesting potential benefits in combining different verification methods. Notably, the correlation between MC verifications and the correlation between CoT verifications are generally the highest, and larger models tend to have a higher correlation between the gaps. Surprisingly, the gaps of any verification method do not positively correlate with generation accuracy, reinforcing the idea that the relative gap may be a more appropriate metric for measuring self-improvement capability.
6.3 IMPROVEMENT VIA ENSEMBLE
The non-overlap property of different verification methods suggests the potential for enhanced verification performance through their combination. We again focus on the rejection sampling setup. We employ logical AND operation, keeping samples only they pass all verification filters. The results are deferred to Appendix E.7. We observe that combining any verifications with non-trivial gaps improves the verification performance (with the exception of CoT for 0.5B model with near O gap). This promising outcome indicates that despite functional similarities, different verification mechanisms can still combined to increase self-improvement efficacy. The consistent improvements across different model sizes also suggest that strategies developed using smaller models can be effectively applied to larger ones, all verifications are valid.
Takeaway on verification mechanisms
fine-grained study on verification reveals several implications for practice:
The distribution of the gaps and optimal verification threshold typically generalize across models.
Verification Consistency:
Verification Distinction: Despite functional similarities, the outputs and gaps of verifica- tion methods show non-trivial differences among each other.
Ensemble Heuristic: Simple verification ensemble heuristics can improve performance.
The consistency result suggests that configurations from smaller models can be applied to larger ones to avoid the costs associated with tuning big models. The discovery that simple ensemble techniques can enhance performance highlights the potential for more sophisticated algorithms to advance self-improvement strategies further.
CONCLUSION AND DISCUSSION
In this paper, we conduct comprehensive and controlled studies on the LLM ieff-improvement framework through multiple model families, tasks, and verification mechanisms. We structure the mathematical framework of the self-improvement process and pinpoint the generation- veriiiiionn gap as a critical metric. Our results reveal several intriguing properties such as the scaling properties of the relative gap, saturation of iterative self-impprvemmnt and enhancement of verification via ensemble methods. These insights are likely to have practical implications for improving pre- traininn post-training, and test- time inference. Additionally, our research opens several promising avenues for future exploration, and defer the list Appendix A.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGEMENT
The authors are grateful to Audrey Huang and Akshay Krishnamurthy for constructive discussion. The authors thank Cyril Zhang for helpful feedback the draft. The authors thank Riccardo Savorgnan, Sohrab Andaz and other members of the Amazon SCOT-RL team for discussion and infrastructure support. HZ is supported by an Eric and Susan Dunn Graduate Fellowship. SK acknowledges the support of the Chan Zuckerberg Initiative Foundation for establishing the Kempner Institute for the Study of Natural and Artificial Intelligence, the Office Naval Research under award N00014-22-1- 22-1-7777 and the National Science Foundation under award #IIS 2229881.
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, lige Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et Gpt-4 technical report. arXiv preprint 1rXxi:23003.87774 2023.
Bo Adler, Niket Agarwal, Ashwath Aithal, Dong H Anh, Pallab Bhattacharya, Annika Brundyn, Jared Casper, Bryan Catanzaro, Sharon Clay, Jonathan Cohen, al. Nemotron-4 340b technical report. arXiv preprint 1rXiv:2406.11704, 2024.
Sina Alemohammad, Josue Casco-Rodrigues Lorenzo Luzi, Ahmed Imtiaz Humayun, Hossein Babaei, Daniel LeJeune, Ali Siahkoohi, and Richard G Baraniuk. Self-consuming generative models go mad. arXiv preprint
2023.
Zachary Ankner, Mansheej Paul, Brandon Cui, Jonathan D Chang, and Prithviraj Ammanabrolu. Critique-out-loud reward models. XrXiv preprint arXiv:2408. 1791, 2024.
Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang. Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. Qwen technical report. arXiv preprint arXiv:2309. 16609, 2023.
Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, al. Constitutional Harmlessness from ai feedback. arXiv preprini
2022
Hritik Bansal, Arian Hosseini, Rishabh Agarwal, Vinh Q Tran, and Mehran Kazemi. Smaller, weaker, yet better: Training IIm reasoners via compute-optimal sampling. arXiv preprint (tti::2008.16333 2024.
Quentin Bertrand, Avishek Joey Bose, Alexandre Duplessis, Marco Jiralerspong, and Gauthier Gidel. On the stability of iterative retraining of generative models on their own data, arXiv preprint arXiv:2310.0044 0040999,, 2023.
Stella Biderman, Hailey Schoelkopf, Quentin Gregory Anthony, Herbie Bradley, Kyle O'Brien, Fric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, et al. Pythia: A suite for analyzing large language models across training and scaling. In International Conference on Machine Learning, pp. 2397-2430. PMLR, 2023.
Martin Briesch, Dominik Sobania, and Franz Rothlauf. Large language models suffer from their own output: An analysis of the self consuming training loop. arXiv preprint arXiv:2311.16822 2023.
Bradley Brown, Jordan Juravsky, Ryan Ehrlich, Ronald Clark, Quoc V Le, Christopher Ré, and Azalia Mirhoseini. Large language monkeys: Scaling inference compute with repeated sampling. arXiv preprint arXiv:2407.21787, 2024.
Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, et al. Sparks of artificial general intelligence: Early experiments with gpt-4. arXiv preprint wrXiv:2303.12712. 2023.
Jonathan D Chang, Kiante Brantley, Rajkumar Ramamurthy, Dipendra Misra, and Wen Sun. Learning to generate better than your 1mm. arXiv preprint arXiv:2306. /816,, 2023.
Xinyun Chen, Maxwell Lin, Nathanael Schärli, and Denny Zhou. Teaching large language models to self-debug. arXiv preprint arXiv:2304.03: /28 2023.

---

Published as a conference paper at ICLR 2025
Zixiang Chen, Yihe Deng, Huizhuo Yuan, Kaixuan Ji, and Quanquan Gu. Self-play fine-tuning converts weak language models to strong language models. arXiv preprint orXiv:2401.01333 2024.
Cheng-Han Chiang and Hung-yi Lee. Can large language models be an alternative to human evaluations? arXiv preprint arXiv:2305.0190 01937, 2023.
Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng. Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E Gonzalez, et al. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality. See https://vicuna. lmsys. org (accessed 14 April 2023), 2(3);6, 2023.
Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. Advances in neural information processing systems, 30, 2017.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14106 14168, 2021.
Michael Collins and Terry Koo. Discriminative reranking for natural language parsing. Computational Linguistics, 31(1):25-70. 2005.
Mehul Damani, Idan Shenfeld, Andi Peng, Andreea Bobu, and Jacob Andreas. Learning how hard to think: Input-adaptive allocation of Im computation. arXiv preprint urXiv:2410.04707, 2024.
Elvis Dohmatob, Yunzhen Feng, and Julia Kempe. Model collapse demystified: The case of regression. arXiv preprint arXiv:2402.077722 2024.
Ricardo Dominguez-Olmedo. Florian E Dorner, and Moritz Hardt. Training on the test task confounds evaluation and emergence. arXiv preprint arXiv:2407.07890 2024.
Hanze Dong, Wei Xiong, Bo Pang, Haoxiang Wang. Han Zhao, Yingbo Zhou, Nan Jiang, Doyen Sahoo, Caiming Xiong, and Tong Zhang. Rlhf workflow: From reward modeling online rlhf. arXiv preprint 20t.::220000086666 2024.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.2178. 2024.
Yann Dubois, Balázs Galambosi, Percy Liang, and Tatsunori B Hashimoto. Length-comtrooledd alpacaeval: A simple way to debias automatic evaluators. arXiv preprint arXiv:2404.04473 2024.
Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding. Jeffrey Hsu, Alain Le Noac'h h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. A framework for few-shot language model evaluation, 07 2024. URL htttps://eeendo...... www.ad/.../rrrrrrro/oooo/ooooose 12608602..
Matthias Gerstgrasser, Rylan Schaeffer, Apratim Dey, Rafael Rafailov, Henry Sleight, John Hughes, Tomasz Korbak, Rajashree Agrawal, Dhruv Pai, Andrey Gromov, et al. Is model collapse in- evitable? breaking the curse of recursion by accumulating real and synthetic data. arXiv preprint arXiv:2404.01413, 2024.
Nate Gillman, Michael Freeman, Daksh Aggarwal, Chia-Hong Hsu, Calvin Luo, Yonglong Tian, and Chen Sun. Self-correcting self-consuming loops for generative model training. arXiv preprint arXiv:2402.070887 2024.
Suriya Gunasekar, Yi Zhang. Jyoti Aneja, Caio César Teodoro Mendes, Allie Del Giomo, Sivakanth Gopi, Mojan Javaheripi, Piero Kauffmann, Gustavo de Rosa, Olli Saarikivi, et al. Textbooks are all you need. arXiv preprint arXiv:2306. 1644, 2023.
12

---

Published as a conference paper at ICLR 2025
Ryuichiro Hataya, Han Bao, and Hiromi Arai. Will large-scat scale generative models corrupt future datasets? In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 20555-20565 2023.
Alex Havrilla, Yuqing Du, Sharath Chandra Raparthy, Christoforos Nalmpantis, Jane Dwivedi- Yu, Maksym Zhuravinskyi, Eric Hambro, Sainbayar Sukhbaatar, and Roberta Raileanu. Teaching large language models to reason with reinforcement learning. arXiv preprint arXiv:2403.0444 04642, 2024.
Michael Haythorpe. Reducing the generalised sudoku problem to the hamiltonian cycle problem. AKCE International Journal of Graphs and Combinatorics, (3(3):272-288) 2016.
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinbardt. Measuring mathematical problem solving with the math dataset. arXiv preprint arXiv:2103. 03874, 2021.
Danny Hemandez, Jared Kaplan, Tom Henighan, and Sam McCandlish. Scaling laws for transfer. arXiv preprint ar1iv:2102.012220 2021.
Audrey Huang, Adam Block, Dylan Foster. Dhruv Rohatgi, Cyril Zhang, Max Simchowitz, Jor- dan T. Ash, and Akshay Krishnamurthy. Self-improvemenn in language models: The sharpening mechanism. In The Thirteenth International Conference on Learning Representations, 2025. URL
https://openrewe wwtpviiwwweee/orrreeee wwwttorumm//eeeee wtwp://wewwefoouiiiiiiaaaaaaweee id=Naaukwcc9..
Jiaxin Huang, Shixiang Shane Gu, Le Hou, Yuexin Wu, Xuezhi Wang, Hongkun Yu, and Jiawei Han. Large language models can self-improve. arXiv preprint arXiv:22 10. 16100,11000000 2022.
Liang Huang and David Chiang. Forest rescoring: Faster decoding with integrated language models. In Proceedings of the 45th annual meeting of the association of computational linguistics, pp- 151, 2007.
Binyuan Hui, Jian Yang, Zeyu Cui, Jiaxi Yang, Dayiheng Liu, Lei Zhang, Tianyu Liu, Jiajun Zhang, Bowen Yu, Kai Dang, et al. Qwen2. 5-coder technical report. arXiv preprint arXiv:2409. /2 186. 2024.
Berivan Isik, Natalia Ponomareva, Hussein Hazimeh, Dimitris Paparas, Sergei Vassilvitskii, and Sanmi Koyejo. Scaling laws for downstream task performance of large language models. arXiv preprint arXiv:2402.044/11 2024.
Dongwei Jiang, Jingyu Zhang, Orion Weller, Nathaniel Weir, Benjamin Van Durme, and Daniel Khashabi. Self-[in] correct: Lims struggle with refining self-generated responses. arXiv preprint arXiv:2404.04298, 2024.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXhv:2001.0836/ 2020.
Kalpesh Krishna, Yapei Chang. John Wieting, and Mohit Iyyer. Rankgen: Improving text generation with large ranking models. arXiv preprini arXiv:2205.0971 09926.. 2022.
Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, et al. Natural questions: a benchmark for question answering research. Transactions of the Association for Computational Linguistics, 7:453-466, 2019.
Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the 29th Symposium on Operating Systems Principles, pp. 1111-626,200 2023.
Haoran Li, Qingxiu Dong, Zhengyang Tang, Chaojun Wang. Xingxing Zhang, Haoyang Huang, Shao- han Huang, Xiaolong Huang, Zeqiang Huang, Dongdong Zhang, et al. Synthetic data (almost) from scratch: Generalized instruction tuning for language models. arXiv preprint arXiv:2402. 13064, 2024.
13

---

Published as a conference paper at ICLR 2025
Yuanzhi Li, Sébastien Bubeck, Ronen Eldan, Allie Del Giorno, Suriya Gunasekar, and Yin Tat Lee. Textbooks are all you need ii: phi-1.5 technical report. arXiv preprint MXXi:233990054633 2023.
Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, Rémi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, al. Competition- code generation with alphacode. Science, 378(6624) 1092-1097, 2022.
Yiming Liang, Ge Zhang, Xingwei Qu, Tianyu Zheng, Jiawei Guo, Xinrun Du, Zhenzhu Yang, Jiaheng Liu, Chenghua Lin, Lei Ma, et al. I-sheep: Self-alignment of Ilm from scratch through an iterative self-enhancement paradigm. arXiv preprint arXiv:2408. 240002000007, 2024.
Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let's verify step by step. arXiv preprint arXiv:2305.20050, 2023.
Tianqi Liu, Yao Zhao, Rishabh Joshi, Misha Khalman, Mohammad Salch, Peter J Liu, and Jialu Liu. Statistical rejection sampling improves preference optimization. arXiv preprint arXiv:2309. 006006677 2023.
Liangchen Luo, Yinxiao Liu, Rosanne Liu, Samrat Phatale, Harsh Lara, Yunxuan Li, Lei Shu, Yun Zhu, Lei Meng, Jiao Sun, et al. Improve mathematical reasoning in language models by automated process supervision. arXiv preprini arXiv:2406.0653 2024.
Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, et al. Self-refine: Iterative refinement with self-feedback. Advances in Neural Information Processing Systems, 36, 2024.
Gonzalo Martínez, Lauren Watson, Pedro Reviriego, José Alberto Hernández, Marc Juarez, and Rik Sarkar. Combining generative artificial intelligence (ai) and the internet: Heading towards evolution or degradation? arXiv preprint arXiv:2303.01255, 2023.
Yu Meng. Jiaxin Huang, Yu Zhang, and Jiawei Han. Generating training data with language models: Towards zero-shot language understanding. Advances in Neural Information Processing Systems, 35:462-477, 2022.
Arkadij Semenovič Nemirovskij and David Borisovich Yudin. Problem complexity and method efficiency in optimization. 1983.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730- 27744, 2022.
Richard Yuanzhe Pang. Weizhe Yuan, Kyunghyun Cho, He He, Sainbayar Sukhbaatar, and Jason Weston. Iterative reasoning preference optimization. arXiv preprint arXiv:2404. /9733, 2024.
Yangjun Ruan, Chris J Maddison, and Tatsunori Hashimoto. Observational scaling laws and the predictability of language model performance. arXiv preprint arXiv:2405. 10938, 2024.
Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: Language agents with verbal reinforcement learning. Advances in Neural Information Processing Systems, 36, 2024.
Avi Singh, John D Co-Reyes, Rishabh Agarwal, Ankesh Anand, Piyush Patil, Peter Liu, James Harrison, Jachoon Lee, Kelvin Xu, Aaron Parisi, et al. Beyond human data: Scaling self-training for problem-solving with language models. arXiv preprint arXiv:2312.06538 2023.
Charlie Snell, Jachoon Lee, Kelvin Xu, and Aviral Kumar. Scaling 11m test-time compute optimally can be more effective than scaling model parameters. arXiv preprint arXiv:2408.03311, 2024.
Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. Learning to summarize with human feedback. Advances in Neural Information Processing Systems, 33:3008-3021, 2020.
14

---

Published as a conference paper at ICLR 2025
Tianxiang Sun, Xiaotian Zhang, Zhengfu He, Peng Li, Qinyuan Cheng, Hang Yan, Xiangyang Liu, Yunfan Shao, Qiong Tang, Xingjian Zhao, et al. Moss: Training conversational language models from synthetic data, arXiv preprint arXhv:2307.15020 7:3, 2023.
Rohan Taori and Tatsunori Hashimoto. Data feedback loops: Model-driven amplification of dataset biases. In International Conference on Machine Learning, pp. 11883-33920 PMLR, 2023,
Rohan Taori, Ishaan Guirajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B Hashimoto. Stanford alpaca: An -osllwiion-following Ilama model, 2023.
Gemini Team, Rohan Anil, Sehastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Sch,lkwyk, Andrew M Dai, Anja Hauth, et al. Gemini: family of highly capable multimodal models. arXiv preprint arXiv:2312. 18005 2023.
Qwen Team. Qwen2. 5: A party of foundation models, 2024.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288 2023.
Peiyi Wang, Lei Li, Zhihong Shao, Runxin Xu, Damai Dai, Yifei Li, Deli Chen, Yu Wu, and Zhifang Sui. Math-shepherd: Verify and reinforce lims step-by-step without human annotations. In Proceedings of the 62nd Annuul Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pp. 9426-9439, 2024a.
Tianlu Wang, Ilia Kulikov, Olga Golovneva, Ping Yu, Weizhe Yuan, Jane Dwivedi- Yu, Richard Yuanzhe Pang. Maryam Fazel-Zarandi, Jason Weston, and Xian Li. Self-taught evaluators. arXiv preprint arXiv:2408.0260 2024b.
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang. Aakanksha Chowdh- ery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. arXiv preprint arXiv:2203. 1111, 2022a.
Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A Smith, Daniel Khashabi, and Hannaneh Hajishirzi. Self-instruct: Aligning language models with self- ref-raenedted instructions. arXiv preprini wxXi:22222.10566 2022b.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thoow of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 15:24824-24837 2022.
Yuxiang Wei, Zhe Wang, Jiawei Liu, Yifeng Ding, and Lingming Zhang. Magicoder: Empowering code generation with oss instruct. In Forty-first International Conference Machine Learning, 2024.
Sean Welleck, Amanda Bertsch, Matthew Finlayson. Hailey Schoelkopf, Alex Xie, Graham Neubig, Ilia Kulikov, and Zaid Harchaoui. From decoding to meta-generatton: Inference-time algorithms for large language models. arXiv preprint arXiv:2406. 16838, 2024.
Yangzhen Wu, Zhiqing Sun, Shanda Li, Sean Welleck, and Yiming Yang. An empirical analy- sis of compute-optimal inference for problem-solving with language models. arXiv preprint arXiv:2408.0072. 2024.
Wenda Xu, Guanglei Zhu, Xuandong Zhao, Liangmmng Pan, Lei Li, and William Wang. Pride and prejudice: Llm amplifies beaf-bias in self-refinement. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 15474-15492, 2024.
An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, et al. Qwen2 technical report. arXiv preprint arXiv:2407.1066 10671, 2024.
15

---

Published as a conference paper at ICLR 2025
Asaf Yehudai, Boaz Carmeli, Yosi Mass, Ofir Arviv, Nathaniel Mills, Assaf Toledo, Eyal Shnarch, and Leshem Choshen, Genie: Achieving human parity in content-groundee datasets generation. arXiv preprint arXhv:240.114667, 2024.
Alex Young, Bei Chen, Chao Li, Chengen Huang, Ge Zhang, Guanwei Zhang, Heng Li, Jiangcheng Zhu, Jianqun Chen, Jing Chang, et al. Yi: Open foundation models by 01. ai. arXiv preprint arXiv:2403.04662, 2024.
Weizhe Yuan, Richard Yuanzhe Pang, Kyunghyun Cho, Sainbayar Sukhbaatar, Jing Xu, and Jason Weston. Self-rewarding language models. arXiv preprint urXiv22401.10020, 2024.
Eric Zelikman, Yuhuai Wu, Jesse Mu, and Noah Goodman. Star: Bootstrapping reasoning with reasoning. Advances in Neural Information Processing Systems, 35:15476-15488, 2022.
Eric Zelikman, Georges Harik, Yijia Shao, Varuna Jayasiri, Nick Haber, and Noah D Goodman. Quiet-star: Language models can teach themselves to think before speaking. arXiv preprint arXiv:2403.09629, 2024.
Di Zhang, Jiatong Li, Xiaoshui Huang, Dongzhan Zhou, Yuqiang Li, and Wanli Ouyang. Accessing gpt-4 level mathematical olympiad solutions via monte carlo tree self-refine with llama-3 8b. arXiv preprint arXiv:2406.073: 073.9394, 2024a.
Lunjun Zhang, Arian Hosseini, Hritik Bansal, Mehran Kazemi, Aviral Kumar, and Rishabh Agarwal. Generative verifiers: Reward modeling as next-token prediction, 2024b. URL https://arxiv. org/abs/244 2408. 15240.
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. Opt: Open pre-trained transformer language models. arXiv preprint (tt::22055000066 2022.
Tong Zhang. From e-entropy to kl-entropy: Analysis of minimum information complexity density estimation. The Annals of Statistics, pp. 2180-2210, 2006.
Xuanchang Zhang, Wei Xiong, Lichang Chen, Tianyi Zhou, Heng Huang, and Tong Zhang. From lists to emojis: How format bias affects model alignment. arXiv preprint arXiv:2409. 117044 2024c.
Yao Zhao, Rishabh Joshi, Tianqi Liu, Misha Khalman, Mohammad Salch, and Peter J Liu. Slic-hf: Sequence likelihood calibration with human feedback. arXiv preprint arXiv:2305. 10425, 2023.
Zihao Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. Calibrate before use: Improving few-shot performance of language models. In International conference machine learning, pp. 12697-12706. PMLR, 2021.
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao WU, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging ilm-as-aajuudge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 36:46595-46623, 2023.
Chunting Zhou, Pengfei Liu, Puxin Xu, Srinivasan Iyer, Jiao Sun, Yuning Mao. Xuezhe Ma, Avia Efrat, Ping Yu, Lili Yu, al. Lima: Less is more for alignment. Advances in Neural Information Processing Systems, 36, 2024.
Banghua Zhu, Evan Frick, Tianhao Wu, Hanlin Zhu, and Jiantao Jiao. Starling-7bs 7b: Improving Ilm helpfulness & harmlessness with rlaif, 2023.
16

---

Published as a conference paper at ICLR 2025
A FUTURE DIRECTIONS
In this section we list several promising avenues for future exploration:
While our scaling analysis is primarily observational (Ruan et al., 2024), pursuing a more extensive scaling law study (Kaplan et al., 2020) based on our preliminary findings could provide robust empirical guidelines.
Our results hint at an inference-time scaling law (Wu et al., 2024) is possible for self-improvement (or with pross-improvement (c.r. Section 4.2)). Identifying compute-optimal methods for improvntroven across different tasks remains a critical challenge.
The decline in the effective diversity of generations during iterative self-improvement presents a sig- nificant obstacle. Developing strategies to mitigate this issue offers considerable empirical benefits.
The distinct non-overlap property of verification mechanisms, despite their functional similarities, suggests that combining compositional verification could significantly enhance self- seff-improvement. Exploring this potential further could yield fruitful results.
B RELATED WORK
Synthetic Data and Self- Trainirainnn. Training LLMs with mixture of "real" data (generated by human) and synthetic data has been the standard protocol nowadays given the limited number of human data and extensive amount of data required as we scale up the LLM training. Initial studies generated synthetic data from more powerful models (Gunasekar et al., 2023; Li et al., 2023; Team et al., 2023; Sun et al., 2023; Taori et al., 2023; Zhu et al., 2023; Wei et al., 2024), while recent approaches involve models training on their own outputs (Achiam et al., 2023; Adler et al., 2024; Dubey et al., 2024; Yang al., 2024; Hui et al., 2024).
On the theoretical front, extensive research has explored the phenomenon of model collapse during self-training and strategies to counter this degenerate behavior (Hataya et al., 2023; Martínez et al., 2023; Bertrand et al., 2023; Briesch et al., 2023; Taori & Hashimoto, 2023; Alemohammad et al., 2023; Dohmatob et al., 2024; Gillman al., 2024).
LLM Self-improvement. One of the most effective strategies to prevent model collapse during self-training the use of a reliable verifier (Gillman al., 2024). In the absence of additional resources like labeled data or external oracle, models can utilize their own verification capabilities. This is particularly effective if the model is more proficient at verification than generation. Numerous studies have proposed variations of self-improvement algorithms based on this principle, resulting in significant practical achievements (Zelikman et al., 2022; Wang et al., 2022b; Huang et al., 2022: Singh et al., 2023; Chen et al., 2023; Madaan et al., 2024; Xu et al., 2024; Yuan et al., 2024; Liang et al., 2024; Wang et al., 2024b; Shinn et al., 2024; Zelikman et al., 2024; Chen et al., 2024; Jiang et al., 2024). Previous research, however, often relied additional data to enhance verification, used surrogate metrics for improvement, or limited their focus to small number of models. In this work, instead of proposing any new algorithm, we aim rigorously analyze the self-improvement phenomenon in a controlled, comprchensive manner. On the theoretical front, Huang et al. (2025) studied celf-improvement using trajectory probability as score, and proved the optimality of SFT (rejection sampling) with certain coverage conditions.
Improving Test-time Inference with Additional Computation. Recent research has demonstrated that the performance of models can be enhanced by allocating more computational resources to inference (Welleck et al., 2024; Damani et al., 2024). This typically leverages the observation that L.MSs can make diverse generations, and with a small probability it can generate high-quality responses (Li al., 2022; Brown et al., 2024; Bansal et al., 2024). Thus with oracle verifier, or with training high-quality reward model, model performance can be improved by simply making multiple generations and selecting the best ones according to the oracle or the reward model (Cobbe al., 2021). There are also works training process- based reward models (Lightman et al., 2023) to improve the model's I'' reasoning results (Luo et al., 2024; Wang et al., 2024a; Zhang et al., 2024a).
Concurrently there also works the test-time scaling law which investigates the computational trade-off between the model size (which determines the number of generations given a computation budget) and final accuracy combined with reward model oracle (Wu et al., 2024; Snell et al., 2024).
17

---

Published as a conference paper at ICLR 2025
The results provide the compute-optimal solution for test-time inferencing with fixed compute budget and a fixed verifier. We believe better understanding of improvemrneenttt can also lead to test-time scaling law without external verifier.
LLM-as-a-Juddee LLM-as-a-judge refers to using an LLM to verify the generation of some other (or the same) LLM (Chiang al., 2023; Zheng et al., 2023; Bubeck al., 2023; Chiang & Lee, 2023; Zhou et al., 2024). Recently the same idea has also been applied to train a generative reward model (Ankner et al., 2024; Zhang et al., 2024b). Having model that verify its own generation is one of the key components of self-improvement, and in this work, we perform a fine-grained study on various types of LLM verification mechanisms.
Reranking Algorithms. The self-imppovemmnt framework we study in this paper relies reweight- ing the generation distribution. Prior self-improvement, the reranking algorithm has already been widely applied in various NLP applications (Collins & Koo, 2005; Huang & Chiang, 2007; Stiennon al., 2020; Cobbe et al., 2021; Krishna et al., 2022; Lightman et al., 2023).
C VERIFICATION MECHANISMS
In this section, we provide more complete description of the verification mechanism we use throughout the paper.
Multiple Choice (MC): Multiple choice verification asks the LM to label responses as "Correct" and "Incorrect". Let pr a verification prompt and denote ( (a, the verifier generating a single token t+,t_ representing the word "Correct" and "Incorrect" respectively. The score uses the logits from these tokens to find the probability of "Correct" conditioned on the next token being "Correct" or "Incorrect":
utility derived from
f(t x. y, prom
@7 v) = f(tt z, y, prom
f(t
x, y. promme)
Chain of Thought (CoT): CoT verification asks the LM to score responses and to provide the CoT. Denote by S C R the set of verification scores and by prom a verification prompt. We can define a utility
af v) == E,, prom [s()]]
where is the verification CoT, and s(z) is the score extracted from the COT. In our experiments consider two versions, CoT-Score with S [10] and CoT-Binary with S = {0. 1).
Tournament (To): The tournament verification does not directly fit the utility framework described Section 2. Rather, this verification procedure involves comparisons of batch of generations to provide modified distribution" Given comparison prompt prom we perform tournament- style elimination over a batch of 2" generations by comparing disjoint pairs in each round until a round the set ()(k) contains 2 remaining generations. These are split into disjoint pairs (y), y;) E ((k), Each pair is compared using the prompt prom and the verifier's output E A, B indicates the preferred generation:
single generation remains. Let y)() = !/1 1/......... U2- be the initial set of generations.
Uu if f( yi Uj. prom ) = A, by
Uwin
if f( yss Mj, prom = B.
where Ywin is the winner of the pairwise comparison and advances to the next round. After each round k, the set of winners (++++) contains half the number of generations. This process is repeated until A r, leaving only one generation, the lone element in y".
"This batch-style distribution weighting also applies strategies like top-k wherein we take the highest utility generations for particular question.
18

---

Published as a conference paper at ICLR 2025
D ABLATION RESULTS
D.1
SAMPLING TEMPERATURE
Qwen-1.5 (t=1)
Qwen-1.5
Qwen-1.5 (t=0.5)
Qwen-1.5 (t=0.5)
5.0
Relative
Relative
-25
-10
-10
-5.0
loge flops (1e21)
MC
CoT.B
CaT-S
Figure 8: With proper verification method (e.g., CoT-S), with two different temperatures within reasonable range, the relative generation-verification gap (Definition 2.2) still scales monotonically with respect to the pre pve-trainng flops. We conjecture that in this case, the relative gap is linear with respect the log the pre-training flops. Again, we do not observe scaling phenomenon for generation-verificttoo gap.
In this section, we tested the robustness our result with different sampling parameters. Specifically, tried temperature 1 and 0.5 for both generation and verification (the default temperature is 0.7). We repeat the scaling experiment on GSM8K with Qwen- 1.5 model family (which contains the number of models), and we record the scaling of gap and relative gap in Figure 8. We observe the same scaling phenomena in both temperatures that the relative gap scales monotonically with the pretrain flops and is almost linear with respect to the log of the pretrain flops. The only exception is the 14B model with temperature 1, but we believe this is due to higher temperature introducing more noise and more samples of generation or verification will recover the perfect trend. We also observe that the improvement in temperature 0.5 setting is less prominent, and we believe this is due to the reduced diversity in the generation with a lower temperature.
D.2 INSTRUCT MODELS
Qwen-1.5-Che (GSM8K)
Uama-2-Chat (GSM8K)
20 Qwen-2 5.ans2..-lnstrrcct.c (MATH)
Relative Gap (%)
20
10
0.
-10
loge flops (1e21)
Qwen-1.5-Chat (GSM8K)
Uama-2-Chat (GSM8K)
Qwen-2 5-Instruct (MATH)
2.0
10
Gap
1.0
-10
log. flops (1e21)
MC
COT-B
CoT-S
Figure 9: Instruct models do not always have the scaling property.
In this section, we present our results on examining the scaling property of the instruct models. Specifically, we tested Qwen- 1.5-Chat and Llama-2-Chat model family on GSM8K and Qwen-2.5- Instruct model family on MATH. We observe that Qwen- 1.5em-155-CC 1.5-Chat models do not have the scaling property. In fact, even the accuracy of the models do not scale monotonically with respect to the
19

---

Published as a conference paper at ICLR 2025
model size (and our accuracy results match the reported number in Bai et al. (2023)). Similarly, does not observe the scaling property on Qwen-2.5 Instructtt model family as well. On the other hand, Llama-2-Che models demonstrate the scaling property but remark that the curve fit is only based on three models.
-owst-t5-Inssruuct
In sense, this result is expected. As discussed in the earlier sections, instruct models have more confounders that may not lead to clear conclusion. In addition, some latest models such as Llama 3.1 already have self-improvement component in the finetune process it unclear if additional self-improvemmet signals will be observed. This result justifies our choice of using base models to study self improveeree
D.3 FIXED VERIFIER
Generation Diversity (Qwen-1.55 1.5-7B, Gold) Generation Diversity (Qwen-1.5-78, Ver72B)
100
90
Pass@k (%)
Pass@k (%)
80
Iteration
Iteration
70
Iteration Iteration
70
Iteration
Iteration
60
60
Iteration
Iteration
20 2 2
2
24
2
26 2
2
2
2
2
2
2
2 2'
Number of Generations (k)
Number of Generations (k)
Figure 10: The change of effective generation diversity along the iterative self-improvement process for Qwen-1.5 7B model, measured by pass@k for different k. Left: using gold verifier. Right: using Qwen-1.5-72B as the verifier.
To achieve better understanding for the reason that effective diversity degrades during self- improvement, in this section we conduct an ablation experiment where we fix the verifier unchanged. We consider use a more powerful model. Following previous protocols, we use Qwen-1.5-4 1,5-7B model on GSM8K. We record the results in Figure 10.
settings, the first use the gold (ground truth) verifier/label and the second is to
We observe that, with gold labels, there is no degradation of the effective diversity along the iterative training. This because with the gold label, all the incorrect generations will be filtered and thus the model will not concentrate on any wrong answers, which is the most intuitive cause of the degradation of the effective diversity. Meanwhile, the same degradation the effective diversity still happens if we use a more powerful but imperfect verifier because in some questions, the model will concentrate on some wrong answers which the verifier labels correct.
D.4 VARIANCE OF THE VERIFIER
onggrizl Shrthrnee seffiattee Vewn 15-1500
Small Varlance Flctton (0aani
2000
100
EROicIl PEETT
Emoercal Nereer
120
Enenccll Here
Enaiccll Veliace
Emyiica Vertace Wuilibation (Dean)
Variene Vuriiautin (De0
Unsktellorana Warfication (Dean15-12))
13000
1000
1082
500-
4000
480
3900
202
Figure 11: Histograms of variance of generation from Qwen-1.5 model family in GSM8K.
20

---

Published as a conference paper at ICLR 2025
In this section, we examine the effect of single generation sample the evaluation of generation- verification gap. The metric we use the variance of the generation. Specially, we tested Qwen- 1.5-72B model on GSM8K, where the main text. For each question, we generate 8 responses, and for each response, we generate 64 verifications with CoT--B generation. Fix each generation, generation as a Bernoulli distribution with parameter p, where is the probability of the verification ending with "correct". For each generation, we measure the empirical variance (p(1 p)) and present the histogram in Figure 1. Note that the variance ranges from (p = or 0) 0.25 (p = 0.5). We observe that the variance behaves differently for the models, but for bigger models, for most generations, the verification variance is near O.
the same sampling hyperparameter (t = 0.7, 0.9) as
treat the distribution of the CoT-BB
E ADDITIONAL RESULTS
E. ADDITIONAL RESULTS FOR SECTION 4.1
Table 3: Gap GSM-8K for all models. For each verification, "top n²" denotes taking the threshold as the n quantile the proxy utility for each prompt, and = n denotes taking the threshold as n for all prompts. All numbers denote the percentage.
MC
CoT
To
Name
Size
Accuracy
top 0.7
top
0.7
S (
8)
(r
round
0.5B
11.31
-0.62
-1.42
-2.77
-3.68
-0.02
-0.01
0.12
0.44
1.8B
32.81
3.06
2.70
-0.01
0.00
1.58
0.95
1.02
-0.44
4B
50.21
4.85
4.63
0.48
0.89
2.36
2.14
2.27
2.08
Qwen-1.5
7B
53.17
7.18
7.42
6.13
4.07
3.59
63.87
-2.04
-1.06
3.16
32B
70.25
-0,22
-1.98
1.72
1,06
3.07
1.84
1.90
-1.03
72B
74.55
4.84
4.75
2.00
3,95
3.55
2.47
2.91
6.40
0.5B
26.19
4.59
3.39
-6.59
0.21
0.64
0.32
Qwen-2
48.82
1.32
2.78
1.29
7B
76.42
4.69
4,66
3,90
2.17
2.36
1.97
2.08
3.86
72B
81.69
2.39
2.38
0.89
2.12
2.51
2.08
2.45
2.20
11.64
2.33
2.20
0.00
0.25
0.99
Ilama-2
21.57
3.45
3.18
-0.13
0.01
0.97
1.01
1.17
70B
48.42
5.14
4.91
4.77
4.16
3.98
3.44
3.45
5.68
Llama-3
45.66
5.34
5.33
-0.50
0.44
2.67
2.10
2.13
4.08
74.19
4.52
1.35
2.72
2.00
8B
49.31
4.68
4.57
-2.25
-0.24
3.37
2.09
2.05
4.78
Llama-3.1
70B
71.71
6.88
6.77
0.93
3.29
2,71
-0.40
6B
$5.53
4.40
4.27
2.98
-0.97
2.01
1.88
1.95
2.24
Yi-1.5
9B
61.04
7.74
7.50
5.61
5.73
2.32
3.61
3.72
7.83
34B
73.71
6.29
6.23
3.34
4.84
2.49
2.86
2.95
3,41
21

---

Published as a conference paper at ICLR 2025
Table 4: Relative gaps on GSM-8K for all models. For each verilication, "top n" denotes taking the threshold as the n quantile of the proxy utility for each prompt, and т denotes taking the threshold for all prompts. All numbers denote the percentage.
MC
CoT
To
Name
Size
Accuracy
top 0.7
top 0.8
T
0.7
T= 0.8
Bin
S (T
8)
( 9)
round
0.5B
11.31
-0.70
-1.60
-2.56
4.15
-0.03
-0.13
0.04
0.59
1.8B
32.81
4.55
4.01
-0.01
-0.01
4.45
3.10
3.15
-0.39
4B
50.21
9.75
9.31
0.96
1.80
8.22
6.92
8.12
5.24
Qwen-1.5
7B
53.17
15.34
15.84
7.87
13.08
14.06
5,50
-0.35
10.26
63.87
-5.64
-15.54
6.52
-2.94
7.77
8.00
9.02
12.19
32B
70.25
-0.75
5.78
3.57
15.31
8.49
.4.55
72B
74.55
19.01
18.65
7.87
15.52
21.82
14.96
16.97
32.06
0.5B
26.19
6.22
4.59
5.16
-8.93
0.44
1.58
1.79
1.31
Qwen-2
1.5B
48.82
11.90
10.30
2.58
4.45
9.96
10.11
5.26
7B
76,42
19.89
9.18
14.34
17.17
72B
81.69
13.07
13.00
4.87
11.57
20.16
14.34
19.92
17.84
7B
11.64
2.64
0.11
0.00
0.25
0.37
1.91
Llama-2
21.57
4.40
0.02
2.45
2.88
70B
48.42
9.97
9.52
9.25
8.07
13.77
12.22
12.07
18.57
45.66
9.62
-0.90
0.80
8.51
6.16
6.70
12.55
Llama-3
70B
74.19
18.08
4.84
13.24
13.74
11.28
Llama-3.1
8B
49.31
8,97
8.77
-.31
-0.47
11.74
7.02
6.89
14.97
70B
71.71
22.83
22.47
19.91
3.09
16.68
12.65
13.59
-1.03
6B
55.53
9.89
9.60
6.70
-2.17
9.10
6.60
6.69
10.64
Yi-1.5
9B
61.04
19.86
19.25
14.40
14.70
10.09
14.29
14.35
24.78
34B
73.71
23.94
23.68
12.70
18.40
15.02
16.69
16.96
8.89
Qwen-1 (GSMBK)
Owen-2 (GSM8K)
(GSMBK)
5.0
2.5
6
2.5
0.0
4
Gap
-2.5
2
-5.0
5.0
5.5
6.0
loge flops (1e21)
MC
Tournament:
CaT-S
Qwen-155 (MATH)
Qwen-2 (MATH)
Qwen-2.5 2.5 (MATH)
Liama-2 (MATH)
Gap (%6)
2
*
loge flops (1e21)
MC CoT-B
CaT-S
Figure 12:
No clear scaling phenomenon of the gap with respect to the pretrain flops. The x-axis is the log of pretrain flops, and the y-axis is the relative gap. MC denotes Multiple Choice verification with quantile threshold 0.8 (top 0.8). CoT-S denotes CoT-Score verification with global threshold 8, and To denotes Tournament verification with 5 rounds.
22

---

Published as a conference paper at ICLR 2025
E.2
ADDITIONAL RESULTS FOR SECTION 4.3
Qwen-1.5
Model
0.5B
1.8B
4B
7B
14B
32B
72B
Generation Accuracy
6.20
11,40
17.16
21.20
26.83
35.79
39.97
MC (top 0.2)
-0.62
-0.02
-1.20
-1.15
-1.99
-0.43
-0.75
MC (r 0.8)
-0.51
-0.32
-0.72
-1.07
-1.21
-0.10
0.32
Qwen-2
Model
0.5B
1.5B
7B
72B
Generation Accuracy
6.51
13.87
29.09
41.45
MC (top 0.2)
-0,06
0.04
0.79
0.28
MC 0.8)
-0.05
0.02
-0.05
-0.05
Llama-2
Model
7B
13B
70B
Generation Accuracy
25.52
41.00
29.09
MC (top 0.2)
-0.96
0.76
0.30
MC (r 0.8)
-0.81
-2.31
-0.44
Llama-3
Model
8B
70B
Generation Accuracy
30.40
45.59
MC (top 0.2)
0.27
0.32
MC (r 0.8)
-0.23
-0.41
Llama-3.1
Model
8B
70B
Generation Accuracy
27.75
45.13
MC (top 0.2)
0.42
0.44
MC 0.8)
-0.59
-0.37
Yi-1.5
Model
6B
9B
34B
Generation Accuracy
22.82
25.94
35.31
MC (top 0.2)
0.09
0.21
0.24
MC (r 0.8)
-0.07
0.61
0.30
Table 5: Gap Natural Question for all models. With non-trivial generation accuracy, all gaps near 0, indicating that the task is -on-improvable.
E.3 ADDITIONAL RESULTS FOR SECTION 4.4
Qwen-1.3 1.5
Model
0.5B
1.8B
4B
7B
14B
32B
72B
Generation Accuracy
0.43
1.00
0.88
0.95
1.57
2.67
2.02
Gap
0.02
-0.03
-0.15
-0.64
0.22
0.07
1.23
Relative Gap
-0.10
-2.80
-1.39
-3.06
0.67
-1.25
1.14
23

---

Published as a conference paper at ICLR 2025
Qwen-2
Model
0.5B
1.5B
7B
72B
IB-Instruct
72B 72B-Instruct
Generation Accuracy
0.66
0.62
2.09
8.82
2.16
8.15
Gap
-0.09
0.04
-0.07
16.99
0.13
22.97
Relative Gap
-0.15
-0.61
-0.01
20.81
0.20
26.40
Llama-2
Model
13B
70B
Generation Accuracy
0.82
0.89
0.86
Gap
-0.13
-0.63
-0.86
Relative Gap
0.45
-2.02
-3.57
Llama-3
Model
8B
70B
Generation Accuracy
1.39
1.63
Gap
-1.10
-0.84
Relative Gap
-15.4
-36.12
Llama-3 1.1
Model
8B
70B
Generation Accuracy
1.11
1.68
Gap
-0.19
5.5
Relative Gap
-452
6.87
Yi-1.5
Model
6B
9B
34B
Generation Accuracy
0.59
1.29
4.48
Gap
-0.60
0.22
-1.75
Relative Gap
-0.94
0.43
-0.77
Table 6: Generation accuracy, gap and relative gap on Sudoku for all models.
E.4 ADDITIONAL RESULTS FOR SECTION 5
In this section we repeat the experiments in Section on the MATH dataset. We observe similar results in Section 5: iterative celf-improvement saturates (in 4 iterations) with gap diminishing 0. We record a partial result at the top of Figure 13. We also observe the same degradation in effective diversity along the iterative self-improvemmnt and record the result in the bottom of Figure 13.
24

---

Published as a conference paper at ICLR 2025
Iterative Self-improvmment (Qwen-1.5, MATH)
Accuracy (%6)
40
78
Gap (%)
30
Round
Generation Diversity (Qwen-1.5-78, MATH)
Iteration
Pass@k (%)
60
iteration
Iteration
Iteration
40
20
29
2
2
2
2
2 2 2
Number Generations (k)
Figure 13: Top:
The generation accuracy and gap along the iterative self-improvement process for Qwen-1.5 model family with CoT--Binary. The horizontal line on the gap plot denotes 0.5%. Bottom: The change of effective generation diversity along the iterative unffimpprveeent process for Qwen-1.5 7B model, measured by pass@k for different k.
E.5 ADDITIONAL RESULTS FOR SECTION 6.1
Qwen-1.5 (GSMBK)
MC Quantile
CoT-S
Gap (%6
100
-10
0.3
0.5
0.9
0.5
0.7
Threshold
0.58
1.00
78
148
328
720
Qwen-2 (GSMBK)
MC Quantile
CaT-S
5
Gap (%))
2
0
0.1
0.3
0.5
0.7
0.9
0.1
0.3
0.5
0.7
0.9
1
Threshold
1.58
Llama-2 (GSMBK)
MC
MC Quantile
CoTS
Gap (%))
0.9
Threshold
7B
138
70B
25

---

Published as a conference paper at ICLR 2025
Llama-3 (GSM8K)
MC
MC Quantile
CoT-S
Gap (96
3
0.1
0.3
0.5
0.7
0.9
0.1
0.3
Threshold
8B
708
Yi-1.5
Quantile
COT-S
GGap (%6
3
Threshold
68
9B
34B
Figure 14: Change in the gap as we vary the threshold for each verification method. We only present the results for MC with global threshold, quantile threshold and CoT-Score, because CoT-Binary" gap does not change as we change the threshold.
Qwen-1.5 Tournament (GSM-BK)
80
Accuracy (%))
60
(%)
40
-2
20
0
2
Round
0.58
-
1.8B
4B
7B
14B
32B
728
Qwen-2 Tournament (GSM-8K)
Accuracy (%)
(%))
60
-4
Round
0.5B
1.5B
72B
Llama-2 Tournament (GSM-8K)
50
Accuracy (%)
40
Gap (%))
30
-2
20
10
0
2
4
Round
7B
13B
708
26

---

Published as a conference paper at ICLR 2025
Llama-3 Tournament (GSM-8K)
Accuracy cy (%)
70
(%6)
60
-2
50
0
2
Round
8B
70B
Llama-3.1 Tournament (GSM-8K)
70
Accuracy (%6
(%)
50
-4
0
2
4
Round
3B
70B
Yi-1.5 Tournament (GSM-8K)
Accuracy (%)
(%)
2
0
2
4
Round
6B
9B
34B
Figure 15: Change in the generation accuracy of the filtered dataset (left) and gap (right) with respect to the round of tournament. The right figure plots the gap with respect to the accuracy from the previous round instead of the base accuracy.
zie tiweeeeeeeeeeee.... 1.324 moNe5
Peitterie (Dem oheed
#0005
20000
10000
4000
L0000
1000
1500
5000
14.00
12:00
13000
2110/0
1000
1500
4500
5000
Scor DisiiaAt 1ww.m-- 2-00, S9PEC)
502% Dattarton LINI-LL-X CENUA)
$12 WAT.RENITTS 00NE3)
512% DUC. 17-15-8 GONEE)
11500
750/4
82000
Toooic
10000
1200
Figure 16: Score distribution of subset of models. The mode of the score for all models is 10.
27

---

Published as a conference paper at ICLR 2025
E.6
ADDITIONAL RESULTS FOR SECTION 6.2
Distribution Gaps (Qwen-1.5-0.58. GSMBK)
1500
-0.017
2000
-0,037
1500
-0,000
#: -0.000000
Frequency
0.014
1000
1000
1000
500
500
0.5
0.0
0.5
-0.5
0.0
0.5
0.5
0.0
0.55
0.5
Gap
MC-Q
MC
CoT-B
CoT-S
Distribution of Gaps (Qwen-1.5-1.88 GSM8K)
0.017
-0,000
0.016
1000
μ:: 0,010
Frequency
0.150
2000
0.002
0.097
a: 0.055
500
500
250
1000
250
0
0.5
0.0
Gap
MC-Q
MC
CoT-B
CoT-S
Distribution of Gaps (Qwen-1.5-4B, GSM8K)
600
0.046
0,009
0.024
1000
μ:: 0.021
Frequency
400
0.148
1000
0.021
0.053
a:
200
-0.5
0.0
0.5
-0.5
0.0
D.5
-0.5
0.0
0.5
-0.5
0.0
0.5
Gap
MC.Q
MC
CoTBB
CoTS
Distribution of Gaps (Qwen-1.5-7B, GSM8K)
Frequency
400
μu: 0.074
0.158
600
0.061
0.156
1000
μ: 0.041 0.062
1000
μ: 0.018
a: 0.046
400
500
500
200
0
Gap
MC:Q
MC
CoT
CaT-5
Distribution of Gaps (Qwen-1.5-14B, GSMBK)
μi: -0.0600
750
μr -0.011
0.018
1000
μ:: 0.017
Frequency
400
0: 0.217
500
a: 0.179
1000
a: 0.032
a: 0.055
500
200
500
-10
0.5
0.0
0.5
0.5
0.0
0.5
0.5
0.0
0.5
0.5
0.0
0.5
Gap
CoT-5
Distribution of Gaps (Qwen-1.5-328, GSM8K)
-0,022
0.011
1000
0.031
#: 0.018
Frequency
750
0.112
a: 1000
σ:
400
500
500
200
250
0
-0.5
-0.5
0.0
0.5
-0.5
0.0
0.5
-0.5
0.0
0.5
Gap
MC-Q
MC
COT-B
CoT-S
28

---

Published as a conference paper at ICLR 2025
Distribution of Gaps (Qwen-1.5-72B, GSM8K)
μr 0. 0.046
0.040
μ: 0.036
Ji: 0.025
Frequency
600
0.127
750
0.112
0.054 1000
d: 0.040
400
200
250
0
Gap
MC-Q
MC
CoT-B
CoT-S
Distribution of Gaps (Qwen-2-0.5B, GSM8K)
μ:: 0.029
1000
μ:: -0.066
1500
μ:: 0.002 0.009
#:: 0.006
Frequency
0.130
0.262
1000
0.030
1000
500
500
500
11...
-0.5
0.0
0.5
-0.5
0.0
0.5
-0.5
0.0
0.5
-0.5
0.0
0.5
Gap
CoT-B
CoT-5
Distribution of Gaps (Qwen-2-1.58, GSM8K)
Frequency
600
0.013
1500
μ:: 0.013
1000
#::
400
0: 0.154
0.205
a: 0.030
a: 0.060
1000
500
200
500
-0.5
-0.5
0.0
0.5
-0.5
0.0
0.5
-0.5
0.0
0.5
Gap
MC-Q
MC
CoT
CaT.S
Distribution of Gaps (Qwen-2-7B, GSM8K)
Frequency
600
JC 0.047
0.126
1000
Ie 0,022
0.165
1000
μ:: 0.024
0.038
1000
#:: 0.020
a: 0.043
400
500
500
200
11
-0.5
0.5
-0.5
0.0
0.5
-0.5
0.0
0.5
-0.5
0.0
0.5
Gap
MC-Q
CoT-S
Distribution of Gaps (Qwen-2-72B, GSM8K)
Frequency
750
JcC. 0.024
0.095
1000
0.060
0.021
1000
0.050
0.025
1000
#: 0.023
0.051
500
500
250
0.5
0.0
0.0
0.5
0.0
0.5
0.5
0.0
0.5
Gap
MC-Q
MC
CoT-B
CoT-S
Distribution of Gaps (Llama-3-8B, GSM8K)
ic. 0. 0.044
μ: 0,004
μ: 0,027
1000
μ: 0.021
Frequency
400
0.159
2000
0.102
600
a: 0.080
a: 0.053
400
500
200
0
Gap
MC-Q
MC
CoT-B
CoT-S
29

---

Published as a conference paper at ICLR 2025
Distribution of Gaps (Llama-3-70B, GSM8K)
1000
Frequency
400
0.044
0.169
1000
0,014
0.039
1000
#: 0.026
a:
500
500
500
0
Gap
MC-Q
MC
CoT-B
CoT-S
Distribution of Gaps (Llama-3.1-88, GSM8K)
600
IF 0.038
μr -0.002
600
μ:: nan nan
#i: 0.021
Frequency
0.155
2000
0.104
750
a: 0.063
400
400
500
200
1000
-0.5
0.0
0.5
-0.5
0,0
0.5
00.5
0,0
0.5
-0.5
0.0
0.5
Gap
CoT-S
Distribution of Gaps (Llama-3.1-708 GSM8K)
Frequency
800
μz: 0.067
0.169
1500
0.009
0.173
1000
μ:: 0.033
0.072
1000
#:: 0.027
0.055
400
1000
500
200
0
-0.5
-0.5
0.0
0.5
0055
0.0
0.5
-0.5
0.0
0.5
Gap
MC-Q
MC
CoT
CaT.S
Distribution of Gaps (Yi-1.5-68, GSMBK)
Frequency
0.122
2000
a: 0.095
-0.010
750
μ::
a: 0.060
1000
#:
0: 0.043
500
500
200
0
0.5
0.5
0.5
0.0
0.5
0.0
0.5
0.5
0.0
0.5
Gap
MC-Q
MC
CoT-B
CoT-5
Distribution Gaps (Yi-1.5-9B, GSM8K)
600
μ: 0.075
μe 0.057
1500
μ:: 0.023
μ:: 0.036
Frequency
400
0.163
0.195
0.031
a: 0.060
1000
500
200
200
-0.5
0.0
0.5
-0.5
0.0
D.5
-0.5
0.0
0.5
-0.5
0.0
0.5
Gap
MC-Q
MC
COT-B
CoT-5
Distribution of Gaps (Yi-1.5-34B, GSM8K)
μ:: 0. 06.062
μ: 0.048
μ: 0.025
μ: 0.029
Frequency
600
0.136
0.114
1000
0.049
1500
a: 0.047
400
500
1000
500
200
500
0
Gap
MC-Q
MC
CoT-B
CaT-S
Figure 17: The empirical distribution of gaps of each verification method of each model on GSMSK. We cluster gaps in bins of intervals with width 0.005. We label the mean (μ) and standard deviation (a) of each distribution.
30

---

Published as a conference paper at ICLR 2025
Verification Correlation Owen-1 5-0.58
Verification ation Correlation Qwen-115-1.. 1--1..
Verification Correlation Qwen-1. 1.5--11--B
1.0
-0.8
0.8
0.8
cxt4
0.03
1.00
0.01
0.07
1.00
0.03
0.6
0:14
1.00
0.6
0.4
-0.2
CaT-
822
-0.04
0.01
1,00
0.10
0.03
1.00
0.24
0.11
1.00
0.2
Co7-8
CoT46
CoT-5
CoT48
Col--
Verification Comelation
Owan-1.5-78
Verification Correlation Qwen-1.5-140
Verification Correittion Qwen-1.5-322
MC
0.39
0.9
0.8
-0.8
0.8
Co148
0.188
0.6
-0.4
Carts
0.39
0:18
1.00
Calls
0.36
0.30
2.00
-0.4
0.41
0.39
1.00
4.5
MC
Cal-B
CaT4
MC
CoT&
Co1-5
MC
CaT4
Colls
Venifictioon Correlation Qwen 1. 5-728
Verifization Correlation
Swen-2-0.5e
Verificatioon Correletion Qwen-2-1.5
MC
0.42
1.00
1.00
0.8
cai-8
0.6
0.6
0.7
0.6
0.4
cat-5
0.42
0.44
1.00
-0.5
0.21
0.09
1.00
0:20
0.07
1.00
0.2
MC
CoT
CoT'
MC
CoTS
CoT5
CoT Merification Correlatinn
MC
CoT
Verification Correlation
Qwen-2-78
Correlation
Qwen-2-776
Ulama- 3-88
MC
1.00
0.35
0.31
1.00
0.39
0.40
1.00
0.10
0.23
0.8
0.9
0.8
9.8
C3748
0.6
0.4
Gat-4
0.31
0.17
1.00
Cof-5
0.40
0.40
1.00
0.5
0.23
0.06
1.00
0.4
MC
Cof-B
CcT-5
MC
CoT-
Cx7-5
MC
CoT-81
Car-S
Correlation
Ulama-3778B
Correlation
LJama-3.1-88
Verification Correistion Liaia-
2.0
MC
1.00
0.46
0.43
1.00
0.12
0.30
1.00
0.45
0 0.49
4.9
0.8
0.8
0.8
Care
0.30
-0.7
-0.6
4.7
-0.4
0.6
0.5
0.43
0.30
1.00
0.30
0.07
1.00
8.2
0.49
0.35
1.00
0.4
MC
CoT-8
CoTS
MC
CoT-
CO7-5
MC
Ca1-8
CoT-S
Verification
1.0
1.0
1.0
MC
1.00
0.08
0.26
1.00
0.47
0.42
1.00
0.31
0.48
0.8
0.8
0.08
-0.6
Co7.8
-0.6
0.4
cr1-3
0.00
1.00
-.2
Caf-s
0.42
0.30
1.00
0.40
0.21
1.00
MC
CDT
COTS
MC
CoTB
COT.5
MC
CoT
CoT.5
Figure 18: The correlation plot of the output of each verification iz.
31

---

Published as a conference paper at ICLR 2025
Gap Correlation- Qwen-1 15-0.88
Gop Correlation- Sowen-1.-1..oo
Cap Correlation Owen- 1.548
MC MC-2
0.36
-0.18
MC-
1.0
0.36
1,00
00.10
-0.22
0.32
0.8
-0.01
1.00
0.00
0.01
0.03
9.8
0.37
1.00
0.38
0.41
-0.05
0.8
9.6
0.6
Gen Act CaTS cora
0.30
Oen CaRs CaT4
0.09
0.00
1.00
0.11
0.06
3.4
0.31
0.38
1.00
0.37
0.03
0..2
0.02
00.12
0.04
1.00
0.02
0.0
9.2
0:35
10.32
1.00
.0.13
0.03
0.06
0.06
1.00
3.0
-0.06
0.00
-0.03
0.03
1.00
0.0
MC-0
MC
Co78
Arc
MC-Q
CoT-8
CoT-5
Gap Correlation
Qwen-1.5-78
Gap Corelation
Qwen-1.5-148
Gap Correlation Qwen Qwen- 1.5-328
NC NCO
NC MCQ
0.77
1.00
0.33
0.35
0.08
0.58
1.00
0.32
0.32
0.19
0.03
1.00
0.50
0.49
0.09
0.8
-0.6
3.6
0.6
Can Acc Da7S caT4
Gen 0.7.5 Cafe
0.44
-0.4
0.50
1.00
-.2
Gee Colte
0.49
0.81
1.00
-0.21
0.2
0.0
.0.04
-0.08
0..00
-0.10
1.00
-0.17
-0.19
-0.14
-0.08
1.00
-0.14
-0.09
-0.20
-0.21
1.00
0.2
MG-Q
MC
Col-8
CaT-5 Gen Acc
MC-4
CoT-
CoT-5
Gan Auc
Col-di
Call5
Gen Acc
Gap Correlation 1-72-115-720
Gap Correlation Qwen 2-1.58
MC
0.65
4.23
Gon koc CaTs CFT MC MCQ
1.00
0.47
0.06
0.04
0.13
2.00
1.00
0.57
MC
1:00
0.45
-0.11
0.6
0.47
1.00
-0.02
-0.10
-0.41
0.75
0.57
1.00
0.14
0.14
0.14
0.6
6en COTS CoTe
0.51
0.45
1.00
0.86
40.99
0.4
0.45
0.86
0.04
0..0
0.16
1.00
0.10
-0.00
CoTS
0.23
0.24
0.2
-0.25
0.0
0.23
.0.21
10.29
.0.27
1.00
-0.13
-0.41
0:30
-0.15
0.24
0.04
0.05
1.00
MC.0
MC
CoTB
CoTS
Gon Acc
MC.0
MC
CoT
CST
Gan Acc
MC Q
MC
CoTB
CoT
Acc
Gap Comelation Qwan-2-78
Correlation
Qwen-2-728
Gap Correlation
Llama-3-88
MC
1.00
0.61
0.48
0.47
40.22
Gen ACTS CoT8 MC MC-
1.00
0.70
0.56
0.57
-0.22
1.0
1.00
0.04
0.19
0.24
-0.16
Sen Acc CATS cot WC
1.00
0.8
MC
0.04
1.00
0.04
0.01
-0.06
0.8
0.6
0.6
CoTe
0.48
0.21
1.00
0.65
0.23
0.4
0.56
0.65
1.00
0.89
0.29
0.4
0.2
0.2
0.0
0.89
0.0
0.24
0.01
0.18
1.00
0.04
0.0
0.22
10.08
-0.23
-0.19
1.00
-0.
.0.22
+0.26
-0.29
-0.27
1.00
-0.16
-0.06
0.06
0.04
1.00
MC-Q
MC
6678
CoT55 Gen Amc
MC-0
MC
G-TB
CGT-
Gen Arc
MC-Q
MC
Co7-81
ExrT-5 Gen Arc
Comelation
Llama-3.1-88
MC 0
1.00
0.63
0.61
0.56
0.19
MC MC-9
1.00
0.10
0.25
0.34
0.11
1.0
1.00
0.45
0.54
0.55
0.19
1.0
NC
0.10
1.00
0.08
0.02
0. 0.03
0.8
Got Acc CoTs care
Gan Kce cats Ca7e
0.61
0.39
1.00
8.76
0.25
0.54
0.18
1.00
0.79
-0.19
0.4
0.2
0.34
0.02
0.24
1.00
0.02
0.0
-0.19
.0.09
-0.25
-0.23
1.00
-0.2
-0.19
-0.01
-0.19
-0.22
1.00
-02
MC-Q
MC
Eoll8
Col-5
Gen Asc
MC-4
MC
CoT-8
Co7-5
Gen Acc
MC-4
MC
Co1-8
Col-5
Gen Acc
MC MS-Q
1.00
0.09
0.22
0.33
0.10
1.0
MC MC-g
1.00
0.71
0.46
0.47
0.10
1.0
1.00
8.71
0.57
0.59
-0.23
1.0
0.09
1.00
6.01
-0.04
0..3
0.71
1.00
0.35
0.39
40
0.8
come MC
0.8
0.6
0.6
Den Arr Gate Ear4
CoT.8
0.57
0.57
1.00
0.70
-0.16
0.4
0.33
-0.04
0.24
1.00
0.04
Dan ALL Cald
0.47
0.39
0.66
1.00
0.
0.13
0.70
1.00
-02
1.00
Gen
.0.33
.0.4
10.16
10.25
1.00
62
MC.
MC
C
CGT
Acn
MC.
CoT
COT.5
Gen.Acc Acc
MC.0
MC
C6T88
CoT5
Gen
Acc
Figure 19: The correlation plot of the gap from each verification and generation accuracy.
32

---

Published as a conference paper at ICLR 2025
E.7 ADDITIONAL RESULTS FOR SECTION 6.3
Table 7: Relative gaps on GSM-8K for all models. For each verification, "top n' denotes taking the threshold the quantile of the proxy utility for each prompt, and denotes taking the threshold as n for all prompts. All numbers denote the percentage
Name
Size
MC
CoT-B
CoT-S
MC+CoT-B
MC+CoT-S
CoT-B+CoT-s
0.5B
-1.62
-0.07
-0.01
-1.65
-1.36
-0.08
-1.42
1.8B
2.36
1.04
0.95
2.16
2.22
1.18
2.14
4B
4.55
2.26
2.14
4.81
4.77
3.50
4.90
Qwen-1.5
7B
7.46
3,98
1.84
9.02
7.88
4.88
9.12
14B
-6.09
1.79
1.69
-5.73
-5.85
2.45
-5.60
32B
-2.30
3.07
1.84
-1.87
-2.11
3.62
-2.00
72B
4.61
3,43
2.47
5.70
5.25
4.46
6.07
0.5B
3.01
0.18
0.64
3.01
3.13
0.72
3.11
1,5B
5.04
0,86
2.78
5.08
7.68
3.15
7.61
Qwen-2
7B
4.68
2.31
1.97
5,25
4.96
3,40
5,46
72B
2.44
2.50
2.28
3.21
3.13
3.35
3.65
7B
2.17
0.13
0.25
2.21
2.42
0.37
2.44
Llama-2
13B
3.19
0.91
0.97
3.47
3.38
1.76
3,21
70B
4.78
3.73
3.44
5.52
5.61
5.79
4.90
8B
4.98
2.59
2.10
5.88
5.81
4.09
5.83
Llama-3
70B
4.37
3.88
2.59
4.91
4.71
4.45
4.98
8B
4.34
3.50
2.09
4.47
4.57
3.27
2.96
Llama-3.1
70B
6.72
3.29
2.71
7.08
7.03
3.94
7.21
6B
4.27
2.01
1.88
4.83
4.72
3.16
4.95
Yi-1.5
9B
7.50
2.32
3.61
7.79
7.87
4,80
8.11
34B
6.23
2.49
2.86
6.32
6.35
3.76
6.41
33

---

Published as a conference paper at ICLR 2025
F DETAILS ON ITERATIVE SELF-IMPRVVEMEN
Here we present the details for RL update: we use Reasoning Preference Optimization (Pang et al., 2024), which is equivalent the RL update under Bradley-Terry Model with SFT regularization. We set the KL regularization coefficient 3 to be 0.05 and the SFT regularization coefficient to be 1.
Algorithm Iterative self-improvement with rejection sampling.
require Model class F, prompt set X, threshold т.
1: Let fo be the pre-trained model.
2: for = 1, 2...... T do
3: For each X, sample N generations y fi-1( (- 2)) and compute the score
s(x, v) = p (2, y).
4: Get the filtered dataset D; = (xx y)
s(x, y) > J..
5: Update the model through MLE to get fr:
arg max JEF
Σ
log f(y z).
(x.y)e,
Table 8: Hyperparameter for iterative self-improvement.
Minibatch size
64
Learning rate
le-6
Optimizer Gradient step
Adam
2000
Max Sequence Length
2048
Data Type
bf16
34

---

Published as a conference paper at ICLR 2025
G GENERATION AND VERIFICATION PROMPTS
Multiple Choice Verification Prompt (GSM8K) / bennopppee open)
Judge the correctness of the following solution of the problem. Answer with either Correct or Incorrect. Problem: [problem]
Solution: (generation) Judge:
Chain of Thoughts Binary Prompt (GSM8K)
Review the following math problem and the attempted solution and verify the correctness of the attempted solution, with judgement of <correct> <incorrect>, Your judgement should follow each criterion below: The final ANSWER is after the phrase "The answer is ANSWER", and verify if the answer is correct with respect to the problem. If there is no such phrase, treat the answer as incorrect. Each solution contains a derivation before the final answer, check the soundness of the derivation as well. Your final judgement should reflect solely the correctness of the final answer, but if there are issues in the derivation, please mention them in your justification.
Problem: (problem)
Attempted Solution: (generation]
After examining the problem and the attempted solution: Briefly justify your judgement, up to 50 words. Conclude with the judgement using the format: "Correctness: <correct> or <incorrect>".
Remember to assess from the math verifier perspective and be critical and verify carefully. Judgement:
Chain of Thoughts Score Prompt (GSM8K)
Review the following math problem and the attempted solution and give a score from to 10 the attempted solution. The final ANSWER is after the phrase "Final Answer: The final answer ANSWER". Give the answer a if there is no such phrase or ANSWER is wrong, and give the answer 10 if both the answer and the derivation are correct. Problem: [problem}
Attempted Solution: (generation]
After examining the problem and the attepmted solution: Briefly justify your score, up to 50 words. Conclude with the score using the format: "Score: <score>". Remember to assess from the math verifier perspective and be critical and verify carefully.
Judgement:
Tournament Prompt (GSM8K)
Review the following math problem and two attempted solutions. Your task is to determine the better solution between the two. Your judgement should follow each criterion below: The final ANSWER is after the phrase "The answer is ANSWER", and you should always prefer correct answers over incorrect answer. Always prefer solutions with the phrase "The answer is ANSWER" over ones without it. If both answers correct or incorrect, you should prefer the one with better reasonings. Problem: (problem} Solution A: !generation
Solution B: {generation2]
After examining the problem and the attempted solutions: Briefly justify your judgement, up to 50 words. Conclude with the judgement using the format: "Preferred solution: <A> or <B>"
Remember to assess from the math verifier perspective and be critical and verify carefully. Judgement:
35

---

Published as a conference paper at ICLR 2025
Sudoku Generation Prompt
You are a Sudoku solver specialized 4x4 puzzles. You will be given string of 16 digits representing an initial 4x4 Sudoku puzzle, where 0 represents an empty cell. Your task is to solve the puzzle and provide the complete solution.
Rules for solving 4x4 Sudoku:
Each row must contain the numbers 1-4 without repetition.
Each column must contain the numbers 1-4 without repetition.
Each 2x2 quadrant must contain the numbers 1-4 without repetition. The solution must maintain all the initial non-zero numbers from the puzzle in their original position.
Input format: ab cd
ef gh
ij kl
mn op
Where each letter represents single digit (0-4), and 0 indicates empty cell. Your tasks: Analyze the given puzzle.
Fill in all empty cells (Os) with appropriate numbers following Sudoku rules.
Provide your solution the same format as the input, maintaining the original structure. Puzzle: (problem)
Answer:
Sudoku Generation Example
Puzzle:
03 21
00 30
04 00
21 00
Answer: I filled A.1 with 4 to complete Row 1. In Quadrant 1, the missing numbers and were assigned to B1 and B2 based on Column constraints. B4 was set to 4 to complete Row 2. In Quadrant C1 was filled with 3. Row missing numbers 1 and 2 were placed in C3 and C4. D4 was assigned to complete Column 4, and D3 was set to 4 to finish Row 4.
Solution:
43 21
12 34
34 12
21 43
36

---

Published as a conference paper at ICLR 2025
Sudoku Verification Prompt
You are a Sudoku validator for 4x4 puzzles. You will be given two strings of 16 numbers each: the first represents the initial Sudoku puzzle, and the second represents the proposed solution. Your task is to determine if the solution correct. Rules for valid 4x4 Sudoku solution:
Each row must contain the numbers 1-4 without repetition.
Each column must contain the numbers 1-4 without repetition.
Each 2x2 quadrant must contain the numbers 1-4 without repetition.
The solution must maintain all the initial numbers from the puzzle.
Input format:
Question: Input format:
ab cd
ef gh
ij kl
mn op
Where each letter represents a single digit (0-4), and O indicates empty cell. Answer: [same format but there should no 0]
Your task:
Check the solution maintains all initial numbers from the puzzle.
Verify that each row, column, and 2x2 quadrant contains 1-4 without repetition. Respond with "Correct" if the solution is valid, or "Incorrect" it's not. Question: (problem} Answer: (generation] Judgment:
Sudoku Verification Example
problem:
03 00
21 00
00 01
10 43
generation:
34 12
21 31
34 21
12 44
correctness: Checking if the solution maintains initial numbers from the puzzle:
Position (Row 1, Column 2): Puzzle has 3, Solution has 3
Correct
Position (Row 2, Column 1): Puzzle has 2, Solution has 2
Correct
Position (Row 2, Column 2): Puzzle has Solution has 1
Correct
Position (Row 3, Column 4): Puzzle has 1, Solution has 1
Correct
Position (Row 4, Column 1): Puzzle has Solution has 1
Correct
Position (Row 4, Column 3): Puzzle has 4, Solution has 4 Correct
Position (Row 4, Column 4): Puzzle has Solution has 3 Correct All initial numbers are maintained.
Checking the rows:
Row 1: Numbers are 4, 3, 1, (No repetition) Correct
Row Numbers are 2, 3, (Number repeats) Invalid
Row 3: Numbers 3, 4, 2, (No repetition) Correct
Row Numbers are 2, 4, (No repetition) Correct
Checking the columns:
Column I: Numbers are 4, 2, 3, 1 (No repetition) - Correct
Column 2: Numbers are 3, 1, 4, 2 (No repetition) Correct
Column 3: Numbers are 1, 3, 2, 4 (No repetition) Correct
Column 4: Numbers are 1, 1, (Number repeats) Invalid
37

---

Published as a conference paper at ICLR 2025
Checking the 2x2 quadrants:
Quadrant 1 (Rows 1-2, Columns 1-2): Numbers are 4, 3, 2, 1 (No repetition) Correct
Quadrant 2 (Rows 1-2, Columns 3-4): Numbers are 1, 3, (Number 1 repeats) - Invalid
Quadrant 3 (Rows 3-4, Columns 1-2): Numbers are 3, 4, 1, 2 (No repetition) Correct
Quadrant 4 (Rows 3-4, Columns 3-4): Numbers are 2, 1, 4, 3 (No repetition) Correct
Conclusion: There are repetitions in Row 2, Column 4, and Quadrant 2, the solution violates Sudoku rules.
Therefore, the response is: Incorrect
H INFRASTRUCTURE STATEMENT
All our inferences are performed on a cluster of Nvidia A100 40GiB nodes, and our iterative self- improvement training experiments are performed on cluster of Nvidia A100 80GiB nodes.
38


Reviews:
[{"review_id": "mhWFF4MfZ5", "reviewer": "Reviewer_eq2C", "summary": "The paper investigates the concept of self-improvement in Large Language Models (LLMs) during pre-training, post-training, and test-time inference. The authors present a framework where the model verifies its own outputs, refines or reweights data based on this verification, and distills the filtered data. Despite empirical successes, there is a lack of fundamental understanding of this process. The study offers a comprehensive, modular, and controlled exploration of LLM self-improvement, introducing a mathematical formulation governed by the \"generation-verification gap.\" Through experiments across different model families and tasks, the authors identify a scaling phenomenon linked to this gap that scales with model pre-training computational efforts. Additionally, the paper examines conditions under which self-improvement is feasible, proposes an iterative procedure for self-improvement, and suggests methods to enhance its performance. The findings not only deepen the understanding of LLM self-improvement but also open new research possibilities regarding its capabilities and limitations.", "questions": "Please Check Weakness.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The research focus of this paper is commendable, as the GV-Gap aids in determining whether a model possesses self-improvement capabilities at any stage (Pre-/Post-training).\n\n2. The overall layout are well-organized.\n\n3. The experiments appear to be extensive and comprehensive. (Respect to Your Hard working)", "weaknesses": "### Deficiencies in Expression\n\n1. The paper lacks a clear definition of the Generation-Verification Gap, with no illustrative diagram and only complex, confusing formulas cluttered with symbols.\n\n2. The authors fail to clearly outline the overall methodological framework or how the experimental process operates. (I have read the experiments three times and even couldn't know how to evaluate the experimental performance! 🤷🏻‍♀️)\n\n### Limitations in Contribution\n\nThe contributions to the community are relatively weak, as most of the conclusions are already known, such as the \"verification gap scales monotonically with the model pre-training flops\" and the \"Saturation Limit and Cause of Saturation.\" \nAs a purely experimental analysis paper, it does not offer new insights, which is regrettably disappointing.\n\n### Experimental Deficiencies\n1. The experiments are primarily based on GSMK8k, which consists only of elementary-level math problems, making the results less convincing.\n2. In Section 3, the experimental setup has predetermined parameters for lm-evaluation-harness tasks. However, it is unclear why so many hyperparameters would affect the GV-Gap instead of being solely determined by differences between the verifier and the generator.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "AavF1lv7Od", "reviewer": "Reviewer_QqKc", "summary": "This work performs an extensive and exhaustive analysis on self-improvement abilities of large language models. First, authors present related work under which the self-improvements capabilities has been studied and formulate a high-level, unified way of representing such framework. Then they propose a novel way of measuring the self-improvement capability using the so called generation-verification gap (GV-gap). Further, they design an experimental setup and provide multiple takeaways that helps to bring the light onto the cases where self-improvement is likely very limited and hypothesize why this might be happening. Finally, authors provided multiple future directions based on their takeaways and findings.", "questions": "* adding more about the limitations coming from only using the base models would make overall takeaways not weaker, but clearer w.r.t. other recent research in this area.\n* overall I think this will be a very useful analysis work for future self-improvement research direction.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* The proposed measure (gv-gap) is clearly defined with real world examples when it comes to different cases of generators and verifiers. This brings good clarify into the overall presentation.\n* The view on self-improvements via the proposed gap brings a new way to quantify the limits of self-improvements, that is a contribution into this research direction and can be used by the research community in future work.", "weaknesses": "* While the definition of the gv-gap is pretty strict, the real world choice of the utility function making the practical measurements very noisy as authors pointed out in the text (which is great!). They have proposed some ways on improving the verification signal, however, that is a weak point in the entire framework that does not present itself when we just measure the model performance before and after self-improvement steps.\n* While I agree that studying base models makes things much easier to handle in paper-size experimental setup, the absence of any signal about the gap changes with instruct based model weakens the provided takeaways. For instance, recent work showed that smaller (8B) instruct models are self-improving very well while in this work there is an opposite signal from the base smaller models that are seemingly lacking reasoning capabilities even under few shot settings.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "JgVjuxTstK", "reviewer": "Reviewer_gkZY", "summary": "The paper provides a mathematical formulation of LLM self-improvement and defines generation-verification gap to measure the potential of self-improvement. Based on the metric, the work conducts extensive experiments in terms of scaling laws, tasks suitable for self-improvement, iterative procedure and better verification attempts.", "questions": "1. The findings on \"effective diversity of generations\" are interesting. Have you conducted further experiments to clarify the underlying causes? For instance, could it be attributed to the training process itself? (Perhaps an experiment involving training on golden labels could provide insights.) Alternatively, could it stem from noise introduced by self-verification? (An experiment to filter out incorrect samples using golden labels or employing stronger evaluators might help determine if this mitigates the issue.)\n2. From the main results on scaling law (Figure 1), the scaling laws seem to be severely influenced by the verification methods. Why do some methods (like MC, To) not exhibit such scaling laws? And whether the other methods share similar findings (like CoT-B in the paper and some others)?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper presents a math formulation and a new metric for studying self-improvement, which I believe will be beneficial for current studies related to LLM self-improvement.\n2. The experiments are comprehensive, analyzing self-improvement from various perspective.\n3. The experimental findings are generally insightful.\n4. The paper is well-written and easy to follow.", "weaknesses": "1. My main concern is about the experiment setup. The main experiments are limited to one benchmark, GSM8k, in the math task. Despite GSM8k is a classical benchmark for mathematics, it is relatively easy for current cutting-edge LLMs and its benchmarking function may decrease due to potential data contamination. Additionally, the self-improvement performance as well as the findings may differ across different datasets or scenarios. Therefore, I think it would be beneficial for the authors to complement experiments on more datasets and scenarios. \n2. The defined metric reveals the capability gap between generation and verification. Empirically the metric is useful in predicting potential self-improvement. But it would be more solid if the authors could analyze the correlations between the metric and the actual performance changes after iterations of self-improvement. Experimental evidences and analyses would provide more understanding of the metric and its potential use.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "DBvrY6HqtY", "reviewer": "Reviewer_1Q5B", "summary": "This paper studies the self-improvement phenomenon of LLMs. The authors have proposed a mathematical framework to formally discuss the LLM self-improving process, and introduced a well-motivated metric called generation-verification gap (GV-gap) for observation. Linear scaling properties between the GV-gap and pretraining FLOPs are illustrated. The authors have adopted three types of distinct tasks for a comprehensive study of the LLM self-improvement process.", "questions": "- Is the RL update illustrated in Example 2.1 adopted in experiments? \n\n- Does the **High-fidelity model update** condition mentioned in Section 2.1 examined / verified in experiments？What's the practical benefit for having this condition (principle)?\n\n- Will different decoding algorithms (e.g., search algorithms) for generation help LLMs to (self-)improve? And will the choices of different generation methods alter the conclusions drawn in this paper?\n\n- I like the experiments with Sudoku (Section 4.4), as this is a setting with answer generation strictly harder than verification (NP-hard > P; assuming NP!=P). However, according to the experimental results, non-trivial gaps are observed only with models of > 70B parameters. I wonder if the authors could conduct additional experiments by sweeping the settings of the Sudoku problems: currently it's of 4x4 matrices; how about 3x3, or even 2x2 sized matrices, where the problems should be much easier? With these experiments, can we have the observation that LLMs with different sizes can self-improve?\n\n- Can the authors **use the mathematical framework proposed in Section 2** to provide insights about why the diversity deteriorates with iterative rejection fine-tuning?\n\n- Further thoughts: if diversity deteriorates, do the authors mean that the self-improving process is in fact harmful in some aspect to LLMs, and eventually the self-improvement is just about some kind of tradeoff? What's the authors' opinion on this?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The authors conducted a timely study on self-improvement of LLMs.\n\n- The proposed mathematical framework about LLM self-improvement is insightful, especially the design of the generation-verification gap (and its relative variant).\n\n- The designed experiments (GSM-8K, NQ, and Sudoku) are representative, as they correspond with the settings of the improvable, the unimprovable, and the \"theoretically improvable, but empirical findings show unimprovable\".\n\n- The scaling property of GV-gap ~ pretraining FLOPs is a nice read.", "weaknesses": "Please see the Questions below.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Oral)