Mind the Gap: Examining the Self-Improvement Capabilities of Large Language Models

Authors: Yuda Song, Hanlin Zhang, Carson Eisenach, Sham M. Kakade, Dean Foster, Udaya Ghai
Primary area: foundation or frontier models, including LLMs
Keywords: llm, self-improvement, synthetic data, post-training, test-time optimization

Abstract:
Self-improvement is a mechanism in Large Language Model (LLM) pre-training, post-training and test-time inference. We explore a framework where the model verifies its own outputs, filters or reweights data based on this verification, and distills the filtered data.  Despite several empirical successes, a fundamental understanding is still lacking. In this work, we initiate a comprehensive, modular and controlled study on LLM self-improvement. We provide a mathematical formulation for self-improvement, which is largely governed by a quantity which we formalize as the **generation-verification gap**. Through experiments with various model families and tasks, we discover a scaling phenomenon of self-improvement -- a variant of the generation-verification gap scales monotonically with the model pre-training flops. We also examine when self-improvement is possible, an iterative self-improvement procedure, and ways to improve its performance. Our findings not only advance understanding of LLM self-improvement with practical implications, but also open numerous avenues for future research into its capabilities and boundaries.

Decision: accept (oral)
Reviewer scores: 6, 8, 8, 6

Reviews and discussion:
<thread>
creation date:
1730500894617

thread_type:
official_review

<thread_object>
creator: reviewer_1q5b

<content>
summary:
This paper studies the self-improvement phenomenon of LLMs. The authors have proposed a mathematical framework to formally discuss the LLM self-improving process, and introduced a well-motivated metric called generation-verification gap (GV-gap) for observation. Linear scaling properties between the GV-gap and pretraining FLOPs are illustrated. The authors have adopted three types of distinct tasks for a comprehensive study of the LLM self-improvement process.

strengths:
- The authors conducted a timely study on self-improvement of LLMs.

- The proposed mathematical framework about LLM self-improvement is insightful, especially the design of the generation-verification gap (and its relative variant).

- The designed experiments (GSM-8K, NQ, and Sudoku) are representative, as they correspond with the settings of the improvable, the unimprovable, and the "theoretically improvable, but empirical findings show unimprovable".

- The scaling property of GV-gap ~ pretraining FLOPs is a nice read.

weaknesses:
Please see the Questions below.

questions:
- Is the RL update illustrated in Example 2.1 adopted in experiments? 

- Does the **High-fidelity model update** condition mentioned in Section 2.1 examined / verified in experiments？What's the practical benefit for having this condition (principle)?

- Will different decoding algorithms (e.g., search algorithms) for generation help LLMs to (self-)improve? And will the choices of different generation methods alter the conclusions drawn in this paper?

- I like the experiments with Sudoku (Section 4.4), as this is a setting with answer generation strictly harder than verification (NP-hard > P; assuming NP!=P). However, according to the experimental results, non-trivial gaps are observed only with models of > 70B parameters. I wonder if the authors could conduct additional experiments by sweeping the settings of the Sudoku problems: currently it's of 4x4 matrices; how about 3x3, or even 2x2 sized matrices, where the problems should be much easier? With these experiments, can we have the observation that LLMs with different sizes can self-improve?

- Can the authors **use the mathematical framework proposed in Section 2** to provide insights about why the diversity deteriorates with iterative rejection fine-tuning?

- Further thoughts: if diversity deteriorates, do the authors mean that the self-improving process is in fact harmful in some aspect to LLMs, and eventually the self-improvement is just about some kind of tradeoff? What's the authors' opinion on this?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730541299460

thread_type:
official_review

<thread_object>
creator: reviewer_gkzy

<content>
summary:
The paper provides a mathematical formulation of LLM self-improvement and defines generation-verification gap to measure the potential of self-improvement. Based on the metric, the work conducts extensive experiments in terms of scaling laws, tasks suitable for self-improvement, iterative procedure and better verification attempts.

strengths:
1. The paper presents a math formulation and a new metric for studying self-improvement, which I believe will be beneficial for current studies related to LLM self-improvement.
2. The experiments are comprehensive, analyzing self-improvement from various perspective.
3. The experimental findings are generally insightful.
4. The paper is well-written and easy to follow.

weaknesses:
1. My main concern is about the experiment setup. The main experiments are limited to one benchmark, GSM8k, in the math task. Despite GSM8k is a classical benchmark for mathematics, it is relatively easy for current cutting-edge LLMs and its benchmarking function may decrease due to potential data contamination. Additionally, the self-improvement performance as well as the findings may differ across different datasets or scenarios. Therefore, I think it would be beneficial for the authors to complement experiments on more datasets and scenarios. 
2. The defined metric reveals the capability gap between generation and verification. Empirically the metric is useful in predicting potential self-improvement. But it would be more solid if the authors could analyze the correlations between the metric and the actual performance changes after iterations of self-improvement. Experimental evidences and analyses would provide more understanding of the metric and its potential use.

questions:
1. The findings on "effective diversity of generations" are interesting. Have you conducted further experiments to clarify the underlying causes? For instance, could it be attributed to the training process itself? (Perhaps an experiment involving training on golden labels could provide insights.) Alternatively, could it stem from noise introduced by self-verification? (An experiment to filter out incorrect samples using golden labels or employing stronger evaluators might help determine if this mitigates the issue.)
2. From the main results on scaling law (Figure 1), the scaling laws seem to be severely influenced by the verification methods. Why do some methods (like MC, To) not exhibit such scaling laws? And whether the other methods share similar findings (like CoT-B in the paper and some others)?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730674650027

thread_type:
official_review

<thread_object>
creator: reviewer_qqkc

<content>
summary:
This work performs an extensive and exhaustive analysis on self-improvement abilities of large language models. First, authors present related work under which the self-improvements capabilities has been studied and formulate a high-level, unified way of representing such framework. Then they propose a novel way of measuring the self-improvement capability using the so called generation-verification gap (GV-gap). Further, they design an experimental setup and provide multiple takeaways that helps to bring the light onto the cases where self-improvement is likely very limited and hypothesize why this might be happening. Finally, authors provided multiple future directions based on their takeaways and findings.

strengths:
* The proposed measure (gv-gap) is clearly defined with real world examples when it comes to different cases of generators and verifiers. This brings good clarify into the overall presentation.
* The view on self-improvements via the proposed gap brings a new way to quantify the limits of self-improvements, that is a contribution into this research direction and can be used by the research community in future work.

weaknesses:
* While the definition of the gv-gap is pretty strict, the real world choice of the utility function making the practical measurements very noisy as authors pointed out in the text (which is great!). They have proposed some ways on improving the verification signal, however, that is a weak point in the entire framework that does not present itself when we just measure the model performance before and after self-improvement steps.
* While I agree that studying base models makes things much easier to handle in paper-size experimental setup, the absence of any signal about the gap changes with instruct based model weakens the provided takeaways. For instance, recent work showed that smaller (8B) instruct models are self-improving very well while in this work there is an opposite signal from the base smaller models that are seemingly lacking reasoning capabilities even under few shot settings.

questions:
* adding more about the limitations coming from only using the base models would make overall takeaways not weaker, but clearer w.r.t. other recent research in this area.
* overall I think this will be a very useful analysis work for future self-improvement research direction.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730793773229

thread_type:
official_review

<thread_object>
creator: reviewer_eq2c

<content>
summary:
The paper investigates the concept of self-improvement in Large Language Models (LLMs) during pre-training, post-training, and test-time inference. The authors present a framework where the model verifies its own outputs, refines or reweights data based on this verification, and distills the filtered data. Despite empirical successes, there is a lack of fundamental understanding of this process. The study offers a comprehensive, modular, and controlled exploration of LLM self-improvement, introducing a mathematical formulation governed by the "generation-verification gap." Through experiments across different model families and tasks, the authors identify a scaling phenomenon linked to this gap that scales with model pre-training computational efforts. Additionally, the paper examines conditions under which self-improvement is feasible, proposes an iterative procedure for self-improvement, and suggests methods to enhance its performance. The findings not only deepen the understanding of LLM self-improvement but also open new research possibilities regarding its capabilities and limitations.

strengths:
1. The research focus of this paper is commendable, as the GV-Gap aids in determining whether a model possesses self-improvement capabilities at any stage (Pre-/Post-training).

2. The overall layout are well-organized.

3. The experiments appear to be extensive and comprehensive. (Respect to Your Hard working)

weaknesses:
### Deficiencies in Expression

1. The paper lacks a clear definition of the Generation-Verification Gap, with no illustrative diagram and only complex, confusing formulas cluttered with symbols.

2. The authors fail to clearly outline the overall methodological framework or how the experimental process operates. (I have read the experiments three times and even couldn't know how to evaluate the experimental performance! 🤷🏻‍♀️)

### Limitations in Contribution

The contributions to the community are relatively weak, as most of the conclusions are already known, such as the "verification gap scales monotonically with the model pre-training flops" and the "Saturation Limit and Cause of Saturation." 
As a purely experimental analysis paper, it does not offer new insights, which is regrettably disappointing.

### Experimental Deficiencies
1. The experiments are primarily based on GSMK8k, which consists only of elementary-level math problems, making the results less convincing.
2. In Section 3, the experimental setup has predetermined parameters for lm-evaluation-harness tasks. However, it is unclear why so many hyperparameters would affect the GV-Gap instead of being solely determined by differences between the verifier and the generator.

questions:
Please Check Weakness.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732158874016

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Response

comment:
First we would like to thank all the reviewers for their constructive comments on new experiments. We believe the new results will greatly improve the paper. 

In this thread we summarize the revision in the draft: we made a few presentational improvements in the main text on the introduction to the experiment protocols and we include the RL results in Figure 3. For the ease of presentation, we aggravate most of the additional results in Appendix A and B, detailed below. We will incorporate the new results into the main text in the final version. All new changes are highlighted with blue. 


### MATH dataset (Appendix A)
As suggested by Reviewer eq2C and gkZY, GSM8K might not be a challenging enough benchmark. Thus we repeat our major experiments on the MATH dataset, a more challenging benchmark with competition level math questions. We record the details in Appendix A. In summary, we observe the same scaling phenomena on relative generation-verification gap with even stronger signals (more than one verification method elicits the scaling phenomena) and the same observations in iterative self-improvement (we have not completed the full iterative training loop for the bigger models and we will update the results once we completed the training). 


### Different Sampling Temperature (Appendix B.1)
As suggested by Reviewer eq2C and 1Q5B, we conduct an ablation on different hyperparameter/decoding strategies. We tested sampling with temperature 1 and 0.5 and we observe the same scaling phenomena holds. 


### Instruct Models (Appendix B.2)
For completeness, we include the scaling study for instruct models as suggested by Reviewer QqKc. We observe the results are more noisy due to the confounders introduced in the post-training process.


### Fixed Verifier (Appendix B.3)
To better understand the decrease in effective diversity, as suggested by Reviewer gkZY, we conduct iterative improvement with the gold label or a more powerful model as the verifier. Our result suggests that the decrease in effective diversity does not happen if we use the gold label as verifier, but it happens in the second setting. This suggests that one reason for this phenomenon is that the model concentrates on some wrong answers that the imperfect verifier labels as correct. 


### Variance of the Verifier (Appendix B.4)
To better understand the noisiness of the verifier, as suggested by Reviewer QqKc, we measure the variance of the verifier in Appendix B.4. We observe that the verifier has near 0 variance on most of the generations.
</content>
</thread_object>

</thread>


Metareview:
metareview:
Claims and findings:  
The paper discusses self improvement in models of different stages of training. 
They offer a way to evaluate the gap between generation and verification
With this way they show how this gap grows with more training, with better or worse model verifying or generating etc.

Strengths:  
General framing yet convincing  
Many experiments to support claims  
Large potential for future improvement