Progress or Regress? Self-Improvement Reversal in Post-training

Authors: Ting Wu, Xuefeng Li, Pengfei Liu
Primary area: learning theory
Keywords: Iterative Self-improvement, Problem-solving AI

Abstract:
Self-improvement through post-training methods such as iterative preference learning has been acclaimed for enhancing the problem-solving capabilities (e.g., mathematical reasoning) of Large Language Models (LLMs) without human intervention. However, as our exploration deepens, it is crucial to critically assess whether these enhancements indeed signify comprehensive progress or if they could lead to unintended regressions. Through rigorous experimentation and analysis across diverse problem-solving tasks, we uncover nuances in the self-improvement trajectories of LLMs. Our study introduces the concept of \emph{self-improvement reversal}, where models showing improved overall accuracy metrics might paradoxically exhibit declines in broader, essential capabilities. We propose a comprehensive evaluative framework to scrutinize the underlying mechanisms and outcomes of post-training self-improvement, aiming to discern between superficial metric improvements and genuine enhancements in model functionality. The findings emphasize the complexity of technological advancements in LLMs, underscoring the need for a nuanced understanding of the \textit{progress or regress} dichotomy in their development.

Decision: accept (poster)
Reviewer scores: 6, 8, 6, 6

Reviews and discussion:
<thread>
creation date:
1730086144301

thread_type:
official_review

<thread_object>
creator: reviewer_2dxd

<content>
summary:
This work goes beyond pass@1 to dive deeper into evaluating self-improvement methods for mathematical reasoning. They show that improvement in benchmark performance of self-improvement showed signs in regression in areas such as output diversity and OOD generalization. They developed a new evaluation framework for current self-improvement methods to evaluate on.

strengths:
- very relevant topic for the conference
- good empirical results and comprehensive experiments
- interesting findings that can inspire future research directions:
	- most methods decline in task performance after ~4 iterations
	- declined abilities in output diversity and OOD generalization
	- lesser powerful models gain more, but more powerful models achieve better max performance
	- M_1 performance effects subsequent iterations

weaknesses:
- (minor) in introduction, it wasn't apparent why the new metrics are important. For example why is diversity needed in solving these reasoning tasks?

questions:
- for table 3, can you label x-axis as `sampling N`? Also what model is used here?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730688161620

thread_type:
official_review

<thread_object>
creator: reviewer_eamz

<content>
summary:
This work studies the post-training steps of supervised fine-tuning and preference optimization during iterative self-improvement, where an LLM improves by training on its own generated solutions. The authors claim that such iterative post-training may show progress according to traditional top-1 test accuracy measures, but may also negatively impact semantic diversity and OOD performance on other tasks. They propose an iterative self-improvement algorithm which, given a training set of inputs and ground truth outputs, iteratively fine-tunes a model on self-generated outputs labelled according to whether they are correct or incorrect, serving as synthetic preferences similar to RLAIF. The authors compare 3 methods of fine-tuning: SFT, DPO, and alternating SFT-DPO. They find that improvement plateaus for most tasks after ~3 iterations before declining, as well as a few other findings. Next, they show that while accuracy on the test set increases over post-training, diversity across a number of metrics may be reduced as well as accuracy on OOD tasks.

strengths:
- Well-motivated and timely. Review of related work is thorough.
- Iterative self-improvement post-training algorithm and most of the analyses seem general enough to be applied to arbitrary domains with minimal changes.
- Interesting reversal results with diversity and OOD generalization decreasing during post-training. Used a variety of diversity metrics.

weaknesses:
- I'm not sure I understand Improvement Sets (Section 5.1). Perhaps I'm missing something here, but in my understanding of figure 3, it seems somewhat trivial that accuracy@N increases with greater N.
- I'm also not sure I understand the point of Group Disparity (Section 5.3). Why not just compare level 1 accuracy vs. level 5 accuracy directly, if level 1 accuracy is increasing and level 5 is decreasing?
- Correct answer coverage seems potentially at odds with solution diversity, so I'm not exactly sure how to resolve these two metrics. Is it possible that better performing models aways have lower solution diversity?
- Many of the results figures could be improved, including with longer captions:
	- Figure 1 - It's currently hard to compare SFT vs. DPO vs. SFT-DPO across (a) - (c) since different rows have different scales for the y-axis.
	- Figure 2 - The left figure took some effort for me to understand. Maybe example questions/answers would help.
	- Figure 3 - The x-axis is unlabeled and the caption is very short. As stated above, I also wasn't sure what the main takeaway of this is.
	- Figure 5 - The bottom y-axis has a very small range [90, 95], and I don't have a good sense for what the scale of these values should be. The caption for this also seems too short, and I don't know what the blue dotted line means.

questions:
- Figure 3: why is it surprising that pass@N accuracy increases with greater N?
- How does correct answer coverage relate to solution diversity? Are the two at odds with each other?
- Is solution diversity inherently desirable in domains like mathematical reasoning? Is it possible that higher performing models on certain domains always have lower diversity?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730720813604

thread_type:
official_review

<thread_object>
creator: reviewer_kgyn

<content>
summary:
This work points out a phenomenon called self-improvement reversal. To be specific, the authors formulate three post-training
paradigms (iterative SFT, iterative DPO, and iterative SFT-DPO) and conduct experiments to examine how various factors (iteration steps, different models, task difficulties) influence self-improvement. Although such strategied shows a general trend of improvement in pass@1
accuracy. They still struggle in some other problems like solutions diversity and OOD generalization. This paper provides a deeper understanding to self-improvement.

strengths:
1. The experiments are well-designed and provide a comprehensive understanding to the internal mechanisms of self-improvement.
2. This paper finds that such pass@1 accuracy metric may lead to a wrong judgment about model performance.
3. The experimental results in Figure 2 are insightful. The relationship between correct answer coverage and iterative post-training methods has not been discussed before.

weaknesses:
1. Some of the conclusions in this paper are as expected, e.g. results from Table 3 are related to [1].

[1] Large Language Monkeys: Scaling Inference Compute with Repeated Sampling

questions:
No more questions.
</content>
</thread_object>

</thread>
<thread>
creation date:
1731053409180

thread_type:
official_review

<thread_object>
creator: reviewer_rhq5

<content>
summary:
This paper analyzes post-training via self improvement loops for LLMs. Their results suggest that there is a sort of "Goodhardt's Law" at play sometimes, where improving the benchmark scores will often lead to decreased performance after some point. They particularly note diversity as an important metric which often drops.

strengths:
Good idea and experiments. It's an important and timely topic. The observation that some metrics like diversity are sometimes unmeasured with these methods is very important.

weaknesses:
I think the writing can be improved somewhat, even if the core ideas are clear.

questions:
Do you expect these results to hold with RL posttraining?
</content>
</thread_object>

</thread>


Metareview:
metareview:
This work studies post-training (specifically, iterative SFT, iterative DPO, and iterative SFT-DPO) and its "self-improvement reversal" problem. Specifically, the work conducts experiments to examine how various factors (iteration steps, different models, task difficulties) influence self-improvement, and shows that although these post-training methods show a general trend of improvement in pass@1 accuracy. They still struggle in some other problems like solutions diversity and OOD generalization. The paper is well-written, with ample results and analysis for this well-motivated problem with practical significance. The paper presents interesting reversal results with diversity and OOD generalization decreasing during post-training, using a variety of diversity metrics. It'd be desirable to include more discussion of why diversity is important in "inference scaling law".