Progress or Regress? Self-Improvement Reversal in Post-training
Authors: Ting Wu
Xuefeng Li
Pengfei Liu
Venue: ICLR 2025 Poster (2025)
Keywords: Iterative Self-improvement
Problem-solving AI
Primary area: learning theory
TL;DR: A complex study of iterative self-improvement in Large Language Models (LLMs), focusing on various fine-tuning strategies and their effectiveness.

Abstract: Self-improvement through post-training methods such as iterative preference learning has been acclaimed for enhancing the problem-solving capabilities (e.g., mathematical reasoning) of Large Language Models (LLMs) without human intervention. However, as our exploration deepens, it is crucial to critically assess whether these enhancements indeed signify comprehensive progress or if they could lead to unintended regressions. Through rigorous experimentation and analysis across diverse problem-solving tasks, we uncover nuances in the self-improvement trajectories of LLMs. Our study introduces the concept of \emph{self-improvement reversal}, where models showing improved overall accuracy metrics might paradoxically exhibit declines in broader, essential capabilities. We propose a comprehensive evaluative framework to scrutinize the underlying mechanisms and outcomes of post-training self-improvement, aiming to discern between superficial metric improvements and genuine enhancements in model functionality. The findings emphasize the complexity of technological advancements in LLMs, underscoring the need for a nuanced understanding of the \textit{progress or regress} dichotomy in their development.

Published as a conference paper at ICLR 2025
PROGRESS OR REGRESS? SELF-LMPROVEMENT REVERSAL IN POST-TRAINING
Ting Wu'-3, Xuefeng Li Pengfei Liu
Fudan University, "Shanghai Jiao Tong University, *Generative Al Research Lab (GAIR)
ABSTRACT
Self- self-improvement through post-training methods such iterative preference learn- ing has been acclaimed for enhancing the problem-solving capabilities (e.g., mathe- matical reasoning) of Large Language Models (LL.M)) without human intervention. However, as exploration deepens, it becomes crucial to assess whether these im- provements genuinely signify progress solving more challenging problems if they could lead to unintended regressions. To address this, we propose a comprehen- sive evaluative framework that goes beyond the superficial pass @ I metric scruti- nize the underlying enhancements of post-trane post-training paradigms for self celr-impooeeeett Through rigorous experimentation and analysis across diverse problem-solving tasks, the empirical results point out the phenomenon of self-im reff-improvement reversal, where models showing improved performance across benchmarks will paradoxi- cally exhibit declines in broader, essential capabilities, like output diversity and out-of-distribution (OOD) generalization. These findings indicate that current self- improvement practices through post-training are inadequate for equipping models to tackle more complex problems. Furthermore, they underscore the necessity of our critical evaluation metrics in discerning the progress or regress dichotomy for self-improving LLMs.
INTRODUCTION
In the rapidly evolving landscape of artificial intelligence (AI), the pursuit of self-improving large language models (LLMs) has garnered significant attention (Singh et al.) 20233 Huang et al. 2023 Sun et 2024). The essence of self-improvement in LLMs lies in their capacity to iteratively refine models' own performance without human intervention (Zelikman et al. 2022 Yuan et al. 2024). This capability is paramount as it holds the promise of fostering the development of more autonomous, adaptable, and efficient AI systems (Silver et al., 2016). Embracing and implementing self-improvement methodologies enables us to push the boundaries of these models' capabilities. ultimately fostering the creation of more sophisticated and versatile Al applications (Significant, Gravitas 2023).
Building on the concept of self-training (Grandvalet & Bengio 2004), wherein models bootstrap their own generated responses for iterative training, a synergetic effect observed and amplified. When models produce superior responses, the quality of the training data used to refine the models improves, subsequently enabling even better responses in future iterations. Such iterative post- training has become the standard paradigm for current self-improving AI (Yuan al., 2224). Notably, STaR (Zelikman et al.) (222) has demonstrated that leveraging model's self-generated reasoning steps for iterative supervised fine-tuning (SFT) effectively enhance its reasoning abilities. Recent studies (Pang et al. 2024) have further revealed that employing iterative preference optimization in LLMs can achieve more performance improvements in reasoning tasks.
However, despite the promising advances in various post-training methods for self-improvement, comprehensive understanding of their effects and underlying mechanisms is still lacking. To address this gap, in this study, we first endeavor to provide comprehensive overview of the main iterative post-training paradigms for self-improvemment, identifying the factors that contribute to consistent performance improvements. We decouple the influencing factors into the initial model,
Corresponding author

---

Published as a conference paper at ICLR 2025
task datasets, the number of iterations, and the specific post-training methods employed. By isolating these variables, our comprehensive experiments and analysis uncover effects the model's performance. This provides actionable insights for practitioners on how to perform iterative self-improvement practices more effectively.
their individual and combined
While our extensive empirical results show that all these iterative post-training methods achieve notable improvements in pass accuracy across various problem-solving benchmarks, the evaluation has been limited to this single and superficial metric. Amidst the quest for self-improvement in LLMs, the persistent question arises: are these iterative post-training methods truly fostering progress, or are they inadvertently leading to regression? Transitioning beyond using pass accuracy as the indicator of improvement, we further develop an evaluative framework equipped with a comprehensive suite of metrics to assess improvement problems, solutions diversity, and OOD capabilities within the iterative process, enabling us to scrutinize the actual improvements beneath self sef--improveeent. Surprisingly, our evaluation results display a paradoxical trend: as pass@) accuracy increases, the proposed metrics exhibit consistent performance declines.
The perceived reversals our evaluative framework prompt a critical reflection on the effectiveness of current self-improvement practices. Through this study, we aim to illuminate the path forward for developing truly self-iip LLMs that balance accuracy, diversity, and robustness. To summarize, work makes three significant contributions follows:
In Sections 3 and we systematically formulate current post-training methodologies and conduct extensive experiments to examine how various factors influence self- improvement in solving challenging tasks. To the best of our knowledge, this work provides the first in-depth overview of these influencing factors.
Systematic Analysis:
Metric Innovation: In Section 5 we propose a comprehensive set of evaluation metrics to better capture the multifaceted nature of L.LM performance in seff-imprrvemmet practices.
Identified Phenomenon: In Section based on the proposed evaluation metrics, we reveal the phenomenon of relf-improvement reversal, where increases pass@) accuracy compromise other essential capabilities such as solution diversity and OOD generalization.
2 BACKGROUND AND RELATED WORK
Training paradigms for LLMs typically consist of two stages: pre-training and post-training (Liu jet al. 2024). Common post-training methods include supervised fine-tuning (Taori et al,, 2023 Wang
ct al. 2023) and preference learning (Ouyang et al. 2022 Lee et al.) (024).. Supervised fine-tuning trains LLMs to produce standard responses for given instructions, while preference learning trains
LLMss align with human preferences for different responses. Both methods, however, rely heavily on extensive
data.
An important question is whether effective LLM post-training be achieved without excessive external feedback. Predating the era of LLMs, the self-training algorithm (Grandvalet & Bengio Goodfellow al. 2014) demonstrated the potential enhance model performance without additional labeled data. Recent studies have revived this concept. employing iterative self-training to facilitate celf-improvement in LLMs without external feedback (Wang al. 2023 Sun et al. 2023). For instance, STaR (Zelikman et al. 2022) shows that iterative training on the model's own reasoning traces for correct answers can help solve increasingly difficult problems. Unlike the iterative nature
of SFT, recent works (Yuan et al.) 2024 Pang et al. 2024) propose iterative preference fine-tuning to aid models self-improving.
In contrast to post-training methods, another line of research explores self-improvemeen through iterative post-prompting during inference (Huang et al., 2023). This approach does not update the model's 'ss parameters but achieves self-improvement by generating reflections on its outputs and adjusting future outputs accordingly (Madaan et al., 2023 Gou et al. (0244). However, as revealed by Huang et al. (2024), post- post-prompting strategies are limited by the model's intrinsic self-correction capabilities, thereby failing to significantly enhance problem-so solving capabilities.
The potential of iterative post-training for self- celf-improvemmet in LLMs remains underexplored. Al- though various cost-training methods have demonstrated promise in general ciarruttio-folloviiii
tasks (Li et al. 2024 Sun al. 2024 Chen et al. 2024 Yuan et al. 2024), they predominantly focus
2

---

Published as a conference paper at ICLR 2025
on aligning models with human values rather than enhancing the models' internal knowledge. A key challenge remains whether LLMs can sustain consistent performance on more complex problem- solving tasks. Recently, Pang et al. (2024) examined iterative preference learning in the context of reasoning tasks, marking the first study to expand beyond instruction tuning.
Despite these advancements, comprehensive overview investigating the effectiveness of various iterative post-training methods for problem-sole solving is still lacking. First, it remains unclear how improvements vary across iteration steps, different base models, task difficulties, and iterative post- training techniques, For practitioners, there is a need for guidelines to help choose the most effective post-training method among the various iterative post- trainnng paradigms. Second, current research only concentrates on maximizing benchmark scores through iterative self-imprrvemmett, there is little exploration of the underlying factors contributing to performance gains. As result, the progress and reliability of different self-improvement methods are not guaranteed.
In this work, we aim to address these two critical issues. Our goal is not only to ensure the effectiveness of various self- eeff-improvement methods but also to ensure that other capabilities are not compromised during the complex self-improvement process.
3 POST-TRAINING FOR SELF-IMPROVEMER
3.1 FORMULATION
Consider training dataset D = [ i= consisting of pairs of queries and their correspond- ing correct responses Y/- A foundation model, denoted My- Our objective to enhance Mo through a self-driven iterative post-training process, leveraging the model's own outputs to refine its capabilities, without reliance on external signals.
Iterative Post- Training The iterative post-training process involves a series of post-training steps, each aimed at using the model's previous outputs to guide its subsequent refinement. These steps designed to foster a continuous loop of self-improvement for the model.
The process is outlined across three main phases follows, where the total number of iterations is denoted as T. and the model employed in the t-th iteration is denoted as M,-1, implying that Mo is used in the first iteration:
Answer sampling: In the t-th iteration, we prompt M,-1 to generate N answers for each query t; in D to form a new self-generated dataset Dieir {( y i. E D.j N]}.
Training set construction: The training set D, in the t-th iteration assembled from Deell without introducing any external data. The approach to constructing the training depends on the specific paradigm of post-training.
Model post-training: Utilizing Dr, the model M,-1 is refined into M,.
It's worth noting that, in the first iteration, we always directly supervised fine-tuning Mo on D to initialize M2 with task-specific knowledge.
Central to these diverse methodologies is the post-training function, symbolized as F. We distinguish among the practices based the nature of F, involving Supervised Fine-tuning (SFT) and Direct Preference Optimization (DPO) (Rafailov et al. 2023), the latter being an effective implementation of preference learning. During the SFT phase, this stage necessitates accurately labeled training data. We derive these correct answers from Deelf to assemble the training dataset:
D. [1,, M) ( 1, (x,, 1 Doell);
where R(x, y) evaluates whether the answer y accurately addresses the question. In our problem- solving task, the conrectness of an answer is verified by its alignment with the response provided in the dataset. While during the DPO phase, for each query q, in dataset D, both correct and incorrect responses from Dielf are paired to construct the training set, allowing for contrastive preference learning:
D. ((x,, . " D²) R, (R(x..1))) O, (1..1) E Deel (xr,1) ) Deelf).
3

---

Published as a conference paper at ICLR 2025
Algorithm Iterative Self- Self-Improvement
Require: training set D {Ia, vs), base model Mo, iteration times T. poop--oooiiiiii training function series (+ ( ())),))))))))))))))))))))))))))) Fr(()) (.))
1: M1 f- SFT (M))))
Initialize base model with task-specific knowledge
2: for = to ] do
3:
Dieli = {[ yy b D.y M=)( [1,
4:
it F.)) = SFT then
5:
D, [( y [R v = (I., 1/ E D
6:
else
7:
Dj (20,, ,", (2 (0)) = 1, R y²) = 0, (2 (2,,8") E Dielf (2,,8)) ) Dielr)
8:
end if
9: Mj F.()M-
end for
3.2 THREE ITERATIVE POST- TRAINING PARADIGMS
Through the implementation of designated self post-training steps (e.g., self-SFT), several distinct iterative post-training paradigms emerge. Our work focuses on three paradigms: (i) iterative SFT, where each cycle consists exclusively of self-SFT steps, (ii) Iterative DPO, characterized by successive self-DPO steps, except for the first iteration which supervised fine-tune the base model MM. and (iii) iterative SFT-DPO, which initiates with self-SFT step and alternates between self-DPO and self-SFT steps to form a complete iterative post-training loop.
We describe the unified procedure in Algorithm
4 EXPERIMENT
As outlined in Algorithm 1 we hypothesize that the key variables--nitiin model (M), task dataset (D), iteration steps (T), and post-training method (F))critically influence model performance during iterative self- celf-improvement. This section explores the impact of these variables on different problem- solving tasks. We aim to determine if models consistently improve with increasing iterations (T) and uncover the trade-off's and comparative advantages of Iterative SFT, Iterative DPO, and Iterative SFT-DPO in enhancing performance across various tasks. Through this analysis, we seek to provide deeper insights into the mechanisms driving iterative seff-improvement.
4.1 EXPERIMENTAL SETUP
Datasets To measure model problem-solving capabilities, we train and test broad spectrum of moblem-solving datasets. We measure general knowledge using the CommonsenseQA (CSQA) (Tal. mor et al. 2019) dataset, assessing mathematical reasoning with the GSM8K (Cobbe et al.) 2021
and MATH (Hendrycks et al.) 2021) dataset, and weigh code generation skills using the MBPP dataset Austin et al.) 2021). Regarding the train-test split, we adhere to (Kojima et al. 2022). utilizing the validation set of CSQA for evaluation. The GSM8K and MATH datasets are employed with their predefined train-test splits. For the MBPP code dataset,
follow the approach outlined by (Austin et al., (021) that utilizes examples of Task IDs 11-510 as the 500 test problems, and the remaining 374 examples ranging Task IDs from 601 to 974 for fine-tuning.
Sampling and Rewarding At the end of training, we sample N=50 outputs for each problem using top p sampling (Holtzman et al. 2020) with p 0.95 and temperature 0.75. Considering the gold labels are provided for the problem-sol binary reward for the output to annotate the preference.
datasets, we use the correctness of final answer as
Training Our experiments primarily leverages three open-source models LLaMA-2-7B Touvron et al. 2023). Mistral-7B (Jiang et al., 2023) and LLaMA3 8B (AI@ Metal 2024), with a fully fine- tuning setting. For the implementation of mefeeence--asse learning, utilize DPO RRfaillvv et al.) 2023) due to its scalability and efficiency. In each iteration, preference data are derived by sampling outputs from the newly updated model, utilizing on-policy sampling strategy. Hence, posit
4

---

Published as a conference paper at ICLR 2025
+2.
40 60
LaM044
23
Taking tradios
Tnihing terations
Tralving henddi
Traking Foratine
Iterative SFT
CSDA
12F
23
Tuen1p tertoor
Traning flevsoo
Teeveg
Thamiin dtore
(b) Iterative DPO
1902 2000000000000
12.42
(c) Iterative SFT-DPO
Figure 1: Pass@1 accuracy across the four benchmarks by performing with the three paradigms: Iterative SFT, Iterative DPO and Iterative SFT-DPO. For each model along with the training iterations, we highlight the optimal result with larger-size marker, the improvement above and final accuracy below.
that this online DPO can be treated effective and representative implementation for preference learning (Tajwar et al. 2024).
Evaluation We use greedy decoding as the temperature set 0 for testing generation. Meanwhile, we utilize zero-shot prompting (Kojima et al. (222) for both answer sampling and evaluations since we find L.LMs finetuned on specific tasks, zero-shot prompting superior to few-shot prompting. More experimental details can be seen in Appendix
4.2 MAIN RESULTS: DECOUPLING THE INFLUENCES OF VARIABLES
We perform the three post-training paradigms with the selected LLMs, training and testing them on the respective tasks. Based on the results shown in figure I how these variables influence the effectiveness of self-improvement.
we delve into the detailed analysis of
Iteration T' Across all methods and datasets, there is a general trend of improvement in pass @ accuracy with increasing iteration steps. This indicates that iterative post-training effectively enhances model performance over time. However, the rate of improvement tends to plateau or even decline slightly after 4-5 iterations. This suggests that current post-training methods struggle to achieve long-lasting improvements, and excessive post-training (beyond a certain number of iterations) may even yield diminishing returns.
Foundation Model M The optimal accuracy improvements across various datasets and post-training methods suggest that LLaMA2-7B demonstrates a relatively higher capacity for improvement under iterative post-training. For instance, on the GSM8K dataset, LLaMA2-7B with Iterative SFT shows an improvement of +12.11 after 5 iterations, whereas LLaMA3-88 exhibits only a moderate gain. This indicates that the more capable Mj is not necessarily the model that achieves the most significant performance gains during the relf-improvement process. However, the most capable model Mi generally achieves the highest optimal accuracy overall. For example, although ILaMA2-7B achieves
5

---

Published as a conference paper at ICLR 2025
Madel
Datset
Corerage
Mr Benaiie SFT
347 Derative DPO
14; hesuive SFT-DPO
A0teet Space
-Crrect Aroven Space
CSQA
[L824
63.36
72.15
33.59
34.11
53.91
52.91
MATH
0.167
5.88
8.1
6.64
7.3
Pat
MBPP
19.0
22.0
CSQA
0.825
73.38
77.81
79.6
Proobiiity
GSMRK
01627
61.29
68.92
70.96
72.18
MATH
11.56
17.58
MBPP
0.629
35.2
35.8
37.0
36.4
CSQA
GSMSK
0.812
66.41
69.06
70.96
75.31
Ansae Space
MBPP
0.545
44.2
44.8
46.4
45.6
Figure 2: Left: The answer distributions of models. PMr and PM; represent the answer distributions of M1 and the optimal model M; (achieving the highest pass@1 @ accuracy) within iterative process. The shaded area indicates the correct answer coverage of M,. Right: For foundation model M and task D, each line lists the correct answer coverage and the optimal pass @ 1 accuracy of M with the three iterative post-training methods. This table aims to display the relationship between correct answer coverage and the efffcctiveness of the post-training method F.
the maximum gains GSM8K with Iterative SFT, it still struggles outperform LLaMA3-8B in terms of absolute optimal accuracy (53.91 vs. 69.06).
Problem-solving Tasks D Models utilizing the three post-training methods all demonstrate notable improvements on the CSQA and GSM8K datasets, while showing more modest gains on the MATH and MBPP datasets. This indicates that, from the perspective of task difficulty, problems the CSQA and GSM8K datasets are relatively easier for the models to resolve. In contrast, the MATH dataset poses significant challenges for 7B models due its complexity. Additionally, the task of code generation, as represented by the MBPP dataset, is also difficult for these foundation models since they were not specifically pretrained on code domains.
Comparative Analysis of Post- Training Methods With foundation model M and task D varying, the best perforreffm iterative method also changes accordingly. For example, for Mistral-7B on the CSQA dataset, Iterative-DPO achieves the highest accuracy improvement of +6.47. However, when applied to the GSM8K dataset, the Iterative SIT-DPO method yields the maximum improvement of 10.99. Therefore, with these identifiable variables characterized, it remains challenging for downstream practitioners to determine the optimal post- pest-training method JF for their specific use case.
Characterizing More Deciding Factor As discussed above, the identifiable variables fail to provide clear clues on the effectiveness of the post- post-training method F when foundation model M and task D change. Upon closer examination of Figure [ we find common thread: regardless of the changes in M and D, models (M1) that perform well on a task after the initial iteration of SFT tend to show substantial improvements with further iterations by performing iterative DPO and iterative SFT-DPO, compared to using Iterative SFT. Conversely, those M, that achieve lower pass 1 accuracy initially exhibit limited gains with iterative DPO. Based on this observation, we hypothesize that M1 's capability to solve the test problems fundamentally influences further improvement trends and optimal improvements of To quantify M, capability on the test set, we introduce Correct Answer Coverage as a measurement, the proportion of the correct answer space that the model's responses occupy. An illustrative display of this coverage shown in Figure 2
Answer Coverage:
Mathematically, we can sample N model's 's outputs to approximate the answer space. As N + oc, these outputs can effectively represent the entire answer space. Therefore, expected accuracy over the N outputs can serve as an unbiased estimate of the correct answer coverage. Formally, we use the following equation to calculate M2 correct answer coverage (for more detailed derivation, please refer to the Appendix C
Neorrett
Correct Answer Coverage E
1M;x == w)
=-
(I)
Drest
N
DE

---

Published as a conference paper at ICLR 2025
03:31
03+41
(a) Iterative SFT
(b) Iterative DPO
C104
83x21
80x1
53-44
03:51
(c) Iterative SFT-DPO
Figure 3: Pass N accuracy of M1 with zero-shot prompting IS(1), for / 2.
As shown in Figure 2 the relationship between correct answer coverage and optimal performance of FF validate our prior observation and hypothesis. The table clearly demonstrates that when the correct answer coverage high (> 0.5), Iterative DPO and Iterative SFT-DPO produce the best-performing Mī. Conversely, when the coverage is lower (S 0.5), Iterative SFT is more effective in achieving the optimal M,. Therefore, correct answer coverage can serve as a key factor in guiding practitioners to choose the most suitable iterative post-training method JF for the specific problem-solving task with fixed foundation model.
CRITICAL EVALUATIONS ON SELF-IMPROVEMENT
Despite the extensive exploration of various post-training practices for self-improvement and deepened understanding of their efficacy, current endeavors remain narrowly focused enhancing performance numbers across these problem-solving benchmarks. Transitioning beyond using pass accuracy the indicator of improvement, examination and reevaluation of iterative self-imppovemmen: discerning whether the improvements constitute genuine progress or merely regression. For brevity, all the results shown in this section is based on the foundation model M as Mistral-7B.
objective in this section is engage in critical
5.1 IMPROVEMENT PROBLEMS
In Figure is evident when t > 1, the pass@1 accuracy M, consistently improves in comparison to Mi. Traditionally, it has been assumed that this improvement indicates the model progressively learning to tackle more challenging problems (Zelikman et al., 2022). However, we posit a nuanced perspective: while an increase in pass@1 I accuracy suggests improvements, does not inherently equate to an increase in model capabilities to solve more difficult problems.
To better gauge how the model problem-solving capabilities evolve overtime, propose to first quan- tify the improvement problems as improvement set (IS) at each iteration. An intuitive improvement between M, and M1 the pass @ accuracy test set, M, correctly answers while Mi fails under greedy decoding to represent IS(t), defined as follows:
the subset of test problems that
TS(() (x Drest Mx(x)= y M1 (x) # y).
7

---

Published as a conference paper at ICLR 2025
Cornect-SntetteRRRT
DotrectDetticct
incorrech SentenceBERT
DeaareecDDetnnct
C504
6SMaK
WATH
HBPP
GSNSK Distinct Equation
Cornet
22
lterative SFT
GSHAK- Distint Equation
Thiiting Iteraiions
(b) Iterative DPO WATH
C504
6SM8K
HBPP
GSNAK Distinct Equation
24
02
60
(c) Iterative SFT-DPO
Figure 4: Diversity of the sampling outputs from Mt within the iterative process.
Then we can prompt M1 with the problems in IS(() and sample N answers for each problem to record the pass@N accuracy of M1. Notably, if M1 exhibits lower pass accuracy even as N increases, it can validate M1 struggles to solve the problems in IS(t) and the iterative process enhances the model's problem-solving abilities.
We apply this evaluative methodology to CSQA, GSM8K, MATH and MBPP datasets with three post-training methods. Generation sampling N varies from 2 2 with the temperature set as 0.75.
Reversal Observation As depicted in Figure 3 contrary to prior assumptions, the rapid increase in pass@N accuracy with increasing N challenges the notion of progressively harder problem-sol Specifically, as N grows, Mj achieves near-perfect pass @ N accuracy on IS(t), suggesting its inherent capacity to tackle the deemed improvement problems.
Selection Optimization for Answer Alignment The empirical findings depicted in Figure 3 offee a critical insight: iterative self-improvement hardly entails the acquisition new problem-solving abilities, but rather the enhancement of the model's 's correct answer selection within its generation space.
5.2 SOLUTIONS DIVERSITY
While pass @1 I accuracy measures the correctness of the final answer, it does not capture the diversity of solutions a model can generate. We posit that model's capacity to produce diverse solutions is indicative of its robustness and flexibility in coblem--olvingg To thoroughly understand the evolution of answer diversity during the process of iterative self-improvement, we employ a combination of Distinct N N-grams (Li et al. 2016) and Sentence-BEi embedding cosine similarity (Reimers Gurevych 2019) to measure mod diversity. These metrics have been shown to correlate well with human assessments of diversity (Tevet Berant 2021). Additionally, for mathematical reasoning, introduce Distinct Equations to measure the diversity of mathematical answers by analyzing the variety of equations in the generated solutions.
Each diversity metric Div takes set of N model outputs, and produces a scalar score representing how diverse the set is. Distinct N-grams measures syntactic diversity by counting the number of unique n-grams (averaged over 1 5). in the output set. The Sentence-BFi metric assesses
8

---

Published as a conference paper at ICLR 2025
Iterative SFT
Iterative DPO
Iterative SFT-DPO
16
NATH Whole ALC.
Paregg ACC
GSNUK
70
GSABK Pacc Acc
14
66
13
66
1.3
65
14
66
1.2
13
S
64
62
93
Group Disperity
94
92
83
62
91
90
90
89
2
2
Figure 5: Pass @ 1 Accuracy of Mr on the MATH Algebra Test Set (Post-Trainins GSM8K).
semantic diversity by embedding each output using a sentence transformer and calculating the average cosine similarity between embeddings. The metric is then minus the average similarity, ensuring that higher scores reflect greater diversity. Distinct Equations, a specialized metric for mathematical reasoning, computes logical diversity by extracting all equations from the outputs and calculating the proportion of unique equations.
At each iteration, we sample N 50 outputs per problem with a temperature 0.75. Outputs are categorized into correct and incorrect based on the final answer's correctness. Then for each problem, we use the metric Die to calculate the average diversity for the correct and incorrect answers.
Reversal Observation Figure 4 presents the diversity results of three post-training methods during the iterative process. All methods show a consistent decrease in diversity, significantly diminishing the diversity of model outputs over iterations, impacting both correct and incorrect answers. This reduction is evident across all three metrics: syntactic, semantic, and logical diversity. Moreover, comparing Iterative SFT and Iterative DPO, is clear that both methods exhibit reduction in diversity, but the extent and pattern of reduction vary. For instance, Iterative DPO maintains slightly higher semantic diversity measured by cosine similarity) over multiple iterations compared to Iterative SFT.
Trade-Off with Output Diversity. The evaluation results highlight a critical trade-off in iterative self-in relf-improvement:: while aiming for higher accuracy, the diversity of outputs, which can be crucial for creativity and robustness in problem-sole solving, compromised. Future approaches should consider strategies to maintain or even enhance diversity while improving accuracy.
5.3 OOD GENERALIZATION
In our pursuit understand the broader implications of iterative self-improvement, is crucial to assess not only the models' performance on specific benchmarks but also their ability to generalize to out-of-disttibution (OOD) tasks. Generalization performance provides insight into the robustness and adaptability of the models when faced with new and varying types of problems.
To evaluate the generalization capability of the models, we conducted iterative GSM8K dataset and then transferred these models to the MATH algebra test set. The MATH algebra test set is organized into five levels of increasing difficulty, providing a comprehensive spectrum to analyze how well the models perform across groups with varied complexities.
on the
For the sake of measuring OOD generalization, we define two metrics as follows defined to facilitate deeper analysis:
Whole Accuracy (Whole Acc.): This metric represents the pass@ @ accuracy across the entire test set, encompassing all difficulty levels from Level to Level 5.

---

Published as a conference paper at ICLR 2025
Group Disparity: This metric quantifies the difference in pass @ I accuracy between the best- performing group (Level I test set) and the worst-performing group (Level test set), thus highlighting disparities in model performance across different difficulty levels. It is calculated using the following equation:
Passs 1 (Level 1) Pass@1
Group Disparity
Pass @1 (Level 1)
A higher value of Group Disparity indicates that the model is performing significantly better on the easier Level while its performance deteriorates on the harder Level group.
Reversal Observation As results shown in Figure with the increase in iterative steps, Iterative SFT and Iterative SFT-DPO can significantly harm the OQD generalization. In contrast, Iterative DPO demonstrates noticeable improvement, which may indicate better generalization the OOD test set, in consistent with the recent findings that DPO can improve OOD generalization (Kirk et al. 2024). However, our more detailed examination of the results across Group Disparity shows Iterative DPO is widening the performances between the easier and harder groups This comparison uncovers the OOD performance improvement from Iterative DPO actually stems from fitting simpler problems, at the expense of solving more complex ones.
Capabilities Collapse All three iterative post-training methods can exacerbate the generalization disparities across groups, inadvertently causing models to focus on easier problems rather than enhancing their ability to solve more complex ones. As discussed in Section 5.2 the decrease in solution diversity during iterations may be the bottleneck leading to reduced OOD generalization and capability collapse. This highlights the intricate nature model capabilities under self-improvement, where capabilities different levels and different facets will compromise each other. Therefore, research developing more sophisticated methods should employ such a comprehensive, fine- inee-grined evaluative framework to monitor post-training processes, an increase in single facet of accuracy does not necessarily represent true self-improvement.
6 EPILOGUE
Conclusion In this paper, we foster a comprehensive understanding of the current landscape of post- training practices in self-improvement. Our evaluation, beyond simple pass @ 1 accuracy, utilizing multifaceted metrics such as improvement problems, solutions diversity and OOD generalization, underscores the necessity for critical examination of both the progressive and regressive effects in current self-improving deeper insights into the true nature of iterative ielloiimpoveeeett with post- post-training, paving the way for more robust and genuinely self-improving LLMs.
post-training methods. By broadening the scope of our analysis, we provide
Limitations and Future Work Despite the comprehensive evaluation and nuanced insights pro- vided by our study, there are several limitations consider. Firstly, while our investigation covers variety of iterative post-training methods, the scope of our experiments is constrained by computa- tional resources, limiting the range of models and tasks could explore. Secondly, our evaluation metrics, although more holistic than traditional measures, may still not capture all dimensions of model performance and behavior, particularly in real-world applications. Thirdly, the iterative nature of our methodologies requires extensive training cycles, which can be computationally expensive and consumin,, potentially limiting their practical applicability in environments with limited resources.
Our future work would like address the limitations identified in this study. Expanding the range of models and tasks, particularly those involving more diverse and complex real-world scenarios, will provide more comprehensive understanding of iterative self-improvement. Additionally, developing more sophisticated and multidimensional evaluation metrics will help in capturing the full spectrum of model capabilities and limitations. Future studies could also explore optimizing the computational efficiency of iterative post-training methods, making them more accessible for broader use. Morcover, investigating the long-term impacts of these methodologies on model robustness and adaptability will be crucial in ensuring sustainable advancements in LLM capabilities.
10

---

Published as a conference paper at ICLR 2025
REFERENCES
A Meta. Introducing meta Ilama 3: The most capable openly available Ilm to date. 2024. URL ctt.e/cccccclcccoo www.co/o........c...oo..oooo...t meta 1lama- 3/
https://ai
Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoe Le, et al. Program synthesis with large language models. arXiv preprint (rXxi:2100807732, 2021.
Zixiang Chen, Yihe Deng, Huizhuo Yuan, Kaixuan Ji, and Quanquan Gu. Self-play fine-tuning converts weak language models to strong language models, 2024.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems, 2021.
lan J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks, 2014.
Zhibin Gou, Zhihong Shao, Yeyun Gong, yelong shen, Yujiu Yang, Nan Duan, and Weizhu Chen. CRITIC: Large language models can self-correct with tool-interactive critiquing. In The Twelfth International Conference on Learning Representations, 2024. URL https: httpt:::///peereviee net bet/Eo=ze
Yves Grandvalet and Yoshua Bengio. Semi-supervised learning by entropy minimization. In L. Saul, Y. Weiss, and L. Bottou (eds.), Advances in Neural Information Processing Systems, volume 17. MIT Press, 2004. URL URL
files/paper/
Dan Hendrycks. Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. NeurIPS, 2021.
Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. The curious case of neural text degeneration. In International Conference Learning Representations, 2020. URL https //openreview. net/ fottte/oorrmm11rryoyyyyyyyr
Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations, 2022. URLhttps: /wwp//ppeeee..eeeeeeeeeeeeee..ee nwt//frum? ld=nzeeKeeFyfF
Jiaxin Huang, Shixiang Gu, Le Hou, Yuexin Wu, Xuezhi Wang, Hongkun Yu, and Jiawei Han. Large language models can self-improve. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 1051-1068, Singapore, December 2023. Association for Computational Linguistics. doi: 10. 18653/ wennp-mmaing. 67
#1/2023.emmll-mma maw.mmmmmmmmm.....m.m URL http:sss/aaalletoooooooooo//////
Jie Huang, Xinyun Chen, Swaroop Mishra, Huaixiu Steven Zheng, Adams Wei Yu, Xinying Song, and Denny Zhou. Large language models cannot self-correct reasoning yet. In The Twelfth International Conference on Learning Represennations, 2024. URL http:tttt///ppeeeeeeet
net net/Eorumt =d hiw.mmmmmmmmmmmmmmmmmmmm
Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b. arXiv preprint arXiv:2310.06825, 2023.
Robert Kirk, Ishita Mediratta, Christoforos Nalmpantis, Jelena Luketina. Eric Hambro, Edward Grefenstette, and Roberta Raileanu. Understanding the effects of RLHF on LLM generalisation and diversity. In The Twelfth International Conference on Learning Representations, 2024. URL https:://ppenrett www.ew.wweeeeeeeeeee.eeeeeeeeeee
net wtt.ee//o/oooooo////////SSSSS/

---

Published as a conference paper at ICLR 2025
Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners. Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Neural Information Processing Systems, 2022. URL ew. net forum? 2 2
Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles, 2023.
Harrison Lee, Samrat Phatale, Hassan Mansoor, Kellie Ren Lu, Thomas Mesnard, Johan Ferret, Colton Bishop, Ethan Hall, Victor Carbune, and Abhinav Rastogi. RLAIF: Scaling reinforcement learning from human feedback with AI feedback, 2024. URL
[Eormm2ddAAxxssD222
Jiwei Li, Michel Galley, Chris Brockett, Jianfeng Gao, and Bill Dolan, A diversity-promoting objective function for neural conversation models. In Kevin Knight, Ani Nenkova, and Owen Rambow (eds.), Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 110-119, San Diego, California, June 2016. Association for Computational Linguistics. doi: 10.18653/v1/ N16-1014, htt.at/ss//aacaatttoooooooo/o///
Xian Li, Ping Yu, Chunting Zhou, Timo Schick, Omer Levy, Luke Zettlemoyer, Jason E Weston, and Mike Lewis. Self-alignment with instruction backtranslation. The Twelfth Internaiional Conference Learning Representations, 2024. httpss://ppeerreeee...
Ad-loijjHJe HH]RSS
Yiheng Liu, Hao He, Tianle Han, Xu Zhang, Mengyuan Liu, Jiaming Tian, Yutong Zhang, Jiaqi Wang, Xiaohui Gao, Tianyang Zhong, Yi Pan, Shaochen Xu, Zihao Wu, Zhengliang Liu, Xin Zhang, Shu Zhang, Xintao Hu, Tuo Zhang, Ning Qiang. Tianming Liu, and Bao Ge. Understanding Ilms: A comprehensive overview from training to inference, 2024.
Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye. Yiming Yang. Shashank Gupta, Bodhisattwa Prasad Majumder, Katherine Hermann, Sean Welleck, Amir Yazdanbakhsh, and Peter Clark. Self-refine: Iterative refinement with self-feedback. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL
Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback, 2022.
Richard Yuanzhe Pang, Weizhe Yuan, Kyunghyun Cho, He He, Sainbayar Sukhbaatar, and Jason Weston. Iterative reasoning preference optimization, 2024.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea
Finn. Direct preference optimization: Your language model is secretly reward model. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL https //openreview.oo nnt/ forum? idd HPSSIIaaa
Nils Reimers and Iryna Gurevych. Sentence-BERG BERT: Sentence embeddings using Siamese BERT- networks. In Kentaro Inui, Jing Jiang, Vincent Ng, and Xiaojun Wan (eds.), Proceedings of the 2019 Conference Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pp. 3982-3992, Hong Kong. China, November 2019. Association for Computational Linguistics. doi: /0.19653/
URL https: http:://aalantt..o. waw.acannnntoooooooooooooooooooa org.D19-1111111111
Soaificant-Graviite. Autogpt: Build & use ai agents. https://githi: com/
Si Signi cantGravitas AutootTT 2023. Accessed: 2023-06-27.
12

---

Published as a conference paper at ICLR 2025
David Silver, Aja Huang, Chris J Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schrittwieser, loannis Antonoglou, Veda Panneershelvam, Mare Lanctot, Sander Dieleman, Dominik Grewe, John Nham, Nal Kalchbrenner, llya Sutskever, Timothy P Lillicrap, Madeleine Leach, Koray Kavukcuoglu, Thore Graepel, and Demis Hassabis. Mastering the game of go with deep neural networks and tree search. Nature, 529(7587):484 489, 2016. doi: 10.1038/ nature 16961.
Avi Singh, John D. Co-Reyes, Rishabh Agarwal, Ankesh Anand, Piyush Patil, Xavier Garcia, Peter J. Liu, James Harrison, Jaehoon Lee, Kelvin Xu, Aaron Parisi, Abhishek Kumar, Alex Alemi, Alex Rizkowsky, Azade Nova, Ben Adlam, Bernd Bohnet, Gamaleldin Elsayed, Hanie Sedghi, Igor Mordatch, Isabelle Simpson, Izzeddin Gur, Jasper Snock, Jeffrey Pennington, Jiri Hron, Kathleen Kenealy, Kevin Swersky, Kshiteej Mahajan, Laura Culp, Lechao Xiao, Maxwell L. Bileschi, Noah Constant, Roman Novak, Rosanne Liu, Tris Warkentin, Yundi Qian, Yamini Bansal, Ethan Dyer, Behnam Neyshabur, Jascha Sohl-Dickstein, and Noah Fiedel. Beyond human data: Scaling self-training for cooblem-solvingg with language models, 2023.
Zhiqing Sun, Yikang Shen, Qinhong Zhou, Hongxin Zhang, Zhenfang Chen, David Cox, Yiming Yang, and Chuang Gan. Principle-driven self-alignment of language models from scratch with minimal human supervision. In seveenh Conference on Neural Information Processing Systems, 2023. URLhttts: http:/:///ppeeeeee..eeee. www.eeeer///... d=w.mm/
Zhiqing Sun, Yikang Shen, Hongxin Zhang. Qinhong Zhou, Zhenfang Chen, David Daniel Cox, Yiming Yang. and Chuang Gan. SALMON: Self-alignment with principle-following reward models. The Twelfth International Conference on Learning Representations, 2024. URL
htttps://ee //tppreve//eeeeeieee net wttp:/ormmmxxxxsssnnnm-ee
Fahim Tajwar, Anikait Singh, Archit Sharma, Rafael Rafailov, Jeff Schneider, Tengyang Xie, Stefano Ermon, Chelsea Finn, and Aviral Kumar. Preference fine-tuning of Ilms should leverage suboptimal, on-policy data, 2024.
Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. Commonsense@A: question answering challenge targeting commonsense knowledge. In Jill Burstein, Christy Doran, and Thamar Solorio (eds.), Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long and Short Pupers). pp. 4 4149-4158, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 10.18653/wI/N1- 1421. URL http:///aaaaattooooo.oooo wwwpoog.oorr.ooooo www.orrggli 9-1-11199110 1421
Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Stanford alpaca: An instruction-followinng llama model.
httpps//giit httpggttttuu...e. com /tt///tttuuuuaa laatsu-lab/aa stanoord_ alpcca 2023.
Guy Tevet and Jonathan Berant. Evaluating the evaluation of diversity in natural language generation. In Paola Merlo, Jorg Tiedemann, and Reut Tsarfaty (eds.), Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguisties: Main Volume, pp. 326-346, Online, April 2021. Association for Computational Linguistics. doi: N0.1188566112223 eacl-main.25 URL https: //acl antholooyyyy..ooo org .ec..eacc.l..............o.cll.n mamecmmmin.2... mm.....n..mm....... 25
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprini 2rXiv:2307.092888 2023.
Yizhong Wang, Yeganch Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith, Daniel Khashabi, and Hannaneh Hajishirzi. Self-instruct: Aligning language models with self-generated instructions. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.). Proceedings of the 61 Annual Meeting of the Association for Computational Linguistics (Volume Long Papers), pp. 13484- 13508, Toronto, Canada, July 2023. Association for Computational Linguistics. doi: 10. 18.15v13/v1/ 2023.acl-logg.755 URL /ttpp:tttp/// ac antto wrooorgg///// 2023 .tte22..clll1.. llng. 754
Longhui Yu, Weisen Jiang, Han Shi, Jincheng YU, Zhengying Liu, Yu Zhang, James Kwok, Zhenguo Li, Adrian Weller, and Weiyang Liu. Metamath: Bootstrap your own mathematical questions for large language models. In The Twelfth International Conference on Learning Representations, 2024. httttss://pperreiii ew. net. wtt.ee/ooorrmmm.eeeeeeaaaaa
13

---

Published as a conference paper at ICLR 2025
Weizhe Yuan, Richard Yuanzhe Pang, Kyunghyun Cho, Sainbayar Sukhbaatar, Jing Xu, and Jason Weston. Self-rewarding language models, 2024.
Eric Zelikman, Yuhuai Wu, Jesse Mu, and Noah Goodman. STar: Bootstrapping reasoning with reasoning. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Neural Information Processing Systems, 2022. URL
net/Forum?id= 2
Yaowei Zheng. Richong Zhang, Junhao Zhang. Yanhan Ye, Zheyan Luo, and Yongqiang Ma. Llamafactory: Unified efficient fine-tuning of 100+ language models. arXiv preprint arXiv:2403.13372, 2024.
13372
14

---

Published as a conference paper at ICLR 2025
APPENDIX
A ALGORITHMIC OVERVIEW OF LLM POST-TRAINING
A.1 SUPERVISED FINE-TUNING
Supervised fine-tuning (SFT) is employed to tailor a pre-trained LLM to specific downstream tasks. Consider the training dataset D = (x) } where x(() is the problem and y is the target response, which the model M parameterized by 0 is trained to generate. The training objective of SFT is to minimize the following negative log-likelihood of the answers:
S (L,,)) log p(yla;;
where ply (2) is the probability of observing the answer given the problem context
A.2 PREFERENCE LEARNING
Preference learning is commonly used train large language models to learn human preferences. The preference learning dataset includes not only problem and target response pairs but also preferences or rankings between different target responses for the given problem. A typical form preference
(i) y where each piece of data contains problem learning data is represented as DD = (x (1) "1/
200), and corresponding preferred and dispreferred responses, denoted y/ and y respectively. Using theoretical model of human discrete choice such the Bradley-Terry model, which relates discrete choices to implicit goodness scores of the underlying options, we can train a reward model with maximum likelihood using this preference data. For the Bradley-Terry model, the reward modeling loss is:
LR D) -B(a.ee. (n) log (ra(z, (x)))))))
In the context LLMs, To (x, 3) is initialized from the SFT model ØSFT- Then, the learned reward function is used to provide feedback to the language model, through the optimization problem described below to train preferences in the language model:
(R maX (g--D.y-movl-)) (R, 3)1 - 3D (noool=)\/sssrr(/l
(6)
where B is parameter controlling the deviation from the base reference policy TsFr- More recently, (Rafailov et al.] (203)) show that the optimal policy for the learned reward can be extracted in closed form, especially skipping the need perform iterative, approximate policy learning. The resulting algorithm, direct preference optimization (DPO), is simpler to tune and less computationally demanding than prior methods, while optimizing the same objective. We therefore DPO the algorithm for the implementation for preference learning in our experiments. The DPO loss for the language model policy Te
(3108g (
me (y/1 (x) TSFT (Yw x) TSFT (y/ x.) )
To (yo x) - B log
LDPO -E wDDD log
(7)
B EXPERIMENTS
B. TRAINING DETAILS
We use fully fine-tuning setting for training LLaMA2-7B, Mistral-7B and LLaMA3-8B models either for supervised and preference fine-tuning, All training experiments are conducted on 8 NVIDIA A100 GPUs, and experiments collectively consumed approximately 2000 A100 GPU hours. Our training codebase is based LLaMA Factory (Zheng et al., 2024), and we vLLM (Kwon ctt al, 2023) framework to perform inference for both CoT sampling and test evaluation. Detailed hyperparameters utilized throughout these experiments documented in Table 1
15

---

Published as a conference paper at ICLR 2025
Type
Parameter
Value
Supervised Fine-Tuuing
Batch Size Learning Rate (LLaMA2-7B}
128
le
Learning Rate [Mistral-7B, LLaMA3-8B)
2c
Learning Rate Scheduler
Cosine
Warm-up Ratio Optimizer Epoch
0.03
AdamW
3
Preference Fine-Tuning
Batch Size
Learning Rate (LLaMA2-7B]
2e -
Learning Rate (Mistral-7B, LLaMA3-8B)
2c
KL Coefficient (3)
0.3
Optimizer
AdamW
Epoch
Sampling Generation
Temperature
0.75
Top-p
0.95
Top
Max. tokens
512
Evaluation Generation
Temperature
Top_k
-1
Max_tokens
512
Table I: Hyperparameters all the experiments.
B.2 DATASET DETAILS
CommonsenseQA (CSQA) (Tammor al.] 2019) offers collection of 5-way multiple-choice questions on commonsense knowledge scenarios. contains 12, 102 questions with training/valida- tion/testing set splits, Due the unavailability correct answers for the testing set, we utilize the validation set comprising 1,221 questions for evaluation, following the practice of (Kojima et al. 2022).
GSM8K (Cobbe et al. 2021 consists of 8.5K high-quality grade school math problems created by human problem writers, with the segmentation into 7.5K training problems and 1K test problems. These problems take between and 8 steps solve, and solutions primarily involve performing a sequence of elementary calculations using basic arithmetic operations (+ / *) to reach the final answer.
MATH (Hendrycks et al. 2021) offers high school math competition problems that span seven subjects including Prealgebra, Algebra, Number Theory, Counting and Probability, Goemetry, In- termediate Algebra and Precalculus. consists of 7,500 and 5,000 samples for training and testing, respectively. Compared to GSM8K, addressing MATH challenges involves more intricate and extended steps.
MBPP (Austin et al. 2021 consists of around 1,000 crowd-soured Python programming problems, designed to be solvable by entry-level programmers, covering programming fundamentals, standard library functionality, and on. Each problem consists of task description, code solution and automated test cases. Following the experimental setup described in (Austin et al. (021). we utilize Task IDs 11-510, comprising 500 problems, as our test set. The remaining 374 problems, ranging from Task IDs 601 to 974, are employed for fine-tuning purposes.
B.3 EVALUATION PROTOCOLS
Zero-shot Prompting We employ zero-shot prompts, as listed in Figure for answer sampling and evaluation tests. Our comprehensive evaluation across all benchmarks demonstrates that zero-shot prompting not only reduces inference costs but also consistently outperforms few-shot prompting in terms of performance. Consequently, when LLMs are fine-tuned for task-specific applications, we advocate for the adoption of zero-shot prompting superior method compared various few-shot
16

---

Published as a conference paper at ICLR 2025
techniques. This perspective aligns with the findings of (u et al. 2024). who also reported the advantages of zero-shot over few-shot prompting for fine-tuned LLMs.
CommonsenseQA
Quastion: fquestion/oAnswer... Let's think step step.In
GSM8K
Question: (questionjnAAsswer..c Let's think step step.In
MATH
Question: (questionjknAnsswer..n Let's think step step.In
MВPP
You expert Python programmer, and here your task: (question) Your code should pass these tests/'nin (test_ list) in[BEGINJin
Figure 6: Zero-shot Evaluation Prompt.
Generation Diversity evaluating natural language generation (NLG) models, two prevalent methods for assessing output diversity the n-gram-based metric and the embedding-basee metric, which embeds generated sentences latent space. In this paper, we adopt distinct n-grams (Li
al. 2016) and Sentence-BERS Embedding Cosine Similarity (Reimers & Gurevych 2019) metrics.
Distinct n-grams is a straightforward yet effective method to quantify the lexical diversity of generated text. This metric calculates the proportion of unique n-grams (sequences within the generated text. The distinct n n-gram measure is typically computed for unigrams, bigrams, trigrams, and sometimes higher-order n-grams. Mathematically. for generated sequence S, distinct-n is defined as:
words)
distinct-n(S) Junique n-grams in S
(8)
ttoal n-grams in S!
This measure provides direct indication of how varied the vocabulary and phrases are within the generated text. In general, higher distinct-n values indicate greater diversity.
Sentence-BEi embedding cosine similarity assesses the semantic diversity of generated sentences. Sentences generated by the model are first encoded into embeddings using Sentence-BEi The cosine similarity between each pair of sentence embeddings is then computed. Cosine similarity between two embeddings u and v is given by:
cos( (u, v)
|v]
The average cosine similarity of all sentence pairs gauges the overall semantic similarity. Lower average cosine similarity indicates higher semantic diversity, as the sentences are less similar in meaning. In our calculations, measure diversity using 1 average cosine similarity, ensuring that higher values reflect greater semantic diversity.
Distinct Equations provides direct indication of how varied the mathematical approaches and solutions are within the generated text. The calculation this metric involves two steps that identifies all equations present in the generated mathematical reasoning steps first, and then computes the ratio of unique equations to the total number of equations. Mathematically, for a set of generated equations E, distinct equations is defined as:
Junique equations in E
Deg total equations in El
(10)
Higher Day values indicate greater diversity in the mathematical reasoning processes.
17

---

Published as a conference paper at ICLR 2025
C EXTRAPOLATION ANALYSIS ON COVERAGE
In Section 4.2
mentioned that correct answer coverage may be deeper factor influencing the subsequent improvements in iterative self improvemrenn Here, we provide detailed explanation of the related concepts involved in this influencing factor, as well the derivation of the calculation for correct answer coverage (in Equation 1 as presented in this paper.
Additionally, emphasize that our consideration of coverage as a deeper factor is preliminary conclusion drawn from summarizing the factors of the model (M), post-training function (F), and task dataset (D) and the empirical results validated in Figure 2 should be noted that need further work and more extensive experiments to both theoretically and empirically validate this observation, as the discussion of correct answer coverage is beyond our work. Our intention here is offer empirical insights and lay the groundwork for future investigations into this aspect of iterative seff-improvement.
Answer space: For given dataset D (test set), all possible (query, answer) pairs form the answer space of the test set. Here, the set of query fixed, and for given query, the number possible answers can be quite large, hence we call the space the answer space. Naturally, the entire answer space can be partition into a correct answer space and an incorrect answer space based on whether the answers are correct. In practical experiments, the correctness of an answer is approximated by whether its final result exactly matches the final result provided by the ground truth in the training set.
Answer distribution: For given model M, its answer distribution refers to the probability dis- tribution of generating answers conditioned on queries from dataset D. For a specific element (gt, (.)...%))...) Dust in the answer space, the generation of this (query, answer) pair occurs in two steps: first, sampling 9. from all queries in D, then model M generates a.y conditioned on gi- Therefore, the probability (g:, aij) is the product of the probability of sampling ş from all queries in D and the probability of model M generating a. conditioned on Considering that all queries should have equal importance, we define that all queries are sampled with the same probability, which is The answer distribution of M can be mathematically linked to model M as follows:
Den
Par(gjayg) del M; (ais))))P())) = (aiyla) Drest
(11)
which Pa represents the model's answer distribution, and M (a:yla)) denotes the probability of model M generating di conditioned qj-
Correct Answer Coverage: As mentioned earlier, the Correct Answer Coverage represents the correctness rate of all answers generated by model M dataset D (training set), It can be calculated using the following mathematical formula:
Correct Answer Coverage
(ar(a, q)
(12)
Correct Answer Space
Although we cannot exhaust the entire answer space and calculate probability distribution to demonstrate the trend of the answer distribution in the progress of self-improvemenn, we can get an unbiased estimate of by sampling answer and calculate the ratio of the number of correct answers to the total number of answers generated by model M for all queries in Diest where N answers are generated for each query, as illustrated in equation The proof of Equation 1 is follows:
18

---

Published as a conference paper at ICLR 2025
Correct Answer Coverage
(M((a
[ Dusl
1
Pas(alg)
1
Par(alg)
[Dessl
1
Ic DE
Σ
PMrlaq= = x)
(Desl
Σ zEDie
Pa x))
(13)
Dieul
No.
E
Diest
N
zEDie
No. N
E(
iDestt
D
E
IM(x, == vl]
Diest
N
DE
ZE
f=1
where C' denotes the correct space, and N represents the number of correct answers generated by model conditioned on the given query v.
D SCALING EXPERIMENTS
To validate that the phenomenon of self-improvement widely exists across different foundation models, ranging from 7B more capable models, in this section, scale the iterative self-improvement practices to LLaMA-2-70BB (Touvron et al. 2023). As observed in Figure 2 Iterative SFT-DPO proves be robust practice that achieves consistent performance improvements regardless of the correct answer coverage being lower or higher. Considering the limitation of GPU resources, we hence set up the scaling experiment with the following parameters: the foundation model M is LLaMA-2-7065 the problem-solving task ' is GSM8K, the post-training function F is Iterative SFT-DPO, and the number of iterations T 5. Additionally, we employ quantized low-rank adaptation (LoRA) (Hu al. 2022) for efficient post-training.
AMMBAACCASO and OOO Turder
GP4R
GCMB Anowar Dharr
GOMaK- Deanns Equation verdit
Cornct hoate
Trandin Reoalies
Semplng Caneratora from
Tuising textions
Trah Baradora
150
Figure 7: Perform iterative SFT-DPO GSM8K (a), and then evaluate M, with proposed metrics:
pass @ N on improvement problems (b), solutions diversity (c, d) and OOD generalization (a).
As shown Figure 7 the performance of LLaMA2-70B on GSM8K demonstrates similar self- improving trend, first descending to optimal @ I accuracy and then declining after the fourth iteration. From Figure (b), it is evident that iterative self-improvement primarily involves the selection of correct answers within its generation space. Additionally, the solution diversity illustrated in Figures (c) and (d) highlights that the trade-off between p accuracy and output diversity is a universal phenomenon, even for highly capable 70B models.
Regarding OOD transfer accuracy shown in Figure (a), we observe that while performing Iterative SFT-DPO for self-improvement GSM8K, I.I L.MMA2-70B exhibits certain degree of OOD gen-
19

---

Published as a conference paper at ICLR 2025
eralization the more challenging MATH test set. This represents opposing trend compared with the whole accuracy of MATH shown in Figure 3 We conjecture that with the scaling of model capacity, the capability of OOD generalization will gradually emerge and evolve within the iterative seff-improvement process.
E
CASE STUDY
In this section, we select one problem from GSM8K test set and record the outputs of Mistral-7b during the process of iterative DPO. This real case can vividly display findings from Section 5 which suggest that iterative sell-improvement primarily manifests as an optimization of answer selection rather than substantial enhancements in problem-solving capabilities. Additionally, we will clearly see how diversity of the model's 's reasoning steps decreases throughout the iterative process.
20

---

Published as a conference paper at ICLR 2025
Itteration 1
Ilteration 2
Greedy Decoding Answer
Gready Decoding Answer
You have 2000- 1850 - 220 calorise left for the day. Simna each serving has 250 caloriee, 5 servings havn
bag of chips has 250 catories serving and there caleries. You have alreaily
Question
250'S colories. This, you can eat 1250:200
calerios have 2800-1830 250 caleriee left: A bag of chips has 1260 calories and you have 200 caloriee left, so you can wat 1250 200 1058 grams of shipe drem 1959
According to its nutritional Info, a bag of chips has 250 calories per serving. a 300g bag has 5 servings, how many grams can you not your daily calorie targel is 2000 and you have already consumee 1800 calories?
grams chips.
daily calorte targed 2050,
Sampling Answers
bag of chipa has servings, and you can 700220
2.8 servings. Bince serving 150g, you can
Sampling Answers
Annotated Answer
you con ean 2000-1888 260 calories more. Since each serving 88 calories, you out 200/59- servings.
If the hegg has 6 servings, ench serving is 25055 a 60 calories. You have aloeady consumedd 1503 calories, so
Eash serving has 259 calorios so 266250 0.8
servings. 0.5 servinga is 0.5 x 300 8. 8.5-300-240 grama. stue 240
You still have 2000 1880 200 calories left for the
If the total calorie target 2000 and I have
consumed 1800 calories then have 2000-1800
servings 130ggh sarvings you 240
Ifa 200g has servings, each
300/5
= 200 calories left to eat. each serving of
00g. If Bogg of chips has 260 calories per serving.
a serving that 300g bag chips servings. hence serving has 300g/5 grams. If can only est serving, 240gi5 grams 48
chips has 250 calories and only have 200
calories left eat, then can only ext 200/250
serving We also know
and you 1258 calories. Your daily calorie targad ln 2000,
It sersing has 260 catories, then the entire bag has
consumed
so you
you have already consamed 1800 calories, then you
can ast 2000-1800 203 calories worth of chips, Since
then 6og of chips have 250 sampo 150 calorien.
est only 60g
calorias the bag chips, 88.10 of the bay There are 500g in the kag. 0.10 of the bag in 35009.16 48g seap 40
203
ent 2081120
1250
shippp 2081168'68 120g vote
Since a bag of chips has 6 corvings and ench sorving has 250 catories, the whole bag has 57255 1250 catories. if you have alrendy concumad 1800 caloriee
1250-208 1050/250 cervings. you out 4 servings 330giserving 1200g chips.
2000- 200 calories. catories per serving caleries.
and your daily caloria targed 2000, then you can siil
1260 calories,
20011250
shird 1200
Since only eat 1.10 a bag. than you can est 0.10"300 in dllg
chips weight 350g
It sersing of chips has 250 catories, then 1860
stre 48
1800/500 servings. you
serving
calories
57250
1250 calories. You have alruady consumed 1850 colories so you have 2000-1880 208 calories left to
calories are left. Since serving dince serving 200:200 0.8 servings
calories,
consume. You can eut 206/250 - 0.5 servings. One serving 150g so you can est B 128 grams
equal to 50 grams, then 0.6 servings a gramssborving 40 grams can bo oalinnatter d)
paus 120
Iteration
Iteration
Iteration 5
Greedy Decoding Answer
Greedy Decoding Answer
Greedy Decoding Answer
a serving has 259 calories, servings will hare 29979 1238 caloriea. you have diresdy consumed
bag has 250
per serving,
IT's serving hes 250 calories, bag has servings, alroady
has
then the
253'5
1930 calories, you can still 2080-1802 Eince
1158 calories. you have already consumed 1860
$800 colories, then
you
calorias, then you can wat 2008-1860 wond cacoriss
2000-1602 208 caleries worth chips. Since each chip has 1259 caloriea, than you can est 20011250 el 0.16 bags of chiips. Sinco a bog has 500g, chon you can 8:18 500 a <8p ships. slaid 48
200/200 servings. eat 0,8'58 a <.0.9" 50 90-4400-4ig acds 40
corving 90p.
worth of chipa. Since each chip has 1250 caloriea, you can ase 20811250 a hage chipa. Sance each bag chips 300g, you can eat 0:100300 d8 grams of
Sampling Answers
chips.
Sampling Answers
serving 25015+1888 caleries.
chips
Sampling Answars
If' 300g bag has servings, each serving 300/5 6eg. If your daily caloris target in 2010 and you have already consamee 1608 calories, you san consurte catories more. Since each serving calories, you consume 60g, you servings
1800 ealoriee, then you have 2008- 18001:200 caloring las
already sumed
If nerving has 250 calorine, then a hag of chipe hax 559120112000 calories. you have alroady oo smeedd
bag of ohipp, grams, this would be 0.8 2.-3300-200 free 240
left for the day. Tharefore, you can eat 200/25040.6 a of
1800 catoriess tham you have 2000-10022 2000-1800-200 calories
the day. Therofore, you can 8.8"350-240 groms,
2000- 1850
If 300g bug has 5 servings, such serving 300/5
9. 0.8 40gg edey 48
calwriss so 60g has 55100 200 cafories. If
<ig. The nutritional info says a serving has 252
have stready consumed
cateries, you have 2000-
60g. bag chips
310g bag has servings, each serving is 20015
chipa 250
caleries por serving.
It's 300g brg has servings, each serving 300/5 your daily colorie target 2009
2003-1004 200 calories left. Since bag chipe 250
have
you calortes
have already conesmee 1603 calories, then
nut 2800-1850- 260 calories worth of chipa dince 60g chips have 250 calortes, then 220/300 0.0 servings chips can be eoten. Therofore, you can ost 8.8"60 40g chips.
calories serving, con eat 200/250 0..
a serving has 250 catories, then the whole bag has 23055 = 1250 calories. you have alraady consamer umed 1800 caloriee, then you can 2000-11000 200)
servings. Since each serving dig you can ext 0,8%60 60 40g of chips. sd80 46
lagg chips are survings in 900g bug. the 300g beg has 2s01s in 1250 caloriee if 300g bag has 1258 calories, and you have alroady consunod 1800 catories, then you can 9255-1850 -350 cateries worth of chips. bilegramm Sinso there are 10800-0. gramss 550
calories serving, and there
calories worth chips. Since there are calorive ting, the Therefore, you equai chipa. edeu
A surving has 250 calories and there are servings in
the ban, so the whole bag hae 2505 1250 calories.
63
You have already so sonsumed 1990 caleries and your
daily target 2000, so you can still come umm 2010. you
260 caleeios. bag has cateries and
you
200 grame.
Figure 8: One case of sampled responses from the test set after iterative DPO training of the Mistral- 7B model on the GSM8K dataset.
21


Reviews:
[{"review_id": "dPoGYSZF3e", "reviewer": "Reviewer_rHQ5", "summary": "This paper analyzes post-training via self improvement loops for LLMs. Their results suggest that there is a sort of \"Goodhardt's Law\" at play sometimes, where improving the benchmark scores will often lead to decreased performance after some point. They particularly note diversity as an important metric which often drops.", "questions": "Do you expect these results to hold with RL posttraining?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "Good idea and experiments. It's an important and timely topic. The observation that some metrics like diversity are sometimes unmeasured with these methods is very important.", "weaknesses": "I think the writing can be improved somewhat, even if the core ideas are clear.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "iZLAyR0PWM", "reviewer": "Reviewer_Kgyn", "summary": "This work points out a phenomenon called self-improvement reversal. To be specific, the authors formulate three post-training\nparadigms (iterative SFT, iterative DPO, and iterative SFT-DPO) and conduct experiments to examine how various factors (iteration steps, different models, task difficulties) influence self-improvement. Although such strategied shows a general trend of improvement in pass@1\naccuracy. They still struggle in some other problems like solutions diversity and OOD generalization. This paper provides a deeper understanding to self-improvement.", "questions": "No more questions.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The experiments are well-designed and provide a comprehensive understanding to the internal mechanisms of self-improvement.\n2. This paper finds that such pass@1 accuracy metric may lead to a wrong judgment about model performance.\n3. The experimental results in Figure 2 are insightful. The relationship between correct answer coverage and iterative post-training methods has not been discussed before.", "weaknesses": "1. Some of the conclusions in this paper are as expected, e.g. results from Table 3 are related to [1].\n\n[1] Large Language Monkeys: Scaling Inference Compute with Repeated Sampling", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Qb2rlblO3K", "reviewer": "Reviewer_eamz", "summary": "This work studies the post-training steps of supervised fine-tuning and preference optimization during iterative self-improvement, where an LLM improves by training on its own generated solutions. The authors claim that such iterative post-training may show progress according to traditional top-1 test accuracy measures, but may also negatively impact semantic diversity and OOD performance on other tasks. They propose an iterative self-improvement algorithm which, given a training set of inputs and ground truth outputs, iteratively fine-tunes a model on self-generated outputs labelled according to whether they are correct or incorrect, serving as synthetic preferences similar to RLAIF. The authors compare 3 methods of fine-tuning: SFT, DPO, and alternating SFT-DPO. They find that improvement plateaus for most tasks after ~3 iterations before declining, as well as a few other findings. Next, they show that while accuracy on the test set increases over post-training, diversity across a number of metrics may be reduced as well as accuracy on OOD tasks.", "questions": "- Figure 3: why is it surprising that pass@N accuracy increases with greater N?\n- How does correct answer coverage relate to solution diversity? Are the two at odds with each other?\n- Is solution diversity inherently desirable in domains like mathematical reasoning? Is it possible that higher performing models on certain domains always have lower diversity?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- Well-motivated and timely. Review of related work is thorough.\n- Iterative self-improvement post-training algorithm and most of the analyses seem general enough to be applied to arbitrary domains with minimal changes.\n- Interesting reversal results with diversity and OOD generalization decreasing during post-training. Used a variety of diversity metrics.", "weaknesses": "- I'm not sure I understand Improvement Sets (Section 5.1). Perhaps I'm missing something here, but in my understanding of figure 3, it seems somewhat trivial that accuracy@N increases with greater N.\n- I'm also not sure I understand the point of Group Disparity (Section 5.3). Why not just compare level 1 accuracy vs. level 5 accuracy directly, if level 1 accuracy is increasing and level 5 is decreasing?\n- Correct answer coverage seems potentially at odds with solution diversity, so I'm not exactly sure how to resolve these two metrics. Is it possible that better performing models aways have lower solution diversity?\n- Many of the results figures could be improved, including with longer captions:\n\t- Figure 1 - It's currently hard to compare SFT vs. DPO vs. SFT-DPO across (a) - (c) since different rows have different scales for the y-axis.\n\t- Figure 2 - The left figure took some effort for me to understand. Maybe example questions/answers would help.\n\t- Figure 3 - The x-axis is unlabeled and the caption is very short. As stated above, I also wasn't sure what the main takeaway of this is.\n\t- Figure 5 - The bottom y-axis has a very small range [90, 95], and I don't have a good sense for what the scale of these values should be. The caption for this also seems too short, and I don't know what the blue dotted line means.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Uyq4iCIXvS", "reviewer": "Reviewer_2Dxd", "summary": "This work goes beyond pass@1 to dive deeper into evaluating self-improvement methods for mathematical reasoning. They show that improvement in benchmark performance of self-improvement showed signs in regression in areas such as output diversity and OOD generalization. They developed a new evaluation framework for current self-improvement methods to evaluate on.", "questions": "- for table 3, can you label x-axis as `sampling N`? Also what model is used here?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- very relevant topic for the conference\n- good empirical results and comprehensive experiments\n- interesting findings that can inspire future research directions:\n\t- most methods decline in task performance after ~4 iterations\n\t- declined abilities in output diversity and OOD generalization\n\t- lesser powerful models gain more, but more powerful models achieve better max performance\n\t- M_1 performance effects subsequent iterations", "weaknesses": "- (minor) in introduction, it wasn't apparent why the new metrics are important. For example why is diversity needed in solving these reasoning tasks?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)