Progress or Regress? Self-Improvement Reversal in Post-training

Authors: ting  wu
xuefeng  li
pengfei  liu
Venue: ICML_cc_2024_Workshop_AI4MATH
Published: 2024-06-01 09:55:46

Abstract: Self-improvement through post-training methods such as iterative preference learning has been acclaimed for enhancing the problem-solving capabilities~(e.g., mathematical reasoning) of Large Language Models~(LLMs) without human intervention. However, as exploration deepens, it becomes crucial to assess whether these improvements genuinely signify progress in solving more challenging problems or if they could lead to unintended regressions. To address this, we propose a comprehensive evaluative framework that goes beyond the superficial pass@1 metric to scrutinize the underlying enhancements of post-training paradigms for self-improvement. Through rigorous experimentation and analysis across diverse problem-solving tasks, the empirical results point out the phenomenon of \emph{self-improvement reversal}, where models showing improved performance across benchmarks will paradoxically exhibit declines in broader, essential capabilities, like output diversity and out-of-distribution~(OOD) generalization. These findings indicate that current self-improvement practices through post-training are inadequate for equipping models to tackle more complex problems. Furthermore, they underscore the necessity of our critical evaluation metrics in discerning the \emph{progress or regress} dichotomy for self-improving LLMs.

Main paper: 
 introduction: 
 In the rapidly evolving landscape of artificial intelligence (AI), the pursuit of self-improving large language models (LLMs) has garnered significant attention (Singh et al., 2023; Huang et al., 2023; Sun et al., 2024) . The essence of self-improvement in LLMs lies in their capacity to iteratively refine models' own performance without hu-Building on the concept of self-training (Grandvalet & Bengio, 2004) , wherein models bootstrap their own generated responses for iterative training, a synergetic effect is observed and amplified. When models produce superior responses, the quality of the training data used to refine the models improves, subsequently enabling even better responses in future iterations. Such iterative post-training has become the standard paradigm for current self-improving AI (Yuan et al., 2024) . Notably, STaR (Zelikman et al., 2022) has demonstrated that leveraging model's self-generated reasoning steps for iterative supervised fine-tuning (SFT) can effectively enhance its reasoning abilities. Recent studies (Pang et al., 2024) have further revealed that employing iterative preference optimization in LLMs can achieve more performance improvements in reasoning tasks.Despite the various post-training methods for selfimprovement, there remains a lack of understanding regarding their effects and underlying mechanisms. Therefore, in this study, we first endeavor to provide a comprehensive overview of the main iterative post-training paradigms for self-improvement, identifying the factors that contribute to consistent performance improvements. We decouple the influencing factors into the initial model, task datasets, the number of iterations, and the specific post-training methods employed. By isolating these variables, our comprehensive experiments and analysis uncover their individual and combined effects on the model's performance. This provides actionable insights for practitioners on how to perform iterative self-improvement practices more effectively.While our extensive empirical results show that all these iterative post-training methods can achieve notable improvements in pass@1 accuracy across various problemsolving benchmarks, the evaluation has been limited to this single and superficial metric. Amidst the quest for self-improvement in LLMs, the persistent question arises: are these iterative post-training methods truly fostering progress, or are they inadvertently leading to regression? Transitioning beyond using pass@1 accuracy as the indicator of improvement, we further develop an evaluative framework equipped with a comprehensive suite of metrics to assess improvement problems, solutions diversity, and OOD capabilities within the iterative process, enabling us to scrutinize the actual improvements beneath self-improvement. Surprisingly, our evaluation results display a paradoxical trend: as pass@1 accuracy increases, the proposed metrics exhibit consistent performance declines.The perceived reversals in our evaluative framework prompt a critical reflection on the effectiveness of current selfimprovement practices. Through this study, we aim to illuminate the path forward for developing truly self-improving LLMs that balance accuracy, diversity, and robustness. To summarize, our work makes three significant contributions as follows:• Comprehensive Analysis: In Sections 3 and 4, we systematically formulate current post-training methodologies and perform extensive experiments to examine how various factors influence self-improvement in solving challenging tasks. To the best of our knowledge, this work is the first to provide an in-depth overview of these influencing factors.• Evaluative Innovations: In Section 5, we propose a holistic set of evaluation metrics to better capture the multifaceted nature of LLM performances in self-improvement practices.• Critical Insights: In Section 5, based on the evaluation metrics, we identify the phenomenon of self-improvement reversal, where increases in pass@1 accuracy comprise other essential capabilities like solution diversity and OOD generalization. 
background and related work: 
 Training paradigms for LLMs typically consist of two stages: pre-training and post-training (Liu et al., 2024) . Common post-training methods include supervised finetuning (Taori et al., 2023; Wang et al., 2023) and preference learning (Ouyang et al., 2022; Lee et al., 2024) . Supervised fine-tuning trains LLMs to produce standard responses for given instructions, while preference learning trains LLMs to align with human preferences for different responses. Both methods, however, rely heavily on extensive humanannotated data.An important question is whether effective LLM posttraining can be achieved without excessive external feedback. Predating the era of LLMs, the self-training algorithm (Grandvalet & Bengio, 2004; Goodfellow et al., 2014) demonstrated the potential to enhance model performance without additional labeled data. Recent studies have revived this concept, employing iterative self-training to facilitate self-improvement in LLMs without external feedback (Wang et al., 2023; Sun et al., 2023) . For instance, STaR (Zelikman et al., 2022) shows that iterative training on the model's own reasoning traces for correct answers can help solve increasingly difficult problems. Unlike the iterative nature of SFT, recent works (Yuan et al., 2024; Pang et al., 2024) propose iterative preference fine-tuning to aid models in self-improving.In contrast to post-training methods, another line of research explores self-improvement through iterative postprompting during inference (Huang et al., 2023) . This approach does not update the model's parameters but achieves self-improvement by generating reflections on its outputs and adjusting future outputs accordingly (Madaan et al., 2023; Gou et al., 2024) . However, as revealed by Huang et al. (2024) , post-prompting strategies are limited by the model's intrinsic self-correction capabilities, thereby failing to significantly enhance problem-solving capabilities.The potential of iterative post-training for self-improvement in LLMs remains underexplored. Although various posttraining methods have demonstrated promise in general instruction-following tasks (Li et al., 2024; Sun et al., 2024; Chen et al., 2024; Yuan et al., 2024) , they predominantly focus on aligning models with human values rather than enhancing the models' internal knowledge. A key challenge remains whether LLMs can sustain consistent performance on more complex problem-solving tasks. Recently, Pang et al. (2024) examined iterative preference learning in the context of reasoning tasks, marking the first study to expand beyond instruction tuning.Despite these advancements, a comprehensive overview investigating the effectiveness of various iterative posttraining methods for problem-solving is still lacking. First, it remains unclear how improvements vary across iteration steps, different base models, task difficulties, and iterative post-training techniques. For practitioners, there is a need for guidelines to help choose the most effective post-training method among the various iterative post-training paradigms. Second, current research only concentrates on maximizing benchmark scores through iterative self-improvement, there is little exploration of the underlying factors contributing to performance gains. As a result, the progress and reliability of different self-improvement methods are not guaranteed.In this work, we aim to address these two critical issues. Our goal is not only to ensure the effectiveness of various self-improvement methods but also to ensure that other capabilities are not compromised during the complex selfimprovement process. 
formulation: 
 Consider a training dataset D = {(x i , y i )} N i=1 , consisting of pairs of queries x i and their corresponding correct responses y i . A foundation model, denoted as M 0 . Our objective is to enhance M 0 through a self-driven iterative post-training process, leveraging the model's own outputs to refine its capabilities, without reliance on external signals.The iterative post-training process involves a series of post-training steps, each aimed at using the model's previous outputs to guide its subsequent refinement. These steps are designed to foster a continuous loop of self-improvement for the model.The process is outlined across three main phases as follows, where the total number of iterations is denoted as T , and the model employed in the t-th iteration is denoted as M t-1 , implying that M 0 is used in the first iteration • Answer sampling: In the t-th iteration, we prompt M t-1 to generate N answers for each queryx i in D to form a new self-generated dataset D self t = {(x i , y j i )|x i ∈ D, j = [1, N ]}.• Training set construction: The training set D t in the t-th iteration is assembled from D self t without introducing any external data. The approach to constructing the training set depends on the specific paradigm of post-training.• Model post-training: Utilizing D t , the model M t-1 is refined into M t .It's worth noting that, in the first iteration, we always directly supervised fine-tuning M 0 on D to initialize M 1 with taskspecific knowledge.Central to these diverse methodologies is the post-training function, symbolized as F. We distinguish among the practices based on the nature of F, involving Supervised Finetuning (SFT) and Direct Preference Optimization (DPO), the latter being an effective implementation of preference learning. During the SFT phase, this stage necessitates accurately labeled training data. We derive these correct answers from D self t to assemble the training dataset:D t = {(x i , y ✔ )|R(x i , y ✔ ) = 1, (x i , y ✔ ) ∈ D self t }where R(x, y) evaluates whether the answer y accurately addresses the question. In our problem-solving task, the correctness of an answer y is verified by its alignment with the response provided in the dataset. While during the DPO phase, for each query q i in dataset D, both correct and incorrect responses from D self t are paired to construct the training set, allowing for contrastive preference learning:D t = {(x i , y ✔ , y ✘ )|R(x i , y ✔ ) = 1, R(x i , y ✘ ) = 0, (x i , y ✔ ) ∈ D self t , (x i , y ✘ ) ∈ D self t } 3.2. Three Iterative Post-Training Paradigms Algorithm 1 Iterative Self-Improvement 1: training set D = {x i , y i }, base model M 0 , iteration times T , post-training function series [F 1 (•), F 2 (•), ..., F T (•)] 1: M 1 ← SFT ((M 0 )|D) 2: for t = 2 to T do 3: D self t = {(x i , y j i )|x i ∈ D, y j i ∼ M t-1 (x i ), j ∈ [1, N ]} 4: if F(•) == SFT then 5: D t = {(x i , y ✔ )|R(x i , y ✔ ) = 1, (x i , y ✔ ) ∈ D self t } 6:else 7:D t = {(x i , y ✔ , y ✘ )|R(x i , y ✔ ) = 1, R(x i , y ✘ ) = 0, (x i , y ✔ ) ∈ D self t , (x i , y ✘ ) ∈ D self t } 8: end if 9: M t ← F t (M t-1 |D t ) 10: end for=0Through the implementation of designated self post-training steps (e.g., self-SFT), several distinct iterative post-training paradigms emerge. Our work focuses on three paradigms: (i) iterative SFT, where each cycle consists exclusively of self-SFT steps, (ii) Iterative DPO, characterized by successive self-DPO steps, except for the first iteration which supervised fine-tune the base model M 0 , and (iii) iterative SFT-DPO, which initiates with a self-SFT step and alternates between self-DPO and self-SFT steps to form a complete iterative post-training loop.We describe the unified procedure in Algorithm 1. 
experiment: 
 As outlined in Algorithm 1, we characterize the key variables-foundation model (M ), task dataset (D), iteration steps (T ), and post-training method (F)-critically influence model performance during iterative self-improvement. This section explores the impact of these variables on different problem-solving tasks. We aim to uncover the trade-offs and comparative advantages of Iterative SFT, Iterative DPO, and Iterative SFT-DPO in enhancing performance across various tasks. Through this analysis, we seek to provide deeper insights into the mechanisms driving iterative selfimprovement. For each model along with the training iterations, we highlight the optimal result with a larger-size marker, the improvement above and final accuracy below.Datasets To measure model problem-solving capabilities, we train and test on a broad spectrum of problem-solving datasets. We measure general knowledge using the Com-monsenseQA (CSQA) dataset (Talmor et al., 2019) , assessing mathematical reasoning with the GSM8K (Cobbe et al., 2021) and MATH (Hendrycks et al., 2021) dataset, and weigh code generation skills using the MBPP dataset (Austin et al., 2021) . Regarding the train-test split, we adhere to (Kojima et al., 2022) , utilizing the validation set of CommonsenseQA for evaluation. The GSM8K and MATH datasets are employed with their predefined train-test splits. For the MBPP code dataset, we follow the approach outlined by (Austin et al., 2021) that utilizes examples of Task IDs 11-510 as the 500 test problems, and the remaining 374 examples ranging Task IDs from 601 to 974 for fine-tuning.Sampling and Rewarding At the end of training, we sample N=50 outputs for each problem using top p sampling (Holtzman et al., 2020) with p = 0.95 and tempera-ture 0.75. Considering the gold labels are provided for the problem-solving datasets, we use the correctness of final answer as a binary reward for the output to annotate the preference.Training Our experiments primarily leverages three opensource models LLaMA-2-7B (Touvron et al., 2023) , Mistral-7B (Jiang et al., 2023) and LLaMA3-8B (AI@Meta, 2024), with a fully fine-tuning setting. For the implementation of preference-based learning, we utilize Direct Preference Optimization (DPO (Rafailov et al., 2023) ) due to its scalability and efficiency. In each iteration, preference data are derived by sampling outputs from the newly updated model, utilizing an online sampling strategy. Hence, we posit that this online DPO can be treated as an effective and representative implementation for preference learning (Tajwar et al., 2024) .Evaluation We use greedy decoding as the temperature set 0 for testing generation. Meanwhile, we utilize zero-shot prompting (Kojima et al., 2022 ) for both answer sampling and evaluations since we find for LLMs finetuned on specific tasks, zero-shot prompting is superior to few-shot prompting. More experimental details can be seen in Appendix B.We perform the three post-training paradigms with the selected LLMs, training and testing them on the respective tasks. Based on the results shown in figure 1 , we delve into the detailed analysis of how these variables influence the effectiveness of self-improvement.Iteration T Across all methods and datasets, there is a general trend of improvement in pass@1 accuracy with increasing iteration steps. This indicates that iterative posttraining effectively enhances model performance over time. However, the rate of improvement tends to plateau or even decline slightly after 4-5 iterations. This suggests that current post-training methods struggle to achieve long-lasting improvements, and excessive post-training (beyond a certain number of iterations) may even yield diminishing returns.Foundation Model M The optimal accuracy improvements across various datasets and post-training methods suggest that LLaMA2-7B demonstrates a relatively higher capacity for improvement under iterative post-training. For instance, on the GSM8K dataset, LLaMA2-7B with Iterative SFT shows an improvement of +12.31 after 5 iterations, whereas LLaMA3-8B exhibits only a moderate gain. This indicates that the more capable M 1 is not necessarily the model that achieves the most significant performance gains during the self-improvement process. However, the most capable model M 1 generally achieves the highest optimal accuracy overall. For example, although LLaMA2-7B achieves the maximum gains on GSM8K with Iterative SFT, it still struggles to outperform LLaMA3-8B in terms of absolute optimal accuracy (53.91 vs. 69.06).Problem-solving Tasks D Models utilizing the three posttraining methods all demonstrate notable improvements on the CSQA and GSM8K datasets, while showing more modest gains on the MATH and MBPP datasets. This indicates that, from the perspective of task difficulty, problems in the CSQA and GSM8K datasets are relatively easier for the models to resolve. In contrast, the MATH dataset poses significant challenges for 7B models due to its complexity. Additionally, the task of code generation, as represented by the MBPP dataset, is also difficult for these foundation models since they were not specifically pretrained on code domains.With foundation model M and task D varying, the bestperforming iterative method also changes accordingly. For example, for Mistral-7B on the CSQA dataset, Iterative-DPO achieves the highest accuracy improvement of +6.47. However, when applied to the GSM8K dataset, the Iterative SFT-DPO method yields the maximum improvement of +10.99. Therefore, with these identifiable variables characterized, it remains challenging for downstream practitioners to determine the optimal post-training method F for their specific use case. achieve lower pass@1 accuracy initially exhibit limited gains with iterative DPO. Based on this observation, we hypothesize that M 1 's capability to solve the test problems fundamentally influences further improvement trends and optimal improvements of F. To quantify M 1 's capability on the test set, we introduce Correct Answer Coverage as a measurement, the proportion of the correct answer space that the model's responses occupy. An illustrative display of this coverage is shown in Figure 2 .Mathematically, we can sample N model's outputs to approximate the answer space. As N → ∞, these outputs can effectively represent the entire answer space. Therefore, expected accuracy over the N outputs can serve as an unbiased estimate of the correct answer coverage. Formally, we use the following equation to calculate M 1 correct answer coverage (for a more detailed derivation, please refer to the Appendix C.):Correct Answer Coverage = E N correct N ≈ 1 |D test | x∈Dtest 1 N N i=1 I[M (x i ) == y] (1)As shown in Figure 2 , the relationship between correct answer coverage and optimal performance of F validate our prior observation and hypothesis. The table clearly demonstrates that when the correct answer coverage is high (> 0.5), Iterative DPO and Iterative SFT-DPO produce the best-performing M * t . Conversely, when the coverage is lower (≤ 0.5), Iterative SFT is more effective in achieving the optimal M * t . Therefore, correct answer coverage can serve as a key factor in guiding practitioners to choose the most suitable iterative post-training method F for the specific problem-solving task with a fixed foundation model. 
critical evaluations on self-improvement: 
 Despite the extensive exploration of various post-training practices for self-improvement and a deepened understanding of their efficacy, current endeavors remain narrowly focused on enhancing performance numbers across these problem-solving benchmarks. Transitioning beyond using pass@1 accuracy as the indicator of improvement, our objective in this section is to engage in a critical examination and reevaluation of iterative self-improvement: discerning whether the improvements constitute genuine progress or merely regression. For brevity, all the results shown in this section is based on the foundation model M as Mistral-7B. In Figure 1 , it is evident when t > 1, the pass@1 accuracy of M t consistently improves in comparison to M 1 . Traditionally, it has been assumed that this improvement indicates the model progressively learning to tackle more challenging problems (Zelikman et al., 2022) . However, we posit a nuanced perspective: while an increase in pass@1 accuracy suggests improvements, it does not inherently equate to an increase in model capabilities to solve more difficult problems.To better gauge how the model problem-solving capabilities evolve overtime, we propose to first quantify the improvement problems as improvement set (IS) at each iteration. An intuitive improvement between M t and M 1 is the pass@1 accuracy on test set, so we use the subset of test problems that M t correctly answers while M 1 fails under greedy decoding to represent IS(t), defined as follows:IS(t) = {x ∈ D test | M t (x) = y ∧ M 1 (x) ̸ = y}. (2)Then we can prompt M 1 with the problems in IS(t) and sample N answers for each problem to record the pass@N accuracy of M 1 . Notably, if M 1 exhibits lower pass@N accuracy even as N increases, it can validate M 1 struggles to solve the problems in IS(t) and the iterative process enhances the model's problem-solving abilities.We apply this evaluative methodology to CSQA, GSM8K, MATH and MBPP datasets with three post-training methods. Generation sampling N varies from 2 1 to 2 6 with the temperature set as 0.75.Reversal Observation As depicted in Figure 3 , contrary to prior assumptions, the rapid increase in pass@N accuracy with increasing N challenges the notion of progressively harder problem-solving. Specifically, as N grows, M 1 achieves near-perfect pass@N accuracy on IS(t) across all evaluated datasets, suggesting its inherent capacity to tackle the deemed improvement problems.Selection Optimization for Answer Alignment The empirical findings depicted in Figure 3 offer a critical insight: iterative self-improvement hardly entails the acquisition of new problem-solving abilities, but rather the enhancement of the model's correct answer selection within its generation space.While pass@1 accuracy measures the correctness of the final answer, it does not capture the diversity of solutions a model can generate. We posit that a model's capacity to produce diverse solutions is indicative of its robustness and flexibility in problem-solving. To thoroughly understand the evolution of answer diversity during the process of iterative self-improvement, we employ a combination of Distinct N-grams (Li et al., 2016) and Sentence-BERT embedding cosine similarity (Reimers & Gurevych, 2019) to measure mod diversity. These metrics have been shown to correlate well with human assessments of diversity (Tevet & Berant, 2021) . Additionally, for mathematical reasoning, we introduce Distinct Equations to measure the diversity of mathematical answers by analyzing the variety of equations in the generated solutions.Each diversity metric Div takes a set of N model outputs, and produces a scalar score representing how diverse the set is. Distinct N-grams measures syntactic diversity by counting the number of unique n-grams (averaged over n = 1 . . . 5) in the output set. The Sentence-BERT metric assesses semantic diversity by embedding each output using a sentence transformer and calculating the average cosine similarity between embeddings. The metric is then 1 minus the average similarity, ensuring that higher scores reflect greater diversity. Distinct Equations, a specialized metric for mathematical reasoning, computes logical diversity by extracting all equations from the outputs and calculating the proportion of unique equations.At each iteration, we sample N = 50 outputs per problem with a temperature of 0.75. Outputs are categorized into correct and incorrect based on the final answer's correctness.Then for each problem, we use the metric Div to calculate the average diversity for the correct and incorrect answers.Reversal Observation Figure 4 presents the diversity results of three post-training methods during the iterative process. All methods show a consistent decrease in diversity, significantly diminishing the diversity of model outputs over iterations, impacting both correct and incorrect answers. This reduction is evident across all three metrics: syntactic, semantic, and logical diversity. Moreover, comparing Iterative SFT and Iterative DPO, it is clear that both methods exhibit a reduction in diversity, but the extent and pattern of reduction vary. For instance, Iterative DPO maintains a slightly higher semantic diversity (as measured by cosine similarity) over multiple iterations compared to Iterative SFT.Trade-Off with Output Diversity. The evaluation results highlight a critical trade-off in iterative self-improvement: while aiming for higher accuracy, the diversity of outputs, which can be crucial for creativity and robustness in problem-solving, is compromised. Future approaches should consider strategies to maintain or even enhance diversity while improving accuracy.In our pursuit to understand the broader implications of iterative self-improvement, it is crucial to assess not only the models' performance on specific benchmarks but also their ability to generalize to out-of-distribution (OOD) tasks. Generalization performance provides insight into the robustness and adaptability of the models when faced with new and varying types of problems.To evaluate the generalization capability of the models, we conducted iterative post-training on the GSM8K dataset and then transferred these models to the MATH algebra test set. The MATH algebra test set is organized into five levels of increasing difficulty, providing a comprehensive spectrum to analyze how well the models perform across groups with varied complexities.For the sake of measuring OOD generalization, we define two metrics as follows defined to facilitate a deeper analysis:• Whole Accuracy (Whole Acc.): This metric represents the pass@1 accuracy across the entire test set, encompassing all difficulty levels from Level 1 to Level 5.• Group Disparity: This metric quantifies the difference in pass@1 accuracy between the best-performing group (Level 1 test set) and the worst-performing group (Level 5 test set), thus highlighting disparities in model performance across different difficulty levels. It is calculated using the following equation: Group Disparity = P ass@1(Level 1) -P ass@1(Level 5) P ass@1(Level 1)A higher value of Group Disparity indicates that the model is performing significantly better on the easier Level 1 while its performance deteriorates on the harder Level 5 group.Reversal Observation As results shown in Figure 5 , with the increase in iterative steps, Iterative SFT and Iterative SFT-DPO can significantly harm the OOD generalization. In contrast, Iterative DPO demonstrates a noticeable improvement, which may indicate better generalization to the OOD test set, in consistent with the recent findings that DPO can improve OOD generalization (Kirk et al., 2024) . However, our more detailed examination of the results across Group Disparity shows Iterative DPO is widening the performances between the easier and harder groups. This comparison uncovers the OOD performance improvement from Iterative DPO actually stems from fitting simpler problems, iat the expense of solving more complex ones.Capabilities Collapse All three iterative post-training methods can exacerbate the generalization disparities across groups, inadvertently causing models to focus on easier problems rather than enhancing their ability to solve more complex ones. As discussed in Section 5.2, the decrease in solution diversity during iterations may be the bottleneck leading to reduced OOD generalization and capability collapse. This highlights the intricate nature of model capabilities under self-improvement, where capabilities at different levels and different facets will compromise each other. Therefore, research developing more sophisticated methods should employ such a comprehensive, fine-grained evaluative framework to monitor post-training processes, as an increase in a single facet of accuracy does not necessarily represent true self-improvement. 
conclusion: 
 In this paper, we foster a comprehensive understanding of the current landscape of post-training practices in selfimprovement. Our evaluation, beyond simple pass@1 accuracy, utilizing multifaceted metrics such as improvement problems, solutions diversity and OOD generalization, underscores the necessity for a critical examination of both the progressive and regressive effects in current self-improving post-training methods. By broadening the scope of our analysis, we provide deeper insights into the true nature of iterative self-improvement with post-training, paving the way for more robust and genuinely self-improving LLMs. 
 
 Appendix: The current landscape of post-training practices for selfimprovement in large language models (LLMs) necessitates a thorough understanding to address both their progressive and regressive effects. In this work, we conduct an in-depth evaluation beyond simple pass@1 accuracy, utilizing multifaceted metrics such as improvement problems, solutions diversity, and OOD generalization. Our findings underscore the critical need for a comprehensive examination of these methods. By expanding our analytical scope, we provide deeper insights into the true nature of iterative selfimprovement through post-training, paving the way for more robust and genuinely self-improving LLMs. This research aims to empower scholars and engineers to develop more reliable and effective post-training strategies, ultimately advancing the field of LLMs and pushing the boundaries of model capabilities to tackle more complex challenges. Sentence-BERT embedding cosine similarity assesses the semantic diversity of generated sentences. Sentences generated by the model are first encoded into embeddings using Sentence-BERT. The cosine similarity between each pair of sentence embeddings is then computed. Cosine similarity between two embeddings u and v is given by:cos(u, v) = u • v ∥u∥∥v∥ (8)The average cosine similarity of all sentence pairs gauges the overall semantic similarity. Lower average cosine similarity indicates higher semantic diversity, as the sentences are less similar in meaning. In our calculations, we measure diversity using 1average cosine similarity, ensuring that higher values reflect greater semantic diversity.Distinct Equations provides a direct indication of how varied the mathematical approaches and solutions are within the generated text. The calculation of this metric involves two steps that identifies all equations present in the generated mathematical reasoning steps first, and then computes the ratio of unique equations to the total number of equations. Mathematically, for a set of generated equations E, distinct equations is defined as:EQUATIONHigher D eq values indicate greater diversity in the mathematical reasoning processes.In Section 4.2, we mentioned that correct answer coverage may be a deeper factor influencing the subsequent improvements in iterative self-improvement. Here, we provide a detailed explanation of the related concepts involved in this influencing factor, as well as the derivation of the calculation for correct answer coverage (in Equation 1) as presented in this paper.Additionally, we emphasize that our consideration of coverage as a deeper factor is a preliminary conclusion drawn from summarizing the factors of the model (M ), post-training function (F), and task dataset (D) and the empirical results validated in Figure ??. It should be noted that we need further work and more extensive experiments to both theoretically and empirically validate this observation, as the discussion of correct answer coverage is beyond our work. Our intention here is to offer empirical insights and lay the groundwork for future investigations into this aspect of iterative self-improvement.Answer space: For a given dataset D (test set), all possible (query, answer) pairs form the answer space of the test set. Here, the set of query is fixed, and for a given query, the number of possible answers can be quite large, hence we call the space as the answer space. Naturally, the entire answer space can be partitioned into a correct answer space and an incorrect answer space based on whether the answers are correct. In practical experiments, the correctness of an answer is approximated by whether its final result exactly matches the final result provided by the ground truth in the training set.Answer distribution: For a given model M , its answer distribution refers to the probability distribution of generating answers conditioned on queries from dataset D. For a specific element (q i , a ij ), q i ∈ D test in the answer space, the generation of this (query, answer) pair occurs in two steps: first, sampling q i from all queries in D, then model M generates a ij conditioned on q i . Therefore, the probability at (q i , a ij ) is the product of the probability of sampling q i from all queries in D and the probability of model M generating a ij conditioned on q i . Considering that all queries should have equal importance, we define that all queries are sampled with the same probability, which is 1Dtest . The answer distribution of M can be mathematically linked to model M as follows: P M (q j , a ij ) def = M (a ijAlthough we cannot exhaust the entire answer space and calculate a probability distribution to demonstrate the trend of the answer distribution in the progress of self-improvement, we can get an unbiased estimate of it by sampling answers and calculate the ratio of the number of correct answers to the total number of answers generated by model M for all queries in D test , where N answers are generated for each query, as illustrated in equation 1. The proof of Equation 1 is as follows:Correct Answer Coverage = C P M (a, q)EQUATIONwhere C denotes the correct answer space, and N cx represents the number of correct answers generated by model conditioned on the given query x. 


Decision: This paper studies self-improvement through post-training methods for LLM problem-solving. 
They propose an evaluation framework to distinguish between superficial metric improvements and genuine enhancement. They conduct experimentation and analysis and find that models showing improved performance across benchmarks will paradoxically exhibit declines in broader and essential capabilities. 
The reviewers agree that this research question is critical and insightful (Reviewer qCiQ), and the findings provide new and significant knowledge to the community (Reviewer LKhf). This paper is sound as it includes extensive experiments. And it is well written.

Reviews: review: paper_summary: paper_summary: Post-training methods, such as iterative preference learning, have received recognition for their ability to enhance the problem-solving capabilities of Large Language Models (LLMs), such as mathematical reasoning, without human intervention. However, upon reviewing the findings, it was discovered that post-training can lead to a phenomenon called self-improvement reversal. Rigorous experimentation and analysis across various problem-solving tasks have revealed empirical results indicating that models demonstrating improved performance on benchmarks may paradoxically experience declines in broader essential capabilities, such as output diversity and out-of-distribution generalization.
strength_weakness: strength_weakness: Figure 1 does not have legend so that it takes more time for reader to know the meaning.
questions: questions: The largest model in the paper is 8B. Is the conclusion the same for lager model, such as 70B?
paper_summary: paper_summary: This paper evaluates a sub-family of self-improvement methods in the field of LLM for math reasoning. It proposes a new systematic multi-facet way to evaluate the model's reasoning capability improvements, and challenges the beliefs of existing works that better PASS@1 accuracy equals to better reasoning capability of the model.
strength_weakness: strength_weakness: The authors conducted a limited literature review. Although the paper's title sounds like a common problem, both the literature review and the experimental setups essentially only focus on iterative self-improvement methods using SFT or DPO, which does not account for all variants of self-improvement methods.

Some references consider adding:\
[1] Learning Math Reasoning from Self-Sampled Correct and Partially-Correct Solutions, ICLR 2023.

Their method "can greatly improve PASS@k, especially for higher k values". However, their approach "does not improve PASS@1, which is expected as learning from multiple targets mainly helps with increasing the diversity of the sampled solutions rather than improving the most-probable solution". This paper shows that self-improvement can indeed **improve diversity** if the self-improvement data includes partially correct solutions, but not simultaneously improving accuracy.

[2] Query and Response Augmentation Cannot Help Out-of-domain Math Reasoning Generalization.

Their paper aligns with the narrative of this paper, saying that even learning from GPT4 **cannot** improve generalization from GSM8K to MATH, let alone learning from the model itself.

Incorporating these related works somewhere in the paper is not difficult. The authors should provide reasons for why they chose to not include these strongly related works in the paper.
questions: questions: Typos: line 368 "mod diversity"; line 408 "iat"; line 611 incorrect latex format; line 741 starting with a comma.