Venue: NAACL 2025

# Mitigating Tail Narrowing in LLM Self-Improvement

via Socratic-Guided Sampling

 Yiwen Ding\({}^{1}\), Zhiheng Xi\({}^{1}\) + * Wei He\({}^{1}\), Zhuoyuan Li\({}^{5}\), Yitao Zhai\({}^{2}\), Xiaowei Shi\({}^{2}\), Xunliang Cai\({}^{2}\), Tao Gui\({}^{3}\), Qi Zhang\({}^{1,4}\), Xuanjing Huang\({}^{1,4}\)

Equal contribution. ++ Work done during internship at Meituan. + Corresponding author. Codes are publicly available at [https://github.com/YiwenDing/Guided-Self-Improvement](https://github.com/YiwenDing/Guided-Self-Improvement). + School of Computer Science, Fudan University 2 Meituan 3 Institute of Modern Languages and Linguistics, Fudan University 4 Key Laboratory of Intelligent Information Processing, Fudan University 5 Macau University of Science and Technology {ywding23,zhxi22}@m.fudan.edu.cn, {tgui, xjhuang}@fudan.edu.cn

###### Abstract

Self-improvement methods enable large language models (LLMs) to generate solutions themselves and iteratively train on filtered, high-quality rationales. This process proves effective and reduces the reliance on human supervision in LLMs' reasoning, but the performance soon plateaus. We delve into the process and find that models tend to over-sample on easy queries and under-sample on queries they have yet to master. As iterations proceed, this imbalance in sampling is exacerbated, leading to a long-tail distribution where solutions to difficult queries almost diminish. This phenomenon limits the performance gain of self-improving models. A straightforward solution is brute-force sampling to balance the distribution, which significantly raises computational costs. In this paper, we introduce **G**uided **S**elf-**I**mprovement (**GSI**), a strategy aimed at improving the efficiency of sampling challenging heavy-tailed data. It leverages Socratic-style guidance signals to help LLM reasoning with complex queries, reducing the exploration effort and minimizing computational overhead. Experiments on four models across diverse mathematical tasks show that GSI strikes a balance between performance and efficiency, while also being effective on held-out tasks1.

Footnote 1: Code as publicly available at [https://github.com/YiwenDing/Guided-Self-Improvement](https://github.com/YiwenDing/Guided-Self-Improvement).

## 1 Introduction

Large language models (LLMs) have demonstrated impressive ability in performing complex reasoning tasks Wei et al. (2022b); Kojima et al. (2022); Zhao et al. (2023). While fine-tuning models on curated data can further boost performance, it relies heavily on human supervision, limiting scalability and generalization Cobbe et al. (2021). To address this, the "self-improvement" paradigm emerges, where models generate multiple reasoning paths, filter out incorrect responses and fine-tune themselves on their own outputs without human intervention Zelikman et al. (2022); Gulcehre et al. (2023); Huang et al. (2023); Singh et al. (2024); Yuan et al. (2024).

Despite the benefits of self-improvement, its performance typically reaches a ceiling after a few iterations Wu et al. (2024). We perform preliminary experiments (SS 4) and find that in reasoning tasks, the most significant gains from self-improvement occur in the first iteration, while subsequent iterations encounter performance bottlenecks or even degradation (Figure 2). Similar performance bottlenecks in synthetic data have also been observed in text generation Shumailov et al. (2023) and image synthesis Alemonammad et al. (2024).

Further, we delve into the self-improvement process and conduct an in-depth analysis (Figure 3) to investigate the underlying causes behind the per

Figure 1: Illustration of distribution during the self-improvement sampling process. **Top:** The long-tail effect intensifies with iterative training on self-generated data. The low-probability data begins to diminish, leading to _tail narrowing_. **Bottom:** Guided sampling balances the distribution by improving tail data sampling efficiency.



formance bottlenecks. On the one hand, complex problems with lengthy reasoning chains tend to amplify hallucinations, making it difficult for models to explore the vast search space and sample correct rationales (Lightman et al., 2024; Zhang et al., 2023; Xie et al., 2023; Xi et al., 2024). Consequently, the models tend to over-sample easy queries and under-sample queries they have yet to master (Tong et al., 2024). On the other hand, as iterations proceed, this imbalance in sampling is exacerbated, leading to a long-tail distribution where solutions to difficult queries almost disappear (Figure 1). This situation is also referred to as _tail narrowing_ or _tail cutting_ in previous studies (Dohmatob et al., 2024b; Shumailov et al., 2023). As a result, the model's self-improvement is limited, since difficult examples are also crucial for further training (Liu et al., 2024).

To address this imbalance, a common approach is to allocate more sampling trials to undersampled, challenging queries (Tong et al., 2024), but this can be considerably more costly. In this paper, we propose **G**uided **S**elf-Improvement (**GSI**), an efficient method that leverages interactive guidance signals inspired by the Socratic method (Chang, 2023; Dong et al., 2023b). Specifically, our approach introduces an additional resampling phase, termed _distribution re-balancing_ for difficult queries, which is applied after the generation step in the self-improvement process. During this phase, we incorporate targeted guidance signals derived from oracle answers, stepwise rationales, and strong teacher supervision. These signals help to narrow the sampling space, reduce sampling difficulty, and minimize hallucinations during reasoning (Xie et al., 2023; Xi et al., 2024). As a result, GSI enables more effective exploration, increases solution coverage for challenging queries (Bansal et al., 2024), and mitigates the issue of tail narrowing during the sampling process.

We perform experiments across four models and six mathematical reasoning tasks, including arithmetic reasoning, abstract algebra, and formal logic. The results demonstrate that GSI mitigates the performance bottlenecks of self-improvement while maintaining computational efficiency. Further analysis shows that this method leads to a more balanced solution distribution and improved model generalization across multiple reasoning tasks. In addition to natural language reasoning, our method has also been proven effective in program-based reasoning (Chen et al., 2023).

Our contributions are summarized as follows:

* We conduct an in-depth study of the self-improvement process, revealing the performance bottlenecks driven by a long-tail distribution of solutions, which results from increasingly imbalanced data sampling.
* To efficiently mitigate the problem of tail narrowing, we introduce the Guided Self-Improvement (GSI) method that employs Socratic-style guidance signals to assist models in exploring solutions for challenging queries.
* We validate our strategy through comprehensive experiments on four backbone models across six mathematical reasoning tasks, demonstrating the effectiveness and efficiency of GSI.

## 2 Related Work

### Self-improvement for LLMs

Self-improvement methods, where models refine themselves using self-generated data, have proven effective in enhancing problem-solving abilities without human intervention (Huang et al., 2023; Zelikman et al., 2022). To ensure the reliability of this process, the generated data is typically filtered using external supervision signals. These signals can be binary rewards, such as correctness checks based on reference answers (Yuan et al., 2023; Zelikman et al., 2022; Tong et al., 2024) or compiler execution feedback (Haluptzok et al., 2023). Alternatively, more nuanced approaches involve scoring Gulcehre et al. (2023) or ranking systems (Dong et al., 2023a), which may be generated by the model itself (Yuan et al., 2024) or external reward models (Hosseini et al., 2024; Qi et al., 2024). Some methods adopt weaker supervision signals, such as majority voting across multiple outputs (Huang et al., 2023).

Once filtered, the high-quality data supports posttraining through methods like SFT (Zelikman et al., 2022; Liang et al., 2024) or preference-based techniques like Direct Preference Optimization (DPO, Yuan et al., 2024). This process is often iterative, allowing models to continually generate new data, filter it, and use it to refine their performance further (Zelikman et al., 2022; Gulcehre et al., 2023; Yuan et al., 2024).

### Distribution Shift in Synthetic Data

The scaling law reveals a predictable increase in model performance as the volume of training data 

grows Kaplan et al. (2020). With the development of LLMs, the demand for vast amounts of high-quality data has surged, leading researchers to rely increasingly on synthetic data. These methods have proven effective across various tasks, from general-purpose chatbots Dubey et al. (2024); Adler et al. (2024) to specialized fields such as mathematical reasoning Yue et al. (2024); Yu et al. (2024).

However, the synthetic data also introduces the risk of _model collapse_, where their performance degrades due to recursive training on model-generated data Shumailov et al. (2023); Alemohammad et al. (2024). This phenomenon arises from _distribution shift_: as models favor high-probability outputs, their results become increasingly uniform, leading to reduced variance. Over time, this shift manifests in the declining diversity of model responses Guo et al. (2024), the disappearance of tail behaviors Dohmatob et al. (2024), and the amplification of systematic bias Yu et al. (2023). Wu et al. (2024) have also observed similar degradation in self-improvement loops, which aligns with the core argument of this paper.

## 3 Preliminaries

### Formulation of Self-improvement

Given a large language model \(M_{0}\) and the original training dataset \(\mathcal{D}=\{(x_{i},r_{i},y_{i})\}^{N}_{i=1}\), where \(x_{i}\) is the problem, \(r_{i}\) is the chain-of-though rationale Wei et al. (2022) and \(y_{i}\) represents the final answer. Each rationale \(r_{i}\) consists of several intermediate steps, i.e., \(r_{i}=[r_{i,1},\ldots,r_{i,L}]\), where \(L\) denotes the number of steps. The self-improvement process enhances the model's reasoning ability through iterative refinement over \(T\) cycles. Each iteration \(t\in[1,T]\) consists of two main steps: _Generate_ and _Improve_.

Generate step.At iteration \(t\), the previous model \(M_{t-1}\) generates multiple reasoning paths for each problem. Specifically, we allocate \(k\) sampling times to each query \(x_{i}\in\mathcal{D}\):

\[(\hat{r}_{i},\hat{y}_{i})=M_{t-1}(x_{i})\]

The newly generated data points form a set \(\mathcal{D}^{{}^{\prime}}=\{(x_{i},\hat{r}_{i}^{j},\hat{y}_{i}^{j})~{}|~{}x_{ i}\in\mathcal{D},j=[1,k]\}\). Each candidate solution \(\hat{y}_{i}^{j}\) is evaluated by a binary reward function \(\text{rf}(y_{i},\hat{y}_{i})\in\{0,1\}\), which verifies correctness based on ground-truth answers \(y_{i}\). Only correct solutions with \(\text{rf}(\cdot)=1\) are filtered to form the high-quality dataset \(\mathcal{D}_{t}\) for the current iteration \(t\).

Improve step.In the \(t\)-th iteration, the model \(M_{t}\) is fine-tuned on the self-generated high-quality dataset \(\mathcal{D}_{t}\). The fine-tuning objective is to minimize the negative log-likelihood (NLL) loss:

\[\mathcal{L}_{\text{SFT}}=-\mathbb{E}_{(x,r)\sim\mathcal{D}_{t}}\sum_{l=1}^{L} \log M(r_{l}~{}|~{}r_{<l},x).\]

By minimizing \(\mathcal{L}_{\text{SFT}}\), the model iteratively improves its ability to generate correct rationales, and this process is repeated for \(T\) iterations to achieve self-improvement. Note that in the first iteration, we directly fine-tune \(M_{0}\) on the original dataset \(\mathcal{D}\) to obtain \(M_{1}\).

### Biased Sampling and Tail Effect

In the self-improvement sampling process, there is a tendency to select higher-quality and more accurate data, which introduces a phenomenon known as sampling bias Alemohammad et al. (2024); Shumailov et al. (2023). This bias often results in the truncation or narrowing of low-probability "tails" in the data distribution.

Figure 2: **Iterative performance in the self-improvement. Experiments are conducted on GSM8K with varying sampling numbers \(k\). Solid markers show the performance of vanilla self-improve, with the solid line fitting these points. The performance plateaus after a few iterations. Hollow markers represent the performance after supplementing tail data, with a dashed line trend. It balances the distribution and alleviates performance bottlenecks.**



To illustrate the effects of sampling bias, consider a one-dimensional Gaussian distribution \(X^{0}\sim\mathcal{N}(\mu,\sigma^{2})\). Let \(\lambda\in[0,1]\) represent the sampling bias parameter and biased sampling from \(\mathcal{N}(\mu,\lambda\sigma^{2})\). When \(\lambda=1\), the sampling is unbiased, maintaining the original variance. Conversely, when \(\lambda=0\), the sampling is derived from the modes of the generative distribution \(M_{t}\) with zero variance.

Initially, the tails of the distribution, which represent low-probability data, begin to diminish due to their low likelihood of being sampled. As the iteration continues, the heavy-tailed data is increasingly excluded, causing a shift in the overall distribution. This phenomenon is referred to as tail narrowing or tail cutting, leading to a more peaked distribution Dohmatob et al. (2024).

## 4 Performance Bottleneck and Tail Narrowing in Self-Improvement

Despite extensive research into self-improvement methods Gulcehre et al. (2023); Singh et al. (2024), the performance dynamics across successive training iterations remain underexplored. To this end, we conduct experiments across four backbone models to investigate the effects of sampling and tail data during iterative training. Tail data refers to samples for which the model rarely generates correct solutions during sampling. These samples lie in the "long tail" of data distributions.

Performance trends.To uncover the relationship between performance trends with sampling, we vary the number of sampling \(k\). Figure 2 reveals the following: (1) Lower sampling times (e.g., \(k=1\) or \(k=2\)) degrade model performance, leading to negative gains. Llama2-7B, Mistral-7B, and Llama3-8B underperform compared to SFT on the original dataset (iteration 1). This decline stems from the models' weaker reasoning abilities, which limit their coverage of challenging queries. Consequently, the models tackle only basic problems, resulting in a degradation or collapse of reasoning ability Shumailov et al. (2023); Dohmatob et al. (2024). (2) With larger sampling numbers \(k\), performance improves across multiple iterations. The most notable gains occur during the first iteration, as shown by the green and orange solid lines in Figure 2. However, after the third iteration, progress halts, eventually reaching a performance plateau.

Impact of tail data.As previously discussed, merely scaling the number of sampling eventually encounters performance bottlenecks. However, challenging low-probability samples, often referred to as tail data, are often considered more crucial for improving model performance Sorscher et al. (2022); Liu et al. (2024); Tong et al. (2024).

To investigate the impact of tail data, we conduct additional experiments targeting these difficult examples. For queries that do not yield a correct response after \(k\) sampling attempts, we supplement them with a golden rationale, ensuring that every query has at least one practically correct response and effectively rebalancing the long-tail distribution. We use hollow markers to represent the performance with tail data. As shown in Figure 2, across different numbers of sampling and model variants, the performance represented by the dashed line exceeds that of the solid line representing vanilla self-improve. The results demonstrate that rebalanced data helps mitigate performance degradation under a small number of sampling attempts (e.g., \(k=1\) or \(k=2\)). Moreover, as \(k\) increases, a

Figure 3: **Comparison of data distributions** between the self-generated and original (SFT) datasets. **(a)** Difficulty distribution across five levels in MATH tasks, with level 1 representing the easiest and level 5 the most difficult. The self-generated data has a lower proportion of difficult problems. **(b)** Length distribution indicates that the self-generated data tends to be shorter compared to the original dataset. **(c)** Perplexity diagram of each training sequence measured with the Llama3-8B, shows that the tails in the self-generated data are diminished.

 

larger number of sampling alleviates performance plateaus and boosts model efficacy to some extent. This study highlights the value of solutions to challenging queries in overcoming the limits of finite and progressively biased sampling.

Emergence of tail narrowing.To analyze the distributional characteristics of tail data, we analyze three dimensions: difficulty, response length, and perplexity, revealing the differences between self-generated and original data. Figure 2(a) shows that model-generated data tends to be at a low level and simpler compared to the original MATH dataset, which contains more challenging queries. Consistently, DART-Math Tong et al. (2024) also identifies a synthesis bias towards easy queries.

Using response length as a complexity indicator, as suggested by Fu et al. (2023), we find that self-generated responses are generally shorter, peaking around \(200\) tokens (Figure 2(b)). This indicates a bias towards simplicity and a lower proportion of complex data.

From a semantic perspective, we computed the perplexity based on Llama3-8B. As shown in Figure 2(c), the self-generated data has lower perplexity, indicating a shift toward more probable and coherent tokens. This reduces the occurrence of rare, complex and diverse tokens in the tail.

These observations indicate a diminishing trend of tail data, termed _Tail Narrowing_Dohmatob et al. (2024). In each iteration, the distribution progressively narrows, hindering performance to challenging queries.

## 5 Guided Self-Improvement

To mitigate the observed tail-narrowing phenomenon, a straightforward solution is to increase the sampling trials for tail data. However, directly tackling these difficult queries from scratch often results in low success rates and higher costs due to repeated failed attempts. Drawing from Socratic-style education Chang (2023); Dong et al. (2023), we introduce guidance-based exploration techniques to improve sampling efficiency, such as learning from demonstrations Schaal (1996); Subramanian et al. (2016). We provide the model with tailored assistance in structured contexts, enabling it to address difficult queries more effectively. The following paragraphs outline four guiding strategies we propose.

Answer-driven.This strategy incorporates the ground-truth answer \(y_{i}\) along with the input query \(x_{i}\) as context to guide the generation process. It helps the model better align with the expected solution, particularly for challenging queries Zelikman et al. (2022). Formally, at each iteration \(t\), instead of inputting only \(x_{i}\) into the model \(M_{t-1}\), we extend the prompt by appending \(y_{i}\) as a hint:

\[(\hat{r}_{i},\hat{y}_{i})=M_{t-1}(x_{i},\text{hint}(y_{i})).\]

This approach helps the model focus on the reasoning process behind the answer, reducing the overall difficulty of the task.

Rationale-driven.In this strategy, we further extend the input by introducing a rationale \(r_{i}\), which helps the model derive the correct reasoning process. Unlike the answer-driven method, providing a rationale offers a more detailed reference for the model to follow, narrowing the exploration space of reasoning paths Yang et al. (2024). Formally, at iteration \(t\), the input to the model \(M_{t-1}\) is augmented as follows:

\[(\hat{r}_{i},\hat{y}_{i})=M_{t-1}(x_{i},\text{hint}(r_{i})).\]

This approach enables the model to handle queries it has yet to master and ensures a higher coverage of solved problems. Importantly, it alleviates the hallucinations when the model tries to provide reasoning paths for problems it doesn't fully understand Lanham et al. (2023), improving the reliability of generated data.

Interactive sampling.Inspired by previous work in the area of Interactive RL Subramanian et al. (2016); Suay and Chernova (2011), we introduce feedback from a stronger model \(M_{s}\) after the model \(M_{t-1}\) fails. Instead of providing hints along with the query, this dynamic process ensures that the model can explore its own solution before receiving external guidance. Formally, after the model \(M_{t-1}\) generates an incorrect answer, we re-sample by giving it both its prior incorrect output and the feedback \(f_{i}\) from \(M_{s}\) as additional context:

\[(\hat{r}^{\mathrm{error}}_{i},\tilde{y}^{\mathrm{error}}_{i})=M_{ t-1}(x_{i}),\] \[f_{i}=M_{s}(x_{i},r_{i},y_{i},\hat{r}^{\mathrm{error}}_{i}),\] \[(\hat{r}_{i},\hat{y}_{i})=M_{t-1}(x_{i},\hat{r}^{\mathrm{error}}_{ i},f_{i}).\]

This feedback includes an analysis and correction of the model's errors, reducing its reliance on the correct answer. Through this interactive process, 

we balance exploration and correction, enabling the model to learn from its mistakes without overly restricting its reasoning path.

State reset.Drawing inspiration from the concept of _state reset_, i.e., going back to intermediate states during problem-solving to refine the approach Chang et al. (2024); Xi et al. (2024), we adopt a strategy where the model is guided step by step with partial rationales. Instead of supplying the full rationale \(r_{i}\) immediately, after \(l\) incorrect attempts, the model is provided with the preceding reasoning steps \(r_{<l}\) from \(r_{i}=[r_{i,1},\ldots,r_{i,L}]\), gradually narrowing down the exploration space:

\[(\hat{r}_{i},\hat{y}_{i})=M_{t-1}(x_{i},\text{hint}(r_{i,<l})).\]

This method reduces the difficulty of queries at a fine-grained level, relieving the model of cognitive overload while still allowing flexibility in the model's solution. Although it increases the number of attempts, the incremental hint helps identify the threshold where the model can solve problems independently with minimal guidance.

## 6 Experiments

### Experimental Setups

Models.We conduct experiments using four widely adopted foundation models, including Llama2-7B-Base Touvron et al. (2023), Llama3-8B-Base Dubey et al. (2024), Deepseek-Math-7B-Base Shao et al. (2024), and Mistral-7B-v0.3 Jiang et al. (2023). For the stronger model in the interactive sampling process, we employ Llama3-70B-Instruct Dubey et al. (2024).

Datasets.We utilize six math reasoning datasets. These include arithmetic reasoning datasets such as GSM8K Cobbe et al. (2021), AQuA Ling et al. (2017), MathQA Amini et al. (2019) and SVAMP Patel et al. (2021), as well as a more challenging dataset MATH Hendrycks et al. (2021). We also include TheoremQA for abstract algebra and formal logic. To evaluate generalization, we choose AQuA, GSM8K, MATH as held-in datasets and MathQA, SVAMP, TheoremQA as held-out datasets. To ensure consistency in answer format across different datasets, we utilize the unified data provided by the MathInstrect dataset Yue et al. (2024) and follow its train-test splits. More dataset statistics can be found in Appendix A.

Implementation details.Following Huang et al. (2023) and Singh et al. (2024), we fine-tune the pre-trained model \(M_{0}\) during the Improve Step of each iteration to prevent overfitting. We set the iteration number \(T=4\) and sampling number \(k=8\). To mitigate the tail-narrowing effect, we identify queries with less than a 50% probability of yielding correct completions as heavy-tailed data. For the tail data, we apply the GSI strategy, resampling up to \(k\) times until the query no longer falls into the tail data. All experiments are performed on 8 A100-80GB GPUs. We run the FFT and Improve Step for 1 epoch. The learning rate is set to \(1\times 10^{-5}\). For sampling and evaluation, we leverage the vLLM Kwon et al. (2023) framework, setting a maximum of \(1024\) output tokens. The temperature is set to \(0.7\) during sampling and \(0\) during evaluation. The prompt templates are detailed in Appendix D.

Baselines.To evaluate the impact of GSI on distributional adjustments, we compare it against SFT and Self-improve variants. We also include baselines with significantly higher sampling trials than GSI.

* **SFT**: Fine-tuning on the original dataset for \(1\) epoch, which corresponds to the first iteration of self-improvement.
* **Self-Improve (\(k=8\))**: In each iteration, we sample \(k=8\) completions per query from the original dataset, filter out correct reasoning paths, and then improve on the self-generated data.
* **Brute-Force Self-Improve** (\(k=64\) or \(k=128\)): Based on Vanilla Self-Improve (\(k=8\)), we add a distribution re-balancing stage for tail data in each iteration. It performs non-guided brute-force sampling of tail-end data up to \(k\) times without additional guidance.

### Main Results

The main results are shown in Table 1. We have the following key findings:

Re-balancing tail data improves coverage and performance of self-improvement.Compared to SFT, vanilla self-improve with \(k=8\) boosts reasoning on held-out datasets but shows only marginal gains, with performance bottlenecks and degradation on held-in datasets. This aligns with our observation (SS 4), where we identify tail narrowing as the primary cause. To mitigate this, Brute-Force Self-Improve (\(k=64\) or \(k=128\)) performs additional sampling on tail data, which 

re-balances the distribution. This adjustment significantly enhances both coverage and overall performance, with further improvements as the number of samples increases. For example, in Llama2-7B, coverage increases from \(52.6\)% to \(80.7\)%, with performance improving from \(21.53\) to \(23.84\). In Mistral-7B, the rebalancing also reverses the observed performance decline. Thus, incorporating a resampling stage for challenging, heavy-tailed data proves essential.

GSI outperforms brute-force sampling with better efficiency.To optimize the amount of sampling computation required for self-improvement, we analyze the sampling efficiency during the Generate step phase. For each query \(x_{i}\in\mathcal{D}\) at iteration \(t\), we perform \(k_{i,t}\) sampling operations. The cumulative sample budget across all queries and \(T\) iterations is defined as: Sample Budget = \(\sum_{x_{i}\in\mathcal{D}}\sum_{t=1}^{T}k_{i,t}\).

When scaling resampling operations from 64 to 128, we observe that the improvement in sampling coverage becomes slower, and the performance gains diminish despite the additional computational costs. This suggests that the model may get trapped in vast search space, particularly when dealing with more challenging queries. In contrast, our strategy GSI, which leverages Socratic-style guidance, achieves more compute-efficient sampling. It outperforms brute-force sampling while using only one-third of the sampling budget. Specifically, on Llama3-8B, the state reset strategy performs \(0.32\)M

\begin{table}
\begin{tabular}{l l r r|r r r r r r r} \hline \multirow{2}{*}{**Models**} & \multicolumn{3}{c|}{**Sample**} & \multicolumn{3}{c|}{**Held-in Datasets**} & \multicolumn{3}{c}{**Held-out Datasets**} \\  & & \multicolumn{1}{c|}{**Budget**} & \multicolumn{1}{c|}{**Coverage**} & \multicolumn{1}{c|}{**Avg.**} & \multicolumn{1}{c|}{**AQuA**} & \multicolumn{1}{c}{**GSMSK**} & \multicolumn{1}{c}{**MATH**} & \multicolumn{1}{c}{**Avg.**} & \multicolumn{1}{c}{**MathQA**} & \multicolumn{1}{c}{**VAMP**} & \multicolumn{1}{c}{**Thm.QA**} \\ \hline \multirow{9}{*}{**Llama2-7B**} & SFT & - & - & 21.00 & 27.17 & 31.31 & 4.52 & 21.08 & 22.08 & 36.80 & 4.38 \\  & SI (\(k=8\)) & \(0.24\)M & \(52.6\)F & \(21.53\) & \(25.59\) & \(31.33\) & \(5.86\) & \(24.46\) & \(27.37\) & \(39.50\) & \(6.50\) \\  & Brute-Force SI (\(k=64\)) & \(1.11\)M & \(77.1\)F & \(23.73\) & \(29.53\) & \(35.86\) & \(5.80\) & \(23.42\) & \(25.76\) & \(39.50\) & \(5.00\) \\  & Brute-Force SI (\(k=128\)) & \(1.67\)M & \(80.7\)F & \(23.84\) & \(27.17\) & \(37.98\) & \(6.36\) & \(24.22\) & \(26.57\) & \(40.60\) & \(5.50\) \\  & \multicolumn{1}{c|}{Guided Self-Improve (\(k=8\))} & & & & & & & & & & \\  & + Answer-driven & \(0.37\)M & \(99.0\)F & \(23.41\) & \(28.34\) & \(35.94\) & \(6.04\) & \(25.27\) & \(25.83\) & \(\mathbf{43.60}\) & \(6.38\) \\  & \multicolumn{1}{c|}{Rationale-driven} & \(0.34\)M & \(99.9\)F & \(24.32\) & \(30.32\) & \(36.16\) & \(6.48\) & \(25.68\) & \(28.11\) & \(41.30\) & \(\mathbf{7.63}\) \\  & \multicolumn{1}{c|}{Ir Interactive Sampling} & \(0.36\)M & \(96.3\)F & \(25.00\) & \(30.71\) & \(37.83\) & \(6.46\) & \(26.25\) & \(27.84\) & \(43.40\) & \(7.50\) \\  & \multicolumn{1}{c|}{* State Reset} & \(0.38\)M & \(82.0\)F & \(\mathbf{25.91}\) & \(\mathbf{31.10}\) & \(\mathbf{40.18}\) & \(6.44\) & \(\mathbf{26.79}\) & \(\mathbf{29.45}\) & \(43.30\) & \(\mathbf{7.63}\) \\ \hline \multirow{9}{*}{**Llama3-8B**} & SFT & - & - & 37.27 & 39.37 & 57.47 & 14.96 & 38.68 & 44.29 & 63.00 & 8.75 \\  & SI (\(k=8\)) & \(0.24\)M & \(68.6\)F & \(36.87\) & \(39.76\) & \(59.14\) & \(11.70\) & \(39.09\) & \(45.63\) & \(62.40\) & \(9.25\) \\  & Brute-Force SI (\(k=64\)) & \(0.85\)M & \(86.47\)F & \(38.16\) & \(38.98\) & \(61.11\) & \(14.40\) & \(38.30\) & \(45.86\) & \(60.90\) & \(8.13\) \\  & Brute-Force SI (\(k=128\)) & \(1.26\)M & \(88.7\)F & \(37.55\) & \(41.34\) & \(61.64\) & \(9.68\) & \(39.32\) & \(45.90\) & \(62.80\) & \(9.25\) \\  & \multicolumn{1}{c|}{Guided Self-Improve (\(k=8\))} & & & & & & & & & & \\  & \multicolumn{1}{c|}{+ Answer-driven} & \(0.31\)M & \(98.5\)F & \(38.71\) & \(42.52\) & \(59.82\) & \(13.80\) & \(40.15\) & \(46.85\) & \(63.60\) & \(\underline{10.00}\) \\  & \multicolumn{1}{c|}{+ Rationale-driven} & \(0.29\)M & \(99.8\)F & \(39.14\) & \(42.91\) & \(60.27\) & \(14.24\) & \(41.08\) & \(47.94\) & \(\mathbf{65.30}\) & \(10.00\) \\  & \multicolumn{1}{c|}{+ Interactive Sampling} & \(0.31\)M & \(97.3\)F & \(39.34\) & \(42.52\) & \(60.05\) & \(15.46\) & \(41.12\) & \(47.24\) & \(64.50\) & \(\mathbf{11.63}\) \\  & \multicolumn{1}{c|}{+ State Reset} & \(0.32\)M & \(90.2\)F & \(\mathbf{41.64}\) & \(\mathbf{46.46}\) & \(\mathbf{62.62}\) & \(\mathbf{15.54}\) & \(\mathbf{41.33}\) & \(\mathbf{49.25}\) & \(65.00\) & \(9.75\) \\ \hline \multirow{9}{*}{**DeepSeek-M**} & SFT & - & - & 50.01 & 60.63 & 60.73 & 28.66 & 21.15 & 21.61 & 37.10 & 4.75 \\  & SI (\(k=8\)) & \(0.24\)M & \(79.8\)F & \(52.22\) & \(56.69\) & \(68.92\) & \(31.06\) & \(49.63\) & \(64.26\) & \(67.50\) & \(17.13\) \\  & Brute-Force SI (\(k=64\)) & \(0.64\)M & \(91.5\)F & \(53.95\) & \(57.87\) & \(70.89\) & \(\mathbf{33.08}\) & \(50.92\) & \(65.36\) & \(70.40\) & \(17.00\) \\  & \multicolumn{1}{c|}{White-Force SI (\(k=128\))} & \(0.93\)M & \(92.2\)F & \(52.43\) 

sampling, which is only one-third of the budget required for brute-force sampling. Moreover, the model shows improved held-in performance from \(37.55\) to \(41.64\) and generalizes well to held-out datasets.

The effectiveness of different strategiesAmong the four strategies, the state reset strategy, which samples from different initial states and generates diverse reasoning paths, performs relatively better. However, the effectiveness of different strategies depends on the model's inherent capabilities. For example, the answer-driven strategy, which provides only the correct value and requires the model to reason backward to generate a rationale, demands advanced reasoning abilities Zelikman et al. (2022). Therefore, this approach yields modest performance gains on weaker models such as Llama2-7B. Further investigation is needed to explore how different models can be optimally paired with various strategies.

## 7 Discussion

### Effectiveness on PoT Reasoning

To fully exploit the potential of diverse reasoning processes, we extend our investigation to Program-of-Thought PoT, Chen et al. (2023) prompting. In the self-improvement process, we utilize PoT rationales for training, then filter data and evaluate performance based on compiler-executed results. As shown in Figure 4, four strategies consistently outperform the self-improvement baseline in program-based reasoning. The state reset strategy on the DeepSeek-Math-7B model shows notable relative gains, with an improvement of up to \(8.7\)%. Similarly, the reference-driven strategy leads to a performance boost of \(8.5\)%.

### Performance of Different Model Sizes

To further explore, we investigate the effectiveness of the proposed strategy across different model sizes. We choose a smaller model, DeepSeek-Coder-1.3B Guo et al. (2024), and a larger model, CodeLlama-13B Roziere et al. (2023), in our experiments. As shown in Table 2, DeepSeek-Coder-1.3B exhibits only marginal improvements when applying the answer-driven strategy compared to the others. The limited improvement may be attributed to the nature of the strategy, which requires the model to reverse-engineer a solution from a given true answer Zelikman et al. (2022). While the final result is provided, deriving a good justification can be challenging for smaller models Wei et al. (2022). However, when scaling up to the 13B model, we observe a pronounced performance boost. The results suggest that GSI is more effective with larger models, which are equipped with advanced reasoning abilities.

### Short-cutting in Generated Rationales

This experiment explores how different guiding strategies influence the quality of rationales. Inspired by Zelikman et al. (2022), we focus on the number of rationale steps. Figure 5 shows that, in most cases, the model's reasoning steps align with the original annotated steps. However, under the rationale-driven strategy, the model is more likely to generate fewer steps than others. Further analysis reveals that providing rationales can cause the model to skip reasoning steps, a phenomenon we refer to as "Hint Short-cutting" Zelikman et al. (2022).

Figure 4: Comparison of average performance on six math tasks using PoT. The percentage of improvement is significant in the held-in datasets.

 

2022). This tendency may weaken the model's ability to think step-by-step, potentially hindering iterative training (Dohmatob et al., 2024a). We show an example in Appendix B, where the model skips critical steps.

### Impact of the Sampling Hyperparameter

We investigate the impact of the hyperparameter \(k\), which determines the number of sampling times allocated to each query. As shown in Table 3, increasing \(k\) yields substantial performance improvements. Notably, the most significant improvements occur up to \(k=8\), after which the gains begin to plateau. Therefore, we choose \(k=8\) as a practical configuration, achieving a balanced trade-off between computational cost and performance gains.

## 8 Conclusion

In this work, we delve into the performance bottlenecks in the self-improvement process of LLMs, identifying the issue of tail narrowing caused by progressively imbalanced data sampling. To mitigate this, we propose Guided Self-Improvement (GSI), a new method that incorporates a distribution re-balancing phase and Socratic-style guidance to enhance solution coverage for challenging queries. Experimental results across multiple models and mathematical reasoning tasks demonstrate the effectiveness of this method in improving reasoning performance while maintaining computational efficiency. We believe GSI offers a promising direction for enhancing the scalability and generalization of self-improving models in the future.

## Limitations

While our work introduces a new approach to mitigating the tail narrowing through GSI, there are still several limitations. First, for computational efficiency, we do not scale the sampling in each iteration. However, we conduct a series of experiments (SS 4) to provide insights into how scaling the number of sampling can boost performance. Second, following prior self-improvement works, we use binary signals for supervision based on final answer checks. However, poor and spurious rationales while yielding correct answers may be utilized, which could hinder the improvement of reasoning ability. Filtering low-quality reasoning paths and ensuring the quality of self-generated data remains an area for further investigation.

## Acknowledgment

The authors wish to thank the anonymous reviewers for their helpful comments. This work was supported by the Science and Technology Commission of Shanghai Municipality (No. 24511103100), National Natural Science Foundation of China (No. 62476061,62206057,62076069), Shanghai Rising-Star Program (23QA1400200), Natural Science Foundation of Shanghai (23ZR1403500), Program of Shanghai Academic Research Leader under grant 22XD1401100. This research was also supported by Meituan.

## References

* Adler et al. [2016] Bo Adler, Niket Agarwal, Ashwath Aithal, Dong H. Anh, Pallab Bhattacharya, Annika Brundyn, Jared Casper, Bryan Catanzaro, Sharon Clay, Jonathan M. Cohen, Sirshak Das, Ayush Dattagupta, Olivier Delalleau, Leon Derczynski, Yi Dong, Daniel Egert, Ellie Evans, Aleksander Ficek, Denys Fridman, Shaona Ghosh, Boris Ginsburg, Igor Gitman, Tomasz Grzegorzek, Robert Hero, Jining Huang, Vibhu Java,

Figure 5: Comparison of the number of rationale steps generated by the model relative to the number of steps used in the ground truth. The red box highlights the occurrence of skip steps in the rationale-driven strategy.

 

Joseph Jennings, Aastha Jhunjhunwala, John Kamalu, Sadar Khan, Olekskii Kuchaiev, Patrick LeGresley, Hui Li, Jiwei Liu, Zihan Liu, Eileen Long, Ameyus Sunil Mahabaleshwarkar, Somshuro Majumdar, James Maki, Miguel Martinez, Maer Rodrigues de Melo, Ivan Moshkov, Deepak Narayanan, Sean Narenthiran, Jesus Navarro, Phong Nguyen, Osvald Nitski, Vahid Noroozi, Guruprasad Nutheti, Christopher Parisien, Jupinder Parmara, Mostofa Patwardy, Krzysztof Pawelec, Wei Ping, Shrimian Prabhmoye, Rajarsish Roy, Trisha Saar, Vasanth Rao Naik Sabavat, Sanjeev Satheesh, Jane Polak Scowcroft, Jason Sewall, Pavel Shamis, Gerald Shen, Mohammad Shoeybi, Dave Sizer, Misha Smelyanskiy, Felipe Soares, Makesh Narsimhan Sreedhar, Dan Su, Sandeep Subramanian, Shengyang Sun, Shubham Toshiwal, Hao Wang, Zhilin Wang, Jiaxuan You, Jiaqi Zeng, Jimmy Zhang, Jing Zhang, Vivenne Zhang, Yian Zhang, and Chen Zhu. 2024. Nemotron-4 340b technical report. _CoRR_, abs/2406.11704.
* Alemohammad et al. (2024) Sina Alemonammad, Josue Casco-Rodriguez, Lorenzo Luzi, Ahmed Imtiaz Humayun, Hossein Babaei, Daniel LeJeune, Ali Siahkoohi, and Richard G. Baraniuk. 2024. Self-consuming generative models go MAD. In _The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024_. OpenReview.net.
* Amini et al. (2019) Aida Amini, Saadia Gabriel, Shanchuan Lin, Rik Koncel-Kedziorski, Yejin Choi, and Hannaneh Hajishirzi. 2019. Mathqa: Towards interpretable math word problem solving with operation-based formalisms. In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume 1 (Long and Short Papers)_, pages 2357-2367. Association for Computational Linguistics.
* Bansal et al. (2024) Hritik Bansal, Arian Hosseini, Rihabh Agarwal, Vinh Q. Tran, and Mehran Kazemi. 2024. Smaller, weaker, yet better: Training LLM reasoners via compute-optimal sampling. _CoRR_, abs/2408.16737.
* Chang (2023) Edward Y. Chang. 2023. Prompting large language models with the socratic method. In _13th IEEE Annual Computing and Communication Workshop and Conference, CCWC 2023, Las Vegas, NV, USA, March 8-11, 2023_, pages 351-360. IEEE.
* Chang et al. (2024) Jonathan D. Chang, Wenhao Zhan, Owen Oertell, Kiante Brantley, Dipendra Misra, Jason D. Lee, and Wen Sun. 2024. Dataset reset policy optimization for RLHF. _CoRR_, abs/2404.08495.
* Chen et al. (2023) Wenhu Chen, Xueguang Ma, Xinyi Wang, and William W. Cohen. 2023. Program of thoughts prompting: Disentangling computation from reasoning for numerical reasoning tasks. _Trans. Mach. Learn. Res._, 2023.
* Cobbe et al. (2023) Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reitichiro Nakano, Christopher Hesse, and John Schulman. 2021. Training verifiers to solve math word problems. _CoRR_, abs/2110.14168.
* Dohmatob et al. (2024a) Elvis Dohmatob, Yunzhen Feng, Arjun Subramonian, and Julia Kempe. 2024a. Strong model collapse. _Preprint_, arXiv:2410.04840.
* Dehmatob et al. (2024b) Elvis Dohmatob, Yunzhen Feng, Pu Yang, Francois Charton, and Julia Kempe. 2024b. A tale of tails: Model collapse as a change of scaling laws. In _Fortyfirst International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024_. OpenReview.net.
* Dong et al. (2023a) Hanze Dong, Wei Xiong, Deepanshu Goyal, Yihan Zhang, Winnie Chow, Rui Pan, Shizhe Diao, Jipeng Zhang, Kashun Shum, and Tong Zhang. 2023a. RAFT: reward ranked finetuning for generative foundation model alignment. _Trans. Mach. Learn. Res._, 2023.
* Dong et al. (2023b) Qingxiu Dong, Li Dong, Ke Xu, Guangyan Zhou, Yaru Hao, Zhifang Sui, and Furu Wei. 2023b. Large language model for science: A study on P vs. NP. _CoRR_, abs/2309.05689.
* Dubey et al. (2019) Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Scheletn, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archit Mitra, Archic Sarvankumar, Artem Korenev, Arthur Hinsyark, Arnu Rao, Aston Zhang, Aurelien Rodriguez, Austen Gregerson, Ava Spataru, Baptite Roziere, Bethany Birron, Binh Tang, Bobbie Chern, Charlotte Caucheteux, Chaya Nayak, Choe Bi, Chris Marra, Chris McConnell, Christian Keller, Christophe Touret, Chunyang Wu, Corinne Wong, Cristian Canton Ferrer, Cyrus Nikolaidis, Damlen Allonsius, Daniel Song, Danielle Pinte, Danny Livshits, David Esibou, Dhruv Choudhary, Dhruv Mahajan, Diego Garcia-Olano, Diego Perino, Dieuwke Hupkes, Egor Lakomkin, Ehab AlbBadawy, Elina Lobanova, Emily Dinan, Eric Michael Smith, Filip Radenovic, Frank Zhang, Gabriel Synnaevre, Gabriele Lee, Georgia Lewis Anderson, Graeme Neil, Gregorie Milion, Guan Pang, Guillem Cucuvell, Hailey Nguyen, Hannah Korcevar, Hu Xu, Hugo Tuovuron, Hiyan Zarov, Imanol Arrieta Ibarra, Isabel M. Kloumann, Ishan Misra, Ivan Evtimov, Jade Copet, Jaewon Lee, Jan Geffert, Jana Vranes, Jason Park, Jay Mahadeokar, Jeet Shah, Jelmer van der Linde, Jennifer Billock, Jenny Hong, Jenya Lee, Jeremy Fu, Jianfeng Chi, Jianyu Huang, Jiawen Liu, Jie Wang, Jicca Yu, Joanna Bitton, Joe Spisak, Jongsoo Park, Joseph Rocca, Joshua Johnston, Joshua Saxe, Junteng Jia, Kalyan Vasaden Ahvala, Kartikeya Upasanti, Kate Flawiak, Ke Li, Kenneth Heafield, Kevin Stone, and et al. 2024. The llama 3 herd of models. _CoRR_, abs/2407.21783.
* Fu et al. (2023) Yao Fu, Hao Peng, Ashish Sabharwal, Peter Clark, and Tushar Khot. 2023. Complexity-based prompting for multi-step reasoning. In _The Eleventh International 

Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023_. OpenReview.net.
* Gulcehre et al. (2023) Caglar Gulcehre, Tom Le Paine, Srivatsan Srinivasan, Ksenia Konyushkova, Lotte Weerts, Abhishek Sharma, Aditya Siddhant, Alex Ahern, Maosen Wang, Chenjie Gu, Wolfgang Macherey, Arnaud Doucet, Orhan Firat, and Nando de Freitas. 2023. Reinforced self-training (rest) for language modeling. _CoRR_, abs/2308.08998.
* the rise of code intelligence. _CoRR_, abs/2401.14196.
* Guo et al. (2024b) Yanzhu Guo, Guokan Shang, Michalis Vazirgiannis, and Chloe Clavel. 2024b. The curious decline of linguistic diversity: Training language models on synthetic text. In _Findings of the Association for Computational Linguistics: NAACL 2024, Mexico City, Mexico, June 16-21, 2024_, pages 3589-3604. Association for Computational Linguistics.
* Haluptzok et al. (2023) Patrick Haluptzok, Matthew Bowers, and Adam Tauman Kalai. 2023. Language models can teach themselves to program better. In _The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023_. OpenReview.net.
* Hendrycks et al. (2021) Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. 2021. Measuring mathematical problem solving with the MAHT dataset. In _Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks 1, NeurIPS Datasets and Benchmarks 2021, December 2021, virtual_.
* Hosseini et al. (2024) Arian Hosseini, Xingdi Yuan, Nikolay Malkin, Aaron C. Courville, Alessandro Sorconi, and Rishabh Agarwal. 2024. V-star: Training verifiers for self-taught reasoners. _CoRR_, abs/2402.06457.
* Huang et al. (2023) Jiaxin Huang, Shixiang Gu, Le Hou, Yuexin Wu, Xuezhi Wang, Hongkun Yu, and Jiawei Han. 2023. Large language models can self-improve. In _Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, EMNLP 2023, Singapore, December 6-10, 2023_, pages 1051-1068. Association for Computational Linguistics.
* Jiang et al. (2023) Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de Las Casas, Florian Bressand, Gianna Lengyel, Guillam Lauple, Lucile Saulinier, Lello Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothee Lacroix, and William El Sayed. 2023. Mistral ?b. _CoRR_, abs/2310.06825.
* Kaplan et al. (2020) Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. 2020. Scaling laws for neural language models. _CoRR_, abs/2001.08361.
* December 9, 2022_.
* Kwon et al. (2023) Woosuk Kwon, Zhuohuan Li, Siyuan Zhuang, Ying Sheng, Liaminin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. 2023. Efficient memory management for large language model serving with pagedattention. In _Proceedings of the 29th Symposium on Operating Systems Principles, SOSP 2023, Koblenz, Germany, October 23-26, 2023_, pages 611-626. ACM.
* Lanham et al. (2023) Tamera Lanham, Anna Chen, Ansh Radhakrishnan, Benoi Steiner, Carson Denison, Danny Hernandez, Dustin Li, Esin Durmus, Evan Hubinger, Jackson Kernion, Kamille Lukosiute, Karina Nguyen, Newton Cheng, Nicholas Joseph, Nicholas Schiefer, Oliver Rausch, Robin Larson, Sam McCandlish, Sandipan Kundu, Saurav Kadavath, Shannon Yang, Thomas Henighan, Timothy Maxwell, Timothy Telleen-Lawton, Tristan Hume, Zac Hatfield-Dodds, Jared Kaplan, Jan Brauner, Samuel R. Bowman, and Ethan Perez. 2023. Measuring faithfulness in chain-of-thought reasoning. _CoRR_, abs/2307.13702.
* Liang et al. (2024)Yiming Liang, Ge Zhang, Xingwei Qu, Tianyu Zheng, Jiawei Guo, Xinrun Du, Zhenzhu Yang, Jiaheng Liu, Chenghua Lin, Lei Ma, Wenhao Huang, and Jianjun Zhang. 2024. I-SHEEP: self-alignment of LLM from scratch through an iterative self-enhancement paradigm. _CoRR_, abs/2408.08072.
* Lightman et al. (2024) Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. 2024. Let's verify step by step. In _The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024_. OpenReview.net.
* August 4, Volume 1: Long Papers_, pages 158-167. Association for Computational Linguistics.
* Liu et al. (2024) Wei Liu, Weihao Zeng, Keqing He, Yong Jiang, and Junxian He. 2024. What makes good data for alignment? A comprehensive study of automatic data selection in instruction tuning. In _The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024_. OpenReview.net.
* Patel et al. (2021) Arkil Patel, Satwik Bhattacharya, and Navin Goyal. 2021. Are NLP models really able to solve simple math word problems? In _Proceedings of the 2021 

Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2021, Online, June 6-11, 2021_, pages 2080-2094. Association for Computational Linguistics.
* Qi et al. (2024) Zhenting Qi, Mingyuan Ma, Jiahang Xu, Li Lyna Zhang, Fan Yang, and Mao Yang. 2024. Mutual reasoning makes smaller lms stronger problem-solvers. _CoRR_, abs/2408.06195.
* Roziere et al. (2023) Baptiste Roziere, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Tal Remez, Jeremy Rapin, Artym Konzhevnikov, Ivan Evtimov, Joanna Bitton, Manish Bhatt, Cristian Canton-Ferrer, Aaron Grattafiori, Wenhan Xiong, Alexandre Defossez, Jade Copet, Faisal Azhar, Hugo Touvron, Louis Martin, Nicolas Usunier, Thomas Scialom, and Gabriel Synnaeve. 2023. Code llama: Open foundation models for code. _CoRR_, abs/2308.12950.
* Schaal (1996) Stefan Schaal. 1996. Learning from demonstration. In _Advances in Neural Information Processing Systems 9, NIPS, Denver, CO, USA, December 2-5, 1996_, pages 1040-1046. MIT Press.
* Setlur et al. (2024) Amrith Setlur, Saurabh Garg, Xinyang Geng, Naman Garg, Virginia Smith, and Aviral Kumar. 2024. RL on incorrect synthetic data scales the efficiency of LLM math reasoning by eight-fold. _CoRR_, abs/2406.14532.
* Shao et al. (2024) Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Mingchuan Zhang, Y. K. Li, Y. Wu, and Daya Guo. 2024. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. _CoRR_, abs/2402.03300.
* Shumailov et al. (2023) Ilia Shumailov, Zakhar Shumaylov, Yiren Zhao, Yarin Gal, Nicolas Papernot, and Ross J. Anderson. 2023. The curse of recursion: Training on generated data makes models forget. _CoRR_, abs/2305.17493.
* Singh et al. (2021) Avi Singh, John D. Co-Reyes, Rishabh Agarwal, Ankesh Anand, Piyush Patil, Xavier Garcia, Peter J. Liu, James Harrison, Jaehoon Lee, Kelvin Xu, Aaron T. Parisi, Abhishek Kumar, Alexander A. Alemi, Alex Rizkowsky, Azade Nova, Ben Adlam, Bernd Bohnet, Gamallemidu Fally Eislyagd, Hanie Sedghi, Igor Mordatch, Isabelle Simpson, Izzeddin Gur, Jasper Snoeb, Jeffrey Pennington, Jiri Hron, Kathleen Kenelay, Kevin Swersky, Kahiteej Mahajan, Laura Culp, Lechao Xiao, Maxwell L. Bileschi, Noah Constant, Roman Novak, Rosanne Liu, Tris Warkentin, Yunqi Qian, Yamini Bansal, Ethan Dyer, Behnam Neyshabbar, Jascha Sohl-Dickstein, and Noah Fielded. 2024. Beyond human data: Scaling self-training for problem-solving with language models. _Trans. Mach. Learn. Res._, 2024.
* December 9, 2022_.
* Suay and Chernova (2011) Halit Bauer Suay and Sonia Chernova. 2011. Effect of human guidance and state space size on interactive reinforcement learning. In _2011 Ro-Man_, pages 1-6. IEEE.
* Subramanian et al. (2016) Kaushik Subramanian, Charles Lee Isbell Jr., and Andrea Lockerd Thomaz. 2016. Exploration from demonstration for interactive reinforcement learning. In _Proceedings of the 2016 International Conference on Autonomous Agents & Multiagent Systems, Singapore, May 9-13, 2016_, pages 447-456. ACM.
* Tong et al. (2024) Yuxuan Tong, Xiwen Zhang, Rui Wang, Ruidong Wu, and Junxian He. 2024. Dart-math: Difficulty-aware rejection tuning for mathematical problem-solving. _CoRR_, abs/2407.13690.
* Touvron et al. (2021) Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almhairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton-Ferrer, Moya Chen, Guillem Cucurull, David Esibbu, Jude Fernandes, Jeremy Fu, Wenyiu Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Sagher Hossini, Rui Hou, Hakan Innan, Marcin Kardas, Viktor Kerkez, Madain Khabsa, Isabel Kloumann, Artem Korenev, Punti Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee,iana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaloylov, Pushkar Mishra, Igor Molbybo, Yixin Nie, Andrew Poulton, Jeremy Rekienstein, Rashi Rungta, Kalyan Saladi, Alan Scheletlen, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaqiang Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Purxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Rambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Sciolam. 2023. Llama 2: Open foundation and fine-tuned chat models. _CoRR_, abs/2307.09288.
* Wei et al. (2022b) Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, Ed H. Chi, Tatsuorni Hashimoto, Oriol Vinyals, Percy Liang, Jeff Dean, and William Fedus. 2022a. Emergent abilities of large language models. _Trans. Mach. Learn. Res._, 2022.
* Wei et al. (2022c) Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022b. Chain-of-thought prompting elicits reasoning in large language models. _Advances in neural information processing systems_, 35:24824-24837.
* Wu et al. (2024) Ting Wu, Xuefeng Li, and Pengfei Liu. 2024. Progress or regress? self-improvement reversal in post-training. _CoRR_, abs/2407.05013.
* Xi et al. (2021) Zhiheng Xi, Wenxiang Chen, Boyang Hong, Senje Jin, Rui Zheng, Wei He, Yiwen Ding, Shichun Liu, Xin Guo, Junzhe Wang, Honglin Guo, Wei Shen, Xiaoran 

Fan, Yuhao Zhou, Shihan Dou, Xiao Wang, Xinbo Zhang, Peng Sun, Tao Gui, Qi Zhang, and Xuanjing Huang. 2024. Training large language models for reasoning through reverse curriculum reinforcement learning. In _Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024_. OpenReview.net.
* 16, 2023_.
* Yang et al. (2024) Zhaorui Yang, Tianyu Pang, Haozhe Feng, Han Wang, Wei Chen, Minfeng Zhu, and Qian Liu. 2024. Self-distillation bridges distribution gap in language model fine-tuning. In _Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2024, Bangkok, Thailand, August 11-16, 2024_, pages 1028-1043. Association for Computational Linguistics.
* Yu et al. (2024) Longhui Yu, Weisen Jiang, Han Shi, Jincheng Yu, Zhengying Liu, Yu Zhang, James T. Kwok, Zhenguo Li, Adrian Weller, and Weiyang Liu. 2024. Metamath: Bootstrap your own mathematical questions for large language models. In _The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024_. OpenReview.net.
* 16, 2023_.
* Yuan et al. (2024) Weizhe Yuan, Richard Yuanzhe Pang, Kyunghyun Cho, Xian Li, Sainbayar Sukhbaatar, Jing Xu, and Jason Weston. 2024. Self-rewarding language models. In _Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024_. OpenReview.net.
* Yuan et al. (2023) Zheng Yuan, Hongyi Yuan, Chengpeng Li, Guanting Dong, Chuanqi Tan, and Chang Zhou. 2023. Scaling relationship on learning mathematical reasoning with large language models. _CoRR_, abs/2308.01825.
* Yue et al. (2024) Xiang Yue, Xingwei Qu, Ge Zhang, Yao Fu, Wenhao Huang, Huan Sun, Yu Su, and Wenhu Chen. 2024. Mammoth: Building math generalist models through hybrid instruction tuning. In _The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024_. OpenReview.net.
* Zelikman et al. (2022) Eric Zelikman, Yuhual Wu, Jesse Mu, and Noah Goodman. 2022. Star: Bootstrapping reasoning with reasoning. _Advances in Neural Information Processing Systems_, 35:15476-15488.
* Zhang et al. (2023) Yue Zhang, Yafu Li, Leyang Cui, Deng Cai, Lemao Liu, Tingchen Fu, Xinting Huang, Enbo Zhao, Yu Zhang, Yulong Chen, Longyue Wang, Anh Tuan Luu, Wei Bi, Freda Shi, and Shuming Shi. 2023. Siren's song in the AI ocean: A survey on hallucination in large language models. _CoRR_, abs/2309.01219.
* Zhao et al. (2023) Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, Yifan Du, Chen Yang, Yushuo Chen, Zhipeng Chen, Jinhao Jiang, Ruiyang Ren, Yifan Li, Xinyu Tang, Zikang Liu, Peiyu Liu, Jian-Yan Nie, and Ji-Rong Wen. 2023. A survey of large language models. _CoRR_, abs/2303.18223.

## Appendix A Dataset Details

For the training set, we randomly select a subset from the extensive datasets provided by MathInstruct (Yue et al., 2024), including AquA, GSM8K, and MATH. The test set is consistent with MathInstruct. The specific data quantities are shown in Table 4. We comply with the license for the use of these datasets in our work.

### Error Patterns

Here, we analyze two classic types of errors in model-generated rationales.

Hint Short-cutting.This occurs when the model is provided with the final answer or rationales as a hint during generation, causing it to skip steps (Zelikman et al., 2022). This leads to responses like those shown in Figure 6 and Figure 7.

Spurious Correctness.This refers to cases where, when given the final answer as a hint, the model fails to generate the correct intermediate reasoning due to its limited capabilities but still copies the correct final answer and misleads the reward function. Responses classified as positive may contain incorrect or irrelevant rationales, and training on such data often incentivizes the model to overfit to spurious correlations, resulting in flat

\begin{table}
\begin{tabular}{l c c c c c c} \hline  & AquA & GSM8K & MATH & MathQA & SVAMP & Thm.QA \\ \hline \# CoT Train & \(3000\) & \(3000\) & \(4000\) & - & - & - \\ \# PoT Train & \(1961\) & \(3000\) & \(4000\) & - & - & - \\ \# Test & \(254\) & \(1319\) & \(5000\) & \(2985\) & \(1000\) & \(800\) \\ \hline \end{tabular}
\end{table}
Table 4: Dataset statistics of the train and test set.

 

or even inverse scaling with more data during finetuning (Setlur et al., 2024). Figure 8 shows a case of spurious correct response.

## Appendix C Case Study

Here, we present two successful cases: GSI (interactive sampling) in Figure 9 and GSI (state reset) in Figure 10, demonstrating the effectiveness of our proposed method.

### Prompt Details

The prompt template of the vanilla self-improve method is presented in Figure 11. The prompt templates of proposed Guided Self-Improvement (GSI) are detailed in Figure 12, 13, 14, 15, and 16.

 



**Question**: Kathryn moved to a new city for the new job she had landed two weeks ago.

Her rent was $1200, 1/2 of what she spent on food and travel expenses in a month. Luckily, she found a new friend Shelby, who moved in with her to share the rent. If her salary was

$5000 per month, how much money remained after her expenses?

Figure 6: An example of **skipped steps** on the GSM8K dataset. The key steps within the original rationales are highlighted in purple, while the corresponding steps generated by Llama2 are indicated in blue. In this case, the model, guided by rationale-driven hints, skips critical steps and directly arrives at a premature conclusion.





**Question**: Andy started out the year weighing 156 pounds. He then grew 3 inches and gained 36 pounds. Andy wasn' happy with his weight and decided to exercise. Over the next 3 months, he lost an eighth of his weight every month. How much less does Andy weigh now than at the beginning of the year?

Andy weighed 156 + 36 = 192 pounds after growing taller. An eighth of his weight is 1/8 x 192 = 24 pounds. Over the next 3 months, Andy lost 24 x 3 = 72 pounds. He now weighs 192 - 72 = 120 pounds. Andy weighs 156 - 120 = 36 pounds less than at the beginning of the year. The answer is 36.

Figure 8: An example of a **hallucinated reasoning process** on the GSM8K dataset. The key steps within the original rationales are highlighted in purple, while the corresponding steps generated by Llama2 are indicated in blue. In this case, the model fails to generate the correct intermediate reasoning but still copies the correct final answer. This behavior can mislead the reward function and introduce noise into the training data in subsequent iterations. Such cases arise mainly with answer-driven hints.





**Question**: Average monthly income of a family of 3 earning members was Rs. 735. One of the earning members died and therefore, the average income came down to Rs 650. The income of the deceased was?

**Choices**: (A) Rs. 692.80 (B) Rs. 905 (C) Rs. 990 (D) Rs. 1385 (E) None

**General Rationales**

Let's think about the multi-choice question.

Income of the deceased

= Totalincome of 3 members - total

income of remaining 2 members.

= 735 x 3 - 650 x 2 Rs.

= 905 Rs.

The answer is B.

* 650 x 3". The mistake lies in the fact that you subtracted the total income of 3 members (after the death) from the total income of 3 members (before the death).

You should have subtracted the total income of 2 members (after the death) from the total income of 3 members (before the death). In other words, you incorrectly used 650 x 3 instead of 650 x 2 in your calculation.

**Corrected Rationales**

Let's solve the multi-choice question step by step.

Income of deceased = Total income of 3 members - total income of remaining 2 member = 735 x 3 - 650 x 2 = 2205 - 1300 = Rs. 905

Theanswer is B.

Figure 9: An example of **interactive sampling** on the AQuA dataset. The key steps within the original rationales are highlighted in purple, while the corresponding steps generated by Llama2 are indicated in blue. In the feedback from the stronger model, errors pointed out are marked in **red**, and corrections are indicated in **green**. In this case, the model successfully corrected its mistake after receiving feedback and arrived at the correct final answer.





**Question**: A married couple opened a savings account. The wife committed to saving $100 every week while the husband committed to saving $225 every month. After 4 months of savings, they decided to invest half of their money in buying stocks. Each share of stocks costs $50. How many shares of stocks can they buy?

Figure 11: Prompt template for vanilla self-improve method. It is also used in the first step of GSI (Interactive Sampling), with the process transitioning into the subsequent interactive steps when incorrect results are sampled.

Figure 10: An example of **state reset** on the GSM8K dataset. The key steps within the original rationales are highlighted in purple, while the corresponding steps generated by Llama2 are indicated in blue. The (Given) highlights the partial rationale provided to the model as a hint. In this case, the model starts from the intermediate steps and gives a different solution, enriching the rationales in the training data.





**Input:**

Below is a math problem, please give a step-by-step answer.

### Question: {question} {answer}

### Your step-by-step answer:

Figure 14: Prompt template for the strong model in GSI (Interactive Sampling) method.

Figure 12: Prompt template for GSI (Answer-driven) method.





**Input:**

Below is a correction to your previous solution. Review this carefully and use it to revise your solution. Ensure that it includes all necessary steps clearly and thoroughly.

### Question: {question}

## Your original wrong answer: {wrong_answer}

### Correction and guidance: {correct_message}

### Your revised, complete step-by-step solution:

Figure 16: Prompt template for GSI (State Reset) method. The partial rationale is truncated from the complete one and placed at the end to guide the model in completing it.

Figure 15: Prompt template for the self-improved model in GSI (Interactive Sampling) method.