Venue: NAACL 2024

# Teaching Language Models to Self-Improve through

Interactive Demonstrations

 Xiao Yu\({}^{\dagger}\) Baolin Peng\({}^{\ddagger}\) Michel Galley\({}^{\dagger}\) Jianfeng Gao\({}^{\dagger}\) Zhou Yu\({}^{\dagger}\)

\({}^{\dagger}\)Columbia University \({}^{\dagger}\)Microsoft Research

{xy2437,zy2416}@columbia.edu

{mgalley,jfgao}@microsoft.com

Now at Tencent AI (baolineng@global.tencent.com).

###### Abstract

The self-improving ability of large language models (LLMs), enabled by prompting them to analyze and revise their own outputs, has garnered significant interest in recent research. However, this ability has been shown to be absent and difficult to learn for smaller models, thus widening the performance gap between state-of-the-art LLMs and more cost-effective and faster ones. To reduce this gap, we introduce TriPosT, a training algorithm that endows smaller models with such self-improvement ability, and show that our approach can improve LLaMA-7B's performance on math and reasoning tasks by up to 7.13%. In contrast to prior work, we achieve this by using the smaller model to interact with LLMs to collect feedback and improvements on _its own generations_. We then replay this experience to train the small model. Our experiments on four math and reasoning datasets show that the interactive experience of learning from and correcting its _own_ mistakes is crucial for small models to improve their performance.

## 1 Introduction

Large language models [5, 8] together with techniques such as few-shot prompting [9] and Chain-of-Thought (CoT) prompting [10, 11] have been shown to be effective in achieving strong performance on various downstream language tasks. More recently, a new way to adapt LLMs to downstream tasks has captured the attention of many researchers, namely to further enhance the LLM's downstream task performance by asking the LLM to provide feedback on its own generations and then use the feedback to revise its outputs [1, 14, 15, 16]. This process is often called "self-improvement", and has proven to be an effective technique to make the LLM's generations more diverse, more precise, or more faithful to a given piece of knowledge [1, 13, 15].

However, Saunders et al. [15]; Huang et al. [16] found that the ability to generate critical feedback or to self-improve is hardly evident in smaller models1. Similarly, Ye et al. [16] found that fine-tuning smaller models (e.g. 7-13B) with self-improvement demonstrations from LLMs can still fail on tasks such as math, reasoning, and factuality. Following these previous works, we performed a similar study on two math and reasoning tasks in Figure 1. We compared the accuracy of the final answer generated by prompting

Figure 1: Compared to LLMs, smaller models have difficulty performing self-improvement on math or logical tasks, such as Multistep Arithmetics and Logical Deduction from the Big-Bench. +_ft_: finetuned on ground-truth rationales; +_SI. prompt_: prompted to perform self-improvement; +_ft SI. demo_ further finetuned +_ft_ on LLM self-improvement demonstrations.



a 175B Codex Chen et al. (2021) to self-improve, with prompting or training a LLMaMA-7B model to self-improve using demonstrations from Codex Ye et al. (2023). In Figure 1, we surprisingly find that _smaller models performed worse_ using prior self-improvement-related methods than simply training on ground-truth step-by-step rationales (+\(f\)t). By comparing the generated solutions from Codex-175B and LLMaM-7B, we find that smaller models, such as LLaMA-7B, not only make more mistakes, but also _different types of mistakes_ compared to an LLM (Table 1 and Appendix B). Due to the smaller model's weaker math and reasoning ability, we believe training on LLM self-improvement demonstrations is less effective, as it forces the smaller model to learn from mistakes not of its own.

Motivated by this finding, we propose TriPosT, a training algorithm that can more effectively train a small model to learn from its mistakes, generate feedback, and improve its performance on math and reasoning tasks. TriPosT is an iterative algorithm consisting of three stages: Interactive **Tra**jectory **E**diting, Data **Post**-processing, and Model **T**raining. Similar to the exploration stage in reinforcement learning, TriPosT first creates improvement demonstrations _using the small model to interact_ with the expert LLMs or relevant Python scripts. Then, TriPosT postprocesses the collected data by filtering out failed improvement attempts, and then re-balances the dataset to disincentivize the model from trying to self-"improve" when it is not needed. Finally, TriPosT replays the post-process dataset Andrychowicz et al. (2018); Schaul et al. (2016), and trains the smaller model using weighted supervised learning. TriPosT repeats entire the process several times. We evaluate our approach on four maths and reasoning datasets from the BIG-Bench Hard Suzgun et al. (2022) collection, and find that TriPosT-trained models can use its learned self-improvement ability to improve their task performance. We also find that TriPosT-trained models achieve better in-domain and out-of-domain performance than models trained using just the ground truth step-by-step rationales and trained using direct LLM demonstrations Saunders et al. (2022); Ye et al. (2023). This paper makes the following contributions:

* We illustrate how prior work Saunders et al. (2022); Ye et al. (2023) can be ineffective in training smaller models to self-improve their performance on math and reasoning tasks.
* We propose TriPosT, an iterative training algorithm that trains a smaller language model to learn to self-improve.
* We show that TriPosT-trained models achieve better performance than models trained using ground-truth rationales or using LLM demonstrations on four math and reasoning datasets from BIG-Bench Hard.

## 2 Approach

TriPosT is an algorithm that trains a small language model to self-improvey by learning from its _own mistakes_. Each iteration of TriPosT consists of three stages. On a high level, we first collect

\begin{table}
\begin{tabular}{l l c} \hline Method & Codex (175B) & LLMaMA+ft (7B) \\ \hline Input & Q: ((4 - 0 ° - 1° 8 + 6) + (0 + 4 + 9 ° 5 ° - 2)) = \\ \hline Attempt: & (1) This equation can be written as ”(A + B)” ... \\  & (2) Let’s calculate A=(4 - 0 ° s– 1 ° 8 + 6) & can be written as “(A + B)” ... \\  & =(4 - (0 ° - 1 ° 8 + 6) & can be written as “(A + B)” ... \\  & =(4 - 0 ° - 1 ° 8 + 6) & can be written as “(A + B)” ... \\  & =(4 - 0 * - 1 ° 8 + 6) & can be written as “(A + B)” ... \\  & =(4 - 0 * - 1 ° 8) + 6) & =(4 - 0 * - 1° 8 + 6) \\  & =(4 - 0 (+ 6) & =(4 - 0 + 8) + 6) \\  & =(4 - 6) & =(4 - 8 + 6) \\  & =2 & — \\  & =(0 + 4 + 9 ° 5 ° - 2) & (3) Let’s calculate B=(0 + 4 + 9 ° 5 ° - 2) \\  & =(0 + 4 + (9 ° 5 ° - 2)) & =(0 + (4 + (9 ° 5))) \\  & =(0 + 4 + (90)) & =(0 + (13)) \\  & =86 & —13 \\  & (4) Then, the final eq i is (A + B) = (- 2 + - 86) = - 88. & (4) Then, the final eq is (A + B) = (2 + 13) = 15. \\  & (Final response) So the answer is –88. & (Final response) So the answer is 15. \\ \hline \end{tabular}
\end{table}
Table 1: Training smaller models using self-improvement demonstrations from LLMs can be ineffective, as models of different sizes make different types and amount of mistakes (highlighted in red). Small models can make simple copying errors, while LLMs can make other arithmetic errors, such as not switching plus or minus signs when adding parentheses. See Appendix B for a more quantitative analysis.

 

a set of improving trajectories by using a smaller model \(M_{\theta}\) to interact with LLMs. We use \(M_{\theta}\) to generate initial attempts and then use a feedback module \(\mathrm{FBK}\) and an improvement module \(\mathrm{IMP}\) to edit parts of the \(M_{\theta}\) generated attempts. This creates a trajectory that includes attempts generated by the small model, with feedbacks and improvements tailored to the small model's capability (Figure 2). Next, we post-process the collected trajectories by 1) using scripts and other heuristics to filter out failed "improvement" attempts; and 2) re-balancing the dataset using both directly correct attempts and the improving trajectories. Finally, we use weighted supervised learning to train a smaller model \(M_{\theta}\) using the post-processed data.

We provide an overview of our algorithm in Figure 2, and detail each of the three stages in Section 2.2, Section 2.3, and Section 2.4, respectively.

### Notation

We denote the entire attempt from a language model to solve a given question as a trajectory \(x\):

\[x=(x^{\rm att}_{0},x^{\rm fb}_{1},x^{\rm att}_{1},x^{\rm fb}_{2},x^{\rm att}_ {2},...,x^{\rm fb}_{m}),\]

where \(x^{\rm att}_{0}\) denotes the initial attempt, and \(x^{\rm fb}_{i},x^{\rm att}_{i}\) denotes the \(i\)-th feedback and updated attempt, respectively. Such a trajectory ends when the last feedback \(x^{\rm fb}_{m}\) contains the phrase "the final response is correct". Therefore, _directly correct_ trajectories take the form of \(x_{\boldsymbol{\check{\check{\check{\check{\prime}}}}}}=(x^{\rm att}_{0},x^{ \rm fb}_{1})\), and _self-improving_ trajectories take the form of \(x_{\rm SI}=(x^{\rm att}_{0},x^{\rm fb}_{1},x^{\rm att}_{1},...,x^{\rm fb}_{m})\) where \(m>1\).

### Interactive Trajectory Editing

In our prior study in Figure 1 and Table 1, we find that it is difficult to elicit a 7B model to perform self-improvement due to its significantly weaker math and reasoning capability compared to LLMs. To address this issue, we use the smaller model \(M_{\theta}\) to first generate an initial attempt2, and then apply a feedback module \(\mathrm{FBK}\) and an improvement module \(\mathrm{IMP}\) to _rewrite parts of the \(M_{\theta}\) trajectories_. Specifically, we first use \(\mathrm{FBK}\) (prompting text-davinci-003 or using a Python script) to generate a feedback \(x^{\rm fb*}_{i}\) based on the first error step it identified for each incorrect attempt. After that, we edit the trajectory by replacing the first feedback that \(M_{\theta}\) and \(\mathrm{FBK}\) disagree on with the \(\mathrm{FBK}\)-generated feedback, creating an edited trajectory:

Footnote 2: We also allow \(M_{\theta}\) to attempt generating feedbacks and improvements, as self-improvement training progresses.

\[(x^{\rm att}_{0},...,x^{\rm att}_{i-1},x^{\rm fb*}_{i}).\]

Finally, we use our improvement module \(\mathrm{IMP}\) (prompting Codex) to generate an improved attempt \(x^{\rm att*}_{i}\) conditioned on the previous \(x^{\rm att}_{i-1}\) and feedback \(x^{\rm fb*}_{i}\), and append it to the trajectory:

\[x_{\rm edited}=(x^{\rm att}_{0},...,x^{\rm att}_{i-1},x^{\rm fbs}_{i},x^{\rm att *}_{i}).\]

Figure 2: Overview of TriPosT algorithm. TriPoST consists of three stages: interactive trajectory editing where we use our \(\mathrm{FBK}\) and \(\mathrm{IMP}\) module to edit trajectories generated by a smaller model \(M_{\theta}\); data post-processing where we filter out erroneous trajectories and create a re-balanced dataset; and model training where we train \(M_{\theta}\) using weighted supervised learning on the post-processed dataset.

 

As an example, if feedback \(x_{i}^{\text{fbs}}\) identifies that the first mistake in \(x_{i-1}^{\mathrm{att}}\) appears in step 3, then step 1-2 in \(x_{i-1}^{\mathrm{att}}\) is kept untouched, and \(\operatorname{IMP}\) is used to generate an improved solution by only changing steps \(\geq 3\). This design is to prevent \(\operatorname{IMP}\) from re-writing the whole attempt from scratch (e.g., generating the gold solution), which would violate our motivation to create trajectories with feedback and improvements that are incremental and tailored to the small model's capability.

We repeat this process, up to a maximum number of iterations, until the last attempt in \(x_{\mathrm{edited}}\) is correct. Otherwise, we discard \(x_{\mathrm{edited}}\) that failed to reach the correct answer.

### Data Post-processing

After the interactive trajectory editing step, we have three types of data: 1) gold step-by-step demonstrations \(x_{\mathrm{gold}}\) for the task, 2) directly correct trajectories \(x_{\boldsymbol{\check{\mathsf{missing}}}}\) generated by \(M_{\theta}\), and 3) edited trajectories \(x_{\mathrm{editted}}\) created using \(M_{\theta},\operatorname{FBK}\), and \(\operatorname{IMP}\).

To make training easier, we first split _all data_ into triplets of _single-step improvement_\(x_{\mathrm{imp}}=(x_{i}^{\mathrm{att}},x_{i}^{\mathrm{fb}},x_{i+1}^{\mathrm{ att}})\) if an attempt \(x_{i}^{\mathrm{att}}\) was incorrect, or into \(x_{\mathrm{T}}=(x^{\mathrm{att}},x^{\mathrm{fb}})\) where the attempt is correct and the trajectory ends with \(x^{\mathrm{fb}}\) containing the phrase "the final response is correct". To learn from expert's correction, \(x_{j}^{\mathrm{att}}\) and \(x_{j}^{\mathrm{fb}}\) may be the edited \(x_{j}^{\mathrm{att*}}\) and \(x_{j}^{\mathrm{fb*}}\), respectively (see Section 2.2). Next, we filter out some \(x_{\mathrm{imp}}\) triplets that contain incorrect feedbacks or improvement steps using some rules (see more in Appendix I). Then, we combine \(x_{\mathrm{T}}\) and filtered \(x_{\mathrm{imp}}\) into a single dataset, and balance them using a hyperparameter \(p\) specifying the proportion of \(x_{\mathrm{imp}}\). We find that this parameter is important for the model to learn to improve its attempt _only when necessary_. This is because we found that training with too many \(x_{\mathrm{imp}}\) can cause the model to attempt self-improvement even when the last attempt is already correct, thus damaging its performance (see Section 4.2 for more details).

### Model Training

Finally, we use supervised learning (SL) to train a smaller model \(M_{\theta}\) on the combined dataset. To promote the model to focus on learning the feedback and improvement steps in \(x_{\mathrm{imp}}\), we use a weighted cross-entropy loss. We weight the loss for all the tokens in \(x_{\mathrm{T}}\) with \(w=1.0\), but with \(w>1.0\) for the tokens that belong to \(x_{i}^{\mathrm{fb}}\) or \(x_{i+1}^{\mathrm{att}}\) in single-step improvement triplets \(x_{\mathrm{imp}}\). We note that we also experimented with masking \(x_{i}^{\mathrm{att}}\)[Zheng et al., 2023], but found it to be less effective than weighted SL in our case. See Appendix E for more empirical analysis and discussions on related techniques.

### TriPosT

In Figure 2 and Algorithm 1 we summarize our TriPoST algorithm. For each of the \(t\) iterations, we first utilize \(M_{\theta}\) to generate its own attempts \(X\), and then use \(\operatorname{FBK}\) and \(\operatorname{IMP}\) to generate and create a set of edited trajectories as described in Section 2.2. Next, we process the newly collected trajectories and the gold task demonstrations \(X_{\mathrm{gold}}\) by first splitting them into a unified format of \(x_{\mathrm{imp}}\) triplet or \(x_{\mathrm{T}}\), and then filtering out erroneous \(x_{\mathrm{imp}}\) data (Section 2.3). Finally, we create a training dataset \(\mathcal{D}\) by balancing the number of \(x_{\mathrm{imp}}\) and \(x_{\mathrm{T}}\) using a hyperparameter \(p\), and finetune \(M_{\theta}\) on \(\mathcal{D}\) using weighted SL. Unless otherwise specified, we repeat this procedure for \(t=3\) iterations, and refer to the model trained using TriPostT with \(t\) iterations as TriPosT(\(t\)).

```
0: Generative language model \(M_{\theta}\)
0:\(\operatorname{FBK}\) and \(\operatorname{IMP}\) modules
0: Gold task demonstrations \(X_{\mathrm{gold}}\)
0: Data buffer \(\mathcal{B}\)
1:for\(t\) iterations do
2:// interactive trajectory editing
3: Gen. trajectories \(X=\{X_{\boldsymbol{\check{\mathsf{missing}}}},X_{\mathbf{k}}\}\) with \(M_{\theta}\)
4: Add correct trajectories \(X_{\boldsymbol{\check{\mathsf{missing}}}}\) to \(\mathcal{B}\)
5:for each incorrect trajectory \(x_{\boldsymbol{\check{\mathsf{missing}}}}\in X_{\boldsymbol{\check{\mathsf{missing}}}}\)do
6: Use \(\operatorname{FBK}\) to generate feedbacks \(x_{i}^{\mathrm{fb*}}\)
7: Replace feedback from \(x_{\boldsymbol{\check{\mathsf{missing}}}}\) with \(x_{i}^{\mathrm{fs}}\)
8: Prompt \(\operatorname{IMP}\) to generate \(x_{i+i}^{\mathrm{att}}\)
9: Repeat until termination cond. reached
10: Add edited trajectory \(x_{\mathrm{editted}}\) to \(\mathcal{B}\)
11:endfor
12:// data post-processing
13: Split \(X_{\mathrm{gold}}\cup\mathcal{B}\) into triplets \(x_{\mathrm{imp}}\) or \(x_{\mathrm{T}}\)
14: Filter \(x_{\mathrm{imp}}\)
15:\(\mathcal{D}=\{x_{\mathrm{imp}},x_{\mathrm{T}}\}\), balanced using \(p\)
16:// model training
17: Train \(M_{\theta}\) on \(\mathcal{D}\) using weighted SL
18:endfor
```

**Algorithm 1**TriPosT Training Algorithm

## 3 Experiments

In this section, we test if our TriPosT can 1) help distill self-improvement ability into a smaller model \(M_{\theta}\), and 2) help \(M_{\theta}\) improve performance on math and reasoning tasks 



### Dataset and Preprocessing

We utilize the BIG-Bench (Srivastava et al., 2023) benchmark to evaluate our approach. BIG-Bench is a collection of more than 200 text-based tasks including categories such as traditional NLP, mathematics, commonsense reasoning, and more.

We perform experiments on four math and reasoning tasks from the challenging BIG-Bench Hard (Suzgun et al., 2022) collection. We consider two _scriptable_ tasks: Multistep Arithmetic and Word Sorting, where a step-by-step solution (rationale) and a feedback can be generated using a script; and two _unscriptable_ tasks: Date Understanding and Logical Deduction, where we prompt an LLM (Codex/text-davinci-003) to generate feedbacks. We prompt Codex as the \({\rm IMP}\) module for all tasks.

For each task, we first collect a set of gold steps-by-step rationales by either scripting a solution for _scriptable_ tasks, or using the CoT prompts from Suzgun et al. (2022) to generate a solution using LLMs. For those LLM-generated rationales, we only keep the correct ones (see Appendix A for more details) for training. Then, to better measure a model's generalization ability, we split each of the 4 tasks further into _seen_ and _unseen_ subtasks. We mainly categorize simpler questions as the _seen_ subtasks to be used for model training. We describe our categorization method in Table 2.

### Models and Baselines

ModelsWe use LLaMA-7B as \(M_{\theta}\) in our main experiments in Table 3. LLaMA (Touvron et al., 2023) is a collection of foundation language models ranging from 7B to 65B that have shown strong performance compared to GPT-3 (175B) on many benchmarks (Zheng et al., 2023; Taori et al., 2023; Peng et al., 2023). Due to the cost of training language models, we use the smallest 7B model. For results with LLaMA-2 models, see Appendix D. For training hyperparameters, see Appendix J.

BaselinesWe compare TriPosT training with three baselines: fine-tuning using self-generated, self-consistent rationales (_LMSI_, Huang et al. (2023)); fine-tuning using only ground truth rationales (_ft rationale_); and fine-tuning using self-improvement demonstrations from LLMs (_ft SI. demo_, similar to Ye et al. (2023)). For better performance, we initialize with the model trained after _ft rationale_ for all methods. Lastly, for a fair comparison, we restrict iterative algorithms such as TriPosT to only have access to the same amount of input prompts as used to train baselines such as _ft rationale_. For more implementation details, see Appendix G and Appendix I.

### Metrics

To measure task performance, we follow prior studies on Big-Bench (Ho et al., 2023; Huang et al., 2023) and report the accuracy of the final answer extracted from the model's output. For each task, we report the accuracy on the seen subtasks and unseen subtasks, and its overall performance. To measure the model's self-improvement ability, we mainly consider two metrics: 1) how often the model tries to self-improve (_SI. Freq._), and 2) how much those of self-improvement attempts contribute to the model's task performance (_SI. Contrib._). We measure _SI. Freq._ as the number of times the model attempted to self-improve divided by the size of the test set, and _SI. Contrib._ as the number of times those improvement attempts actually reached the correct final answer.

### Main Results

Table 3 summarizes TriPosT's evaluation results on the four datasets. First, we find _LMSI_(Huang et al., 2023) to be roughly on-par with _ft. rationale_ only when the performance of the base model (i.e., _ft. rationale_) is already high on the training questions (the _seen_ subtask). This is understandable, as _LMSI_ was originally designed for LLM (e.g.,

\begin{table}
\begin{tabular}{l l l l l} \hline Dataset & Criterion & Example & _seen_ subtask & _unseen_ subtask \\ \hline Multistep Arithmetic & nesting depth (\(d\)) and & Q: (**2*** 2+1) + (3* 1 - 1)) & \(l=\{3,4\}\times d=\{2\}\) & \(l=\{3,4\}\times d=\{3\}\) and \\  & number of operands (\(l\)) & \(//l=3,d=2\) & \(l=\{5,6\}\times d=\{2,3\}\) \\ Word Sorting & number of words to sort (\(l\)) & Q: orange apple banana pear & \(l=\{2,3,...,7\}\) & \(l=\{8,9,....,16\}\) \\  & & \(//l=4\) & \(l=\{1,2\}\) & \(l\geq 3\) \\ Date Understanding & number of steps to solve (\(l\)) & Q: Today is 01/02, what’s the date yesterday? \(//l=1\) & \(l=\{1,2\}\) & \(l\geq 3\) \\ Logical Deduction & number of options (\(l\)) & Q: John runs ... Who runs fastest? Options: (A).. (B).. (C).. \(//l=3\) & \(l=\{7\}\) \\ \hline \end{tabular}
\end{table}
Table 2: Categorization of the datasets into seen and unseen tasks. _seen_ tasks are chosen to be easier and are used for training. Example questions are abbreviated, for complete examples please refer to Appendix A.



PaLM-540B) to improve on tasks where it can already achieve a reasonable performance. Next, we find _ft SI demo_ to slightly degrade the model's performance across all tasks, which we believe is due to the capability mismatch between the LLM demonstrator and the small LM learner (Section 1). This forces the small LM to learn from "advanced" errors not from its own (Table 1 and Appendix B). Finally, we see that in all tasks, TriPosT-trained models performs the best in all metrics. In general, we also observe improvement in the performance of TriPosT-trained models as the number of iterations \(t\) increases.3 We believe this is because, during the process of learning to self-improve, the model also learns to better understand the tasks by learning from its _own mistakes_[29, 7]. This enables the model to not only generate better initial attempts, but also improve its self-improvement ability.

Footnote 3: For a comparison against LMSI with more than \(t=1\) iteration, please see Appendix H.

In Table 4, we further explore the contribution of \(M_{\theta}\)'s self-improvement ability by describing how its overall performance improved. We find that in two out of the four datasets, TriPosT-trained models generate an more accurate initial attempt than the baselines (denoted as _Directly Correct_), and in all cases, TriPosT-trained models had measurable self-improvement contributions in both seen and unseen tasks (cf. Figure 1 and Table A4). This suggests that TriPosT-training can 1) help the model better understand the tasks and generate better initial attempts, and 2) help distill self-improving ability into the model. We believe that the combination of both factors improve the model's overall performance in Table 3.

### TriPosT-auto

In Table 5, we explore another way of training \(M_{\theta}\) with TriPosT. Instead of re-balancing the training dataset using a fixed \(p\) as in Section 3.4, we can simply include all the edited improvement tuples \(x_{\rm imp}\) and the directly correct attempts \(x_{\rm T}\) generated by \(M_{\theta}\). We denote this method as TriPosT-auto, as it automatically "balances" its training data to be proportional to its current performance, because \(p\) can be interpreted as how often the model's attempts were incorrect and needed editing. TriPosT-auto training included no less \(x_{\rm imp}\) compared to TriPosT (but generally more \(x_{\rm T}\), resulting in \(p<0.43\)), and we find that the model now rarely attempts to self-improve. However, this unexpectedly leads to even better overall performance, especially on _unscriptable_ tasks. We believe this indicates that 1) learning to always generate a useful feedback and the corresponding improvement is

\begin{table}
\begin{tabular}{l l c c c c} \hline \hline \multicolumn{2}{c}{Dataset} & \multicolumn{3}{c}{SI. Contrib.} \\  & seen & unseen & total & Directly Correct & Total Acc. \\ \hline Multistep Arithmetic & 1.39 & 0.28 & 1.67 & 20.83 & 22.50 \\ Word Sorting & 1.85 & 0.52 & 2.37 & 27.44 & 29.82 \\ Date Understanding Logical Deduction & 1.95 & 1.29 & 3.25 & 33.76 & 37.01 \\ \hline \hline \end{tabular}
\end{table}
Table 4: Analyzing how TriPosT-trained models improved the overall task performance. Total accuracy is first decomposed into attempts that are directly correct (_Directly Correct_) and attempts with self-improvement (_SL Contrib._). _SI. Contrib._ is then further decomposed into its accuracy contribution on the seen and unseen subtasks.

\begin{table}
\begin{tabular}{l l c c c c c c c c c c c} \hline \hline \multirow{2}{*}{Method} & \multicolumn{3}{c}{Multistep Arithmetic\({}^{\ddagger}\)} & \multicolumn{3}{c}{Word Sorting\({}^{\dagger}\)} & \multicolumn{3}{c}{Date Understanding} & \multicolumn{3}{c}{Logical Deduction} \\ \cline{3-13}  & & seen & unseen & total & seen & unseen & total & seen & unseen & total & seen & unseen & total \\ \hline \multirow{11}{*}{Prediction} & LMSI & 10.83 & 0.00 & 4.33 & 67.72 & 5.56 & 26.83 & 14.55 & 9.09 & 12.99 & 61.11 & 20.00 & 48.10 \\  & ft rationale & 39.75 & 1.48 & 16.78 & 73.49 & 5.82 & 28.50 & 33.35 & 21.21 & 29.87 & 62.69 & 8.67 & 45.78 \\  & ft SI. demo & 29.17 & 0.00 & 11.67 & 53.54 & 1.98 & 19.26 & 27.27 & 18.18 & 24.68 & 54.63 & 15.00 & 41.67 \\ \cline{1-1} \cline{2-13}  & TriPosT\((t=1)\) & 41.67 & 0.84 & 17.17 & 74.02 & 5.16 & 28.23 & 32.73 & 13.64 & 27.27 & 57.88 & **22.00** & 46.52 \\ \cline{1-1}  & TriPosT\((t=2)\) & 49.58 & 1.39 & 20.67 & 74.02 & **7.14** & 29.55 & 35.46 & 25.00 & 32.47 & 58.80 & 18.00 & 45.25 \\ \cline{1-1}  & TriPosT\((t=3)\) & **52.50** & **2.50** & **22.50** & **77.17** & 5.95 & **29.82** & **40.00** & **29.55** & **37.01** & **63.89** & 15.00 & **48.42** \\ \hline \hline \end{tabular}
\end{table}
Table 3: Overall performance of TriPosT on four BIG-Bench hard datasets. For each dataset, we train our models on the _seen_ tasks, and evaluate their performance on both _seen_ and _unseen_ tasks. For all TriPosT runs, we use the same hyperparameters (e.g., \(p=0.43\)). Total accuracy (_total_) is accuracy weighted based on the number of test samples. \({}^{\dagger}\) denotes that the task uses scripted rationalée/feedback. Results are averaged over three runs.

 

_harder_ than learning to directly generate a correct attempt, and 2) using LLM-generated feedbacks, which covers more error cases than a Python script, is effective in improving a model's performance.

## 4 Analysis

To investigate the factors that can influence how TriPosT-trained models learned to attempt self-improvement, we focus our analysis on the Multistep Arithmetic and Logical Deduction datatset. We also mainly study TriPosT with \(p=0.43\), which has both a measurable self-improvement contribution and improvement in its task performance (see Table 3 and Table 4)4.

Footnote 4: In practice, we implement \(p\) by specifying the ratio of the number of “self-improvement samples vs. directly correct samples vs. gold samples”. For example, a ratio of \(1.5:1.0:1.0\) corresponds to \(p=0.43\).

### Ablation Studies

We perform ablation studies for each of the three stages in TriPosT to better understand their contribution to model's overall performance. In Table 6, we report the task accuracy when: interaction between \(M_{\theta}\) and LLM is removed, so that \(M_{\theta}\) is distilled with purely LLM demonstrations (_-interaction_); data filtering is removed (_-filtering_); dataset balancing is changed to using its own performance (+_auto-balance_); and the weights for SL are changed to be the same for all tokens (_-weighed SL_). We find that all components are important for TriPosT to work well, and the choice of fixing \(p\) presents a trade-off between a model's self-improvement ability and its task performance (notibly, both TriPosT and TriPosT-auto improve upon the baselines).

### Proportion of SI. Training Data

In Table 7, we investigate how much improvement demonstration (\(x_{\mathrm{imp}}\)) is needed to elicit a measurable self-improvement contribution from \(M_{\theta}\). We find that when a large proportion (e.g. \(p=0.70\)) of the training data contains \(x_{\mathrm{imp}}\), the model often _attempts_ to self-improve but does not always result in an overall better performance. This is because many of the "improvement" attempts result in failures (e.g. changing an already correct attempt to become an incorrect one), and the best performance is achieved typically when \(p\) is low. Despite this, we find that for all other cases with \(p\leq 0.43\), TriPosT-trained model achieved a better performance than the baseline methods (see Table 4).

### Number of TriPosT Iterations

In most of our experiments, we trained TriPosT up to \(t=3\) iterations. This is because we found that LLMs and our Python scripts start to struggle with generating feedback or improving \(M_{\theta}\) attempts after three iterations. In Figure 3, we present

\begin{table}
\begin{tabular}{l c c c c} \hline \hline \multirow{2}{*}{Method} & \multicolumn{2}{c}{Multistep Arithmetic} & \multicolumn{2}{c}{Logical Deduction} \\ \cline{2-5}  & SI. Contrib. & Total Acc. & SI. Contrib. & Total Acc. \\ \hline TriPosT & **1.67** & 22.50 & **8.86** & 48.42 \\ -interaction & 0.28 & 11.67 & 0.00 & 41.67 \\ -filtering & 0.33 & 20.67 & 7.59 & 48.27 \\ auto-balance & 0.00 & **24.33** & 0.63 & **56.96** \\ -weighted SL & 0.00 & 21.33 & 1.90 & 43.67 \\ \hline \end{tabular}
\end{table}
Table 6: TriPosT ablation studies.

\begin{table}
\begin{tabular}{l l c c c c c c c c c c c} \hline \hline \multirow{2}{*}{Method} & \multicolumn{2}{c}{Multistep Arithmetic1} & \multicolumn{4}{c}{Word Sorting†} & \multicolumn{4}{c}{Date Understanding} & \multicolumn{4}{c}{Logical Deduction} \\ \cline{2-11}  & SI. Freq & SI. Cont. & total & SI. Freq & SI. Cont. & total & SI. Freq & SI. Cont. & total & SI. Freq & SI. Cont. & total \\ \hline TriPosT(\(t=1\)) & 0.00 & 0.00 & 17.17 & 1.58 & 0.52 & 28.23 & 0.00 & 0.00 & 27.27 & 8.86 & 2.85 & 46.52 \\ TriPosT(\(t=2\)) & 1.33 & 1.11 & 20.67 & 2.90 & 0.52 & 29.55 & 1.94 & 0.65 & 32.47 & **29.72** & **11.39** & 45.25 \\ TriPosT(\(t=3\)) & **3.67** & **1.67** & 22.50 & **4.38** & **2.37** & 29.82 & **10.38** & **3.25** & 37.01 & 23.42 & 8.86 & 48.42 \\ \hline TriPost-autot(\(t=1\)) & 0.00 & 0.00 & 20.00 & 0.00 & 0.00 & 30.34 & 0.00 & 0.00 & 32.47 & 1.90 & 0.63 & 51.27 \\ TriPosT-autot(\(t=2\)) & 0.00 & 0.00 & 23.33 & 0.00 & 0.00 & 29.55 & 0.00 & 0.00 & 56.82 & 0.63 & 0.00 & 55.06 \\ TriPosT-autot(\(t=3\)) & 0.00 & 0.00 & **24.33** & 0.00 & 0.00 & **30.34** & 0.00 & 0.00 & **68.83** & 0.63 & 0.63 & **56.96** \\ \hline \hline \end{tabular}
\end{table}
Table 5: Overall performance of TriPosT without explicit re-balancing. TriPosT-auto uses the same training procedure as TriPosT, except that the proportion of \(x_{\mathrm{imp}}\) used for training is determined automatically using the model’s current task performance.

 

how the number of self-improving trajectories collected (\(x_{\rm imp}\), after filtering) changes as TriPosT iteration increases. We found that as \(M_{\theta}\) improves its performance over time, it 1) poses a greater challenge for our \(\mathrm{FBK}\) module to generate feedback and/or the \(\mathrm{IMP}\) module to generate improvement, and 2) generates fewer incorrect attempts for TriPosT to edit. This is especially impactful for Multistep Arithmetic, as our feedback scripts can only consider a fixed number of error types. This also shows that even LLMs can struggle at generating useful feedbacks or correct improvements, which supports our findings in Section 3.5 that learning to generate feedback and improvements may be harder than to directly generate a correct solution. Lastly, we note that TriPosT can, in principle, be applied as an online RL algorithm, where one does not restrict the input prompts to be a fixed set as in Section 3. We believe this could be beneficial to improve the model's performance and genearlization ability beyond TriPosT(\(t=3\)).

## 5 Related Work

Prompting LLMs to Self-ImproveRecently, many work Bai et al. (2022); Madaan et al. (2023) have discovered LLM's capability to self-improve by letting it revise its own answer after prompting it to generate feedbacks. Following these work, Yang et al. (2022); Peng et al. (2023a); Shinn et al. (2023); Schick et al. (2022); Yang et al. (2023) has utilized such a capability to improve LLM's performance on various tasks. For example, Yang et al. (2022) recursively prompts an LLM to generate a longer story, and Madaan et al. (2023) iteratively prompts an LLM to improve its answers on a wide range of tasks such as sentiment reversal and dialogue response generation. More generally, Yang et al. (2023) finds that LLMs can be prompted to act as an "optimization function", which can be used to automatically perform prompt engineering. Our work focuses on distilling the self-improvement ability of LLMs into a smaller model, which was initially not capable of self-improvement (Figure 1).

Training LMs to Self-ImproveBesides prompting methods, recent work also explored approaches to train a LM to self-improve. LMSI Huang et al. (2023) trains LMs (e.g., PaLM-540B) with self-generated, self-consistent answers to improve their task performance, yet we found such method ineffective for small LMs. Many work such as Paul et al. (2023); Welleck et al. (2022); Madaan et al. (2021); Yasunaga and Liang (2020); Du et al. (2022) considered using multiple small LMs to generate feedback and improvement, which also relates to model ensemble methods Dietterich (2000). For example, Welleck et al. (2022) trains a "corrector" to improve answers generated by a given fixed generator. This method gathers improved attempts by sampling from the generator and pairing high-scoring attempts with low-scoring ones. It also does not provide reasonings (e.g., feedbacks) for each improvement. Paul et al. (2023) first trains a feedback model by using a set of predefined rules that perturbs an original solution, and then trains a separate model to generate answers conditioned on the feedback. Our work leverages LLMs to train a single model capable of generating both feedback and improvement, and also does not require any predefined rules (e.g., using LLMs as the \(\mathrm{FBK}\) module). Saunders et al. (2022); Ye et al. (2023) has attempted to equip a single small model to self-improve by training on LLM demonstrations, but found that it had little to no effect for small models on math/reasoning tasks. Our work presents analyses of how these previous methods can fail, and proposes TriPosT that can train a small model to self-improve and achieve better task performance.

Knowledge DistillationLearning from experts' demonstrations or reasoning (e.g., from GPT-4) has shown to be successful at improving the performance of smaller models in various tasks Mukherjee et al. (2023); Laskin et al. (2022); Peng et al. (2023b); Ho et al. (2023); Ye et al. (2023); Huang et al. (2023); Jung et al. (2024). Distillation methods Hinton et al. (2015); Ba and Caruana (2014) generally train a target model using expert demonstrations unaware of the target model's capability. While Tri

Figure 3: Improvement demonstrations become more difficult to collect as TriPosT iteration increases.

 

PosT also use LLMs to demonstrate generating a feedback or an improvement, these demonstrations are always conditioned on the output of the smaller model. In this view, our approach combines merits from reinforcement learning with knowledge distillation techniques, where small models are distilled with demonstrations that are created by its own exploration augmented by LLMs' supervision.

## 6 Conclusion

We introduce TriPosT, a training algorithm that distills the ability to self-improve to a small model and help it achieve better task performance. TriPosT first creates improving trajectories using interactions between a smaller model and an LLM, then post-process the collected trajectories, and finally train the smaller model to self-improve using weighted SL. We evaluated TriPosT on four math and reasoning tasks from the Big-Bench Hard collection and found that it can help small models achieve better task performance. In our analysis, we find that 1) the interactive process of learning from and correcting its _own_ mistakes is crucial for small models to learn to self-improve and 2) learning to always generate a useful feedback and a corresponding improvement can be much harder than learning to directly generate a correct answer. These findings suggest that other data formats, beyond the traditional (input, answer) pair, could be better suited for training a language model to solve a downstream task. We believe this also opens new possibilities for future work to leverage LLMs to improve the performance of smaller, faster models.

## 7 Limitations

Model SizesIn all of our experiments, we used a single A100 and mainly tested TriPosT on 7B models, the smallest in the LLaMA-1 and LLaMA-2 family Touvron et al. (2023, 2023). However, with the recently introduced flash attention technique Dao et al. (2022); Dao (2023) which can be used to reduce memory usage during training, we plan to extend our experiments to use models with more than 7B parameters.

DatasetsWe focused our experiments on math and reasoning tasks because 1) prior work Ye et al. (2023) had found it difficult to train a 7-13B to self-improve on those tasks and 2) measuring performance improvement is more well defined (for example, as compared to creative story writing). However, we note that as TriPosT is task agnostic, in theory it can be applied to other tasks such as knowledge-grounded dialogue generation Yoshino et al. (2023) or dialogue safety Dinan et al. (2019). We intend to leave this for future work.

LLM UsageWhile attempts for some tasks can be parsed and evaluated using a Python script (e.g., multistep arithmetic and word sorting), it quickly becomes unmanageable for tasks where reasonings mostly take the form of free text (e.g., date understanding and logical deduction). Therefore, we use LLMs such as GPT-3 and Codex (and ChatGPT, see Appendix F), which are highly performant at a reasonable cost. Specifically, we mainly use text-davinci-003 as the feedback module and Codex as the improvement module, as we found this to be the most cost-performant configuration in our experiments.

However, since the ability of LLMs to generate feedback or improvements is _crucial_ for TriPosT to collect training data, this presents a trade-off between the cost of using more performant LLMs (e.g., GPT-4) and the training outcome of TriPosT, for example on harder tasks such as GSM8k Cobbe et al. (2021). We hope that with advances in making LLMs more available Zhang et al. (2022), such a trade-off would diminish.

## 8 Ethical Considerations

Our work describes an algorithm to improve small models' performance on math and reasoning tasks, by distilling them the ability to self-improve using interaction records with LLMs. Generally, while most algorithms are not designed for unethical usage, there is often potential for abuse in their applications. In our experiments, we apply TriPosT to four math and reasoning tasks from the Big-Bench Hard collection Suzgun et al. (2022). However, because training algorithms are typically task-agnostic, it is possible to use them for unethical tasks, such as scamming and generating harmful responses Welbl et al. (2021); Gehman et al. (2020). We do not condone the use of TriPosT for any unlawful or morally unjust purposes.

## References

* Andrychowicz et al. (2018) Marcin Andrychowicz, Filip Wolski, Alex Ray, Jonas Schneider, Rachel Fong, Peter Welinder, Bob McGrew, Josh Tobin, Pieter Abbeel, and Wojciech Zaremba. 2018. Hindsight experience replay 

* Ba and Caruana (2014) Lei Jimmy Ba and Rich Caruana. 2014. Do deep nets really need to be deep?
* Bai et al. (2021) Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, Carol Chen, Catherine Olsson, Christopher Olah, Danny Hernandez, Dawn Drain, Deep Ganguli, Dustin Li, Eli Tran-Johnson, Ethan Perez, Jamie Kerr, Jared Mueller, Jeffrey Ladish, Joshua Landau, Kamal Ndousse, Kamille Lukosite, Liane Lovitt, Michael Sellitto, Nelson Elhage, Nicholas Schiefer, Noemi Mercado, Nova DasSarma, Robert Lasenby, Robin Larson, Sam Ringer, Scott Johnston, Shauna Kravec, Sheer El Shbow, Stanislav Fort, Tamera Lanham, Timothy Teles-Lawton, Tom Conferly, Tom Henighan, Tristan Hume, Samuel R. Bowman, Zac Hatfield-Dodds, Ben Mann, Dario Amodei, Nicholas Joseph, Sam McCandlish, Tom Brown, and Jared Kaplan. 2021. Constitutional ai: Harmlessness from ai feedback.
* Brown et al. (2020) Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ranesch, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesese, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. In _Advances in Neural Information Processing Systems_, volume 33, pages 1877-1901. Curran Associates, Inc.
* Chen et al. (2019) Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Pande de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fotiios Chantzis, Elizabeth Barnes, Ariel Herbert-Voss, William Hebgen Guess, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Sucihr Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Josh Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katic Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojciech Zaemba. 2021. Evaluating large language models trained on code.
* Chowdhery et al. (2022) Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vinodkumar Prabhakaran, Emily Reif, Nan Du, Ben Hutchinson, Reiner Pope, James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Levskaya, Sanjay Ghemawat, Sunpia Dev, Henry Michaewski, Xavier Garcia, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito, David Luan, Heyoutae Lim, Barret Zoph, Alexander Spiridonov, Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omernick, Andrew M. Dai, Thuanmalayan Sankaranarayana Pilali, Marie Pellart, Attor Lewkowycz, Eric Moreira, Rewon Child, Oleksander Polozov, Katherine Lee, Zongwei Zhou, Xuezhi Wang, Brennan Saeta, Mark Diaz, Orhan Firat, Michele Catasta, Jason Wei, Kathy Meier-Hellstern, Douglas Eck, Jeff Dean, Slav Petrov, and Noah Fiedel. 2022. Palm: Scaling language modeling with pathways.
* Cobbe et al. (2021) Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reitichiro Nakano, Christopher Hesse, and John Schulman. 2021. Training verifiers to solve math word problems. _arXiv preprint arXiv:2110.14168_.
* Dao (2023) Tri Dao. 2023. FlashAttention-2: Faster attention with better parallelism and work partitioning. _ArXiv_.
* Dao et al. (2022) Tri Dao, Daniel Y. Fu, Stefano Ermon, Attri Rudra, and Christopher Re. 2022. FlashAttention: Fast and memory-efficient exact attention with IO-awareness. In _Advances in Neural Information Processing Systems_.
* Dietterich (2000) Thomas G. Dietterich. 2000. Ensemble methods in machine learning. In _International Workshop on Multiple Classifier Systems_.
* Dinan et al. (2019) Emily Dinan, Samuel Humeau, Bharath Chitagunita, and Jason Weston. 2019. Build it break it fix it for dialogue safety: Robustness from adversarial human attack.
* Du et al. (2022) Wanyu Du, Zhe Myung Kim, Vipul Raheja, Dhruv Kumar, and Dongyeop Kang. 2022. Read, revise, repeat: A system demonstration for human-in-the-loop iterative text revision. In _Proceedings of the First Workshop on Intelligent and Interactive Writing Assistants (In2Writing 2022)_, pages 96-108, Dublin, Ireland. Association for Computational Linguistics.
* Gehman et al. (2020) Samuel Gehman, Suchin Gururangan, Maarten Sap, Yejin Choi, and Noah A. Smith. 2020. RealToxicityPrompts: Evaluating neural toxic degeneration in language models. In _Findings of the Association for Computational Linguistics: EMNLP 2020_, pages 3356-3369, Online. Association for Computational Linguistics.
* He et al. (2019) Zhiyuan He, Danchen Lin, Thomas Lau, and Mike Wu. 2019. Gradient boosting machine: A survey.
* Hinton et al. (2015) Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. 2015. Distilling the knowledge in a neural network.
* Ho et al. (2023) Namgyu Ho, Laura Schmid, and Se-Young Yun. 2023. Large language models are reasoning teachers 

Jiaxin Huang, Shixiang Gu, Le Hou, Yuexin Wu, Xuezhi Wang, Hongkun Yu, and Jiawei Han. 2023. Large language models can self-improve. In _Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing_, pages 1051-1068, Singapore. Association for Computational Linguistics.
* Jung et al. (2024) Jaehun Jung, Peter West, Liwei Jiang, Faeze Brahman, Xinming Lu, Jillian Fisher, Taylor Sorensen, and Yejin Choi. 2024. Impossible distillation: from low-quality model to high-quality dataset & model for summarization and paraphrasing.
* Kanai et al. (2023) Sekitoshi Kanai, Shin'ya Yamaguchi, Masanori Yamada, Hiroshi Takahashi, Kentaro Ohno, and Yasutoshi Ida. 2023. One-vs-the-rest loss to focus on important samples in adversarial training.
* Katharopoulos and Fleuret (2019) Angelos Katharopoulos and Francois Fleuret. 2019. Not all samples are created equal: Deep learning with importance sampling.
* Kojima et al. (2023) Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. 2023. Large language models are zero-shot reasoners.
* Laskin et al. (2022) Michael Laskin, Luyu Wang, Junhyuk Oh, Emilio Parisotto, Stephen Spencer, Richie Steigerwald, DJ Strouse, Steven Hansen, Angelos Filos, Ethan Brooks, Maxime Gazeau, Himanshu Sahni, Satinder Singh, and Volodymyr Mnih. 2022. In-context reinforcement learning with algorithm distillation.
* Lightman et al. (2023) Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. 2023. Let's verify step by step.
* Loshchilov and Hutter (2019) Ilya Loshchilov and Frank Hutter. 2019. Decoupled weight decay regularization.
* Madaan et al. (2023) Aman Mahaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Utri Alono, Nouhan Dziri, Shrimi Prabhumoye, Yiming Yang, Shahsank Gupta, Bodhisattwa Prasad Majumder, Katherine Hermann, Sean Welleck, Amir Yadambakhsh, and Peter Clark. 2023. Self-refine: Iterative refinement with self-feedback.
* Madaan et al. (2021) Aman Madaan, Niket Tandon, Dheeraj Rajagopal, Peter Clark, Yiming Yang, and Eduard Hovy. 2021. Think about it! improving defeasible reasoning by first modeling the question scenario.
* Mukherjee et al. (2023) Subhabrata Mukherjee, Arindam Mitra, Ganesh Jawahar, Sahaij Agarwal, Hamid Palangi, and Ahmed Awadallah. 2023. Orca: Progressive learning from complex explanation traces of gpt-4.
* OpenAI (2022) OpenAI. 2022. OpenAI: Introducing ChatGPT.
* OpenAI (2023) OpenAI. 2023. GPT-4 technical report.
* Ouyang et al. (2022) Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. 2022. Training language models to follow instructions with human feedback.
* Paul et al. (2023) Debjit Paul, Mete Ismayilkada, Maxime Peyrard, Beatriz Borges, Antoine Bosselut, Robert West, and Boi Faltings. 2023. Refiner: Reasoning feedback on intermediate representations.
* Peng et al. (2022) Baolin Peng, Michel Galley, Pengcheng He, Hao Cheng, Yujia Xie, Yu Hu, Qiuyuan Huang, Lars Liden, Zhou Yu, Weizhu Chen, and Jianfeng Gao. 2023a. Check your facts and try again: Improving large language models with external knowledge and automated feedback.
* Peng et al. (2023b) Baolin Peng, Chunyuan Li, Pengcheng He, Michel Galley, and Jianfeng Gao. 2023b. Instruction tuning with gpt-4.
* Rasley et al. (2020) Jeff Rasley, Samyam Rajbhandari, Olatunji Ruwase, and Yuxiong He. 2020. Deepspeed: System optimizations enable training deep learning models with over 100 billion parameters. In _Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining_, KDD '20, page 3505-3506, New York, NY, USA. Association for Computing Machinery.
* Saunders et al. (2022) William Saunders, Catherine Yeh, Jeff Wu, Steven Bills, Long Ouyang, Jonathan Ward, and Jan Leike. 2022. Self-critiquing models for assisting human evaluators.
* Volume 2_, IJCAI'99, page 1401-1406, San Francisco, CA, USA. Morgan Kaufmann Publishers Inc.
* Schaul et al. (2016) Tom Schaul, John Quan, Ioannis Antonoglou, and David Silver. 2016. Prioritized experience replay.
* Schick et al. (2022) Timo Schick, Jane Dwivedi-Yu, Zhengbao Jiang, Fabio Petroni, Patrick Lewis, Gauter Izacard, Qingfei You, Christoforos Nalmapantis, Edouard Grave, and Sebastian Riedel. 2022. Peer: A collaborative language model.
* Shinn et al. (2023) Noah Shinn, Federico Cassano, Beck Labash, Ashwin Gopinath, Karthik Narasimhan, and Shuryu Yao. 2023. Reflexion: Language agents with verbal reinforcement learning.
* Srivastava et al. (2023) Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abuwal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R. Brown, Adam Santoro, Aditya Gupta, Adria Garriga-Alonso, and et al. 2023. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models.
* Suzgun et al. (2020) Mirac Suzgun, Nathan Scales, Nathanaael Scharli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc V. Le, Ed H. Chi, 

Denny Zhou, and Jason Wei. 2022. Challenging big-bench tasks and whether chain-of-thought can solve them.
* Taori et al. (2023) Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. 2023. Stanford alpaca: An instruction-following llama model. [https://github.com/tatsu-lab/stanford_alpaca](https://github.com/tatsu-lab/stanford_alpaca).
* Tourron et al. (2023) Hugo Tourron, Thibaut Lavril, Gautier Lacard, Xavier Martinet, Marie-Anne Lachaux, Timothee Lacroix, Baptiste Roziere, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023a. Llama: Open and efficient foundation language models.
* Touvron et al. (2021) Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajiwal Bhargava, Shruti Bhosale, Dan Bilkel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esibou, Jude Fernandes, Jeremy Fu, Wenyu Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inanc, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isshel Kloumann, Artem Korenev, Punit Singh Kouara, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molbyog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Runga, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Purxin Xu, Zheng Yan, Hiyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialoom. 2023b. Llama 2: Open foundation and fine-tuned chat models.
* Wang et al. (2022) Qizhou Wang, Feng Liu, Bo Han, Tongliang Liu, Chen Gong, Gang Niu, Mingyuan Zhou, and Masashi Sugiyama. 2022. Probabilistic margins for instance reweighting in adversarial training.
* Wang et al. (2020) Yusen Wang, Difan Zou, Jinfeng Yi, James Bailey, Xingjun Ma, and Quanquan Gu. 2020. Improving adversarial robustness requires revisiting misclassified examples. In _International Conference on Learning Representations_.
* Wei et al. (2023) Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Bian Ichter, Fei Xia, Ed Chi, Quoc Le, and Deny Zhou. 2023. Chain-of-thought prompting elicits reasoning in large language models.
* Welbl et al. (2021) Johannes Welbl, Amelia Glaese, Jonathan Uesato, Sumanath Dathathri, John Mellor, Lisa Anne Hendricks, Kirsty Anderson, Pushmeet Kohli, Ben Coppin, and Po-Sen Huang. 2021. Challenges in detoxifying language models. In _Findings of the Association for Computational Linguistics: EMNLP 2021_, pages 2447-2469, Punta Cana, Dominican Republic. Association for Computational Linguistics.
* Welleck et al. (2022) Sean Welleck, Kiming Lu, Peter West, Faeze Brahman, Tianxiao Shen, Daniel Khashabi, and Yejin Choi. 2022. Generating sequences by learning to self-correct.
* Yang et al. (2023) Chengrun Yang, Xuezhi Wang, Yifeng Lu, Hanxiao Liu, Quoc V. Le, Denny Zhou, and Xinyun Chen. 2023. Large language models as optimizers.
* Yang et al. (2022) Kevin Yang, Yuandong Tian, Nanyun Peng, and Dan Klein. 2022. Re3: Generating longer stories with recursive reprompting and revision.
* Yasunaga and Liang (2020) Michihiro Yasunaga and Percy Liang. 2020. Graph-based, self-supervised program repair from diagnostic feedback.
* Ye et al. (2023) Seonghyeon Ye, Yongrae Jo, Doyoung Kim, Sungdong Kim, Hyeonbin Hwang, and Minjoon Seo. 2023. Selfee: Iterative self-revising llm empowered by self-feedback generation. Blog post.
* Yoshino et al. (2023) Koichiro Yoshino, Yun-Nung Chen, Paul Crook, Satwik Kottur, Jinchao Li, Behnam Hedayatnia, Seungwhan Moon, Zhenceng Fei, Zekang Li, Jinchao Zhang, Yang Feng, Jee Zhou, Seokhwan Kim, Yang Liu, Di Jin, Alexandros Papangelis, Karthik Gopalakrishnan, Dilek Hakkani-Tur, Babak Damavandi, Alborz Geramifard, Chiori Hori, Ankit Shah, Chen Zhang, Haizhoul Li, Joao Sedoc, Luis F. D'Haro, Rafael Banchs, and Alexander Rudnicky. 2023. Overview of the tenth dialog system technology challenge: DistC10. _IEEE/ACM Transactions on Audio, Speech, and Language Processing_, pages 1-14.
* Zhang et al. (2022a) Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, Todor Mihalyo, Myle Ott, Sam Sheifer, Kurt Shuster, Daniel Simig, Punit Singh Koruz, Anjal Sridhar, Tianlu Wang, and Luke Zettlemoyger. 2022a. Opt: Open pre-trained transformer language models.
* Zhang et al. (2023) Tianjun Zhang, Fangchen Liu, Justin Wong, Pieter Abbeel, and Joseph E. Gonzalez. 2023. The wisdom of hindsight makes language models better instruction followers.
* Zhang et al. (2022b) Zhisong Zhang, Emma Strubell, and Eduard Hovy. 2022b. A survey of active learning for natural language processing. In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_, pages 6166-6190, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
* Zheng et al. (2023) Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric. P Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. 2023. Judging llvm-as-a-judge with mt-bench and chatbot arena.

 

More Details on Datasets and Preprocessing

We use four tasks from the Big-Bench Hard collection (Suzgun et al., 2022) for our experiments: _multistep arithmetic_, _word sorting_, _date understanding_, and _logical deduction_. Since these tasks do not provide ground truth step-by-step rationale, we either generate them using a script (for _multistep arithmetic_ and _word sorting_), or prompt Codex (Chen et al., 2021) in a few-shot setting using examples from Suzgun et al. (2022). For rationales generated using prompting, we only keep the ones that reached the correct answer and passed a simple consistency check (e.g. for multiple choice questions, we ensure that the final selected choice in the last step appeared in the second last step). We provide example rationales used for each task in Table A8, Table A9, Table A10, and Table A11. Since Big-Bench (Srivastava et al., 2023) did not provide an official training/validation/test split, we generated our own splits with statistics shown in Table A11.

## Appendix B Analyzing Errors Made by Codex and LLaMA-7B

To detail the different type and amount of errors made by an LLM (e.g., Codex) and a smaller model (e.g., LLaMA-7B), we manually examine incorrect attempts generated by the two models in the Multistep Arithmetics dataset. We use Codex with few-shot prompting, and LLaMA-7B after supervised finetuning on ground-truth step-by-step solutions (denoted as _LLaMA+ft_). We randomly sample 50 generated attempts with incorrect answers, and carefully review each step in those attempts. For each incorrect step, we apply the principle of error-carried-forward and categorize the first error encountered according to Table A2.

We present our analysis in Figure A1 and Table A3. Figure A1 shows that calculation errors take up more than 50% of the time for both Codex and the finetuned LLaMA-7B. However, Codex also makes many algebraic errors (such as forgetting to change sign after adding brackets), while LLaMA-7B often hallucinates by adding or deleting terms from previous calculations. Furthermore, Table A3 shows that, compared to the fine-tuned LLaMA-7B, Codex generates longer solutions while producing fewer errors per step. These findings suggest that supervised finetuning a smaller LM (e.g., LLaMA-7B) based on correcting LLM-generated errors may be inefficient, as it forces the smaller model to learn from attempts and mistakes very different from its own (see Section 1 and Appendix C for more details).

 

loss to emphasize learning the improvement-related tokens (\(x_{i}^{\text{fb}}\) or \(x_{i+1}^{\mathrm{att}}\)) of each training sample. In Table A6, we find that using a weight too low (\(w=1.0\)) can result in the model rarely attempting to self-improve, while using a weight too high (\(w=3.0\)) does not result in better performance. We believe that this has a similar effect of adjusting \(p\) in Section 4.2: some incentive is needed for the model to learn to self-improve, while too much emphasis on trying to self-improve can result in a worse performance.

While we also experimented with alternatives such as masking easier tokens (\(x_{i}^{\mathrm{att}}\) in a single-step improvement triplet), we believe there is a rich set of techniques that can be used to train the model to focus on harder inputs. This includes boosting algorithms [7, 8], automatic loss reweighing methods [59, 10, 11], as well as importance-sampling based methods [10]. We leave this for future work as it is orthogonal to our main contributions.

## Appendix F Prompting Details

Besides prompting to generate rationales (e.g. for _date understanding_), we also use prompting to gen 

erate feedbacks and improvements given the initial attempt. For scriptable tasks such as _multistep arithmetic_ and _word sorting_, we use a script to generate the feedback by first parsing each step in the attempt, and check their correctness/consistency with other steps using a set of predefined rules. This is similar to Welleck et al. (2022), but we also generalize this to unscriptable tasks such as _date understanding_ and _logical deduction_ by few-shot prompting GPT-3 (text-davinci-003) Brown et al. (2020) and Codex Chen et al. (2021) to generate feedbacks and improvements. We found that being able to generate useful feedback is critical for gathering successful improvement trajectories, and we discovered that ChatGPT OpenAI (2022) is less effective than GPT-3 or Codex in our case. We provide examples of the feedbacks generated for each task in Table A12, and the prompts used to generate feedback or improvements in Table A13, Table A14, Table A15, and Table A16. Note that we used a form-type of prompting for generating feedback because it can more easily ensure that our (formatted) feedback will contain all the elements we need.

When an answer is correct, we manually attach the phrase "Step 1 to step x is correct, and the final response is also correct." as the termination feedback, where "x" is the last step number. This termination condition is also used during inference.

## Appendix G More Details on Baselines

LmsiHuang et al. (2023) proposed LMSI, a method to improve PaLM-540B Chowdhery et al. (2022) on math and reasoning tasks by training it on self-generated and consistent step-by-step rationales. First, LMSI generates multiple step-by-step solutions using a high temperature (\(\tau=1.2\)). Then, LMSI only keeps the answers that are self-consistent (by majority voting) in the final answer. Finally, LMSI further augments these solutions with mixed formats, such as removing all the intermediate steps and only keep the final answer. To be comparable with other methods in Table 3 that have access to the ground truth answer, we modify the second step to only keep the answers that are correct. In addition, since small models such as LLaMA-7B performed poorly in these tasks without fine-tuning, we perform LMSI after training the model on the collected silver step-by-step solutions in Appendix A.

If SI demoFollowing Ye et al. (2023), _ft. SI demo_ finetunes a model on LLM-generated self-improvement demonstrations. For all tasks, we experimented with LLMs \(\in\{\text{ChatGPT},\text{Codex}\}\) and reported one with better performance (often Codex). In details, we first prompt a LLM (e.g. Codex) to generate an initial attempt, and then re-used TriPosT with the same LLM as the FBK and IMP to generate a feedback and an improvement. For a fair comparison in Table 3, we also balanced 

the collected data using the same \(p=0.43\) as with TriPosT. Finally, train the small LM using (unweighted) SL on the collected data.

## Appendix H Running LMSI(\(t>1\))

LMSI described in Huang et al. (2023) was not applied as an iterative algorithm. However, since LMSI training only relies on self-generated and self-consistent answers, it can be _ran iteratively_ similar to TriPosT. We present this comparison in Table A7, and find that LMSI(\(t\geq 1\)) struggles when the base model (_ft rationale_) has a weak task performance. We believe this is because LMSI is mainly a self-training algorithm designed for LLMs such as PaLM-540B Chowdhery et al. (2022), which can often generate correct or near-correct solutions. However, TriPosT is a training algorithm designed for smaller LMs, where models learns to self-improve from its interaction records with expert LLMs.

## Appendix I Implementation Details

We combine techniques from prompting-based self-improvement Madaan et al. (2023); Bai et al. (2022) and active learning Zhang et al. (2022b); Lightman et al. (2023) to collect a set of self-improving trajectories. Specifically, we first either use a script or few-shot prompting (see Appendix F for more details) to gather _feedbacks_ on a given attempt, and then use prompting to generate _improvements_ conditioned on the previous attempt, the feedback, and all the steps in the previous attempt before the first error step (see Tables A13 to A16 for example). This is to ensure that the improved attempt is making modifications on the previous attempt, rather than creating an entirely new attempt.

To edit the original attempt given the script/LLM-generated feedback, we 1) find the first \(x^{\text{fb*}}_{i}\) feedback that differs from the \(M_{\theta}\)-generated feedback \(x^{\text{fb}}_{i}\) (usually \(i=1\)); 2) replace \(x^{\text{fb*}}_{i}\) with \(x^{\text{fb}}_{i}\); 3) remove all the attempts, feedback, and improvement after after \(x^{\text{fb}}_{i}\) from the trajectory. After this, we prompt an LLM in the improvement module \(\operatorname{IMP}\) to generate an improvement as described above and in Appendix F.

To filter out some of the unhelpful feedbacks or incorrectly "improved" attempts, we mainly check 1) whether the final attempt reached the correct answer; 2) if there is at least one difference between the previous attempt and the improved attempt; and 3) if the final answer is consistent with the second last step. We only keep the data that have passed all checks. The effect of this filtering is discussed in our ablation studies in Section 4.1.

## Appendix J Model/Training hyperparameters

In our main experiments, we used a single A100 GPU with DeepSpeed Rasley et al. (2020) Zero-2 optimization. We used AdamW Loshchilov and Hutter (2019) as the optimizer. For each iteration of the TriPosT algorithm, we train the model for 4 epochs (line 17). We use a linear learning rate schedule with 20% of warmup steps and a peak learning rate of 1e-6. We use a maximum sequence length of 1024 tokens, batch size = 1 and gradient accumulation step = 4. On average, three iterations of TriPosT take about 12 hours to train.

 



**Example Question and Generated Rationale for Multistep Arithmetic**

\begin{tabular}{l} _Q:__((7 - 1 + -1 + -4) - (-7 + 7 *3 *.9)) =_ \\ _Answer: Let's think step by step. Recall that the order of operations in mathematics is as follows:_ \\ _(1) Parentheses, (2) exponents, (3) multiplication and division (from left to right), (4) addition_ \\ _and multiplication (from left to right). So, remember to always compute the expressions inside parentheses or brackets first._ \\ _parentheses or brackets first._ \\ _(1) This equation can be written as "(A - B)", where A = (7 - -1 + -1 + -4) and B = (-7 + 7 * 3 * -9)._ \\ _(2) Let's calculate A = (7 - -1 + -1 + -4) = (((7 -1) + -1) + -4) = ((8 + -1) + -4) = (7 + -4) = 3._ \\ _(3) Let's calculate B = (-7 + 7 * 3 * -9) = (-7 + ((7 * 3) * -9)) = (-7 + (21 * -9)) = (-7 + -189) = -196._ \\ _(4) Then, the final equation is (A - B) = (3 - -196) = 199._ \\ _(Final response) So the answer is 199._ \\ \end{tabular} Table A8: Examples of generated rationales for Multistep Arithmetic using Python scripts. Question and prompts are written in _gray italics_, and the generated step-by-step rationale is shown in black.

\begin{table}
\begin{tabular}{l} _Q:__Sort the following words alphabetically:_ \\ _List: sioux fortescue purloin percept helmsman_ \\ _Attempt: Let’s think step by step._ \\ (1) The first letter: "sioux"=“s”, “fortscue”=‘f” (6), “purloin”=“p” (16), “percept”=‘p” (16), “helmsman”=“h” (8) \\ (2) We now have: (6) “fortescue” < (8) “helmsman” < (16) [“purloin” ? “percept”] < (19) “sioux”. \\ (3) Now sort this subpart [“purloin” ? ”percept”] by looking at their second letters: “purloin”=“u” (21), “percept”=“e” (5). \\ (3.1) We now have: (5) “percept” < (21) “purloin”. \\ (3.2) Hence, we have “percept” < “purloin”. \\ (4) Hence, we have “fortescue” < “helmsman” < “percept” < “purloin” < “sioux”. \\ (Final response) So the answer is: fortescue helmsman percept puloin sioux. \\ \end{tabular}
\end{table}
Table A9: Examples of generated rationales for Word Sorting using Python scripts. Question and prompts are written in _gray italics_, and the generated step-by-step rationale is shown in black.





**Example Question and Generated Rationale for Logical Deduction**

_Q: The following paragraphs each describe a set of three objects arranged in a fixed order. The statements are logically consistent within each paragraph. On a shelf, there are three books: a white book, a green book, and an orange book. The green book is to the right of the white book. The orange book is the rightmost. Options: (A) The white book is the leftmost. (B) The green book is the leftmost. (C) The orange book is the leftmost. Attempt: Let's think step by step. Let "??" represent 0 or more objects, and "?"_

_represent exactly 1 object._

**(1) The green book is to the right of the white book: "(left) ?? white ?? green ?? (right)". (2) The orange book is the rightmost: "(left) ?? orange (right)". (3) There are in total three books: a white book, a green book, and an orange book. (4) Combining (1) and (2) we get the following ordering: "(left) ?? white ?? green ?? orange (right)". (5) Combining (3) and (4) we get the following ordering: "(left) white green orange (right)". (6) According to this ordering, the leftmost book is the white book. (Final response) So the answer is (A).





\begin{table}
\begin{tabular}{l l} \hline Task & Example \\ \hline Multistep Arithmetic1 [FOOTNOTE:1]Footnote 1: The first letter: “cole”=“c” (3), ”estonia"=”e” (5), ..., ”codea”=“c” (3).
2 We now have: (2) ”brian’ (<) (3) [“cole” ?



\begin{table}
\begin{tabular}{l l} Prompt Type & Prompt Example \\ Get Improvement & Q: (((-9 * 3 + -6) * (-2 + 7 * -6)) - ((-4 - -6 + 6) * (4 * -4 + 8))) = \\ Answer: Let’s think step by step. Recall that the order of operations in mathematics is as follows: (1) Parentheses, (2) exponents, (3) multiplication and division (from left to right), (4) addition and multiplication (from left to right). So, remember to always compute the expressions inside parentheses or brackets first. (1) This equation can be written as ”((A * B) - (C * D))”, where A = (-9 * 3 + -6), B = (-2 + 7 * -6), C = (-4 - -6 + 6) and D = (4 * -4 + 8). (2) Let’s calculate A = (-9 * 3 + -6) = ((-9 * 3) + -6) = (-27 + -6) = -33. (3) Let’s calculate B = (-2 + 7 * -6) = (-2 + (7 * -6)) = (-2 + -42) = -44. (4) Let’s calculate C = (-4 - -6 + 6) = ((-4 -6) + 6) = (2 + 6) = 8. (5) Let’s calculate D = (4 * -4 + 8) = ((4 * -4) + 8) = (-16 + 8) = -8. (6) Then, the final equation is ((A * B) - (C * D)) = ((-33 * -44) - (8 * -8)) = (1424 - (8 * -8)) = (1424 - 64) = 1360. (Final response) So the answer is 1360.  Feedback: Step (1) to step (5) are correct. In step (6) the part “((-33 * -44) - (8 * -8)) = (1424 - (8 * -8))” is incorrect. This is because there is a calculation error, since (-33 * -44) should be 1452. Updated Answer: Let’s think step by step. Recall that the order of operations in mathematics is as follows: (1) Parentheses, (2) exponents, (3) multiplication and division (from left to right), (4) addition and multiplication (from left to right). So, remember to always compute the expressions inside parentheses or brackets first. (1) This equation can be written as ”((A * B) - (C * D))”, where A = (-9 * 3 + -6), B = (-2 + 7 * -6), C = (-4 -6 + 6) and D = (4 * -4 + 8). (2) Let’s calculate A = (-9 * 3 + -6) = (-9 * 3) + -6) = (-27 + -6) = -33. (3) Let's calculate B = (-2 + 7 * -6) = (-2 + (7 * -6)) = (-2 + -42) = -44. (4) Let’s calculate C = (-4 -6 + 6) = (-4 -6) + 6) = (2 + 6) = 8. (5) Let’s calculate D = (4 * -4 + 8) = ((4 * -4) + 8) = (-16 + 8) = -8. (6) Then, the final equation is ((A * B) - (C * D)) = ((-33 * -44) - (8 * -8)) = (1452 - (8 * -8)) = (1452 - -64) = 1516. (Final response) So the answer is 1516. [END] \\  & & \\ \end{tabular}
\end{table}
Table A13: Prompt used for generating improvement for Multistep Arithmetic. The generated feedback is then formatted as “Step 1 to step {earliest error step - 1} is correct. In step {earliest error step} the part ‘{error segment}’ is incorrect. This is because ’{error reason}’.” In general, we used three-shot prompting. Parts that will be generated are highlighted in blue. **Due to limited space, we present one example used for each task.** Please refer to our code repository for the full prompt.





\begin{table}
\begin{tabular}{l l} Prompt Type & Prompt Example \\ Get Improvement & Q: Sort the following words alphabetically: \\ List: cosh inferno every foyer coterie mcfadden halifax \\ Answer: Let’s think step by step. \\  & (1) The first letter: “cosh”=“c” (3), ”inferno”=‘i” (9), “every”=“e” (5), “foyer”=“f” (6), \\ “cotetrie"=“c” (3), “mcfdaden”=“m” (13), “halifax”=“h” (8). \\  & (2) We now have: (3) “cosh” ? "cotorie" < (5) “every” < (6) “foyer” < (8) “halfix” < (9) “inferno” < (13) “mcfdden”. \\  & (3) Now sort this subpart [”cosh” ? "cotarie"] by looking at their second letters: \\ “cosh”=”o” (15), “cotericie”=“0” (15). \\
**(3.1)** We now have: (15) [“cosh” ? "coterie”]. \\
**(3.2)** Now sort this subpart [”cosh” ? "cotericie"] by looking at their third letters: \\ “cosh”=’’s” (19), “cotericie”=“c” (3). \\
**(3.3)** We now have: (3) “cotericie” < (19) “cosh”. \\
**(3.4)** Hence, we have “cotericie” + “cosh”. \\
**(4)** Hence, we have “cotericie” + “cosh” + “every” + “foyer” + “halfix” + “inferiorn” + “mcfadden”. \\ (Final response) So the answer is: cotericic osh every foyer halfiax inferno mcfadden. [END] \\ Feedback: Step (1) to (3.1) are correct. In step (3.2) the part ”cotericie”=“c” (3) ” is incorrect. \\ This is because the third letter of “cotericie” should be “t”. \\ Updated Answer: Let’s think step by step. \\ \hline (1) The first letter: “cosh”=“c” (3), ”inference”=“i” (9), “every”=“e” (5), “foyer”=“f” (6), \\ “cotericie”=“c” (3), “mcfadden”=“m” (13), “hailifax”=”h” (8). \\ (2) We now have: (3) [”cosh” ? "cotericie”] < (5) “every” < (6) “foyer” < (8) “halfix” < (9) ”inferno” < (13) ”mcfadden”. \\ (3) Now sort this subpart [”cosh” ? "cotericie"] by looking at their second letters: \\ “cosh”=”o” (15), “cotericie”=“0” (15). \\ (3.1) We now have: (15) [”cosh” ? "cotericie"]. \\ (3.2) Now sort this subpart [”cosh” ? "cotericie"] by looking at their third letters: \\ “cosh”=”s” (19), “cotericie”=“1” (20). \\ (3.3) We now have: (19) “cosh” ! (20) “cotericie”. \\ (3.4) Hence, we have “cosh” ! (21) Hence, we have “cosh” ! (22) Hence, we have “cosh” ! (23) Here, we have “cosh” ! (24) Hence, we have “cosh” ! (25) Note that “cosh” ! (26) Hence, we have “cosh” ! (27) Hence, we have “cosh” ! (28) Hence, we have “cosh” ! (29) “(20). \\ (3.3) We now have: (19) “cosh” ! (20) “cotericie”. \\ (3.4) Hence, we have “cosh” ! (21) Hence, we have “cosh” ! (22) Hence, we have “cosh” ! (23) Here, we have “cosh” ! (24) Hence, we have “cosh” ! (25) Note that “cosh” ! (26) Hence, we have “cosh” ! (27) Hence, we have “cosh” ! (28) Hence, we have “cosh” ! (29) “(20). \\ (3.3) We now have: (19) “cosh” ! (20) “cotericie”. \\ (3.4) Hence, we have “cosh” ! (27) Hence, we have “cosh” ! (28) Hence, we have “cosh” ! (29) “(20). \\ (3.3) We now have: (19) “cosh” ! (20) “cotericie”. \\ (3.4) Hence, we have “cosh” ! (27) Hence, we have “cosh” ! (28) Hence, we have “cosh” ! (29) “(20). \\ (3.3) Hence, we have “cosh” ! (29) “cosh” ! (28) Hence, we have “cosh” ! (29) “(20). \\ (3.4) Hence, we have “cosh” ! (28) Hence, we have “cosh” ! (29) “(20). \\ (3.3) We now have: (19) “cosh” ! (20) “cotericie”. \\ (3.4) Hence, we have “cosh” ! (28) Hence, we have “cosh” ! (29) “(20). \\ (3.4) Hence, we have “cosh” ! (29) “(20). \\ (4) Hence, we have “cosh” ! (29) “(20). \\ (4) Hence, we have “cosh” ! (29) “(20). \\ (4) Hence, we have “cosh” ! (29) “(20). \\ (4) Hence, we have “cosh” ! (29) “(20). \\ (4) Hence, we have “cosh” ! (29) “(20). \\ (4) Hence, we have “cosh” ! (29) “(20). \\ (4) Hence, we have “cosh” ! (29) “(20). \\ (4) Hence, we have “cosh” ! (29) “(20). \\ (4) Hence, we have “cosh” ! (29) “(20). \\ (4) Hence, we have “cosh ! (29) “(20). \\ (4) Hence, we have “cosh ! (29) “(20). \\ (4) Hence, we have “cosh ! (29) “(20). \\ (4) Hence, we have “cosh ! (29) “(20). \\ (4) Hence, we have “cosh ! (29) “(20). \\ (4) Hence, we have “cosh ! (29) “(20). \\ (4) Hence, we have “cosh ! (29) “(20). \\ (4) Hence, we have “cosh ! (29) “(20). \\ (4) Hence, we have “cosh ! (29) “(20). \\ (4) Hence, we have “cosh ! (29) “(20). \\ (4) Hence, we have “cosh ! (29) “(20). \\ (4) Hence, we have “cosh ! (29) “(20). \\ (4) Hence, we have “cosh ! (29) “(20). \\ (4) Hence, we have “cosh ! (29) “(20). \\ (4) Hence, we have “cosh ! (29) “(20). \\ (4) Hence, we have “cosh ! (29) “(20). \\ (4) Hence, we have “cosh ! (29) “(20). \\ (4) Hence, we have “cosh ! (29) “(20). \\ (4) Hence, we have “cosh ! (29) “(20). \\ (4) Hence, we have “cosh ! (29) “(20). \\ (4) Hence, we have “cosh ! (29) “(20). \\ (4) Hence, we have “cosh ! (29) “(20). \\ (4) Hence, we have “cosh ! (29) “(20). \\ (4) Hence, we have “cosh ! (29) “(20). \\ (4) Hence, we have “cosh ! (29) “(20). \\ (4) Hence, we have “cosh ! (29) “(20). \\ (4) Hence, we have “cosh ! (29) “(20). \\ (4) Hence, we have “cosh ! (29) “(20). \\ (4) Hence, we have “cosh ! (29) “(20). \\ (4) Hence, we have “cosh ! (29) “(20). \\ (4) Hence, we have “cosh ! (29) “(20). \\ (4) Hence, we have “cosh ! (29) “(20). \\ (4) Hence, we have “cosh ! (29) “(20). \\ (4) Hence, we have “cosh ! (29) “(20). \\ (4) Hence, we have “cosh ! (29) “(20). \\ (4) Hence, we have “cosh ! (29) “(20). \\ (4) Hence, we have “cosh ! (29) “(20). \\ (4) Hence, we have “cosh ! (29) “(20). \\ (4) Hence, we have “cosh ! (29) “(20). \\ (4) Hence, we have “cosh ! (29) “(20). \\ (4) Hence, we have “cosh ! (29) “(20). \\ (4) Hence, we have “cosh ! (29) “(20). \\ (4) Hence, we have “cosh ! (29) “(20). \\ (4) Hence, we have “cosh ! (29) “(20). \\ (4) Hence, we have “cosh ! (29) “(20). \\ (4) Hence, we have “cosh ! (29) “(20). \\ (4) Hence, we have “cosh ! (29) “(20). \\ (4) Hence, we have “cosh ! (29) “(20). \\ (4) Hence, we have “cosh ! (29) “(20). \\ (4) Hence, we have “cosh ! (29) “(20). \\ (4) Hence, we have “cosh ! (29) “(20). \\ (4) Hence, we have “cosh ! (29) “(20). \\ (4) Hence, we have “cosh ! (29) “(20). \\ (4) Hence, we have “cosh ! (29) “(20). \\ (4) Hence, we have “cosh ! (29) “(20). \\ (4) Hence, we have “cosh ! (29) “(20). \\ (4) Hence, we have “cosh ! (29) “(20). \\ (4) Hence, we have “cosh ! (29) “(20). \\



\begin{table}
\begin{tabular}{l l} \hline Prompt Type & Prompt Example \\ \hline Get Feedback & Q: Yesterday was 12/31/1929. Today could not be 12/32/1929 because December has only \\  & 31 days. What is the date tomorrow in MM/DD/YYYY? Options: (A) 12/12/1929 (B) 01/01/1930 (C) 01/02/1998 (D) 01/02/1885 (E) 01/02/1930 (F) 12/23/1929 Answer: Let’s think step by step. (1) If yesterday was 12/31/1929, then today is 01/01/1930. (2) The date tomorrow is 01/02/1930. (Final response) So the answer is (F). Earliest error step: (Final response) Error segment: “the answer is (F)” Error reason: (F) 12/23/1929 is inconsistent with the result ”01/02/1930” in step (2). [END] Q: Yesterday was 12/31/1929. Today could not be 12/32/1929 because December has only \\  & 31 days. What is the date tomorrow in MM/DD/YYYY? Options: (A) 12/12/1929 (B) 01/01/1930 (C) 01/02/1998 (D) 01/02/1930 (E) 01/02/1930 (F) 12/23/1929 Answer: Let’s think step by step. (1) If yesterday was 12/31/1929, then today is 01/01/1930. (2) The date tomorrow is 01/02/1930. (Final response) So the answer is (F). Feedback: Step (1) to step (2) are correct. In step (Final response) the part “the answer is (F)” is incorrect. This is because (F) 12/23/1929 is inconsistent with the result ”01/02/1930” in step (2). Updated Answer: Let’s think step by step. (1) If yesterday was 12/31/1929, then today is 01/01/1930. (2) The date tomorrow is 01/02/1930. (Final response) So the answer is (B). [END] \\ \end{tabular}
\end{table}
Table A15: Prompt used for generating feedback and improvement for Date Understanding. The generated feedback is then formatted as “Step 1 to step {first error step - 1} is correct. In step {first error step} the part ‘{error part}’ is incorrect. This is because ’{error reason}’.” In general, we used three-shot prompting. Parts that will be generated are highlighted in blue. **Due to limited space, we present one example used for each task.** Please refer to our code repository for the full prompt.





\begin{table}
\begin{tabular}{l l} \hline Prompt Type & Prompt Example \\ \hline Get Feedback & Q: The following paragraphs each describe a set of three objects arranged in a fixed order. \\  & The statements are logically consistent within each paragraph. On a branch, there are three birds: a hummingbird, an owl, and a falcon. The falcon is to the right of the owl. The hummingbird is to the left of the owl. \\  & Options: \\  & (A) The hummingbird is the second from the left. \\  & (B) The owl is the second from the left. \\  & (C) The falcon is the second from the left. \\  & Answer: Let’s think step by step. Let “??” represents 0 or more objects, and ”?” represents exactly 1 object. \\  & (1) The falcon is to the right of the owl: “(left) ??owl ??falon ??(right)". \\  & (2) The hummingbird is to the left of the owl: “(left) ??humingbird ??owl ??(right)”. \\  & (3) There are in total three birds: a hummingbird, an owl, and a falcon. \\  & (4) Combining (1) and (2) we get the following ordering: “(left) ??hummingbird ??owl ??falcon ??(right)”. \\  & ?? (right)”. \\  & (5) Combining (3) and (4) we get the following ordering: “(left) hummingbird owl falcon (right)”. \\  & (6) According to this ordering, the bird that is second from the left is the falcon. \\  & (Final response) The falcon is the second from the left. So the answer is (C). \\  & Earliest error step: (6) \\  & Error season: It is inconsistent with the ordering in step (5), which established that \\  & “(left) hummingbird owl falcon (right)”. [END] \\ Get Improvement & Q: The following paragraphs each describe a set of three objects arranged in a fixed order. \\  & The statements are logically consistent within each paragraph. On a branch, there are three birds: a hummingbird, an owl, and a falcon. The falcon is to the right of the owl. \\  & the left of the owl. \\  & Options: \\  & (A) The hummingbird is the second from the left. \\  & (B) The owl is the second from the left. \\  & (C) The falcon is the second from the left. \\  & Answer: Let’s think step by step. Let “??” represents 0 or more objects, and ”?” represents exactly 1 object. \\  & (1) The falcon is to the right of the owl: “(left) ??owl ??falcon ??(right)”. \\  & (2) The hummingbird is to the left of the owl: “(left) ??hummingbird ??owl ??(right)”. \\  & (3) There are in total three birds: a hummingbird, an owl, and a falcon. \\  & (4) Combining (1) and (2) we get the following ordering: “(left) ?? hummingbird ??owl ??falcon ??(right)”. \\  & ?? (right)”. \\  & (5) Combining (3) and (4) we get the following ordering: “(left) hummingbird owl falcon (right)”. \\  & (6) According to this ordering, the bird that is second from the left is the falcon. \\  & (Final response) The falcon is the second from the left. So the answer is (C). \\  & Feedback: Step (1) to (5) are correct. In step (6) the part “the bird that is second from the left is the falcon” is incorrect. This is because it is inconsistent with the ordering in step (5), which established that “(left) hummingbird owl falcon (right)”. \\  & Updated Answer: Let’s think step by step. Let “??” represents 0 or more objects, and ”?” represents exactly 1 object. \\  & (1) The falcon is to the right of the owl: “(left) ??owl ??falcon ??(right)”. \\  & (2) The hummingbird is to the left of the owl: “(left) ??hummingbird ??owl ??(right)”. \\  & (3) There are in total three birds: a hummingbird, an owl, and a falcon. \\  & (4) Combining (1) and (2) we get the following ordering: “(left) ?? hummingbird ??owl ??falcon ??(right)”. \\  & (5) Combining (3) and (4) we get the following ordering: “(left) hummingbird owl falcon (right)”. \\  & (6) According to this ordering, the bird that is second from the left is the owl. \\  & (Final response) The owl is the second from the left. So the answer is (B). [END] \\ \end{tabular}
\end{table}
Table A16: Prompt used for generating feedback and improvement for Logical Deduction. The generated feedback is then formatted as “Step 1 to step { first error step - 1} is correct. In step { first error step} the part ‘{error part}’ is incorrect. This is because ’{error reason}’.” In general, we used three-shot promptting. Parts that will be generated are highlighted in blue. **Due to limited space, we present one example used for each task.** Please refer to our code repository for the full prompt.