Teaching Language Models to Self-Improve by Learning from Language Feedback (2024)

Abstract: Aligning Large Language Models (LLMs) with human intentions and values is crucial yet challenging. Current methods primarily rely on human preferences, which are costly and insufficient in capturing nuanced feedback expressed in natural language. In this paper, we present Self-Refinement Tuning (SRT), a method that leverages model feedback for alignment, thereby reducing reliance on human annotations. SRT uses a base language model (e.g., Tulu2) to generate initial responses, which are critiqued and refined by a more advanced model (e.g., GPT-4-Turbo). This process enables the base model to self-evaluate and improve its outputs, facilitating continuous learning. SRT further optimizes the model by learning from its self-generated feedback and refinements, creating a feedback loop that promotes model improvement. Our empirical evaluations demonstrate that SRT significantly outperforms strong baselines across diverse tasks and model sizes. When applied to a 70B parameter model, SRT increases the win rate from 9.6% to 25.8% on the AlpacaEval 2.0 benchmark, surpassing well-established systems such as GPT-4-0314, Claude 2, and Gemini. Our analysis highlights the crucial role of language feedback in the success of SRT, suggesting potential for further exploration in this direction.

heading: 1 Introduction
text: Recent advances in Large Language Models (LLMs) have revolutionized the field of natural language processing. These models have demonstrated remarkable capabilities in various tasks such as open-ended generation, question answering, and mathematical reasoning (Brown et al., 2020; Chowdhery et al., 2023; Ouyang et al., 2022; Bubeck et al., 2023). However, despite their impressive performance, LLMs occasionally generate content that can be untruthful or harmful. This
*Corresponding author.
highlights the need to align language models with human intentions and values to ensure safe and controllable deployment (Weidinger et al., 2021; Bai et al., 2022b; Perez et al., 2022).
Many current methods to align LLMs, such as Reinforcement Learning from Human Feedback (RLHF) and Direct Preference Optimization (DPO), rely on human preferences (Christiano et al., 2017; Jaques et al., 2019; Stiennon et al., 2020; Rafailov et al., 2023). These preferences are typically expressed through rankings or scores. However, there are two main challenges with these methods. Firstly, annotating human preferences is expensive and time-consuming, which makes it difficult to scale these techniques. For instance, Scheurer et al. (2023) spent $40k to annotate approximately 60K feedback instances for a single summarization task. Secondly, simple rankings or scores may not fully capture the subtlety and complexity of human preferences. This can limit the depth of feedback provided to models for improvement (Myers et al., 2021; Casper et al., 2023). Humans, on the other hand, can derive insights from minimal language feedback, indicating that there is potential for more sophisticated and efficient alignment methods (Scheurer et al., 2022; Chen et al., 2023).
6090
In response to these challenges, we introduce a new method for aligning language models, which we call Self-Refinement Tuning (SRT). SRT obviates the need for human preferences with a twostage learning process. In the first stage, SRT employs a powerful model like GPT-4 to critique and refine the outputs from a base model, such as Tulu2 (Ivison et al., 2023). The base model is then finetuned on critiques and refinements, enabling selfevaluation and improvement. In the second stage, SRT further boosts the model by learning from selffeedback. Specifically, SRT uses the fine-tuned model to generate model preferences, i.e., pairs of outputs and refinements. Subsequently, SRT optimizes the model on these preferences using DPO (Rafailov et al., 2023).
We evaluate SRT on open-ended generation, question answering, and mathematical reasoning. Empirical results show that SRT consistently outperforms baseline models of sizes from 7B to 70B, with an average performance enhancement of 3.7 to 4.0 points. These improvements are obtained using merely 22K feedback instances annotated by GPT-4 Turbo. Figure 1 shows that the SRT significantly improves Tulu2 models using modelgenerated feedback. The strongest model trained with SRT attains a 25.8% win rate against GPT4 Turbo on the AlpacaEval 2.0 benchmark. This performance surpasses established systems such as GPT-4 0314, Mistral Medium, Claude, and Gemini. Our analysis confirms that the success of SRT primarily stems from its language feedback feature, which identifies weak areas and offers valuable suggestions for improvement.

heading: 2 Related Work
text: This section briefly reviews two critical areas in the alignment of LLMs: learning from AI feedback and learning to self-improve. Our work intersects with these domains and addresses existing challenges.
Learning from AI Feedback. Learning from human feedback is the key to the success of stateof-the-art language models such as GPT-4 (OpenAI, 2023) and Gemini (Google, 2023). However, acquiring high-quality human feedback is both costly and time-consuming (Christiano et al., 2017; Jaques et al., 2019; Stiennon et al., 2020; Rafailov et al., 2023). This has led to a growing interest in harnessing AI-generated feedback to enhance LLMs (Lee et al., 2023; Roit et al., 2023; Hu et al., 2024). For instance, Hu et al. (2024)
employs LLMs to annotate ranking-based preferences. Our research diverges from this approach by utilizing a more comprehensive range of feedback, encompassing identified weaknesses and proposed improvements. A similar approach is Reinforcement Learning from AI feedback (RLAIF, Bai et al., 2022b), which uses LLMs to generate critiques and refinements. However, RLAIF encounters challenges such as computational inefficiency and unstable training dynamics due to the inherent complexities of reinforcement learning techniques (Casper et al., 2023; Shen et al., 2023). We address these challenges by unifying the generation of feedback and refinement into instruction-following, thereby simplifying the training process.
Learning to Self-Improve. Using Large Language Models (LLMs) to identify and correct their own errors has become increasingly popular. This self-improvement capability has significantly enhanced performance across various tasks, such as question-answering, code generation, and mathematical reasoning (Shinn et al., 2023; Madaan et al., 2023; Chen et al., 2024). Pan et al. (2023) conducted a comprehensive survey of this field. However, these approaches rely on critique and refinement skills that are generally lacking in opensource language models (Valmeekam et al., 2023; Huang et al., 2023). This underscores the urgent need to train open-source models for selfimprovement. Unlike previous methods that train separate models for generation, critique, and refinement (Yasunaga and Liang, 2020; Welleck et al., 2022), our approach employs a single model for all tasks, facilitating knowledge transfer across them. Additionally, our method is designed for the general alignment of LLMs, in contrast to prior methods that are task-specific (Wang and Li, 2023; Yu et al., 2023; Lu et al., 2023).

heading: 3 Methodology
text: Figure 2 presents the two-stage Self-Refinement Tuning (SRT) process. SRT aims to enhance the capabilities of a base language model (denoted as Mbase) by harnessing learning from AI feedback (LAIF), thereby reducing the reliance on human feedback. In the first stage, we train Mbase to selfimprove by learning from feedback and refinements annotated by more powerful models. In the second stage, we further optimize the model using its selfgenerated feedback.

heading: 3.1 Training Models for Self-Improvement

heading: 3.1.1 Collecting Feedback and Refinements
text: In the first stage, we collect the training data for self-improvement from the interaction of two models. Specifically, Mbase interacts with a stronger model, denoted as Mcritic. Given a set of instructions x = [x1, ..., xN ], Mbase generates initial responses y = [y1, ..., yN ]. Mcritic then provides feedback f = [f1, ..., fN ] on these responses. As delineated in Table 1, we instruct Mcritic to generate 1) analyses of the weaknesses, 2) overall quality scores ranging from 1 to 10, and 3) suggestions for enhancing the responses. Subsequently, Mcritic generates the refinements (i.e., improved responses) r = [r1, ..., rN ] according to its previous feedback.
The critique-refinement process can be iterative: after the first iteration, we obtain a set of instructions, initial outputs, critiques, and refinements. We then use the refinements as the inputs for the next iteration and continue the process until the quality of the outputs no longer improves. Intriguingly, we find that a single iteration of refinement is typically adequate, with additional iterations contributing only marginal improvements. See Section 4.1 for more details of the process of collecting the
feedback and refinement.

heading: 3.1.2 Self-Improvement Training
text: After collecting the feedback and refinements, we fine-tune Mbase to identify and rectify its own errors or undesirable behaviors, thereby improving its performance. To facilitate this, we reformat the collected data into sequences of sentences as follow: Instruction→ Response → Feedback → Refinement By doing so, we can train the model using a language modeling objective. This method is somewhat akin to Chain of Hindsight (Liu et al., 2023), where the model is trained to produce outputs from provided instructions and feedback. However, our approach differs in training the model to generate the feedback and refinements sequentially instead of merely predicting the outputs. More formally, we want to optimize:
L = − 1 N
∑
i=1
logP (yi, fi, ri|xi) (1)
where N is data size, xi is the input instruction, yi is the initial output, fi is the feedback, and ri is the best refinement with the highest score. The input instruction is masked from the loss calculation during the training phase. This results in a
self-improvement model, denoted by Mself , which can provide feedback on its outputs and refine them until they are satisfactory. In Section 4.2, we show that Mself significantly outperforms Mbase.

heading: 3.2 Scaling SRT through Self-Feedback
text: We now describe how Mself can be used to scale the learning from AI feedback. Specifically, we leverage Mself to generate preference data and for further model optimization. Our approach is similar to Self-Rewarding (Yuan et al., 2024) but deviates from the strategy for acquiring superior responses. Rather than sampling multiple responses and selecting the optimal one, we directly generate an improved response via self-refinement, thereby enhancing efficiency.
More concretely, we use model Mself to generate initial responses y’ = [y′1, ..., y′N ] from a distinct instruction set x’ = [x′1, ..., x′N ]. The model critiques and refines these responses into feedback f’ and refinements r’. We filter to ensure refinements exceed initial response quality, based on model-assigned scores. This yields preference pairs of initial responses and refinements, which can used for DPO training (Rafailov et al., 2023).

heading: 4 Experiments

heading: 4.1 Implementation Details
text: In this subsection, we outline the specifics of SRT training, which includes the choice of base models, the procedure for feedback annotation and cleaning, and the comprehensive model training process.
Models. To evaluate the efficacy of SRT, we employ three fine-tuned LLaMA2 models as our base models. These include Tulu2-7B, Tulu2-13B, and Tulu2-70B, all trained on the Tulu-2 Mixture (Ivison et al., 2023). Given that these datasets are partially distilled from models such as GPT-4, they can also be referred to as dSFT, an acronym for distilled Supervised Fine-Tuning (Tunstall et al., 2023). In the first stage of SRT, we use Tulu2-7B to generate initial responses and employ GPT-4 Turbo (gpt-4-1106-preview) to generate language feedback and refinements. The GPT-4 critic exhibits a 78.9% agreement rate with human preferences on the HH-RLHF dataset (Bai et al., 2022a). We then fine-tune all base models on these refinements. In the second stage, these fine-tuned models independently generate feedback and refinements for DPO training, which we refer to as sDPO (selffeedback DPO). To measure the effectiveness of self-feedback, we further compare our models with other DPO variants trained on the UltraFeedback (Cui et al., 2023) dataset. These include Tulu2DPO-7B, Tulu2-DPO-13B, and Tulu2-DPO-70B. We refer to these models as dDPO, an acronym for distilled DPO (Tunstall et al., 2023).
Details on Feedback and Refinements Annotation. In our two-stage process, we initially select 25,000 instructions from the Tulu-2 Mixture, followed by 75,000 instructions for the second stage. We adopt the approach of Li et al. (2023a), choosing examples of a single conversation turn to streamline the refinement process. For generating responses, we use a sampling temperature of 0.7, while for feedback and refinement generation, we set the temperature to 0 and restrict the maximum new tokens to 2,048. We use the prompt template shown in Table 1 for annotating feedback and refinements. Figure 3 shows the score distribution of the initial outputs of Tulu2-7B and their subsequent refinements. On average, the refinements enhance the score by 1.5 points compared to the initial outputs. Table 2 shows one iteration of critique-and-refinement is sufficient.
Post-processing of Feedback and Refinements. Our goal is to compile instances that form a consistent sequence of instruction-response-feedbackrefinement. However, even sophisticated models like GPT-4 Turbo occasionally fail to conform strictly to the format requirements. As a result, we apply the following filtering rules:
Rule #1: The feedback should include potential weaknesses, overall scores, and suggestions1.
Rule #2: The quality of the refinement should not be lower than that of the initial response.
Note that we evaluate the initial responses and their subsequent refinements independently. This is important as the critic sometimes gives higher scores when it has been conditioned with prior feedback, regardless of the quality of the refinement. After applying these filters, we obtain 22K and 63K valid feedback and refinements for the initial and subsequent stages, respectively.
Training Details. The base models are trained using the Tulu-2 Mixture dataset, following the same settings as Ivison et al. (2023). In the first stage of SRT, we fine-tune the base models for five epochs. We concatenate each instruction’s initial response, feedback, and refinement into sequences. Following Ivison et al. (2023), we set the maximum sequence length to 8,192 but use a smaller global batch size of 32. We use a cosine learning rate scheduler with a peak learning rate of 1e-5 and 10% warmup steps. In the second stage, we train our DPO models for one epoch with a β value of 0.01. We maintain the same optimizer and learning rate scheduler as the first stage but adjust the maximum learning rate to 5e-7 to ensure training stability.
1We identify these elements by looking for keywords such as ‘###Feedback,’ ‘###Overall Score,’ and ‘###Improved Response.’
We use the HuggingFace TRL library2 to train our models, and use FlashAttention-2 (Dao, 2023) and DeepSpeed Zero-3 (Rajbhandari et al., 2020) to speedup training.
Evaluation Details. We evaluate SRT across open-ended generation, reasoning, and questionanswering. For open-ended generation, we test SRT on the AlpacaEval benchmark (Li et al., 2023b), which consists of 805 instructions. We report the win rate of our models compared to GPT3.5 (text-davinci-003). Since this baseline may be outdated, we also compare our models with GPT-4 Turbo (gpt-4-1106-preview). We use the default configuration provided in the official library3. For reasoning, we evaluate our models on GSM8K (Cobbe et al., 2021) and Big-Bench-Hard (BBH, Suzgun et al., 2022). We report the exact match score (EM) on the test sets, using few-shot chain-ofthought prompting strategies identical to those used by Ivison et al. (2023). For question-answering, we test with TydiQA (Clark et al., 2020), a multilingual benchmark that covers 11 different languages. In line with Ivison et al. (2023), we report the F1 score under the Gold Passage (GP) setting, where the passage containing the answer is provided. To assess the “self-evaluation" capability of SRT, we test our models on the HH-RLHF dataset (Bai et al., 2022a), comparing model-predicted scores with human preferences. We evaluate using a sample of 500 data points from the HH-RLHF test set. Unless stated otherwise, we always select the refined responses as the outputs for the SRT models. We also limit the output refinement to one iteration, as additional iterations provide minimal benefits and significantly slow down the model’s generation.
2https://github.com/huggingface/trl 3We use ‘alpaca_eval_gpt4’ as the judge for comparing GPT-3.5 and use ‘weighted_alpaca_eval_gpt4_turbo’ for GPT4 Turbo.

heading: 4.2 Main Results
text: Table 3 summarizes the experimental results on four benchmarks. The results indicate that SRT consistently improves model performance across various tasks and model sizes. In the first stage, models trained with language feedback from GPT4 significantly outperform the Tulu2 baselines. On average, the first stage of SRT enhances the performance by a margin of 3.7 to 4.0 across different model scales. Models trained during this stage are comparable to the Tulu2-DPO baselines on AlpacaEval but use considerably fewer feedback instances (22K vs 64K). These findings underscore the efficacy of training models to assess and refine their outputs by learning from the feedback of more advanced models.
In the second stage, SRT further boosts performance by scaling feedback with self-generated responses and refinements. The most notable improvement is observed on the AlpacaEval task. SRT’s second stage is more effective with larger models (13B and 70B) but less with smaller ones (7B), which see an average drop of 1.0 points. This suggests that the inherent capabilities of a model may restrict its ability to learn from self-feedback.
Additionally, the enhancements observed in reasoning tasks, including GSM8K and BBH, are relatively slight, especially for the 7B and 13B models. This could be attributed to the limited reasoning capabilities present in these models.

heading: 4.3 Results on AlpacaEval 2.0
text: GPT-3.5 may be a relatively weak baseline for evaluating our strongest models. For example, both SRT 70B and Tulu2-DPO 70B models achieve win rates over 95%. Therefore, we further evaluate these models on the more challenging AlpacaEval 2.0 benchmark, which employs GPT-4 Turbo as the baseline for calculating win rates. Table 4 presents the results of 13B and 70B models and several proprietary models. We can see that SRT models significantly outperform Tulu2 models. Notably, our strongest model, SRT 70B (stage 2), achieves a win rate of 25.8%, outperforming Tulu2-DPO 70B by a substantial margin of 9.0%. The model
surpasses well-established models such as GPT-4 0314, Mistral Medium, Claude 2, and Gemini Pro. We also find that SRT encourages models to produce more verbose outputs, which could influence the observed performance improvement. However, despite generating shorter responses, our SRT 13B models achieve superior results compared to Tulu2DPO models.

heading: 4.4 Results on HH-RLHF
text: Table 5 presents a comparison of feedback accuracy among different models on the HH-RLHF test set. Feedback accuracy is evaluated based on the agreement of model outputs with human preferences. Specifically, for each pair of responses, the feedback is considered ‘correct’ if the score of the ‘chosen’ response exceeds that of the ‘rejected’ response. Remarkably, GPT-4 Turbo surpasses all other models, achieving an agreement rate of 78.9%. Intriguingly, during the initial stage of SRT, the 7B model outperforms larger models, reaching an agreement rate of 74.4%. However, after the second-stage fine-tuning of SRT, the accuracy of the 7B and 13B models significantly declines, while the 70B models show slight changes.

heading: 5 Analysis
text: This section provides an in-depth examination of the factors that affect the performance of our methods. Initially, we evaluate the efficacy of self-refinement in SRT by contrasting it with the widely adopted re-ranking approach. Subsequently, we explore the influence of various elements in SRT, encompassing language feedback, the quality of refinements, and the quantity of training data. Through these analyses, we aim to deepen the understanding of our methods.

heading: 5.1 Self-Refinement vs. Re-Ranking
text: Re-ranking is a widely adopted technique to bolster the performance of text generation systems. Given that our models can precisely assess the quality of their outputs, they can be effectively leveraged for re-ranking. In this study, we contrast the efficacy of self-refinement and re-ranking using the AlpacaEval test set. For self-refinement, we employ greedy decoding and refine the response once. For re-ranking, we sample 16 distinct responses using a temperature of 0.7. These responses are then re-ranked based on the scores predicted by the model itself. We then task GPT-4 for comparing the two sets of responses using the settings described in Section 4.1. Figure 4 illustrates a direct comparison between self-refinement and re-ranking across 16 candidate responses. It is evident that self-refinement consistently surpasses re-ranking in performance across various models. This advantage becomes more pronounced with an increase in model size. Given the significant cost associated with generating multiple responses, self-refinement emerges as a more efficient and cost-effective approach for improving the performance of language models.
0 5 10 15 20 #Training Samples (K)
5
10
15 20 W in R at e (% ) v s. GP T4
Tu rb
o SRT (stage 1)
0 20 40 60 #Training Samples (K)
10
15
20
25 SRT (stage 2)
13B 70B
Figure 5: Win rates against GPT-4 Turbo by varying numbers of training samples for SRT models.

heading: 5.2 Impact of Language Feedback
text: A unique characteristic of SRT is its integration of language feedback, which offers insights into potential weaknesses, overall scores, and suggestions for improvement. Here, we ablate these components to investigate their influence on the final performance. We evaluate 13B models trained during SRT’s first stage on AlpacaEval and compare these models with GPT-3.5. The results are presented in Table 6. We find that removing any part of the feedback leads to a drop in performance. Also, weaknesses and suggestions are almost equally important, while the score seems less necessary. Removing all feedback results in a significant performance drop of 5.1 points, which is even inferior to the performance achieved by training solely with refinement. To summarize, these findings emphasize the crucial role of language feedback in enhancing SRT’s performance.

heading: 5.3 Impact of Training Data Size
text: In our primary experiments, we use 22K and 63K training samples for the first and second stages of SRT, respectively. We now delve further into the impact of data volume on the performance of SRT. We experiment with the challenging AlpacaEval 2.0 benchmark and replicate the training settings from Section 4.1. Figure 5 illustrates the results of 13B and 70B models. As depicted, the performance almost monotonically increases with the variation in the number of training samples. Models of different sizes and stages exhibit diverse convergence speeds. Increasing the training data results in a more significant performance gain on the 70B models. However, we also find the 13B model’s performance slightly deteriorates when the data volume increases from 36K to 48K. We hypothesize that this could be due to the presence of noise. To validate this hypothesis, we further analyze the impact
7B Models 13B Models 70B Models70
75
80
85
90
95
W in
R at
e vs
. G PT
-3 .5 Low-quality
Medium-quality High-quality
Figure 6: AlpacaEval results (vs. GPT-3.5). The low-, medium-, and high-quality refinements have scores of 6, 7, and 8, respectively.
of data quality on the performance of SRT.

heading: 5.4 Impact of Refinement Quality
text: In our primary experiment, we merely filter refinements that are of lower quality (reflected by overall scores) than the initial responses. Here, we delve deeper into this issue by conducting experiments with more fine-grained control over the refinement quality. To investigate this, we train models with varying levels of refinement quality and compare their performances. Specifically, we select samples of 2,000 refinements each from low-, medium-, and high-quality categories, corresponding to scores of 6, 7, and 8, respectively. As illustrated in Figure 6, there is a clear monotonic increase in the performance of SRT in line with the quality of refinements. The results hold with varying model sizes, suggesting that SRT can be further augmented by enhancing the refinement quality.

heading: 6 Conclusion
text: We have presented Self-Refinement Tuning (SRT) for aligning language models using language feedback. SRT initially teaches language models to assess and enhance their outputs by learning from feedback provided by more advanced models. Subsequently, SRT optimizes these models further by learning from their self-generated feedback. The primary benefits of SRT are twofold: (1) it obviates the need for human-annotated preferences, and (2) it obtains promising performance across a wide range of tasks and model sizes. In our analysis, we find that both high-quality language feedback and refinements are crucial for language models to learn to self-improve. Collectively, we demonstrate that SRT is an effective and efficient strategy for improving the performance of language models.

heading: Acknowledgement
text: This work was supported in part by the National Science Foundation of China (No.62276056), the Natural Science Foundation of Liaoning Province of China (2022-KF-16-01), the Fundamental Research Funds for the Central Universities (Nos. N2216016 and N2316002), the Yunnan Fundamental Research Projects (No. 202401BC070021), and the Program of Introducing Talents of Discipline to Universities, Plan 111 (No.B16009).
Limitations
While SRT has demonstrated promising results across a variety of tasks, it is not without its limitations. A significant constraint lies in its dependency on a powerful critic model to provide feedback and refinements. Even state-of-the-art language models, such as GPT-4-Turbo, can sometimes generate feedback or refinements that are inaccurate or suboptimal. For instance, the refined responses from GPT-4 yield an approximately average score of 7, indicating substantial room for improvement.
Moreover, one of the inherent limitation of selfrefinement is it greatly increase the output length of language models. Although we find a single iteration of self-refinement is typically sufficient (in Table 2), it still approximately doubles the output length of our models. The lengthy outputs lead to higher computational costs during decoding.
In the future, we will develop more accurate and efficient feedback mechanisms, optimize the selfrefinement process to control output length, and explore ways to improve the quality of refinements.
Ethical Considerations
SRT enhances language models by incorporating feedback and refinements from both open-source and commercial models. This approach minimizes the reliance on human annotations but also might introduces potential risks. Notably, SRT can lead to unintended issues such as overfitting and the reinforcement of existing biases in the initial models. Therefore, it is essential to develop more robust and trustworthy alignment methods to ensure these language models are safe, fair, and controllable. Additionally, as the model improves, it may become increasingly complex and difficult to interpret, posing challenges in applications where transparency and interpretability are crucial. To mitigate these issues, future work could integrate ethical guidelines or external validation tools into the SRT process.

heading: Future Work
text: In the future, we will develop more accurate and efficient feedback mechanisms, optimize the selfrefinement process to control output length, and explore ways to improve the quality of refinements. Ethical Considerations
SRT enhances language models by incorporating feedback and refinements from both open-source and commercial models. This approach minimizes the reliance on human annotations but also might introduces potential risks. Notably, SRT can lead to unintended issues such as overfitting and the reinforcement of existing biases in the initial models. Therefore, it is essential to develop more robust and trustworthy alignment methods to ensure these language models are safe, fair, and controllable. Additionally, as the model improves, it may become increasingly complex and difficult to interpret, posing challenges in applications where transparency and interpretability are crucial. To mitigate these issues, future work could integrate ethical guidelines or external validation tools into the SRT process