A Critical Evaluation of AI Feedback for Aligning Large Language Models

Authors: archit  sharma
sedrick  keh
eric  mitchell
chelsea  finn
kushal  arora
thomas  kollar
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-15 20:56:36

Abstract: Learning from AI feedback (LAIF) is a popular paradigm for improving the instruction-following abilities of powerful pre-trained language models. LAIF first performs supervised fine-tuning (SFT) using demonstrations from a teacher model and then further fine-tunes the model with reinforcement learning (RL) or direct preference optimization (DPO), using feedback from a critic model. While recent popular open-source models have demonstrated substantial improvements in performance from the RL step, in this paper we question whether the complexity of this RL step is truly warranted for AI feedback. We show that the improvements of the RL step are virtually entirely due to the widespread practice of using a weaker teacher model (e.g. GPT-3.5) for SFT data collection than the critic (e.g., GPT-4) used for AI feedback generation. Specifically, we show that simple supervised fine-tuning with GPT-4 as the teacher outperforms existing LAIF pipelines. More generally, we find that the gains from LAIF vary substantially across base model families, test-time evaluation protocols, and critic models. Finally, we provide a mechanistic explanation for when SFT may outperform the full two-step LAIF pipeline as well as suggestions for making LAIF maximally useful in practice.

Main paper: 
 introduction: 
 As the raw capabilities of open-source large language models (LLM) improve through pre-training at a large scale [Touvron et al., 2023a ,b, Jiang et al., 2023 , 2024 , Bai et al., 2023 , Bi et al., 2024] , methods for effectively 'aligning' these models; i.e., steering them to follow user instructions effectively and safely, has garnered increasing attention. Supervised fine-tuning (SFT) with large datasets of user queries and human-written responses is one popular approach. Further refining a model fine-tuned with SFT using reinforcement learning from human feedback (RLHF; Christiano et al. [2017] ) has been shown to further improve the quality of model responses, as judged by humans [Ouyang et al., 2022 , Stiennon et al., 2020] . However, collecting the data for SFT and RLHF is expensive, requiring human annotations for both the SFT and preference rankings over several candidate responses to a query to be used as feedback for the RL stage. Given the high cost of data collection as well as the high level of disagreement among human annotators, many works such as Bai et al. [2022] and Lee et al. [2023] have replaced the human annotators in both the SFT and RLHF stages with strong language model annotators such as GPT-4 [Achiam et al., 2023] . The corresponding techniques, supervised model distillation [Taori et al., 2023 , Chiang et al., 2023 , Ding et al., 2023] and learning from AI feedback (LAIF; Bai et al. [2022] , Lee et al. [2023] , Cui et al. [2023] ), have proven effective in training state-of-the-art open source models [Tunstall et al., 2023 , Ivison et al., 2023 , Intel, 2023 , Teknium, 2023] . However, plugging in LLMs to substitute human annotators in the same RLHF pipeline may not be the best way to leverage such LLMs. First, LLMs are often better at generating answers for instructions than discriminative tasks [West et al., 2023] , and labeling preferred answers for AI feedback is an example. Further, while humans may find comparing answers incurs lower cognitive cost than producing answers themselves, for modern LLMs, the cost of generating a preference comparison or ranking may actually be higher than simply generating a demonstration, owing to the longer input context when comparing multiple responses. Therefore, while LAIF may seem like a more convenient variant of RLHF prima facie, we ask: what kind of data from strong LLMs is more effective for learning instruction-following: completions for SFT or AI feedback for RL?Figure 1 : Supervised fine-tuning (SFT) on strong teachers can accounts for improvements from learning from AI feedback (LAIF). LAIF from strong models such as GPT-4 can result in substantially better instruction-following LLMs than supervised SFT alone on popular datasets such as ShareGPT [Chiang et al., 2023] constructed using GPT-3.5 completions (GPT-3.5 SFT + GPT-4 AIF). However, simply performing SFT on completions from GPT-4 can result in a better model (GPT-4 SFT), suggesting that improvement in performance from LAIF is partly because the default ShareGPT completions are from a weak teacher (GPT-3.5). Furthermore, LAIF (GPT-4 SFT + GPT-4 AIF) does not result in a significantly better model compared to GPT-4 SFT alone.To answer this question, we compare the effectiveness of the LAIF pipeline with doing SFT on demonstrations directly generated from the annotator language model. In our experiments, we align a variety of pre-trained base language models, both with SFT and LAIF, using the prompts from the ShareGPT dataset [Chiang et al., 2023] . We use SFT demonstrations from three teacher language models, namely, GPT-3.5, GPT-4, and Claude, and use two strong language models as critics, GPT-4 and Claude, for collecting AI feedback. We evaluate our fine-tuned models using AlpacaEval [Li et al., 2023] . The experimental setup is described in further detail in Section 4. In our experiments, we find that two conditions are necessary for LAIF to significantly outperform SFT: (a) a sufficiently strong pre-trained base model and, (b) a capability mismatch between the teacher used for the SFT data collection and the critic used for collecting AI feedback. The latter condition has surprising consequences: if the target completions are sufficiently performant and capability gap between models used for SFT and AI feedback is minimal, then simply doing SFT can suffice. We observe this in Figure 1 , where SFT on the completions generated by GPT-4 outperforms GPT-3.5 SFT + GPT-4 feedback. This suggests that LAIF from a strong critic may be compensating for a weak teacher in popular SFT datasets like ShareGPT that are generated using GPT-3.5, and we may be overestimating the effectiveness of AI feedback. We further try to analyze these findings in a more principled way in Section 5, and provide a possible mechanistic intuition in a simplified bandit setting for why SFT might perform comparably or even outperform LAIF.Finally, Section 6 provides practical suggestions and discussions based on the experimental and theoretical insights in the previous sections. First, one should account for the distribution of completions in instruction-tuning datasets [Taori et al., 2023 , Chiang et al., 2023 , Ding et al., 2023] before evaluating AI feedback based methods. Further, we should consider updated instruction-tuning datasets created using more recent state-of-the-art LLMs, like GPT-4 (as of February 2024). Second, while our experiments suggest that SFT on strong teacher completions can be just as effective as LAIF, we caution readers into generalizing these findings to RLHF setting and highlight why preference-based RL might be superior when collecting human feedback due to inherent cognitive load of collecting demonstrations. Third, our experiments further support the importance of pre-training for effective instruction-following, both in terms of absolute performance and also in its ability to fine-tune from AI feedback. Finally, we also provide possible ways to improve the effectiveness of AI feedback and hypothesize how AI feedback could improve over SFT over target distributions from strong teacher LLMs. Overall, AI feedback has several desirable features that make it a compelling avenue for scalable oversight and effective alignment, but current evaluation may be overestimating the significance of improvements in instruction-following for open-source LLMs and we hope that this paper will inspire investigation into more effective LAIF approaches. 
related work: 
 Modern language models rely on a two-stage pipeline that first learns representations with large-scale unsupervised learning and then adapts, or fine-tunes, those representations to the task of interest; this strategy initially proved effective for isolated word embeddings [Collobert et al., 2011] and more recently has been adopted for sequence-level representations [Devlin et al., 2019, Radford and Narasimhan, 2018] . Virtually all of the most powerful language models are pre-trained with a simple maximum likelihood objective over a large, unsupervised dataset [Radford and Narasimhan, 2018 , Radford et al., 2019 , Brown et al., 2020 , Touvron et al., 2023a,b] . Owing to the drastically smaller computational demands of fine-tuning, considerably more diversity exists in the objectives used for fine-tuning, including supervised objectives such as imitation [Collobert et al., 2011 , Devlin et al., 2019 , Radford et al., 2019 , Raffel et al., 2020 , Taori et al., 2023] and ranking losses [Yuan et al., 2023] as well as reinforcement learning from heuristic rewards [Paulus et al., 2018] , learned rewards trained from human reward scores [Böhm et al., 2019] or preference annotations [Ziegler et al., 2020 , Ouyang et al., 2022 , Bai et al., 2022 , Rafailov et al., 2023] .Simultaneous with the development of new fine-tuning objectives, several works have focused on the source of data used for fine-tuning. While human-annotated data has been widely used [Collobert et al., 2011 , Devlin et al., 2019 , Radford and Narasimhan, 2018 , Ouyang et al., 2022 , Conover et al., 2023] , more scalable synthetic data sources have recently been explored, using language models themselves to generate some or all of the fine-tuning data. Bai et al. [2022] , Chiang et al. [2023] , Taori et al. [2023] , Teknium [2023] show that outputs generated by a high-quality language model can be distilled into a smaller language model through supervised imitation. Bai et al. [2022] also show that a large language model can effectively generate preferences over model samples, which are ultimately used to train a reward model used for further RL-based fine-tuning, which is referred to as learning from AI feedback (LAIF). Tunstall et al. [2023] , Ivison et al. [2023] , Intel [2023] further explore using model-generated datasets with various combinations of supervised and reinforcement learning-based fine-tuning objectives, suggesting that LAIF can produce improvements over purely supervised fine-tuning from AI feedback. However, notably, commonly-used datasets for supervised training [Taori et al., 2023 , Ding et al., 2023 , Geng et al., 2023] contain substantial amounts of data generated by GPT-3.5 [OpenAI, 2023] , while the more recently-collected AI datasets for LAIF contain annotations generated by the stronger GPT-4 model [Ding et al., 2023] . The ramifications of this discrepancy, and how the effectiveness of LAIF is affected by the teacher and critic quality is the focus of this work. 
algorithmic overview of llm fine-tuning: 
 The dominant approaches to fine-tuning LLMs as general dialogue agents are supervised fine-tuning (SFT) and preference-based reinforcement learning from either human or AI feedback (RLHF or LAIF, respectively).Supervised fine-tuning. Consider a dataset D SFT = {x, y}, where x is one or more turns of dialogue history1 and y is the target response (a sequence of tokens) for this dialogue history, which the model is trained to generate. Specifically, an autoregressive language model p θ parameterized by θ is trained to minimize the negative log likelihood of the responses, given the histories:EQUATIONThe targets y may be written by either humans or strong LLM teachers.RLHF & LAIF. Motivated by the difficulty of gathering high-quality target responses y at scale from humans, an alternative to purely supervised fine-tuning collects preference annotations over pairs of candidate responses y, y ′ to a dialogue history x. Typically, a model fine-tuned with SFT is used to generate the pair of responses, 2 and humans for RLHF or strong LLM critics for LAIF annotate which better responds to the input according to some set of criteria. We refer to the preferred response as y w and the dispreferred response as y l , producing a dataset of the form D p = {x i , y i w , y i l }. Using a theoretical model of human discrete choice such as the Bradley-Terry model [Bradley and Terry, 1952] , which relates discrete choices to implicit goodness scores of the underlying options, we can train a reward model with maximum likelihood using this preference data. For the Bradley-Terry model, the reward modeling loss is:EQUATION= -E (x,yw,y l )∼Dp log σ (r ϕ (x, y w ) -r ϕ (x, y l ))where σ denotes the sigmoid, and r ϕ denotes the reward function with parameters ϕ to fit the goodness score of a completion y for an input prompt x. The equality between the two RHS expressions is due to choice of the Bradley-Terry model. Using a reward model trained with this objective, the final step is to fine-tune a generative language model policy π θ that generates high-scoring responses. Typically, π θ is initialized from a model trained with SFT π SFT , and regularized to keep the KL divergence between the π θ and π SFT small, giving the final policy search objectiveEQUATIONwhere D is a dataset of unlabeled prompts and β controls the strength of the KL regularization.The original form of RLHF for language models [Böhm et al., 2019 , Ziegler et al., 2020] used relatively expensive online reinforcement learning algorithms such as A2C [Mnih et al., 2016] or PPO [Schulman et al., 2017] . More recently, Rafailov et al. [2023] show that the optimal policy for the learned reward can be extracted in closed form, essentially skipping the need to perform iterative, approximate policy learning. The resulting algorithm, direct preference optimization (DPO), is simpler to tune and less computationally demanding than prior methods, while optimizing the same objective. We therefore use DPO as the algorithm for LAIF in our experiments. The DPO loss for the language model policy π θ isEQUATIONwhich trains the language model policy π θ directly, without the need to first train a separate reward model r ϕ . 
experiments: 
 In light of increasing usage of LAIF, the primary question we want to investigate is whether preferencebased RL from AI feedback, produced by blackbox LLMs, like GPT-4 is more effective at aligning language models compared to simple supervised fine-tuning on completions from these LLMs. To this end, we first explain the experimental setup for comparing LAIF and SFT in 4.1. With this experimental setup, we observe in 4.2 that LAIF can result in substantially more capable models than those obtained by instruction-tuning on current public SFT datasets. However, we also find that using a stronger teacher for SFT consistently matches or outperforms LAIF. This surprising result spurs deeper investigation into effectiveness of LAIF, where we find that using a stronger SFT distribution also limits any further gains from AI feedback.Datasets & Models. To systematically compare SFT and LAIF on their effectiveness for instruction following, it is important to control the dataset of instructions used to train both the methods. To this end, we fix the dataset of prompts to be single-turn instructions derived from ShareGPT [Chiang et al., 2023] . We consider a variety of performant pre-trained LLMs for our experiments: Llama [Touvron et al., 2023a,b] , Mistral [Jiang et al., 2023] and Mixtral [Jiang et al., 2024] , Yi [01.AI, 2023] and DeepSeek [Bi et al., 2024] .Terminology. We will use the strong LLMs like GPT-4 in three different roles: as a teacher where the LLM generates a target completion for an instruction, as a critic where the LLM gives feedback on which completion it prefers for a given instruction, and similarly as an evaluator where it labels the preferred completion on a held out set of instructions. For LAIF, SFT is an important precursor, so we SFT on 10% of the total prompts, and LAIF is done on the remaining 90%. For the other settings, the full set of prompts are used for SFT. While LAIF improves the performance compared to SFT on the default ShareGPT completions, SFT on GPT-4 completions consistently matches or outperforms LAIF.Setup. When fine-tuning these pre-trained models with SFT, we consider the default target distribution in ShareGPT, which is sampled from GPT-3.5 as the teacher. We also generate completions from GPT-4 (gpt-4-0314) and Claude (claude-v1) for the same set of instructions For LAIF, we consider the following pipeline:1. Split the training prompts for SFT and LAIF. Fine-tune the base model on completions from teacher M teacher to minimize L SFT . 2. For prompts in the AIF split, create a dataset by sampling two completions for every prompt and label the preferred completion using M critic to create D p . 3. Fine-tune the instruction-tuned model using L DPO .Prompt split for SFT and AIF. For a controlled study comparing AIF and SFT, one would ideally compare the performance of a base LLM fine-tuned to minimize L DPO on a preference dataset D p containing all prompts, to that of a base LLM fine-tuned to minimize L SFT on all prompts. However, prior works have found instruction-tuning to be a necessary precursor for successful RL fine-tuning [Ouyang et al., 2022 , Touvron et al., 2023b , Rafailov et al., 2023 , Tunstall et al., 2023] , which motivates the inclusion of first step in the pipeline. However, we still want to predominantly evaluate the effectiveness of AIF, therefore we want to use majority of the prompts for AIF. We observe in Figure 6 that when we evaluate the SFT-only instruction following performance of Llama7B as instruction prompts are scaled, performance improves rapidly as instruction-tuning data is increased initially, but the improvement is minimal as the SFT data is further increased. This also corroborates observations in prior work [Zhou et al., 2023 , Touvron et al., 2023b] . Therefore, we use 10% of the available prompts for the SFT stage and the rest of them to generate the AIF dataset. This provides a strong initialization for LAIF while still leaving a large disjoint set of prompts to use for training on AI feedback.Response Pairs for AI Feedback. To sample response pairs for AI feedback, we construct preference pairs by using one completion from M teacher and one completion from π SFT . In a typical RL pipeline, the preference dataset is generated by sampling two completions from π SFT for every prompt. However, we find that using the completions from M teacher as one of the inputs in the preference pair results in better performing models. Thus, we evaluate LAIF in the most favorable possible circumstances by using this scheme to construct preference pairs. Evaluation protocol. We evaluate all the instruction-tuned models using AlpacaEval [Li et al., 2023] , where an evaluator model M eval compares the outputs of the current model with the outputs of instruction-tuned GPT-3 (text-davinci-003), and these preference labels are averaged over 805 instructions to give a win rate for the model. This evaluation protocol essentially compares how well a given model adheres to the preferences of M eval . The evaluation questions in AlpacaEval have no overlap with our training prompts from ShareGPT. To minimize discrepancy between the critic preferences and evaluation preferences, we will use the same model for both, that is M eval = M critic , reducing concerns about overoptimization [Gao et al., 2022] . Further, we also use the same prompt template for labeling training preferences and computing evaluation win rates. For all the experiments, we restrict M critic to be GPT-4 or Claude, and the prompt template for them is shown in Appendix B.Further details about dataset construction, hyperparameters for SFT and DPO can be found in Appendix A. We use DPO for all the experiments in this section for its computational simplicity and effectiveness, but we also provide additional PPO based experiments in Appendix D. Comparing ShareGPT SFT and LAIF. First, we compare fine-tuning various base LLMs with SFT on the default target completions in ShareGPT, i.e, M teacher = GPT-3.5. We use M critic = GPT-4 in Figure 2 and M critic = Claude in Figure 3 for AI feedback. We find that LAIF can substantially improve instruction following performance over instruction-tuning on GPT-3.5 across all base models. For both GPT-4 and Claude feedback, we find that instructionfollowing performance can improve by over 10% in absolute win rate. The only exception is the LAIF with GPT-4 feedback for Llama 7B where the improvement is far more modest; we investigate this outlier later in greater depth. Using completions from M critic for SFT.While LAIF provides substantial improvement over SFT, there is an important discrepancy: The default SFT samples in ShareGPT are from a relatively weaker model, while the AI feedback is collected from the stronger model M critic (GPT-4 or Claude). What if we use the stronger LLM M critic as a teacher? We evaluate LLMs fine-tuned with SFT on GPT-4 completions in Figure 2 and on Claude completions in Figure 3 , and we find that these SFT models consistently outperform or match LAIF for all base models. This strongly suggests the performance of SFT models was limited by the quality of the instruction-tuning data; accounting for the SFT distribution is important when evaluating models using AI feedback.Figure 4 : SFT on strong distribution minimizes any improvements from LAIF. We consider LAIF starting with GPT-4 completions as the target distribution for SFT in the first step, and also to generate preference labels for RL. We find surprisingly minimal improvements in performance when compared to SFT on completions sampled from GPT-4 for all prompts. This experiment suggests that SFT on the target completions from the strong critic can minimize any further gains from LAIF.We note that this phenomenon is not self-evident apriori. One would expect that a model trained with an explicitly preference-based objective, in the favorable setting where the same ranking model is used both as critic and evaluator, would perform better on the fundamentally preferencebased AlpacaEval evaluation. Instead, we find that a model trained on SFT, where the objective is less congruent with the evaluation, performs comparably or better. We offer possible explanations to this in Section 5.What if we use completions from M critic in LAIF too? A natural question to ask is how does LAIF performs when we use completions from M critic during the SFT step of LAIF (ie Step 1). In this experiment, the completions from M critic on 10% of the prompts are used to train the SFT model π SFT , and pairs of M critic and π SFT completions are used for the rest of the prompts to form the AI feedback data. These preference pairs are labeled by the same M critic . While the SFT on 10% M oracle completions is substantially better than before, we find that benefits of the complete LAIF pipeline are dramatically diminished, providing little to no benefit over SFT alone, when M critic is used as teacher too. This finding holds when using either GPT-4 in Figure 4 and Claude in Figure 7 as the teacher/critic and for various base LLMs.The results in this section suggest the following hypothesis: AI feedback is effective when there is a discrepancy between the SFT distribution and the evaluator, that is, M teacher is worse than M critic . This is implied by the observation that LAIF pipeline, with SFT targets generated by M critic , performs similarly or worse than simple SFT with M critic as the teacher too. But, the same LAIF pipeline substantially improves the model when M teacher is worse than M critic (i.e., GPT-3.5 as the teacher and GPT-4 as the critic).Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?Answer: [Yes] Justification: Section 4, Appendix A and we have provided an anonymized repository for our code.Guidelines:• The answer NA means that the paper does not include experiments.• If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not. • If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable. • Depending on the contribution, reproducibility can be accomplished in various ways.For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed. • While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. (d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility.In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results. 
possible mechanistic explanations for the ineffectiveness of laif: 
 The experiments in Section 4 suggest a surprising result that SFT on the right target distribution can match or even outperform LAIF, especially when one considers that the preferences used during training and evaluation are generated by the same model and using the same prompting template. The LAIF objective of maximizing the (implicit) reward implied by the training preferences is closely aligned with the evaluation objective of generating the preferred completion. Why then would a model fine-tuned to maximize the log-likelihood on potentially suboptimal completions outperform LAIF on this evaluation? Note, even the completions sampled directly from M critic are not necessarily optimal for the reward function implied by the evaluator, and optimizing L SFT on a finite dataset does not guarantee the recovery of the optimal policy [Ross et al., 2011] . In this section, we consider some empirically motivated hypotheses for why LAIF might be ineffective compared to SFT in our current experiments.Current base LLMs are insufficiently responsive to AI feedback. Most base LLMs improve substantially from GPT-4 feedback when instruction-tuned on GPT-3.5 completions in ShareGPT in Figure 1 , except for Llama models. Particularly for Llama 7B, where the absolute performance is also low, LAIF seems to be relatively ineffective at improving performance (see both Figure 1 and Figure 4 ). There are two possible reasons: either the preference dataset generated by Llama 7B (after SFT) is not informative enough to learn from, or the base model has limited ability to improve from RL fine-tuning. We consider the following experiment to resolve this: We fine-tune the Llama 7B SFT model with DPO on the preference dataset generated from samples from Mistral 7B, fine-tuned with SFT on GPT-3.5. Similarly, we fine-tune Mistral 7B with SFT on the preference dataset using samples from Llama 7B. Surprisingly, we find in Table 1 that performance after swapping preference datasets is close to the original performance of the models. That is, Mistral 7B performs nearly as well training on preferences generated by Llama 7B as it would training on its own preferences. This suggests that the performance of a model when learning from preferences may be tied to the base LLM itself, rather than exclusively the exploration performed during the sampling of the preference data responses. DPO fine-tunes a LLM by a solving a classification problem, where the model learns to discern preferred completions from dispreferred completions based on preference labels. Improvements in instructionfollowing performance when solving this classification performance may be tied to the underlying representation space of the models. Based on the fact that LAIF on Llama 7B does not improve substantially over SFT on GPT-3.5 responses alone, a possible hypothesis is that all the current base LLMs may provide limited improvements over GPT-4 SFT because the representation space does not improve further for instruction-following by improving the RLHF objective, even though the preference data with right base LLM can lead to more performant instruction-following model than SFT.Completions sampled from SFT models are substantially poorer than completions sampled from M oracle . Oracle models like GPT-4 and Claude are high-quality instruction-following LLMs; thus, the completions generated by these LLMs would be higher-quality than the completions generated by SFT models considered in our experiments. Assuming that M critic completions rank sufficiently higher than π SFT completions according to M eval 's ranking over all possible responses, would it better be to imitate the completions from M critic or improve the LLM via via LAIF?Consider an illustrative bandit problem in Figure 5 with a discrete action space of size 100, where the model completions from a strong teacher distribution completions (hyphenated black) rank around the 80th percentile under some 'true' reward function (light dashed line), whereas the model completions from a weak student π s (hyphenated blue) rank around the 40th percentile. The SFT policy (red) is learned by minimizing L SFT on 500,000 actions sampled from the teacher distribution. For LAIF (yellow), we sample 500,000 pairs of completions and label preferences on each pair of completions using the true reward. We fit a reward function by minimizing L BT and compute the optimal LAIF policy analytically as π r ϕ (a) ∝ π s (a) exp(r ϕ (a)/β), where r ϕ (•) is the learned reward function (light solid line), π s denotes the initial student distribution, and the temperature β = 0.1. We find that while the SFT policy has a higher variance than the LAIF policy, it has a higher expected reward by simply imitating the actions sampled from the teacher. On the other hand, we observe that while LAIF policy improves the student policy, it is still substantially suboptimal. underperforms relative to SFT on a strong teacher in a synthetic bandit problem with 100 possible actions. We assume the completions from the teacher (black) rank relatively highly (centered around 80th percentile). The improvements in LAIF (yellow) are limited because the actions sampled for labeling preferences are tied to the initial student distribution (blue). In this scenario, where the teacher distribution is sufficiently stronger than the student distribution, simple SFT on the teacher's samples (red) may be more effective than LAIF on samples from a weak student. The actions are sorted by their true reward, which is used to generate a teacher labeled preference dataset over samples from the student.We make several observations about this result.First, the student fails to outperform the teacher due to inadequate exploration; the preference dataset simply does not contain any high-quality responses to reinforce. Therefore, even though the teacher policy is suboptimal under the evaluation reward, imitating the teacher still outperforms LAIF with preferences annotated by M critic . Finally, while theoretically one could improve the LAIF policy further by setting a lower β, in practice, we are limited by the imperfect learned reward which can be exploited as the temperature is decreased.However, this experiment also suggests that LAIF may be able to learn a policy that improves over the SFT policy if the samples from the student policy include actions above the 80th percentile. Further, with the small temperature values used in practice (e.g., Rafailov et al. [2023] ), the LAIF policy may have much lower variance than policies trained with SFT. Nonetheless, if the mode is located around high reward completions, the LAIF policy can have higher expected return. In contrast, the SFT policy may end up with higher variance by virtue of minimizing an imitation loss and thus lower expected reward.3 Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material? Answer: [Yes] Justification: We have shared all the code and data used in the anonymized code repository. Guidelines:• The answer NA means that paper does not include experiments requiring code. Justification: While large language models (LLMs) are an incredibly important tool with substantial social implications, this paper provides a critical study of alignment with automatic AI feedback, which does not introduce any additional social and ethical considerations beyond those for LLMs. Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
going forward: 
 In light of our results, we suggest several points of consideration for future implementations of LAIF.Improve Datasets for Instruction-Tuning. In Section 4.2, we showed how the performance gains of LAIF over the SFT might be partially attributed to mismatch between the evaluator (M critic ) and the model used to collect the SFT data (M teacher ). Just like ShareGPT [Chiang et al., 2023] , numerous other AI-generated instruction-tuning SFT datasets such as Alpaca [Taori et al., 2023] , Self-Instruct [Wang et al., 2022] , UltraChat [Ding et al., 2023] are collected using relatively weaker GPT-3.5 models. It is important to consider and account for the instruction-tuning distribution when studying RLHF and LAIF, as our experiments show that it can drastically change the conclusions. The next generation of distilled models can also benefit from improved instruction-tuning datasets sampled from more recent and performant LLMs. Further, versioning and continual updating of these datasets will reduce drift in quality between SFT and AI feedback datasets. Some recent datasets are collecting better instruction-tuning datasets, such as the GPT4-LLM dataset Peng et al. [2023] , also demonstrating better instruction-tuned models. More such datasets would not only result in stronger SFT fine-tuned models but would also help the community study the new and existing AI feedback mechanisms, which often rely on state-of-the-art models as evaluators, in a fairer setting.Implications for RLHF. As motivated in the introduction and shown by our experiments, using LLMs for generating completions or providing feedback in the RLHF pipeline introduces new considerations for data collection and algorithms to use for optimization, specifically that SFT on the right target distribution can be much more performant. While one may be tempted to conclude that SFT may suffice in case of human completions and human feedback, the performance gains of RLHF over SFT are well documented Ouyang et al. [2022] , Bai et al. [2022] , Rafailov et al. [2023] . We postulate that this dichotomy is due to how data collection works for RLHF and LAIF. In RLHF, the completions for the SFT stage are collected by humans, whereas most LAIF instruction tuning SFT datasets use LLMs for completion generation. 4 Humans, when asked to generate completions, may not be sufficiently incentivized to generate high-quality completions for instructions, especially when collecting data at scale. For example, writing detailed and insightful answers for every question may be a demanding task, and humans may also not be sufficiently informed to write such answers. However, humans may implicitly prefer longer detailed answers [Singhal et al., 2023] , and thus, when giving feedback on completions for RLHF, may encourage such behavior. This discrepancy between the quality of human completions and what humans prefer may explain the effectiveness of RLHF over SFT.Additionally, in terms of data collection efficiency for human feedback, it is easier for humans to generate preference labels as opposed to generating completions. However, in the case AI feedback, the computational cost of collecting preferences may be higher than generating completions as the length of the context for collecting preferences is usually considerably longer than total length (context and generation) of the output while collecting completions.Is AI Feedback Simply Ineffective? In this paper, our limited claim is that adapting preference-based learning from the RLHF framework by replacing humans with AI might lead to sub-optimal outcomes from the perspectives of both downstream performance and cost-effectiveness. Our experiments suggest that the performance gains of LAIF depend upon several factors that can impact final performance in non-obvious ways, such as the base pre-trained model size and class, the SFT data distribution, and the quality of the exploration performed in the generated responses in the preference data. Nonetheless, our discussion in Section 5 suggests that if (1) preference datasets contain adequate exploration and ( 2) AI feedback provides high-quality preference labels, learning from AI feedback may be a promising and scalable approach to training capable instruction-following models. Moreover, alignment can have broader goals than improving instruction following abilities [Ouyang et al., 2022 , Bai et al., 2022] , in which case AI feedback can be leveraged for inducing safe and desirable behavior in a cheap and automated fashion. 
conclusion: 
 In this paper, we critically evaluate the current prevalent paradigm of LAIF, which replaces humans with strong LLM annotators for labeling feedback. We observe in our experiments that current gains observed by open-source LAIF models might be an artifact of the capability mismatch between the models generating SFT data and the AI feedback, and effectiveness of LAIF for strong-instruction following LLMs may be overestimated. We also showed that LAIF gains do not translate universally across models, evaluators, and oracle models. We our analysis is limited, we provide some mechanistic insight into this behavior via a simplified bandit setting. Finally, we provide some practical suggestions and discuss the implications of these results for future LAIF and RLHF research. Concretely, we recommend periodic versioning and regular updates to AI-generated instruction finetuning datasets with the release of more powerful language models. More design and investigation is needed to extract the best possible performance from LAIF methods. 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)? Answer: [NA] Justification: We do not release models. We modify publicly available instruction-tuning and preference datasets that were already available. Guidelines:• The answer NA means that the paper poses no such risks. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: A.1 Data ProcessingWe use ShareGPT [Chiang et al., 2023] for all our experiments. We filter the prompt to only single-turn conversations, which gives us about 46,000 (instruction, completion) pairs, where the completions are sampled from GPT-3.5. To preprocess the dataset, we use truncate every prompt to a maximum of 256 tokens, and truncate the completion such that the total length of the prompt and completion combined does not exceed 512 tokens. We use the following template for all our instruction-tuning experiments:\n\nHuman: {instruction}\n\nAssistant: {completion}<eos>For SFT runs, we train the models on 9 epochs and evaluate every 3 epochs. From here, we select the best checkpoint to report. We use a batch size of 8 and conduct a hyperparameter sweep for learning rate across {1e-7, 5e-7, 1e-6}. We found some models to converge faster than others. For instance, for the Mistral-7B and Mixtral-8x7B models, converged within 3 epochs, so we conducted our hyperparameter sweep over 3 epochs and evaluated the model every epoch.For Prompt template used by GPT-4 in AlpacaEval [Li et al., 2023] : 
systemIn Figure 4 , we evaluate the performance of various models when both completions and AI feedback is generated by GPT-4. We conduct a similar experiment with Claude in Figure 7 , where the a Llama 7B is trained is fine-tuned with SFT on 10% of the prompts, followed by LAIF on 90% of the prompts with AI feedback from Claude. We find that a model instruction-tuned using LAIF underperforms a model fine-tuned using SFT on Claude completions on all prompts, replicating the observation in a similar setup with GPT-4.Figure 7 : SFT on strong distribution minimizes any improvements from LAIF. We consider LAIF starting with Claude completions as the target distribution for SFT in the first step, and also to generate preference labels for RL. We find SFT on Claude completions for all prompts (SFT 100%) outperforms LAIF, where we observe no improvement from LAIF. This experiment suggests that SFT with the right target distribution can potentially minimize any further gains from LAIF, even when starting with SFT on completions from the oracle model.In this section, we present attempt to verify if our findings also hold in the RLAIF setting, i.e., where the policy is trained using an online reinforcement learning algorithm such as PPO, rather than an offline equivalent such as DPO. The standard RLAIF pipeline has three stages: 1.) SFT fine-tuning, 2.) reward training, and 3.) PPO fine-tuning. Following the same protocol as Bai et al. [2022] and ?, we adapt the standard RLHF pipeline [Christiano et al., 2017] for AIF, by replacing humans with a strong AI model, both as a teacher for SFT fine-tuning data collection and as a critic for preference collection for reward modeling. we discuss our experimental setup in detail.Data: Similar to our DPO-AIF experiments, we use the 10% prompts and completions from the ShareGPT [Chiang et al., 2023] for SFT fine-tuning, and the remaining 90% of the prompts for reward modeling and PPO fine-tuning. We use the same prompts for reward modeling and PPO fine-tuning to negate the concerns about reward overoptimization [Gao et al., 2022] .To preprocess the dataset, we use truncate every prompt to a maximum of 256 tokens, and truncate the completion such that the total length of the prompt and completion combined does not exceed 512 tokens. We use the following template for all all three stages of the PPO training.\n\nHuman: {instruction}\n\nAssistant: {completion}<eos>We limit the generations during preference data collection and the PPO training to a maximum of 512 tokens, including the prompt.SFT Fine-tuning: We start with a pre-trained Mistral [Jiang et al., 2023] model, and fine-tune it using the single-turn instructions from the ShareGPT data [Chiang et al., 2023] . We use the same SFT checkpoints for DPO and LAIF experiments to ensure fair comparison.Reward Modeling: For reward modeling, we use the same response pairs that were used for DPO training, i.e., we sample one completion in the pair from M teacher and the other from π SFT , i.e., the pair contains one completion each sampled from GPT-3.5 and from the SFT fine-tuned Mistral model. We use GPT-4 as the critic for preference data collection. We hold the last 512 preference data instances for as the validation set and use the remaining preferences for training the reward model.We trained the reward model using the Adam optimizer with the default hyper parameters, the global batch size of 8, and tried the following learning rates: {1e-4, 5e-5, 1.41e-5, 5e-6}, and trained for 5 epochs. We selected the checkpoint with the best validation accuracy. Our best reward model was trained with 5e-5 learning rate and had a validation accuracy of 84.18%. Figure 8 shows the reward modeling evaluation accuracy across steps for different learning rates. PPO Fine-Tuning: We use the same prompts and split from the reward modeling stage for PPO fine-tuning. The prompts are truncated to a maximum length of 256, and limit the total sequence length to 512. We use the last 512 prompts for validation. We trained our model with the Adam optimizer with the default hyper parameters, and used the adaptive KL controller from Ouyang et al. [2022] . We did the hyper parameter search on two dimensions, the initial KL coefficient (init_kl_coef), and the learning rate (lr), and used the reward model score for the checkpoint selection.We use LoRA for PPO fine-tuning to fit our computational constraints. We used the following hyperparameters: r = 16, α = 32, and a dropout of 0.05. 2 : AlpacaEval score delta between the SFT and the best checkpoint according to the validation reward score for various configurations tested.Results. Table 2 shows the improvement in the AlpacaEval win rate over the SFT checkpoint when fine-tuning using PPO. In our experiments, we observe that the PPO-AIF gives at most 3.36% points improvement over the SFT model. PPO-AIF yields considerably less than the improvement (3.361 points) when compared to DPO-AIF (14.43 points), though we expect that this may be an artifact of our limited computational budget. However, this is substantially less (≈ 20 points) than what we achieve by just doing SFT with a data from a strong teacher such as GPT-4. Hence, the our observation that doing SFT with a stronger teacher (GPT-4) often yields better performance than doing AIF with a stronger critic (GPT-4) on a weaker SFT model (GPT-3.5 SFT), stands in the case of PPO-AIF.• The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition. • Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material. • Theorems and Lemmas that the proof relies upon should be properly referenced.In many cases, x contains a 'system prompt' prefix that is used to steer the high-level behaviors of the language model across dialogues.Or larger sets, but for simplicity, we consider pairs here.However, the relationship between policy variance and quality depends on whether evaluation is performed by expected reward (for which a single very bad response can 'spoil' an otherwise strong model) or win rates (for which the contribution of a single very bad response to overall model scoring is limited).There are some exceptions to this such as the OpenAssistant Conversations DatasetKöpf et al. [2023]. 


Decision: The paper carefully studies the current practice of Learning from AI Feedback (LAIF) for fine-tuning large language models and show that the benefits of LAIF (distinct from learning from human feedback) are likely over-stated. All of the reviewers found the experiments comprehensive and convincing, and the paper's highlighting of the need for better Supervised Fine-Tuning datasets (should be drawn from LLMs of the same quality as the LLMs used to assign preferences) is likely to be of value to the LLM fine-tuning and distillation community.

The reviewers raised several questions that led to insightful discussions with the authors. All of the reviewers' questions were eventually answered satisfactorily, and incorporating all of the discussions into a revised version will substantially strengthen the paper.

Reviews: review: paper_summary: paper_summary: Learning from AI Feedback (LAIF) has become a popular alternative for improving the instruction-following abilities of large language models (LLMs). Despite its popularity, many unresolved questions remain regarding the actual improvements gained through LAIF. The authors address some of these questions, with a focus on where exactly the improvement in the LAIF pipeline is coming from 

The authors found that many of the improvements in LAIF are attributed to the differences in the weak teacher LLM (that provides the SFT data) and a strong critic LLM (that provides the preference data). Their empirical evidence demonstrated this issue across a wide range of base, critic, evaluator, and teacher models.

Moreover, the authors suggested two potential explanations for LAIF's ineffectiveness: either the preference dataset is not sufficiently informative, or the base model has inherent limitations. Finally, the authors offer several insightful suggestions for future research in the area of LAIF.
strength_weakness: strength_weakness: - Some of the observations in the paper are straightforward.
- A few more experiments should be included in the paper to complete some of its conclusions.
- The 10% rule doesn't always hold true. In Figure 3 and Figure 4, SFT 10% performs worse than SFT 100%. A better split could improve SFT performance when doing SFT + LAIF, which is important based on the paper's conclusions.
- The LAIF ablation experiments for addressing the LAIF ineffectiveness, either as preference data or as the base model, have issues. The author samples data from Llama or Mistral and trains on the data using Llama or Mistral as the baseline model. However, the results could suffer from overoptimization problems [1].

[1] Scaling Laws for Reward Model Overoptimization by Gao et al 2022
questions: questions: - Line 64: If the discrepancy between SFT and AI feedback is minimal, then doing SFT can suffice. I am trying to understand what this statement implies; it seems pretty straightforward. If there is no gap between the two, then you do not need the second step.
- Line 206: Missing citation, using completions from M_{teacher} as one of the inputs in the preference pair results, was observed in [1].
- Also, why not use the M_{teacher} to generate both responses to ensure that the preference data is high quality? DPO [2] mentioned this setting in the "DPO outline" section; essentially, you can train on \pi_{ref} on the preferred completions.
- Line 252: Would you agree that M_{critic} does not fully capture the quality of the preference dataset? If so, then comparing M_{critic} versus M_{teacher} is a little odd because M_{teacher} fully affects the quality of the SFT dataset, whereas M_{critic} only partially affects the quality of the dataset. If M_{critic} generated both the chosen and rejected, then you be certain that generations are high quality and the M_{critic} learning signal is good.
- Figures 4 and 7 show that the 10% threshold is ideal for certain models and settings. Have you experimented with a different percentage threshold? 
- Missing citations [3], [4], [5]

[1] Coactive Learning for Large Language Models using Implicit User Feedback by Tucker et al. ICML 2024

[2] Direct Preference Optimization: Your Language Model is Secretly a Reward Model by Rafailov et al. NeurIPS 2023

[3] Starling-7B: Increasing LLM Helpfulness & Harmlessness with RLAIF by Zhu et al. 2023

[4] UltraFeedback: Boosting Language Models with High-quality Feedback by Cui et al. 2023

[5] Peering Through Preferences: Unraveling Feedback Aquisitiosn for aligning Large Language models by Bansal et a. 2024 ICLR
limitations: limitations: Yes, the authors have adequately addressed the limitations.
paper_summary: paper_summary: The paper evaluates the extent to which AI feedback is helpful in aligning large language models (LLMs) within the commonly used two-step method of improving pre-trained LLMs. This method involves first performing supervised fine-tuning (SFT) and then fine-tuning with reinforcement learning (RL) or direct preference optimization (DPO) using preference feedback. The findings indicate that, in some cases, SFT may outperform the two-step LAIF approach.
strength_weakness: strength_weakness: - I am not entirely certain about the claim that “SFT on strong distribution minimizes any improvements from LAIF.” While this was the case for 3 out of the 4 result settings (including both figures 4 and 7), it is difficult to assert this as a general truth. Could the authors rephrase the claim to be more nuanced?
- In the analysis, the authors make general claims that may not hold true in all cases. For example, “AI feedback is effective when there is a discrepancy between the SFT distribution and the evaluator,”, the analysis lacks numerical values, and the claims are not nuanced, even though they do not hold in all cases. However, I do like the conclusion, where the authors emphasize that the claim does not hold true in all cases.
- It would be interesting to see if the same hypotheses hold in more general LLM settings, such as multi-turn instructions and multi-modal foundation models.

Minor things
- Typo in line 331 “via via LAIF”
questions: questions: - In Table 1, are the values in brackets variances or confidence intervals? How many repeated runs were conducted?
- When the paper uses phrases like “weaker SFT target distribution,” how exactly is a target distribution determined to be weaker or stronger?
- It seems that the difference in target distribution is based solely on the percentage of total examples used. It would be interesting to see if the diversity of examples affects the improvements from LAIF and to what extent. For instance, what if the total number of examples remains the same, but the diversity of examples in terms of their distribution in an embedding space is different?
limitations: limitations: - Yes, the authors have sufficiently addressed the limitations.
paper_summary: paper_summary: The paper explores the current experimental set up for learning from AI feedback (LAIF), specifically using demonstrations generated by a LLM that is weaker than the LLM used to assign preference labels both for training and evaluation. By comparing training several different base LLMs on demonstrations generated by weaker (GPT3.5) versus stronger (GPT4, Claude) LLMs both in the absence of LAIF and followed by LAIF, the results suggest that current evidence for the benefits of LAIF (noted to be distinct from learning from human feedback - LHF) is over stated. The conclusion and take away is to better construct the SFT datasets such that the available demonstrations come from LLMs of the same quality level as those used to assign preference labels.
strength_weakness: strength_weakness: - Some of the motivation for the experiment set up is incongruous. The authors motivate the paper around the LAIF paradigm (not using LLMs as human proxies for algorithm development), and call out why the claims in the paper should be considered separate from LHF. However, they also highlight the importance of having the same critic as evaluator. In practice the final evaluators for a LLM trained with LAIF methods are humans. Therefore, it would be good to have experiments where humans are the evaluators or the critic and evaluator are not the same LLM. 
- A clear distinction between LAIF and LHF is made throughout the paper to highlight that the take aways are for LAIF only. However, it seems some of the take aways discussed in "Current base LLMs are insufficiently responsive to AI feedback" (pg. 7) could apply to LHF. Using strong LLMs as a proxy for humans is valid, and when the experiments are looked at from the perspective of the LAIF as proxies for LHF, more conclusions can be drawn made. Why would the hypotheses about representational space mismatches not be relevant to LHF?
- The experiment evaluating training the base LLM on demonstrations from Claude prior to LAIF is only run on llama. It would be helpful to at least also see results for Mistral 7B so that more of a comparison can be drawn against Figure 3. 
- Different experiments are run with different numbers of base LLMs. The same base LLMs should be used across all main experiments, those whose results are in Figures 2, 3, 4, and 7. If fewer base LLMs are used in some experiments, a justification should be provided. 
- The SFT results with GPT and Claude should not be separately compared against the LAIF results using the 10% GPT-3.5 demonstrations versus the GPT-4/Claude demonstrations. 
- The story around the section "Completions samples from SFT models are substantially poorer than completions sampled from M_{oracle}" should be better motivated with a clear, strong take away.
questions: questions: - Which version of Claude was used?
- Why not use AlpacaEval 2.0? Why evaluate against GPT-3 (text-davinci-003)?
- How do these findings connect and relate to "Preference Fine-Tuning of LLMs Should Leverage Suboptimal, On-Policy Data" (https://arxiv.org/pdf/2404.14367)? 
- How did you decide which base models to use in each experiment?
limitations: limitations: The limitations are limited. There is a single sentence saying that there is only a limited analysis. However, other limitations should be discussed and addressed. For example, what are scenarios under which the analysis/results that was done may not hold? The current experimental set up assumes (implicitly) that the goal is to copy/mimic/match the performance of GPT4/Claude. However, what if someone does not want a final LLM that approximates the distribution of GPT4/Claude? What about in the base of bootstrapping versus distilling? The answers to the questions above are kind of scattered across the paper, but it would be beneficial to have a clear section discussing the boundaries of the work.

Comments: comment: Thank you for addressing my concerns and questions! I will keep my current score.
Thank you for addressing my question and concerns. I will raise my score accordingly
Thank you for addressing my concerns and questions! I will keep my current score.