A Critical Evaluation of AI Feedback for Aligning Large Language Models
Authors: Archit Sharma
Sedrick Keh
Eric Mitchell
Chelsea Finn
Kushal Arora
Thomas Kollar
Venue: NeurIPS 2024 poster (2024)
Keywords: reinforcement learning from human feedback
ai feedback
alignment
direct preference optimization
Primary area: natural_language_processing
TL;DR: SFT can be just as effective as AI feedback when using strong teacher models.

Abstract: Learning from AI feedback (LAIF) is a popular paradigm for improving the instruction-following abilities of powerful pre-trained language models. LAIF first performs supervised fine-tuning (SFT) using demonstrations from a teacher model and then further fine-tunes the model with reinforcement learning (RL) or direct preference optimization (DPO), using feedback from a critic model. While recent popular open-source models have demonstrated substantial improvements in performance from the RL step, in this paper we question whether the complexity of this RL step is truly warranted for AI feedback. We show that the improvements of the RL step are virtually entirely due to the widespread practice of using a weaker teacher model (e.g. GPT-3.5) for SFT data collection than the critic (e.g., GPT-4) used for AI feedback generation. Specifically, we show that simple supervised fine-tuning with GPT-4 as the teacher outperforms existing LAIF pipelines. More generally, we find that the gains from LAIF vary substantially across base model families, test-time evaluation protocols, and critic models. Finally, we provide a mechanistic explanation for when SFT may outperform the full two-step LAIF pipeline as well as suggestions for making LAIF maximally useful in practice.

A Critical Evaluation of AI Feedback for Aligning Large Language Models
Archit Sharma"
Sedrick Keh*
Eric Mitchell"
Chelsea Finn"
Kushal Arora
Thomas Kollar!
'Stanford University Toyota Research Institute architsh@cs. stanford....... edu
Abstract
Learning from AI feedback (LAIF) a popular paradigm for improving the instruction-followingg abilities of powerful pre-trained language models. LAIF first performs supervised fine-tuning (SFT) using demonstrations from a teacher model and then further fine-tunes the model with reinforcement learning (RL) direct preference optimization (DPO), using feedback from a critic model. While recent popular open-source models have demonstrated substantial improvements performance from the RL step, in this paper we question whether the complexity of this RL step is truly warranted for AI feedback. We show that the improvements of the RL step are virtually entirely due to the widespread practice of using a weaker teacher model (e.g. GPT-3.5) for SFT data collection than the critic (e.g.. GPT-4) used for AI feedback generation. Specifically, we show that simple supervised tunint with GPT-4 as the teacher outperforms existing LAIF pipelines. More generally, find that the gains from LAIF vary substantially across base model families, test-time evaluation protocols, and critic models. Finally, we provide a mechanistic explanation for when SFT may outperform the full two-step LAIF pipeline as well as suggestions for making LAIF maximally useful in practice. Code is available at: atttps//githin. com/ wtw.aaaaaaaraa7///// /aporrarl
Introduction
As the raw capabilities of open source large language models (LLM)) improve through pre-training at large scale [Touvron et al.] 2023a b) Jiang et al. 2023 2024 Bai et al. 2023 Bi al. 2024). methods for effectively "aligning" these models; i.e., steering them to follow user instructions effectively and safely, has garnered increasing attention. Supervised fine-tuning (SFT) with large datasets of user queries and human-written responses is one popular approach. Further refining model fine-tuned with SFT using reinforcement learning from human feedback (RLHF: Christiano et al. (2017) has been shown to further improve the quality of model responses, as judged by humans [Ouyang et al.) 2022, Stiennon et al. 2020]. However, collecting the data for SFT and RLHF is expensive, requiring human annotations for both the SFT and preference rankings over several candidate responses to a query to be used as feedback for the RL stage. Given the high cost of data collection as well as the high level of disagreement among human annotators, many works such as Bai al. 2022 and
Lee et al. 2023 have replaced the human annotators in both the SFT and RLHF stages with strong language model annotators such GPT4 (Achiam et al.] 2023] The corresponding techniques,
supervised model distillation [Taori et al. 2023 Chiang el al. 2023 Ding et al. 2023) and learning
from A/ feedback (LAIF: Bai al. 2022 Lee [2023] Cui
[0233 have proven effective
in training state-of-the-at open source models Tunstaal et al. 2023 Vison el al. 2023 Intel 2023
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

However, plugging in LLMs to substitute human annotators in the same RLHF pipeline may not be the best way to leverage such LLMs. First, LLMs are often better at generating answers for instructions than discriminative tasks (West et al., 2023], and labeling preferred answers for Al feedback an example. Further, while humans may find comparing answers incurs lower cognitive cost than producing answers themselves, for modern LLMs, the cost of generating a preference comparison or ranking may actually be higher than simply generating a demonstration, owing to the longer input context when comparing multiple responses. Therefore, while LAIF may seem like a more convenient variant of RLHF prima facie, we ask: what kind of data from strong completions for SFT feedback for RL?
Teknium 2023].
LLMs is more effective for learning cnstructionfolloviie To answer this question, we compare the effec- tiveness of the LAIF pipeline with doing SFT on demonstrations directly generated from the an- notator language model. In our experiments, align a variety of pre-trained base language mod- els, both with SFT and LAIF, using the prompts from the ShareGPT dataset [Chiang et al., [2023]. We use SFT demonstrations from three teacher language models, namely, GPT-3.5, GPT-4, and Claude, and use two strong language models as critics. GPT-4 and Claude, for collecting AI feedback. We evaluate our fine-tuned models using A Apaa [Li et al. 22023] The exper- imental setup described in further detail in Section 4 In our experiments, we find that two conditions are necessary for LAIF signif- icantly outperform SFT: (a) a sufficiently strong pre-trained base model and, (b) capability mis- match between the teucher used for the SFT data collection and the critic used for collecting Al feedback. The latter condition has surprising consequences: if the target completions are suf- ficiently performant and capabillity gap between models used for SFT and AI feedback min- imal, then simply doing SFT suffice. We observe this in Figure [ where SFT on the com- pletions generated by GPT-4 outperforms GPT- 3.5 SFT + GPT-4 feedback. This suggests that LAIF from a strong critic may be compensating for a weak teacher in popular SFT datasets like ShareGPT that are generated using GPT-3.5, and we may be overestimating the effectiveness of AI feedback. We further try to analyze these findings in a more principled way in Section [S] and provide a possible mechanistic intuition in a simplified bandit setting for why SFT might perform comparably or even outperform LAIF.
Fine-tuning Mistral 7B using ShareGPT Prompts
Alpaca Eval Winrate % (GPT-4)
65-
GFT-3.5 SFT
GFT SFT AP
GPT-4 SFT
GPT-4 SFT
GGP-44 ARl
Figure 1: Supervised fine-tuning (SFT) on strong teachers accounts for improvements from learn- ing from AT feedback (LAIF). LAIF from strong mod- els such as GPT-4 can result in substantially better instruction-followwe
M.MMs than supervised SFT alone on popular datasets such as ShareGPT [Chiang et al. 2023 constructed using GPT-3.5 completions (GPT- 3.5 SFT + GPT-4 AIF). However, simply performing SFT on completions from GPT-4 can result in a better model (GPT-4 SFT), suggesting that improvement in performance from LAIF is partly because the default ShareGPT completions are from a weak teacher (GPT- 3.5). Furthermore, LAIF (GPT-4 SFT GPT-4 AIF) does not result in significantly better model compared to GPT-4 SFT alone.
Finally, Section 6 provides practical suggestions and discussions based on the experimental and theo- retical insights in the previous sections. First, one should account for the distribution of completions in instruction-tuning datasets (Tooii et al. 2023 Chiang et al. 2023 Ding et al. 2023 before evalu- ating Al feedback based methods. Further, we should consider updated instruction-tuning datasets created using more recent state-of-the-ar art LLMs, like GPT-4 (as of February 2024). Second, while our experiments suggest that SFT on strong teacher completions can be just as effective as LAIF, we caution readers into generalizing these findings RLHF setting and highlight why preference-based RL might be superior when collecting human feedback due to inherent cognitive load of collecting demonstrations. Third, our experiments further support the importance of pre- training for effective both in terms of absolute performance and also in its ability to fine-tune from AI feedback. Finally, we also provide possible ways to improve the effectiveness of AI feedback and hypothesize how AI feedback could improve over SFT over target distributions from strong teacher LLMs. Overall, AI feedback has several desirable features that make a compelling avenue for scalable oversight and effective alignment, but current evaluation may be overestimating the

---

significance of improvements in instruction-followe for open-source LLMs and we hope that this paper will inspire investigation into more effective LAIF approaches.
2
Related Work
Modern language models rely on two-stage pipeline that first learns representations with large-scale unsupervised learning and then adapts, or fine-tunes, those representations to the task of interest; this strategy initially proved effective for isolated word embeddings [Collobert et al.] [011] and
more recently has been adopted for sequence-level representations (Devlin et al. |019| |
Radford and
Narasimhan 2018). Virtually all of the most powerful language models are pre-trained with simple maximum likelihood objective over a large, unsupervised dataset [Rafford and Narasimhan 20118) Radford et al. 2019 Brown et al. 2020 Touvron et al. 2023a Owing to the drastically smaller computational demands of fine-tuning, considerably more diversity exists in the objectives used for fine-tuning, including supervised objectives such imitation [Collobert et al., 2011 Devlin et al.
2019
Radford et al. 2019 Raffel et al.) 2020 Taori et al. 20233] and ranking losses Yuan et al.
2023 as well as reinforcement learning from heuristic rewards [Paulus et al. 2018). learned rewards
trained from human reward scores [Böhm et al. 2019) or preference annotations [Ziegler et al. 2020
Ouyang al. 2022 Bai et al. 2022 Rafailov al. 2023]
Simultaneous with the development of new fine-tuning objectives, several works have focused on the source of data used for fine-tuning, While nunan-an data has been widely used (Collobert)
ct al.
2011 I Devlin et al. 2019 Radford and Narasimhan 2018 Ouyang et al. 2022 Conover et al. more scalable synthetic data sources have recently been explored, using language models themselves to generate some or all of the fine-tuning data. Bai et al. 20022] Chiang et al. (2023),
2023
Taori et al. 2023). , Teknium [023] show that outputs generated by high-quality language model can be distilled into a smaller language model through supervised imitation. Bai et al. [2022] also
show that a large language model can effectively generate preferences over model samples. which are
ultimately used to train a reward model used for further R[.-based fine-tuning, which is referred to as
learning from AI feedback (LAIF). Tuustaal et al. 2023 Ivison al.) 20233] Intel [2023] further
explore using model-generater datasets with various combinations of supervised and reinforcement learning-basee fine-tuning objectives, suggesting that LAIF can produce improvements over purely supervised fine-tuning from Al feedback. However, notably, commonly-used datasets for supervised
training [Taori et al., 2023 Ding al. 2023, Geng et al., 2023] contain substantial amounts of while the more aecently-collecter AI datasets for LAIF contain annotations generated by the stronger GPT-4 model (Ding et al. 2023). The ramifications of this discrepancy, and how the effectiveness of LAIF is affected by the teacher and critic quality is the
data generated by G.5 OpenAl|[2223
focus of this work.
3 Algorithmic Overview of LLM Fine-tuning
The dominant approaches to fine-tuning LLMs as general dialogue agents are supervised fine-tuning (SFT) and preference- based reinforcement learning from either human or AI feedback (RLHF or LAIF, respectively).
more turns of dialogue history[ and is the target response (a sequence of tokens) for this dialogue history, which the model trained to generate. Specifically, an autoregressive language model pe parameterized by 0 is trained to minimize the negative log likelihood of the responses, given the histories:
Supervised fine-tuning. Consider dataset DSFT {xr v). where is
LSFT - (x.)) log (y x))
The targets y may be written by either humans or strong L.M teachers.
RLHF& & LAIF. Motivated by the difficulty of gathering high-quality target responses at scale from humans, an alternative to purely supervised fine-tuning collects preference annotations over pairs of candidate responses y, y to a dialogue history æ. Typically, a model fine-tuned with SFT is used to generate the pair of responses 2 and humans for RLHF or strong LLM critics for LAIF annotate
In many cases, x contains 'system prompt' prefix that is used to steer the high-level behaviors of the language model across dialogues.
"Or larger sets, but for simplicity, we consider pairs here.
3

---

which better responds to the input according to some set of criteria. We refer to the preferred response as You and the dispreferred response as yt, producing a dataset of the form D, f vi }. Using a theoretical model of human discrete choice such as the Bradley-Terry model [Bradley and Terry 1952], which relates discrete choices to implicit goodness scores of the underlying options, we can train a reward model with maximum likelihood using this preference data. For the Bradley-Terry model, the reward modeling loss is:
CBT -E (a. Po > y1 2)
(2)
-E (a.. log (ro(a, re v1)
where a denotes the sigmoid, and To denotes the reward function with parameters & to fit the goodness score of a completion y for an input prompt z. The equality between the two RHS expressions is due to choice of the Bradley-Tere -erry model. Using a reward model trained with this objective, the final step fine-tune a generative language model policy no that generates high-scoring responses. Typically, To is initialized from a model trained with SFT TSFT, and regularized to keep the KL divergence between the Tyy and SSFT small, giving the final policy search objective
max E (.y-..y- [ro y) _KL (me(-)æ)) TSFT (arr(((17)) (-1=)]]
(4)
where D a dataset of unlabeled prompts and | controls the strength of the KL. regularization.
The original form of RLHF for language models [Böhm et al., 2019. Ziegler et al.] [0200] used relatively expensive online reinforcement learning algorithms such A2C [Mnih et al. 20166] or PPO [Schulman et al. 2017). More recently, |Rafailov et al. (2023) show that the optimal policy for the learned reward can be extracted in closed form, essentially skipping the need perform iterative, approximate policy learning. The resulting algorithm, direct preference optimization (DPO), simpler to tune and less computationally demanding than prior methods, while optimizing the same objective. We therefore use DPO as the algorithm for LAIF in our experiments. The DPO loss for the language model policy Tg
To (y x)) #SFT(ye / xr)
( x))
LDoO - -. log BBog
B log
#sFT (y/ xx)
which trains the language model policy no directly, without the need to first train separate reward model rg.
4 Experiments
In light of increasing usage of LAIF, the primary question we want to investigate is whether preference- based RL from Al feedback, produced by blackbox LLMs, like GPT-4 more effective at aligning language models compared to simple supervised fine-tuning on completions from these LLMs. To this end, we first explain the experimental setup for comparing LAIF and SFT in 4.1 With this experimental setup, observe in 4.2 that LAIF result in substantially more capable models than those obtained by instruction-tuning on current public SFT datasets. However, we also find that using stronger teacher for SFT consistently matches or outperforms LAIF. This surprising result spurs deeper investigation into effectiveness of LAIF, where we find that using a stronger SFT distribution also limits any further gains from Al feedback.
4.1 LAIF Setup and Data Scaling for SFT
Datasets & Models. To systematically compare SFT and LAIF on their effectiveness for instruction following, it is important to control the dataset of instructions used to train both the methods. To this end, we fix the dataset of prompts to be single-turn instructions derived from ShareGPT [Chiang et al.
2023
We consider a variety of performant pre-trained LLMs for our experiments: Llama Touvron
et al. 2023a b). Mistral [Jiang et al. 2023 and Mixtral (Jiang et al. 2024 Yi [01.A] 2023 and
DeepSeek Bi et al. 2024
Terminology. We will use the strong LLMs like GPT-4 three different roles: as a teacher where the L.MM generates target completion for an instruction, as critic where the LLM gives feedback on which completion prefers for given instruction, and similarly as an evaluator where it labels the preferred completion held out set of instructions.

---

Fine-tuning Base LLMs on ShareGPT Prompts (GPT-4 version)
Alpaca Eval Winrate % (Measured by GPT-4)
80
SFT w/ GPT-3:5 Idefault)
w/ GPT-4 preferences
70
GPT-4
65
60
50
40
DeepSeek 7B
Yi 6B
LLaMA1 7B
LLaMA2 138
Mistral 7B
Mixtral 8x7B
Figure 2: SFT can perform comparably or better than LAIF across various model sizes and classes. The same set of prompts are used for all three settings and for each model, and the oracle LLM either generates a completion (for SFT) or a preference label on two completions (for LAIF). For LAIF, SFT an important precursor, on the remaining 90%. For the other settings, the full set of prompts are used for SFT. While LAIF improves the performance compared to SFT on the default ShareGPT completions, SFT on GPT-4 completions consistently matches or outperforms LAIF.
SFT on 10% of the total prompts, and LAIF is done
Setup. When fine-tuning these pre-trained models with SFT, we consider the default target distribution in ShareGPT, which is sampled from GPT-3.5 5 as the teacher. We also generate completions from GPT- 4 (gpt-4-0314) and Claude (claude-vil v1) for the same set of instructions For LAIF, we consider the following pipeline:
1. Split the training prompts for SFT and LAIE. Fine-tune the base model on completions from teacher Muencher to minimize LSFT-
2. For prompts in the AIF split, create a dataset by sampling two completions for every prompt and label the preferred completion using Meriiti create D,,-
3. Fine-tune the instruction-tuned model using LDPO-
Prompt split for SFT and AIF. For controlled study comparing AIF and SFT, one would ideally compare the performance of base LLM fine-tuned to minimize LDPO on preference dataset Dp containing all prompts, to that of base LLM fine-tuned to minimize LSFT on all prompts. However, prior works have found instruction-tuning to be a necessary precursor for successful RL fine-tuning [Ouyang et al.. 2022 Touvron al., 2023b Rafailov et al. 2023 Tunstall al. 2023), which motivates the inclusion of hirst step in the pipeline. However, we still want to predominantly evaluate the effectiveness of AIF, therefore we want to use majority of the prompts for AIF. We observe in Figure 6 that when we evaluate the SFT-only instruction following performance of Llama7B as instruction prompts are scaled, performance improves rapidly as instruction-tunin data is increased initially, but the improvement is minimal as the SFT data is further increased. This also corroborates observations in prior work [Zhou et al. [023 Touvron et al.) 2023b]. Therefore, we use 10% of the available prompts for the SFT stage and the rest of them to generate the AIF dataset. This provides a strong initialization for LAIF while still leaving a large disjoint set of prompts to use for training on AI feedback.
Response Pairs for AI Feedback. To sample response pairs for AI feedback, we construct preference pairs by using one completion from Mwecher and one completion from Wser- In a typical RL pipeline, the preference dataset is generated by sampling two completions from नşF for every prompt. However, we find that using the completions from Miescher as one of the inputs in the preference pair results in better performing models. Thus, we evaluate LAIF in the most favorable possible circumstances by using this scheme to construct preference pairs.
5

---

Evaluation protocol. We evaluate all the Instruction-tunee models using AlpacaEval (Li et al. 2023]. where an evaluator model Meval compares the outputs of the current model with the outputs of instruction-tunet GPT-3 instructions to give a win rate for the model. This evaluation protocol essentially compares how well given model adheres to the preferences of Meval- The evaluation questions in AlpacaEval have no overlap with our training prompts from ShareGPT. To minimize discrepancy between the critic preferences and evaluation preferences, we will use the same model for both, that is Meval Meeitiee reducing concerns about overoptimization [Gao et al. 2022]. Further, we also use the same prompt template for labeling training preferences and computing evaluation win rates. For all the experiments, we restrict Menti to be GPT-4 or Claude, and the prompt template for them is shown in Appendix B
and these preference labels are averaged over 805
Further details about dataset construction, hyperparameters for SFT and DPO can be found in Appendix A We use DPO for all the experiments this section for its computational simplicity and effectiveness, but we also provide additional PPO based experiments in Appendix D
4.2 Fine-tuning under Different Target Distributions
Comparing ShareGPT SFT and LAIF, First, we compare fine-tuning various base LLMs with SFT on the default target completions in ShareGPT, i.e Micactee = GPT-3.5. We use Martte GPT-4 in Figure 2 and MMrntic Claude in Figure Al feedback. We find that LAIF can substantially improve instruction fol- lowing performance over instruction-tunie tunning on GPT-3.5 across all base models. For both GPT-4 and Claude feedback, we find that instruction- following performance can improve by over 10% in absolute win rate. The only exception is the LAIF with GPT-4 feedback for Llama 7B where the improvement is far more modest; we investigate this outlier later in greater depth. Using completions from Meritie for SFT. While LAIF provides substantial improvement over SFT, there is an important discrepancy: The default SFT samples in ShareGPT are from a rel- atively weaker model, while the AI feedback is collected from the stronger model Maritic (GPT-4 or Claude). What we use the stronger LLM Meritic as teacher? We evaluate LLMs fine-tuned with SFT on GPT-4 completions in Figure and on Claude completions in Figure 3 and we find that these SFT models consistently outperform or match LAIF for all base models. This strongly suggests the performance of SFT models limited by the quality of the instruction-tune for the SFT distribution is important when evaluating models using Al feedback.
Fine-tuning Base LLMS on ShareGPT Prompts (Claude version)
ul Glale moce
of
7B Figure 3: We make a similar observation that SFT performs comparably to LAIF when Claude as is used an oracle. LAIF with AT feedback from does not significantly outperform SFT on Claude comple- tions, and the performance improvement from LAIF is explained by the use of a weaker SFT target distribution (GPT-3.5). The results for effectiveness of SFT may apply more generally to strong LLMs bcyond GPT-4.
data; accounting
We note that this phenomenon is not self-evident apriori. One would expect that a model trained with an explicitly preference-bai based objective, in the favorable setting where the same ranking model is used both as critic and evaluator, would perform better on the fundamentally preference- based AlpacaEval evaluation. Instead, we find that a model trained on SFT, where the objective is less congruent with the evaluation, performs comparably or better. We offer possible expla- nations to this in Section
ng
Stronger [GPT-41
SFT kl GPT4 1200% of total ecumpies)
Alpaco Sone Untroza %
GPT4 evamplest
perternces
Nistral 7B
Figure 4: SET on strong distribution minimizes any improvements from LAIE We consider LAIF starting with GPT-4 completions as the target distribution for SFT in the first step, and also to generate preference la- bels for RL. We find surprisingly minimal improvements performance when compared to SFT on completions sampled from GPT-4 for all prompts. This experiment suggests that SFT on the target completions from the strong critic can minimize any further gains from LAIF
What if we use completions from Meriti in LAIF too? A natural question to ask is how does LAIF performs when we use completions from Meniti during the SFT step of LAIF (ie Step 1). this experiment, the completions from Mantic 10% of the prompts are used to

---

train the SFT model USFT, and pairs of Mesui and TSFT completions are used for the rest of the prompts to form the AI feedback data. These preference pairs are labeled by the same Manitic- While the SFT on 10% Muracle completions is substantially better than before, we find that benefits of the complete LATF pipeline are dramatically diminished, providing little to no benefit over SFT alone, when Meritte is used as teacher too. This finding holds when using either GPT-4 in Figure 4 and Claude in Figure 7 as the teacher/critic and for various base LLMs.
The results in this section suggest the following hypothesis: AI feedback is effective when there is a discrepancy between the SFT distribution and the evaluator, that is, Muscher is worse than Mentie- This is implied by the observation that LAIF pipeline, with SFT targets generated by MMentie performs similarly or worse than simple SFT with Mentte as the teacher too. But, the same LAIF pipeline substantially improves the model when Meecher is worse than Mariti (i.c., GPT-3.5 as the teacher and GPT-4 as the critic).
Possible Mechanistic Explanations for the Ineffectiveness of LAIF
The experiments in Section 4 suggest a surprising result that SFT on the right target distribution can match or even outperform LAIF, especially when one considers that the preferences used during training and evaluation are generated by the same model and using the same prompting template. The LAIF objective of maximizing the (implicit) reward implied by the training preferences is closely aligned with the evaluation objective of generating the preferred completion. Why then would a model fine-tuned to maximize the log-likelihood
potentially suboptimal completions outperform LAIF on this evaluation? Note, even the completions sampled directly from Menise are not necessarily optimal for the reward function implied by the evaluator, and optimizing Lsrr on finite dataset does not guarantee the recovery of the optimal policy [Ross et al. (201T. In this section, we consider some empirically motivated hypotheses for why LAIF might be ineffective compared to SFT in our current experiments.
Current base LLMs are insufficiently responsive to AI feedback. Most base LLMs improve substantially from GPT-4 feedback when instruction-tunen on GPT-3.5 completions in ShareGPT in Figure D except for Llama models. Particularly for Llama 7B, where the absolute performance is also low, LAIF seems to be relatively ineffective at improving performance (see both Figure T and Figure 4 There are two possible reasons: either the preference dataset generated by Llama 7B (after SFT) is not informative enough to learn from, or the base model has limited ability to improve from RL fine-tuning. We consider the following experiment to resolve this: We fine-tune the Llama 7B SET model with DPO on the preference dataset generated from samples from Mistral 7B, fine-tuned with SFT on GPT-335. Similarly, we fine-tune Mistral 7B with SFT on the preference dataset using samples from Llama 7B. Surprisingly, we find in Table I that performance after swapping preference datasets is close the original performance of the models. That is. Mistral 7B performs nearly as well training on preferences generated by Llama 7B as it would training on its own preferences.
This suggests that the performance of model when learning from preferences may be tied to the base LLM itself. rather than exclusively the exploration performed during the sampling of the preference data responses. DPO fine-tunes a LLM by a solving a classification problem, where the model learns discern preferred com- pletions from dispreferred completions based on preference labels. Improvements in instruction- following performance when solving this clas- sification performance may be tied to the under- lying representation space of the models. Based on the fact that LAIF on Llama 7B does not im- prove substantially over SFT on GPT-3.5 responses alone, possible hypothesis is that all the current base L.MSs may provide limited improvements over GPT-4 SFT because the representation space does not improve further for instruction-follovinn by improving the RLHF objective, even though the preference data with right base I.LM can lead to more performant cnstruction-followiii model than SFT.
Preference Distribution D, Llama 7B
Mistral 7B
Base LLM
Llama 7B
45.0 (1.76)
47.6 (1.76)
Mistral 7B
68.5 (1.64)
71.1 (1.60)
Table I: LAIF with preference data responses sampled from different model than the base model being fine- tuned. We find that the final performance after fine- tuning is affected more by the choice of the base LLM, as Mistral 7B reaches a similar performance when fine- tuning on preferences over Llama 7B responses, whereas Llama7B does not improve significantly when trained on preferences over responses generated by Mistral 7B.
7

---

Completions sampled from SET models are substantially poorer than completions sampled from Moracl- Oracle models like GPT-4 and Claude are high-quality instruction-folee following LLMs; thus. the completions generated by these L.LSs would be higher-quality than the completions generated by SFT models considered in our experiments. Assuming that Meniie completions rank sufficiently higher than "SFT completions according to .Meva's 's ranking over all possible responses, would it
better be to imitate the completions from Merite or improve the LLM via via LAIF?
Consider an illustrative bandit problem in Figure 5 with a discrete action space of size 100, where the model completions from a strong teacher distribution completions (hyphenated black) rank around the 80th percentile under some 'true' reward function (light dashed line). whereas the model completions from a weak student ne (hyphenated blue) rank around the 40th percentile. The SFT policy (red) is learned by minimizing LSFT on 500,000 actions sampled from the teacher distribution. For LAIF (yellow), we sample 500,000 pairs of completions and label preferences on each pair of completions using the true reward. We fit a reward function by minimizing LET and compute the optimal LAIF policy analytically as Tr, (a) X T. (a) (a)(((())))) where (-)-) the learned reward function (light solid line), T. denotes the initial student distribution, and the temperature 3 = 0.1. We find
that while the SFT policy has a higher variance than the LAIF policy, it has a higher expected reward by simply imitating the actions sampled from the teacher. On the other hand, we observe that while LAIF policy improves the student policy, it is still substantially suboptimal.
We make several observations about this result. First, the student fails to outperform the teacher due to inadequate exploration; the preference dataset simply does not contain any high-quality responses to reinforce. Therefore, even though
Strong Teacher
Weak Student
25
Policy Probability Mass
0.10
RLAIF
SFT
the teacher policy is suboptimal under the eval- uation reward, imitating the teacher still out- performs LAIF with preferences annotated by Manite- Finally, while theoretically one could improve the LAIF policy further by setting a lower 3, in practice, we are limited by the im- perfect learned reward which can be exploited as the temperature is decreased.
0.08
Learned Reward True Reward
2.0
0.06
155
0.04
1.0
0.02
0.5
0.00
0.0
0
20
40 60
80
100
Action Index (IA) 100)
However, this experiment also suggests that LAIF may be able to learn policy that improves over the SFT policy if the samples from the stu- dent policy include actions above the 80th per- centile. Further, with the small temperature val- used in practice (e.g.. Rafailov et al. (2023), D. the LAIF policy may have much lower variance than policies trained with SFT. Nonetheless, if the mode is located around high reward comple- tions, the LAIF policy have higher expected return. In contrast, the SFT policy may end up with higher variance by virtue of minimizing an imitation loss and thus lower expected reward [
Figure 5: Fine-tuning a weak student with LAIF underperforms relative SFT on strong teacher in a synthetic bandit problem with 100 possible actions. We assume the completions from the teacher (black) rank relatively highly (centered around 80th percentile). The improvements in LAIF (yellow) are limited hecause the actions sampled for labeling preferences are tied to the initial student distribution (blue). In this scenario, where the teacher distribution is sufficiently stronger than the student distribution, simple SFT on the teacher's 's samples (red) may be more effective than LAIF on samples from a weak student. The actions are sorted by their true reward, which is used to generate a teacher labeled preference dataset over samples from the student.
6 Going Forward
In light of our results, we suggest several points of consideration for future implementations of LAIF
Improve Datasets for cirruction-Tuningg In Section 4.2 we showed how the performance gains of LAIF over the SFT might be partially attributed to mismatch between the evaluator (Merine) and the model used to collect the SFT data (Musschr)). Just like ShareGPT [Chiang et al. 2023
numerous other Al-generated instruction-tuning SFT datasets such as Alpaca [Taori et al. 2023
Ienf-untt (Wang et al. 2022]. UltraChat (Ding al.
2023 are collected using relatively weaker
Howeeer, the relationship between policy variance and quality depends on whether evaluation is performed by expected reward (for which single very bad response can 'spoil' an otherwise strong model) or win rates (for which the contribution of a single very bad response to overall model scoring limited).
8

---

GPT-3.5 models. It is important to consider and account for the instruction-tuna distribution when studying RLHF and LAIF, as our experiments show that it can drastically change the conclusions. The next generation of distilled models can also benefit from improved instruction-tunigg datasets sampled from more recent and performant L.MMs. Further, versioning and continual updating of these datasets will reduce drift in quality between SFT and Al feedback datasets. Some recent datasets are collecting better instruction-tumme tuuning datasets, such as the GPT4-LLM dataset Peng et al. [2023), also demonstrating better instruction-tunee models. More such datasets would not only result in stronger SFT fine-tuned models but would also help the community study the new and existing AI feedback mechanisms, which often rely ofate-of-the-ar models evaluators, in a fairer setting.
Implications for RLHF. As motivated in the introduction and shown by our experiments. using LLMs for generating completions or providing feedback in the RLHF pipeline introduces new considerations for data collection and algorithms to use for optimization, specifically that SFT on the right target distribution can be much more performant. While one may be tempted to conclude that SFT may suffice in case of human completions and human feedhack, the performance gains of (uuuunn
RLHF over SFT are well documented Ouyang et al. [2022] Bai et al. 20222, Rafailov et al. [2023]. We postulate that this dichotomy is due to how data collection works for RLHF and LAIF. In RLHE, the completions for the SFT stage are collected by humans, whereas most LAIF instruction tuning SET datasets use LLMs for completion generation. 4 Humans, when asked to generate completions, may not be sufficiently incentivized to generate nigh-quality completions for instructions, especially when collecting data at scale. For example, writing detailed and insightful answers for every question may be demanding task, and humans may also not be sufficiently informed to write such answers. However, humans may implicitly prefer longer detailed answers (Singhal et al. 2023), and thus, when giving feedback on completions for RLHF, may encourage such behavior. This discrepancy between the quality of human completions and what humans prefer may explain the effectiveness of RLHF over SFT.
Additionally, in terms of data collection efficiency for human feedback, it easier for humans to generate preference labels as opposed to generating completions. However, in the case A/ feedback, the computational cost of collecting preferences may be higher than generating completions as the length of the context for collecting preferences is usually considerably longer than total length (context and generation) of the output while collecting completions.
Is AT Feedback Simply Ineffective? In this paper, our limited claim is that adapting preference- learning from the RLHF framework by replacing humans with Al might lead to sub-optimal outcomes from the perspectives of both downstream performance and cost-effictivveness. Our experiments suggest that the performance gains of LAIF depend upon several factors that can impact final performance in non-obvious ways, such as the base pre-trained model size and class, the SFT data distribution, and the quality of the exploration performed the generated responses in the preference discussion in Section 5 suggests that if (1) preference datasets contain
data. Nonetheless, adequate exploration and (2) AI feedback provides high-quality preference labels, learning from AI feedback may be promising and scalable approach to training capable cnstruction-olllwiinn models. Moreover, alignment can have broader goals than improving instruction following abilities (Ouyang et al.) 2022 Bai et al.) 2022]. in which case AI feedback can be leveraged for inducing safe and desirable behavior in a cheap and automated fashion.
7 Conclusion
In this paper, we critically evaluate the current prevalent paradigm of LAIF, which replaces humans with strong LLM annotators for labeling feedback. We observe in our experiments that current gains observed by open-source LAIF models might be an artifact of the capability mismatch between the models generating SFT data and the AI feedback, and effectiveness of LAIF for strong-instruction following LLMs may be overestimated. We also showed that LAIF gains do not translate univer- sally across models, evaluators, and oracle models. We our analysis is limited, we provide some mechanistic insight into this behavior via simplified bandit setting. Finally, we provide some practical suggestions and discuss the implications of these results for future LAIF and RLHF research. Concretely, we recommend periodic versioning and regular updates to Al-generated instruction fine- tuning datasets with the release of more powerful language models. More design and investigation is needed to extract the best possible performance from LAIF methods.
'There exceptions this such the OpenAssistant Conversations Dataset Köpf al.
2023

---

8 Acknowledgements
We would like to thank Adrien Gaidon, Rares Ambrus, Achal Dave, Jean Mercat and Igor Vasiljevic for their insight and advice during the early development of this work.
10

---

References
0L.II. Yr: Building the next generation of open-source and bilingual 1lms. 2023. URL https
//github. com/01-ai/y
J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt,
S. Altman, Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 2023.
Bai, S. Bai, Y. Chu, Z. Cui, K. Dang, Deng, Fan, W. Ge, Y. Han, F. Huang, et al. Qwen technical report. arXiv preprint arXiv:2309.166000 2023.
Y. Bai, A. Jones, K. Ndousse, A. Askell, A. Chen, N. DasSarma, D. Drain, S. Fort, D. Ganguli, T. Henighan, et al. Training helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint wixx:i220.008000 2022.
X. Bi, D. Chen, G. Chen, S. Chen, D. Dai, C. Deng, H. Ding, K. Dong, Du, Z. Fu, et al. Deepseek I1m: Scaling open-source language models with longtermism. arXiv preprint arXiv:2401.02954. 2024.
F. Böhm, Y. Gao, C. M. Meyer, O. Shapira, I. Dagan, and I. Gurevych. Better rewards yield better summaries: Learning to summarise without references.
K. Inui, Jiang, Ng, and X. Wan, editors. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 31 3110-3120, Hong Kong. China, Nov. 2019. Association for Computational Linguistics. doi: 10.186533v1/D19-111
URL httpp::/aclaatthlll.e www.ttooooooooooooooooo-o org//19-1307
R. A. Bradley and M. Terry. Rank analysis of incomplete block designs: the method of paired comparisons. Biomeirika, 39(3/4):324-344 1952. doi:
T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, Askell, S. Agarwal, A. Herbert- Voss. G. Krueger, Henighan, R. Child, A. Ramesh, D. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. Chess, J. Clark, C. Berner, S. McCandlish, A. Radford, I. Sutskever, and D. Amodei. Language models are few-shot learners. In H. Larochelle, M. Ranzato, R. Hadsell, M. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 1877-1901. Curran As- neurips wt/pcc/pp/aae...ccee f files/ www.ssspaeeee//a//ea paper/20201 Paper.
sociates, Inc., 2020. https::////::ttttteeeeee Paper..daappppppppp pdf
W.-L. Chiang, Z. Li, Z. Lin, Y. Sheng, Z. Wu, H. Zhang, L. Zheng, S. Zhuang, Y. Zhuang, J. Gonzalez, et al. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality. See hups://vicuna. Imsys. org (accessed 14 April 2023), 2023.
F. Christiano, J. Leike, T. Brown, M. Martic, S. Legg, and D. Amodei. Deep reinforcement learning from human preferences. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017. URL https: / htttps://prooceedingg www..ps..r.cmmm cctpppppprrcpprr..pppppppppp
hhie:ppppeeo......cceccaaaaaaaee Pape Paper.
R. Collobert, Weston, L. Bottou, M. Karlen, K. Kavukcuoglu, and P. Kuksa, Natural language processing (almost) from scratch. J. Mach. Learn Res., 124m nov 2011. ISSN 1532-4435.
M. Conover, M. Hayes, Mathur, Xie, J. Wan, S. Shah, Ghodsi, Wen- dell, M. Zaharia, and R. Xin. Free dolly: Introducing the world's first truly open instruction-tuned 11m, 2023. URL https: //www www.wrdaaaabccc.coo com////lgg///22/////22// dolly-first-opent www.-roooeeree.eme mimeerciaallnn lyaiaalllllleeeeeeaaaeeaeaaaelle www.llaableeeeeeee
11m
G. Cui, I.. Yuan, N. Ding, G. Yao, W. Zhu, Y. Ni, G. Xie, Z. Liu, and M. Sun. Ultrafeedback: Boosting language models with high-quality feedback. arXiv preprint arXiv:2310.01377, 2023.
Devlin, M.-W. Chang, K. Lee, and K. Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In J. Burstein, C. Doran, and T. Solorio, editors, Proceedings of the 2019 Conference of the North American Chapter of the Association for Compu- tational Linguistics: Human Language Technologies. Volume I (Long and Short Papers), pages
11

---

4171-4186, Minneapolis. Minnesota, June 2019. Association for Computational Linguistics. doi: 10.18653/v1/N11-1444 /RL. org -wwg/NN9-20
1423
N. Ding, Chen, Xu, Y. Qin, Z. Zheng, Hu, Liu, Sun, and Zhou. Enhancing chat lan- guage models by scaling high-quality instructional conversations, arXiv preprini www.iv:2305.14235: 2023.
L. Gao, J. Schulman, and J. Hilton. Scaling laws for reward model overoptimization, 2022.
X. Geng, A. Gudibande, H. Liu, E. Wallace, P. Abbeel, S. Levine, and D. Song. Koala: A dialogue model for academic research. Blog post, April 2023. URL httph://baaatt betpe///bab.bbbeeleere.e. berkeley edu/ htte//2223////////ee//
Intel.
Supervised
fine-tuning and direct preference optimization on intel
gaudi2. 2023.
URL
the pe-pppaaccicoooo stwpree-----uupeeidee--------e
and weapiing-nndd--deeeedeeeee ont.a
hww--aabaaaaaaaana gaudi2-as a11
H. Ivison, Y. Wang. V. Pyatkin, N. Lambert, M. Peters, P. Dasigi, J. Jang, D. Wadden, N. A. Smith, Beltagy, et al. Camels in a changing climate: Enhancing Im adaptation with tulu 2. arXiv
preprint arXiv22311.10702 2023.
A. Q. Jiang, A. Sablayrolles, A. Mensch, C. Bamford, D. Chaplot, D. d. I. Casas, F. Bressand, G. Lengyel, G. Lample. L. Saulnier, et al. Mistral 7b. arXiv preprint arXiv:2310.06825. 2023,
A. Q. Jiang, A. Sablayrolles, A. Roux, A. Mensch, B. Savary, C. Bamford, D. S. Chaplot, D. d. I. Mixtral of experts. arXiv preprint arXiv:2401.0400 2024.
Casas, E. Hanna, F. Bressand,
A. Köpf, Y. Kilcher, D. von Rütte, S. Anagnostidis, Z.-R. Tam, K. Stevens, A. Barhoum, N. M. Duc, O. Stanley, R. Nagyfi, S. ES, S. Suri, D. Glushkov, A. Dantuluri, A. Maguire, C. Schuhmann, H. Nguyen, and A. Mattick. Openassistant conversations democratizing large language model alignment, 2023.
H. Lee, S. Phatale, H. Mansoor, K. Lu, T. Mesnard, C. Bishop, V. Carbune, and A. Rastogi. Rlaif: Scaling reinforcement learning from human feedback with ai feedback. arXiv preprint arXiv:2309.0000 00267, 2023.
X. Li, T. Zhang, Y. Dubois, Taori, I. Gulrajani, C. Guestrin, P. Liang, and T. B. Hashimoto. Alpacaeval: An automatic evaluator of esstuction-following models. https: thub. com/ tatsu tatsu-lab/alpan evall 2023.
V. Mnih, A. P. Badia, M. Mirza, A. Graves, T. Lillicrap. T. Harley, D. Silver, and K. Kavukcuoglu. Asynchronous methods for deep reinforcement learning. In M. F. Balcan and K. Q. Weinberger, editors, Proceedings of The 33rd International Conference on Machine Learning, volume 48 of Proceedings of Machine Learning Research, pages 1928 1937, New York, New York, USA, 20-22 Jun 2016. PMLR. URL
htwp://innnnnii...e///eeeeeeeeee mlr prwss Stt.arprreevvv///mmeeeee htt/v/////m/////// html
OpenAl. Introducing ChatGPT - openai.com. http::///opeai..oom//e chattpt 2023. [Accessed 30-01-2024].
L. Ouyang, Wu, Jiang, D. Almeida, C. Wainwright, Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, et al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35:27730-27744 2022.
R. Paulus, C. Xiong, and R. Socher. A deep reinforced model for abstractive summarization. In International Conference on Learning Representations, 2018. URL https:://ppenreriirr net/ Corum?id=HkACIne
B. Peng, C. Li, P. He, M. Galley, and J. Gao. Instruction tuning with gpt-4, 2023.
A. Radford and K. Narasimhan. Improving language understanding by generative pre-training. 2018. URL http::tttt:/aaat semanticscholar wwtp:/orgs/////eeee 49313245
A. Radford, Wu, R. Child, D. Luan, Amodei, and Sutskever. Language models are unsupervised multitask learners. 2019.
12

---

R. Rafailov, A. Sharma, E. Mitchell, S. Ermon, C. D. Manning, and C. Finn. Direct preference optimization: Your language model is secretly reward model. arXiv preprint wrXiv:2305.18290 2023.
C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y. Zhou, W. Li, and P. J. Liu. Exploring the limits of transfer learning with unified text-to-text transformer. J. Mach. Learn. Res., 21(1), jan 2020. ISSN 1532-4435.
S. Ross, G. Gordon, and Bagnell. A reduction of imitation learning and structured prediction to no-regret online learning. In Proceedings of the fourteenth international conference on artificial intelligence and statistics, pages 627-635. JMLR Workshop and Conference Proceedings, 2011.
J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv: 170.003377 2017.
Singhal, Goyal, J. Xu, and G. Durrett. A long way to go: Investigating length correlations in rlhf. arXiv preprint arXiv:2310.337/00 2023.
N. Stiennon, Ouyang, J. Wu, D. Ziegler, R. Lowe, C. Voss, A. Radford, D. Amodei, and P. F. Christiano. Learning summarize with human feedback. Advances in Neural Information Processing Systems, 33:3008-3021, 2020.
R. Taori, Gulrajani, T. Zhang, Dubois, X. Li, C. Guestrin, P. Liang, and T. B. Hashimoto. Stanford alpaca: An instruction-following llama model, 2023.
Teknium. 0penherress2..-mmisttt77bb (penlierrmes-2.5-MMsttalll
2023.
URL http::/huggingfccc.cooeeeees
H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozière, N. Goyal, E. Hambro, F. Azhar, al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302. 13971, 2023a.
H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y. Babaei, N. Bashlykov, S. -Batra, P. Bhargava, S. Bhosale, al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint wxXv:23377.09288, 2023b.
L. Tunstall, E. Beeching, N. Lambert, N. Rajani, K. Rasul, Belkada, Huang, L. von Werra, C. Fourrier, N. Habib, et al. Zephyr: Direct distillation of Im alignment. arXiv preprint arXiv:2310.16944 2023.
Y. Wang, Y. Kordi, S. Mishra, A. Liu, N. A. Smith, D. Khashabi, and H. Hajishirzi. Self-instruct: Aligning language model with self generated instructions, 2022.
West, X. Lu, N. Dziri, Brahman, Li, J. D. Hwang, Jiang, J. Fisher, A. Ravichander, K. Chandu, et al. The generative ai paradox:" what can create, it may not understand". arXiv preprint arXiv:2311.0003 2023.
H. Yuan, 7. Yuan, C. Tan, W. Wang, S. Huang, and F. Huang. RRHF: Rank responses align language models with human feedback. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL htttt:://ppenreeee wwt.e//wwww..eeoooorrreeaaaaaaaa
C. Zhou, P. Liu, P. Xu, S. Iyer, J. Sun, Y. Mao, X. Ma, A. Efrat, P. Yu, L. Yu, et al. Lima: Less is more for alignment. arXiv preprint arXiv:2305.11206 2023.
D. M. Ziegler, N. Stiennon, J. Wu, T. Brown, A. Radford, D. Amodei, Christiano, and G. Irving. Fine-tuning language models from human preferences, 2020.
13

---

A Detailed Experimental Setup and Hyperparameters
A.1 Data Processing
We use ShareGPT [Chiang al. 2023] for all experiments. We filter the prompt to only single-turn conversations, which gives us about 46,000 (instruction, completion) pairs, where the completions are sampled from GPT-3.5. To preprocess the dataset, use truncate every prompt to maximum of 256 tokens, and truncate the completion such that the total length of the prompt and completion combined does not exceed 512 tokens. We the following template for all our instruction-tuning experiments:
An An\nHuan:: (instruction) \n lnlnssssttnn:: {completion))eoss)
A.2 Training Settings and Hyperparameters
For SFT runs, we train the models on 9 epochs and evaluate every epochs. From here, we select the best checkpoint to report. We use a batch size of 8 and conduct a hyperparameter sweep for learning rate across (1e-7, 5c-7, lc-6]. We found some models converge faster than others. For instance, for the Mistral-7B and Mixtral-8x7B models, converged within 3 epochs, we conducted our hyperparameter sweep over epochs and evaluated the model every epoch.
For DPO, we select the best SFT checkpoint and train on top of it. We do DPO training for 3 epochs and evaluate every epoch. Learning rate for DPO runs is fixed at 5e-7 and beta at 0.05 for all models. We have also released the code and datasets for reproducing the experiments. Training was done on A A100 80GB instances and took around hour per epoch for 7B model when trained 100% of the training examples.
B Preference Labeling and Evaluation Prompt Templates
Prompt template used by GPT-4 in AlpacaEval
Li ct al. 2023
<I <mm_startt |>system
You are a helpful instruction-folloring assistant.
< imimeddd>>
iii_start/>user
Select the output (a) (b) that best matches the given instruction. Choose your preferred output, which can be subjective. Your answer should ONLY contain: Output (a) Output (b). Here's example:
Example:
## Instruction:
Give description of the following job: "ophthalmologist"
## Output (a)
An ophthalmologist is a medical doctor who specializes in the diagnosis \ and treatment of eye diseases and conditions.
## Output (b):
An ophthalmologist is a medical doctor who pokes and prods at your eyes \
while asking you to read letters from chart.
## Which best, Output (a) or Output (b)?
Output (a)
Here the answer is Output (a) because it provides a comprehensive and accurate description the job of an ophthalmologist. In contrast, output (b) is more of joke.
# Task:
14

---

Now is the real task, do not explain your answer, just say Output (a) or Output (b).
## Instruction: (instruction)
## Output (a)
foutput_1}
## Output (b): [output_2]
## Which is best, Output (a) Output (b)? < 

Prompt template for Claude:
Human: Select the output (a) (b) that best matches the given instruction.
Choose your preferred output, which can be subjective. Your answer should ONLY contain: Output (a) Output (b). Here's example:
Example:
## Instruction:
Give description of the following job: "ophthalmologistt"
## Output (a):
An ophthalmologist is medical doctor who specializes in the diagnosis \ and treatment of eye diseases and conditions.
Output (b):
An ophthalmologist is medical doctor who pokes and prods at your eyes \
while asking you to read letters from chart.
## Which is best, Output (a) or Output (b)? Output (a)
Here the answer is Output (a) because it provides comprehensive and accurate description of the job of an ophthalmologist. In contrast, output (b) is more \
of joke. Now is the real task, remember to only include Output (a) Output (b)
in your answer, not the explanation.
Task:
Now is the real task, do not explain your answer, just say Output (a) Output (b).
## Instruction:
{instruction)
## Output (a) {output_1) 1}
## Output (b): (output_2)
## Which is best, Output (a) or Output (b)? ?
Assistant:
15

---

Data Scaling Curve for SFT ShareGPT (GPT3.5 completions)
60
Alpaca Eval Winrate
40
20
Winrates evaluated using GPT-4
Winrates evaluated using Claude
10
25
50
100
Number of Training Examples for SFT (% of Total Examples)
Figure 6: The performance improvements from increasing the number of training points SFT on 100% of the training prompts yields minimal improvements over SFT 10% of the training prompts. Hence, for our LAIF setting, we first perform SFT on only 10% of the training examples, and we use the remaining for LAIF.
C Additional Results
In Figure we evaluate the performance of various models when both completions and AI feedback generated by GPT-4. We conduct similar experiment with Claude in Figure where the Llama 7B is trained is fine-tuned with SFT on 10% of the prompts, followed by LAIF on 90% of the prompts with AI feedback from Claude. We find that model instruction-uuned using LAIF underperforms model fine-tuned using SFT on Claude completions on all prompts, replicating the observation in similar setup with GPT-4.
SFT and LAIF Using Completions From Stronger Oracle Models (Claude)
80-
Alpaca Winrate
55
50-
45
SFT wi Cloude (10% total examplest
LNI Cleuce preferences
SFT Claude (103% total unamplas)
LLaMA 7B
Figure 7: SFT strong distribution minimizes any improvements from LAIF, We consider LAIF starting with Claude completions as target distribution SFT the first step, also to generate preference labels for RL. We find SFT on Claude completions for all prompts (SFT 100%) outperforms LAIF, where we observe no improvement from LAIE This experiment suggests that SFT with the right target distribution can potentially minimize any further gains from LAIF, even when starting with SFT completions from the oracle model.
D PPO-based RLAIF Experiments
In this section, we present attempt to verify if our findings also hold the RLAIF setting, i.e., where the policy is trained using online reinforcement learning algorithm such PPO, rather than an
16

---

offline equivalent such as DPO. The standard RLAIF pipeline has three stages: 1.) SFT fine-tuning,
2.) reward training. and 3.) PPO fine-tuning. Following the same protocol as Bai al. 2022] and we adapt the standard RLHF pipeline [Christiano et al. 2017 for AIF, by replacing humans with strong Al model, both as a teacher for SFT fine-tuning data collection and as a critic for preference collection for reward modeling. Next, we discuss our experimental setup in detail.
Experimental Setup
Similar to our DPO-AIF experiments, we use the 10% prompts and completions from the ShareGPT [Chiang et al. 2023 for SFT fine-tuning, and the remaining 90% of the prompts for reward modeling and PPO fine-tuning. We use the same prompts for reward modeling and PPO fine-tuning negate the concerns about reward overoptimization [Gao al. 2022].
Data:
To preprocess the dataset, we use truncate every prompt to a maximum 256 tokens, and truncate the completion such that the total length of the prompt and completion combined does not exceed 512 tokens. We use the following template for all all three stages of the PPO training.
An\nHuman:
(instruction) innnsssssant:: {completion))ees))
We limit the generations during preference data collection and the PPO training maximum of 512 tokens, including the prompt.
SFT Fine-tuning:
We start with pre-trained Mistral [Jiang al.) 2023 model, and fine-tune it using the single-turn instructions from the ShareGPT data [Chiang et al. 2023]. We use the same SFT checkpoints for DPO and LAIF experiments to ensure fair comparison.
For reward modeling, use the same response pairs that were used for DPO training, i.c. we sample one completion in the pair from MMencher and the other from TSFT, i.e., the pair contains one completion each sampled from GPT-3.5 and from the SFT fine-tuned Mistral model. use GPT-4 as the critic for preference data collection. We hold the last 512 preference data instances for as the validation set and use the remaining preferences for training the reward model. We trained the reward model using the Adam optimizer with the default hyper parameters, the global batch size of 8, and tried the following learning rates: [lc-4, 5c-5, 1.41e-5, 5e-6), and trained for epochs. We selected the checkpoint with the best validation accuracy. Our best reward model was trained with 5e-5 learning rate and had validation accuracy of 84.18% Figure shows the reward modeling evaluation accuracy across steps for different learning rates.
Reward Modeling:
Steps vs Reward Modeling Evaluation Accuracy
0.85
Reward Modeling Evaluation Accuracy
0.80
0.79
0.75
0.70
0.65
0.60
0.58
5r=5
0.55
fr.1.1e-5
Ir=5e-5
0.50
Ir=1e-4
20
40
60
80
100
Step
Figure 8: The highest point each LR curve highlighted. Our best reward model the model trained with 1r=5e-5, which reaches an accuracy of 84. 18%.
PPO Fine-Tuning: We use the same prompts and split from the reward modeling stage for PPO fine-tuning. The prompts truncated to maximum length of 256, and limit the total sequence
17

---

length to 512. We use the last 512 prompts for validation. We trained our model with the Adam optimizer with the default hyper parameters, and used the adaptive KL controller from Ouyang et al. 2022 We did the hyper parameter search on two dimensions, the initial KI. coefficient (init coef), and the learning rate (Ir), and used the reward model score for the checkpoint selection.
We use LoRA for PPO fine-tuning to fit our computational constraints. We used the following hyperparameters:
16, 32, and a dropout of 0.05. Figure 2 and Figure [0 show the training and validation reward score evolution during the course of the training. We find that the reward score improves over the baseline SFT model for almost all hyperparameter combinations. We select the checkpoint with the best validation reward score for the downstream AlpacaEval evaluation.
Train Reward Score over Steps
Init KL Coeff: 1 Ir: 1e-6
Init KL Coeff: 1 5e-7
Init [in_Kl_CCe KL_Coeff: 0.1 Ir: 5e-7
Init KL Coeff: 0.1 Ir: 1e-7
Init KL Coeff: 1 5e-6
1.8
Init KL KleCff 1 Ir: 1e-5
Train Reward Score
1.6
50
100
150
200
250
300
350
Steps
Figure 9: Train reward over PPO iterations.
Validation Reward Score over Steps
Init KL, Coeff: 1 Ir: 1e-6
Init. KL Coeff: 1 5e-7
Init KL_Coeff: 0.1 Ir: 5e-7
1.8
Init KL, Coeff: 0.1 Ir: 1e-7
KL_ Coeff: 1 5e-6
Validation Reward Score
Init KL_Coeff: 1 Ir: 1e-5
1.6
1.4
50
100
150
200
250
300
350
Steps
Figure 10: Validation reward over PPO iterations.
18

---

init kl coef
Ir
AlpacaEval Delta
1.00E-06
-0,9
5.00E-07
2.307
0.1
5.00FF07
1.767
0.1
1.00E-07
2.506
5. 5.00E-06
3.361
Table 2:
AlpacaEval score delta between the SFT and the best checkpoint according to the validation reward score for various configurations tested.
Results. Table shows shows the improvement in the AlpacaEval win rate over the SFT checkpoint when fine-tuning using PPO. In our experiments, we observe that the PPO-AIF gives at most 3.36% points improvement over the SFT model. PPO-AIF yields considerably less than the improvement (3.361 points) when compared to DPO-AIF (14.43 points), though we expect that this may be an artifact of our limited computational budget. However, this is substantially less (= 20 points) than what we achieve by just doing SFT with a data from strong teacher such as GPT-4. Hence, the our observation that doing SFT with a stronger teacher (GPT-4) often yields better performance than doing AIF with stronger critic (GPT-4) on weaker SFT model (GPT-3.5 SFT), stands in the case of PPO-AIF.
19

---

NeurIPS Paper Checklist
Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the
paper's contributions and scope?
Answer: [Yes]
Justification: Section backs our claims with experiments. Guidelines:
The answer NA means that the abstract and introduction do not include the claims
made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes]
Justification: We provide some discussion in Section Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification: No theoretical results presented. Guidelines:
The answer NA means that the paper does not include theoretical results. All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
20

---

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: Section Appendix A and we have provided an anonymized repository for our code.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (c.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors welcome describe the particular way they provide for reproducibilityy In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results. 5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We have shared all the code and data used in the anonymized code repository.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips. ec/
public/ public/guides
SoddeeSubiissionPP ootoooooooyyy for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
21

---

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Section 4 and Appendix A Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: We have tried our best to report error bars and statistical significance, but we are limited by the computational cost of collecting teacher datasets repeatedly. Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall
run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula,
call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
It is OK to report I-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
Question: For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Answer: [No]
Justification: We do not provide the resources for every experiment, but we have provided
the details of our computational resources in Appendix A Guidelines:
22

---

The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification: [NA]
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there special consid-
eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: While large language models (LLMs) are incredibly important tool with substantial social implications, this paper provides a critical study of alignment with auto- matic AI feedback, which does not introduce any additional social and ethical considerations beyond those for LLMs. Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended
(e.g., disinformation. generating fake profiles, surveillance), fairness considerations (e.g.. deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)? Answer: [NA]
Justification: We do not release models. We modify publicly available Instruction-tuning and preference datasets that were already available.
Guidelines:
The answer NA means that the paper poses such risks.
23

---

Released models that have high risk for misuse or dual-use use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We have cited relevant sources throughout the paper. Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g.. website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the pataasets.ppprsstttt package should be provided. For popular datasets, paperswi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should provided. If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
New Assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: We provide a complete documentation in our repository alongside the paper. Guidelines:
The answer NA means that the paper does not release assets. Researchers should communicate the details the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: Human subjects were not involved in any study. Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
24

---

According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: Human subjects were not involved in any experiment. Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with
human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
25


Reviews:
[{"review_id": "lFouYcHRr8", "reviewer": "Reviewer_Sb9E", "summary": "Learning from AI Feedback (LAIF) has become a popular alternative for improving the instruction-following abilities of large language models (LLMs). Despite its popularity, many unresolved questions remain regarding the actual improvements gained through LAIF. The authors address some of these questions, with a focus on where exactly the improvement in the LAIF pipeline is coming from \n\nThe authors found that many of the improvements in LAIF are attributed to the differences in the weak teacher LLM (that provides the SFT data) and a strong critic LLM (that provides the preference data). Their empirical evidence demonstrated this issue across a wide range of base, critic, evaluator, and teacher models.\n\nMoreover, the authors suggested two potential explanations for LAIF's ineffectiveness: either the preference dataset is not sufficiently informative, or the base model has inherent limitations. Finally, the authors offer several insightful suggestions for future research in the area of LAIF.", "questions": "- Line 64: If the discrepancy between SFT and AI feedback is minimal, then doing SFT can suffice. I am trying to understand what this statement implies; it seems pretty straightforward. If there is no gap between the two, then you do not need the second step.\n- Line 206: Missing citation, using completions from M_{teacher} as one of the inputs in the preference pair results, was observed in [1].\n- Also, why not use the M_{teacher} to generate both responses to ensure that the preference data is high quality? DPO [2] mentioned this setting in the \"DPO outline\" section; essentially, you can train on \\pi_{ref} on the preferred completions.\n- Line 252: Would you agree that M_{critic} does not fully capture the quality of the preference dataset? If so, then comparing M_{critic} versus M_{teacher} is a little odd because M_{teacher} fully affects the quality of the SFT dataset, whereas M_{critic} only partially affects the quality of the dataset. If M_{critic} generated both the chosen and rejected, then you be certain that generations are high quality and the M_{critic} learning signal is good.\n- Figures 4 and 7 show that the 10% threshold is ideal for certain models and settings. Have you experimented with a different percentage threshold? \n- Missing citations [3], [4], [5]\n\n[1] Coactive Learning for Large Language Models using Implicit User Feedback by Tucker et al. ICML 2024\n\n[2] Direct Preference Optimization: Your Language Model is Secretly a Reward Model by Rafailov et al. NeurIPS 2023\n\n[3] Starling-7B: Increasing LLM Helpfulness & Harmlessness with RLAIF by Zhu et al. 2023\n\n[4] UltraFeedback: Boosting Language Models with High-quality Feedback by Cui et al. 2023\n\n[5] Peering Through Preferences: Unraveling Feedback Aquisitiosn for aligning Large Language models by Bansal et a. 2024 ICLR", "limitations": "Yes, the authors have adequately addressed the limitations.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper is well-written and easy to follow.\n- The authors are addressing a significant problem.\n- The experiments were really well designed and performed.\n- The authors show the robustness of their observation by running experiments across several models and dataset splits.\n- The authors not only identified the problem in LAIF but also provided some possible explanations that enhanced the reader's understanding of it.\n- LAIF is an important path forward for improving LLM instruction following capability. Therefore, as outlined in the paper, it is important to systematically identify the problems in LAIF so that researchers can address them.\n- The authors address most of my obvious internal questions and thoughts on various experiment details and design decisions.", "weaknesses": "- Some of the observations in the paper are straightforward.\n- A few more experiments should be included in the paper to complete some of its conclusions.\n- The 10% rule doesn't always hold true. In Figure 3 and Figure 4, SFT 10% performs worse than SFT 100%. A better split could improve SFT performance when doing SFT + LAIF, which is important based on the paper's conclusions.\n- The LAIF ablation experiments for addressing the LAIF ineffectiveness, either as preference data or as the base model, have issues. The author samples data from Llama or Mistral and trains on the data using Llama or Mistral as the baseline model. However, the results could suffer from overoptimization problems [1].\n\n[1] Scaling Laws for Reward Model Overoptimization by Gao et al 2022", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> 10% rule doesn’t always hold\n\n> Figures 4 and 7 show that the 10% threshold is ideal for certain models and settings. Have you experimented with a different percentage threshold?\n\nThanks for noting this. Using an increasingly larger split for SFT would weaken our claims compare LAIF with SFT (for example, consider using 99% of the prompts for SFT and 1% for LAIF). We heuristically chose a split which provides enough SFT data to provide a good initialization for LAIF, while using most of the prompts for LAIF. We will revise the text to note that this heuristic may not be optimal.\n\n> Line 64: If the discrepancy between SFT and AI feedback is minimal, then doing SFT can suffice. I am trying to understand what this statement implies; it seems pretty straightforward. If there is no gap between the two, then you do not need the second step.\n\nThat is the right interpretation → if you do not see any substantial improvement from LAIF, it was not needed in the first place. We will rephrase this sentence to be clearer.\n\n> Line 206: Missing citation, using completions from M_{teacher} as one of the inputs in the preference pair results, was observed in [1].\n\n> Missing citations [3], [4], [5]\n\nThanks for the pointers. We will add the missing citations in the next revision.\n\n> Also, why not use the M_{teacher} to generate both responses to ensure that the preference data is high quality? DPO [2] mentioned this setting in the \"DPO outline\" section; essentially, you can train on \\pi_{ref} on the preferred completions.\n\nThis is a great suggestion, and would be good to explore better in future work. There are two concerns:\n- If generating a completion for SFT from the teacher costs 1 unit of supervision per prompt, and generating a preference label (given two completions) costs 1 unit of supervision, our current setup for LAIF uses 2 units of supervision per prompt, compared to SFT which only gets 1 unit per prompt. Generating both the completions using the teacher would mean 3 units of supervision per prompt. We ideally want to compare SFT and LAIF with equivalent amounts of supervision, but our setup is already unfairly biased to LAIF.\n- Our preliminary experiments find that sampling both the completions from the teacher underperforms using our current scheme 1 completion from the teacher and 1 from the model. DPO [1] discusses how samples not from the reference model can hurt the performance. There may be ways to improve the pipeline when both the samples are from the teacher (for example, training pi_{ref} on both preferred and dispreferred completions), it requires deeper exploration beyond the scope of this paper.\n\n[1] Direct Preference Optimization: Your Language Model is Secretly a Reward Model by Rafailov et al. NeurIPS 2023", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "yABwM6ZtKs", "reviewer": "Reviewer_pKmA", "summary": "The paper evaluates the extent to which AI feedback is helpful in aligning large language models (LLMs) within the commonly used two-step method of improving pre-trained LLMs. This method involves first performing supervised fine-tuning (SFT) and then fine-tuning with reinforcement learning (RL) or direct preference optimization (DPO) using preference feedback. The findings indicate that, in some cases, SFT may outperform the two-step LAIF approach.", "questions": "- In Table 1, are the values in brackets variances or confidence intervals? How many repeated runs were conducted?\n- When the paper uses phrases like “weaker SFT target distribution,” how exactly is a target distribution determined to be weaker or stronger?\n- It seems that the difference in target distribution is based solely on the percentage of total examples used. It would be interesting to see if the diversity of examples affects the improvements from LAIF and to what extent. For instance, what if the total number of examples remains the same, but the diversity of examples in terms of their distribution in an embedding space is different?", "limitations": "- Yes, the authors have sufficiently addressed the limitations.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper includes comprehensive experiments and provides a detailed analysis along with hypotheses explaining the results.\n- These experiments encompass a wide range of different LLMs and settings, offering good quantitative metrics and analyses.\n- I particularly appreciate the bandit experiments; despite their simplicity, they effectively convey the core ideas and strongly support the paper's claims.", "weaknesses": "- I am not entirely certain about the claim that “SFT on strong distribution minimizes any improvements from LAIF.” While this was the case for 3 out of the 4 result settings (including both figures 4 and 7), it is difficult to assert this as a general truth. Could the authors rephrase the claim to be more nuanced?\n- In the analysis, the authors make general claims that may not hold true in all cases. For example, “AI feedback is effective when there is a discrepancy between the SFT distribution and the evaluator,”, the analysis lacks numerical values, and the claims are not nuanced, even though they do not hold in all cases. However, I do like the conclusion, where the authors emphasize that the claim does not hold true in all cases.\n- It would be interesting to see if the same hypotheses hold in more general LLM settings, such as multi-turn instructions and multi-modal foundation models.\n\nMinor things\n- Typo in line 331 “via via LAIF”", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for finding our experiments comprehensive and appreciating our bandit experiments! We answer your questions and concerns below:\n\n> I am not entirely certain about the claim that “SFT on strong distribution minimizes any improvements from LAIF.” While this was the case for 3 out of the 4 result settings (including both figures 4 and 7), it is difficult to assert this as a general truth. Could the authors rephrase the claim to be more nuanced?\n\n> In the analysis, the authors make general claims that may not hold true in all cases …\n\nThanks for the suggestions. We will revise the text to be nuanced, and point the readers to conclusions explicitly for takeaways. We will revise the specific statement to be more nuanced: “We found LAIF to provide minimal improvement when SFT was a dataset of completions from a strong teacher, in 3 out of the 4 cases”.\n\n> In Table 1, are the values in brackets variances or confidence intervals? How many repeated runs were conducted?\n\nThe confidence intervals were computed by the automated evaluation in AlpacaEval, which evaluates models on 800 questions (we will clarify this in our text). Our computational budget did not allow for repeated runs per model/dataset/teacher, and we favored distributing our budget over more models and teachers than repeated runs with a smaller set of models and teachers.\n\n> When the paper uses phrases like “weaker SFT target distribution,” how exactly is a target distribution determined to be weaker or stronger?\n\nThanks for bringing this up, and we recognize the impreciseness of the notion of strength in this context. The strength of the target distribution in this context is used to refer to the strength of the teacher model, and GPT-3.5 is generally agreed to be worse at instruction following compared to GPT-4 (both subjectively, but also benchmarks like LMSys and AlpacaEval).\n\n> It would be interesting to see if the diversity of examples affects the improvements from LAIF and to what extent. For instance, what if the total number of examples remains the same, but the diversity of examples in terms of their distribution in an embedding space is different?\n\nThat’s a great suggestion, and we agree it would be good to explore in future work!", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "b5MgP9hu27", "reviewer": "Reviewer_6Bqz", "summary": "The paper explores the current experimental set up for learning from AI feedback (LAIF), specifically using demonstrations generated by a LLM that is weaker than the LLM used to assign preference labels both for training and evaluation. By comparing training several different base LLMs on demonstrations generated by weaker (GPT3.5) versus stronger (GPT4, Claude) LLMs both in the absence of LAIF and followed by LAIF, the results suggest that current evidence for the benefits of LAIF (noted to be distinct from learning from human feedback - LHF) is over stated. The conclusion and take away is to better construct the SFT datasets such that the available demonstrations come from LLMs of the same quality level as those used to assign preference labels.", "questions": "- Which version of Claude was used?\n- Why not use AlpacaEval 2.0? Why evaluate against GPT-3 (text-davinci-003)?\n- How do these findings connect and relate to \"Preference Fine-Tuning of LLMs Should Leverage Suboptimal, On-Policy Data\" (https://arxiv.org/pdf/2404.14367)? \n- How did you decide which base models to use in each experiment?", "limitations": "The limitations are limited. There is a single sentence saying that there is only a limited analysis. However, other limitations should be discussed and addressed. For example, what are scenarios under which the analysis/results that was done may not hold? The current experimental set up assumes (implicitly) that the goal is to copy/mimic/match the performance of GPT4/Claude. However, what if someone does not want a final LLM that approximates the distribution of GPT4/Claude? What about in the base of bootstrapping versus distilling? The answers to the questions above are kind of scattered across the paper, but it would be beneficial to have a clear section discussing the boundaries of the work.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- The paper is well written and easy to follow. The arguments are well outlined to make the motivations, benefits of the work, and the learnings clear.\n- The issue of ensuring that current and widely used experimental designs are correctly constructed is a vital contribution. The potential need for algorithmic adjustments to learn from AI feedback versus human feedback is something that could kick off a whole new avenue of research and investigation in the field. \n- This is probably one of the few papers I have seen that attempts to dig into differences in LAIF performance gains for different base models. While the investigation is cursory, it does provide some evidence and hypotheses that can help direct future investigations, either by these authors or others.", "weaknesses": "- Some of the motivation for the experiment set up is incongruous. The authors motivate the paper around the LAIF paradigm (not using LLMs as human proxies for algorithm development), and call out why the claims in the paper should be considered separate from LHF. However, they also highlight the importance of having the same critic as evaluator. In practice the final evaluators for a LLM trained with LAIF methods are humans. Therefore, it would be good to have experiments where humans are the evaluators or the critic and evaluator are not the same LLM. \n- A clear distinction between LAIF and LHF is made throughout the paper to highlight that the take aways are for LAIF only. However, it seems some of the take aways discussed in \"Current base LLMs are insufficiently responsive to AI feedback\" (pg. 7) could apply to LHF. Using strong LLMs as a proxy for humans is valid, and when the experiments are looked at from the perspective of the LAIF as proxies for LHF, more conclusions can be drawn made. Why would the hypotheses about representational space mismatches not be relevant to LHF?\n- The experiment evaluating training the base LLM on demonstrations from Claude prior to LAIF is only run on llama. It would be helpful to at least also see results for Mistral 7B so that more of a comparison can be drawn against Figure 3. \n- Different experiments are run with different numbers of base LLMs. The same base LLMs should be used across all main experiments, those whose results are in Figures 2, 3, 4, and 7. If fewer base LLMs are used in some experiments, a justification should be provided. \n- The SFT results with GPT and Claude should not be separately compared against the LAIF results using the 10% GPT-3.5 demonstrations versus the GPT-4/Claude demonstrations. \n- The story around the section \"Completions samples from SFT models are substantially poorer than completions sampled from M_{oracle}\" should be better motivated with a clear, strong take away.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for finding our paper well written, and our contributions for experimental design vital. We address your concerns and questions below:\n\n\n>  However, it seems some of the take aways discussed in \"Current base LLMs are insufficiently responsive to AI feedback\" (pg. 7) could apply to LHF. Using strong LLMs as a proxy for humans is valid, and when the experiments are looked at from the perspective of the LAIF as proxies for LHF, more conclusions can be drawn made. Why would the hypotheses about representational space mismatches not be relevant to LHF?\n\nThanks for noting this and we agree, and it is possible that some of the base models are insufficiently responsive to HF too (and may partly explain why reward model accuracies for human feedback are low, ~70%). However, in LAIF, the discrepancy between representational strength of the teachers (like GPT-4, Claude) and models (Llama/Mistral-7B) is clearer, and we wanted to be conservative in our claims.\n\n> The story around the section \"Completions samples from SFT models are substantially poorer than completions sampled from M_{oracle}\" should be better motivated with a clear, strong take away.\n\nWe will revise the text to include the takeaway: “Better samples from the student models, generated using better prompts or CoT and similar techniques may generate higher quality samples and allow for LAIF to be more effective”. Please let us know if rephrasing this would be better.\n\n> Which version of Claude was used?\n\nWe used Claude-v1. Claude-v2 was found to have a poorer correlation with human judgment in the original AlpacaEval study.\n\n> How do these findings connect and relate to \"Preference Fine-Tuning of LLMs Should Leverage Suboptimal, On-Policy Data\" (https://arxiv.org/pdf/2404.14367)?\n\nThe findings are complementary, as the main message of the cited paper is how to sample data for labeling or query reward model in RLHF, whereas our paper analyzes how effective automatic mechanisms such as AI feedback (loosely, our paper studies the choice of reward model and the paper cited in the question studies the data used during optimization).\n\n> How did you decide which base models to use in each experiment?\n\nOur academic access to Anthropic API expired and was not renewed, so our studies with Claude are more limited as compared to GPT-4.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper carefully studies the current practice of Learning from AI Feedback (LAIF) for fine-tuning large language models and show that the benefits of LAIF (distinct from learning from human feedback) are likely over-stated. All of the reviewers found the experiments comprehensive and convincing, and the paper's highlighting of the need for better Supervised Fine-Tuning datasets (should be drawn from LLMs of the same quality as the LLMs used to assign preferences) is likely to be of value to the LLM fine-tuning and distillation community.

The reviewers raised several questions that led to insightful discussions with the authors. All of the reviewers' questions were eventually answered satisfactorily, and incorporating all of the discussions into a revised version will substantially strengthen the paper.
Decision: Accept (poster)