AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedback
Authors: Yann Dubois
Xuechen Li
Rohan Taori
Tianyi Zhang
Ishaan Gulrajani
Jimmy Ba
Carlos Guestrin
Percy Liang
Tatsunori Hashimoto
Venue: NeurIPS 2023 spotlight (2023)
Keywords: Instruction-Following
Reinforcement Learning from Human Feedback
Artificial General Intelligence
Large Language Models
TL;DR: We propose and validate a simulator that enables research on learning from human feedback at a low cost.

Abstract: Large language models (LLMs) such as ChatGPT have seen widespread adoption due to their ability to follow user instructions well.
Developing these LLMs involves a complex yet poorly understood workflow requiring training with human feedback. Replicating and understanding this instruction-following process faces three major challenges: the high cost of data collection, the lack of trustworthy evaluation, and the absence of reference method implementations. We address these bottlenecks with AlpacaFarm, a simulator that enables research and development for learning from feedback at a low cost. First, we design LLM based simulator for human feedback that is 45x cheaper than crowdworkers and displays high agreement with humans. Second, we identify an evaluation dataset representative of real-world instructions and propose an automatic evaluation procedure. Third, we contribute reference implementations for several methods (PPO, best-of-n, expert iteration, among others) that learn from pairwise feedback. Finally, as an end-to-end validation of AlpacaFarm, we train and evaluate eleven models on 10k pairs of human feedback and show that rankings of models trained in AlpacaFarm match rankings of models trained on human data. As a demonstration of the research possible in AlpacaFarm, we find that methods that use a reward model can substantially improve over supervised fine-tuning and that our reference PPO implementation leads to a +10% win-rate improvement against Davinci003.

AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedback
Yann Dubois" Stanford
Xuechen Li Stanford
Rohan Taori* Stanford
Tianyi Zhang* Stanford
Ishaan Gulrajani Stanford
Jimmy Ba University Toronto
Carlos Guestrin
Percy Liang
Tatsunori B. Hashimoto
Stanford
Stanford
Stanford
Abstract
Large language models (LLMs) such as ChatGPT have seen widespread adoption due to their ability to follow user instructions well. Developing these LLMs involves a complex yet poorly understood workflow requiring training with human feedback, Replicating and understanding this instruction-followin process faces three major challenges: the high cost data collection, the lack of trustworthy evaluation, and the absence of reference method implementations. We address these challenges with AlpacaFarm, a simulator that enables research and development for learning from feedback at low cost. First, we design LLM prompts to simulate human feedback that are 45x cheaper than crowd-workers and display high agreement with humans. Second, propose automatic evaluation and validate against human instructions obtained on real-world interactions. Third, we contribute reference implementations for several methods (PPO, best-of-n, expert iteration, and more) that learn from pairwise feedback. Finally, as an end-to-end validation of AlpacaFarm, real human feedback and show that the rankings of models trained in AlpacaFarm match the rankings of models trained on human data. As a demonstration of the research possible in AlpacaFarm, we find that methods that use a reward model can substantially improve over supervised fine-tuning and that our reference PPO implementation leads to a +10% improvement in win-rate against Davinci003. We release AlpacaFarm https: //tg:://tttttt thub. com/tatsu 1ab/ aalaaalpaaaaaaaaaaaaaaaaaaaaa farmm
train and evaluate eleven models 10k pairs of
Introduction
Large language models (LLMs) [10, 13, 48] have demonstrated unprecedented capabilities in follow- ing diverse and open-ended instructions [49, 5, 40). These achievements have often been attributed to the fine-tuning of pretrained I.LMs using human feedback, but this process remains poorly understood due to the lack of published information on the training methods from I.I.M vendors. For example, was recently revealed that only the Davinci003 model in the instruct series of OpenAI models used reinforcement learning (RL) with the PPO algorithm [47], leading some to question the importance of RL in the training process. Understanding and improving these methods requires open and transparent replications of the training process, but this remains challenging due to the cost and complexity associated with methods that learn from human feedback.
Our goal facilitate research and development on instruction following models and methods that learn from human feedback. We identify three main challenges: the high cost of data annotation, the lack of automated evaluation for model development, and the absence of validated implementations
"qqual contribution; random order. Contact fyanndubs, lxuechen, rtaori, tz58)@stanford.edd
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

best method human feedback
Proposs new mechods
Train methods simulation
Alpaca Farm
API
PPQ
Dev Model
feedback for model samplees
compute w n-rrtte against baseline
3.
Expert lteration
+ Learning from Feedback
Baseline Model
Automatic Evaluation
Compare Reference Methods
Feedback Source
Experiment Cost
Reproducible
Alpaca Farm
$70
Hours
Human Feedback
$3,150
Days
X
Figure AlpacaFarm is simulation sandbox that enables fast and cheap experimentation on methods that learn from human feedback. It simulates human feedback with API LLMs, provides validated evaluation protocol, and offers a suite of reference method implementations. Researchers can iterate on model development and transfer their methods to training on human data to maximize performance.
of existing methods. To address these three challenges, we introduce AlpacaFarm (Figure 1), a simulation sandbox that enables experimentation at a low cost. Using AlpacaFarm, researchers can rapidly iterate on method development in simulation and transfer these insights to build high- performance systems with actual human feedback.
For the first challenge of data annotation costs, AlpacaFarm simulates human annotators with API LLMs that are faster and lower cost. To collect simulated feedback data, design prompts for API LLMs (c.g. GPT-4) that enable us to simulate human pairwise comparisons at cost that 45x cheaper than crowd-workers, and tune these prompts to faithfully capture many aspects of human annotators, such as their quality judgments, inter-annotator variability, and stylistic preferences.
For the second challenge of automated evaluation, we design an automatic evaluation protocol that aims to quantify system performance on simple but realistic real-world human instructions. Improving evaluations of open-ended instruction following has been challenging due to the cost and non-replicability
human evaluation, the lack of real human interaction data, and the diversity instructions from user interactions (Alpaca Demo [69]) as a reference for simple but real human interactions and show that we can combine existing evaluation datasets mimic this evaluation. Quantitative evaluations of system rankings our evaluation data show a high correlation with system rankings on the Alpaca Demo instructions.
of natural human instructions. To address this,
For the third challenge of missing reference implementations, we implement and test several popular learning algorithms including PPO [62], expert iteration [2], and Quark [42], and release reference implementations. We show that, among the methods we studied, PPO with surrogate reward model the most effective training-time method in our leaderboard, improving the win-rate against Davinci003 of instruction fine-tuned LLaMA 7B model from 44% to 55%. Other baselines that have been validated on simpler tasks fall short comparison, highlighting the importance of testing these algorithms in real instruction-followii setting.
As an end-to-end evaluation of the AlpacaFarm, we compare eleven methods trained and evaluated in AlpacaFarm with the same methods trained and evaluated on actual human feedback. We show that the method rankings obtained from developing on AlpacaFarm closely agree with the method rankings obtained from training actual human data (Spearman correlation of 0.98) and that the best method AlpacaFarm leads to substantial gains with human feedback. Finally, we find that AlpacaFarm can replicate qualitative behaviors of human feedback such as over-optimization of the reward model, suggesting that AlpacaFarm serves as an effective way for researchers to rapidly study and develop methods that learn from human feedback.

---

2 Background & problem statement
To begin, introduce the instruction following task and the setrwise-comparison based human feedback setting that we study. With this background, we formally define the goals of developing a low-cost simulator for studying instruction following and learning from human feedback.
2.1 Learning follow instructions
In the instruction following task [49, 6, 76, 40], we are presented with user instructions 2 E X (e.g. "Tell me something about Alpacas"), ") and our goal is to develop a model py that generates high-quality responses y py .) as judged by an unobserved human reward function R X y R.
While there are a rich set of methods that learn from human feedback to directly optimize R (see Appendix A), in this work we focus the setting of learning from pairwise feedback (LPF) due to its central role in recent instruction-followinn L.MSs [49]. The starting point of this process model that is fine-tuned on eestructionfolllwwiie SFT (yy 4)) The LPF process then involves taking pairs of samples from pa SFT querying humans for Pa
demonstrations (I. y), which we denote as
which sample within each pair is better, and learning from this pairwise feedback. Since all methods start from the SFT base, we use pe for notational simplicity.
Learning from pairwise feedback (LPF). More formally, we define the pairwise feedback dataset
(5) ()) , In this notation, human annotator rates two candidate as D pairwise (1x00) Mo + 31
responses y/o 3/1 ~ Pe (y x)) for the instruction x. These binary ratings z E {0, 1} are assumed to be generated according to their unobserved reward R and indicates a (potentially stochastic) y1-s).
comparison for the better response ys, where R(x, y2)
Many algorithms have been proposed to learn on D pairwiss. Some algorithms like RLHF [18, 49] learn surrogate reward function as the learning signal and some operate more directly on D pairwise We defer the discussion of different learning algorithms to Section 3.4.
Pairwise evaluation. Once intructtion-ollowwiin models are trained, researchers need to evaluate these models. One common approach for evaluating models is pairwise model evaluation [15, 22, 7], which performs pairwise comparisons on outputs generated by the model pa and a reference model
Pref- Concretely, we collect pairwise preference for two models (((x(1), Ve Vref 200))))) which is aggregated by computing the average win-rate the percentage of times po is preferred to pret- Researchers can then compare LPF models by their win-rate against the same reference model.
(j) (i)
2.2 Problem statement
The goal of AlpacaFarm is to provide three key components that enable rapid research and develop- ment of instruction following models: low-cost pairwise feedback generators, automated evaluations for methods development, and reference implementations for comparison and modification. With these three components, researchers can develop new methods in simulation and transfer these insights build perffrrrffo systems on actual human feedback.
For pairwise feedback, we substitute human preference judgements Phoman Phuman Ir, No, 91) with simulated preference žsim ~ Psim xt, 3o,y1) using API LLMs.. Our goal is to construct Psim that is both low-cost and faithfully captures different aspects of human preference feedback, such as quality judgments, inter-annotator agreement rates, and stylistic preferences.
For evaluations, we evaluate system outputs using the pairwise preference simulator and identify evaluation datasets that reflect natural human-LLM, interactions. The goal for our evaluations is to ensure that system rankings on the new evaluation dataset closely match both human rankings and rankings on instructions from real usage of the Alpaca Demo.
For reference methods, we develop and evaluate six LPF methods. Our goal will be to provide simple and working implementations that provide substantial improvements on both simulated and human feedback data. This will allow researchers to build upon and compare to competitive baselines in complex instruction followiin environment.
3

---

AlpacaFarm combines these three components into a simulation framework for learning from pairwise feedback. We evaluate the complete system by an end-to-ent workflow of developing methods in simulation and transferring the insights to the real world.
Concretely, will each method M the simulated preferences (called Msim) and evaluate with simulated rankings Psim- In parallel, we will run M on human preferences (called Mhuman) and evaluate with human rankings Phuman- We consider AlpacaFarm to be successful if the simulated method rankings correlate well with the human method rankings. The rest of this work will present the details of the pairwise feedback and evaluation design (Section 3), evaluate these designs (Section 4), and analyze the different reference methods we implemented in the AlpacaFarm (Section 5).
3 Constructing the AlpacaFarm
In this section, we detail how we construct the AlpacaFarm. In Section 4, we then validate our design choices by comparing the LPF workflow with human feedback and evaluation.
3.1 Instruction following data
Before defining the details of how we simulate pairwise feedback, we must first specify a large and diverse set of instructions x upon which we can build the rest of AlpacaFarm. We opt to use the Alpaca data [69] starting point due to its large size (52k (x, y) examples) and the non-trivial instruction following capabilities of models trained on this data.
We repurpose the Alpaca data into splits suitable for learning from human feedback by following a similar splitting ratio as [49]. We created four splits (42k in total), leaving 10k for the future:
Supervised finetuning (SFT) split: 10k data for fine-tuning the base instruction-followin LLM used in subsequent steps.
Pairwise preference (PREF) split: 10k instructions on which we collect pairwise feedback data.
Unlabeled split: 20k unlabeled instructions used in algorithms such as PPO.
Validation split: 2k data for development and tuning.
3.2 Designing simulated pairwise preference Psim
We now describe the design of our simulated annotator for pairwise preferences. Our core proposal is to design the simulator (= Ir yo: y1 by prompting OpenAl API LLMs. While using LLMs as a proxy for annotators has become increasingly popular [15, 39], using LLMS as part of simulation environment poses major additional challenges. Our simulated preferences psim must not only have a high agreement with human preferences Phuman: must also capture other qualitative aspects of human feedback such as inter- and intra-annotator inconsistencies. Intuitively, the noise and variability in pairwise feedback are key parts of the challenge in the LPF problem and we find that ignoring these factors leads to simulator that diverges substantially from real-world behavior (Section 4.3).
Basic GPTT4 prompt design. To start with, we design prompts by providing an annotation guideline, feeding in-context examples, and leveraging batch generation to save on costs, As first baseline, we query GPT-4 with a single prompt (denoted as p pfPT-4 and we find GPT- has a high agreement rate with human annotators (65%; see results Section 4.3). However, we find that this simple baseline of GPT fails to capture the variability in human annotation and can lead to qualitatively different results for method development, especially for reward optimiiiinittion (Section 4.3).
Simulating human variability. To more completely emulate human annotators, we modify the basic simulated annotator design to capture annotator variability in two ways. First, we emulate inter- annotator variability in the simulated pairwise preference Psim by mimicking a pool of annotators. We design different annotators by querying different API LLMs and varying the prompts with different formats, batch sizes, and in-context examples. In the end, created 13 simulated annotators which we describe fully in Appendix D. Second, we emulate intra-annotator variability by directly injecting random noise and flipping the simulated preference 25% of the time.
With these ingredients, we come up with a simulated preference yam that meets our requirement of agreement and variability. Overall, annotating 1000 outputs using simulated preference only costs $6.

---

which is 50x cheaper than human annotation. In Section 4, we collect actual human preference and quantitatively verify the agreement and variability of our simulated preference.
3.3 Designing an automatic evaluation
For researchers to develop LPF methods in the AlpacaFaamm we want to support them with an automatic evaluation so they can quickly iterate while reliably comparing methods. To replace the usual human evaluation, there are two challenges. First, how do quantify the quality of different models? Second, what instructions can we use that are representative human interactions?
Evaluation protocol. To quantify the quality of an LLM po. we measure the win-rate of that LLM against reference model, i.e, the expected number of times that the output from is preferred the output of reference model Piet on the same instruction æ. The benefits of using simulated win-rates are that it provides a metric that is easy to understand, is comparable across methods, and can reuse the routine we built for pairwise feedback. We use the 13 simulated annotators described in Section 3.2 without injecting the additional noise (as adding uniform noise does not change model rankings) and denote this preference simulator as Pam- For the reference model, we use Davinci003 as it is well-studied system that performs similarly to the models we fine-tune.
Evaluation data. Instruction following requires diverse coverage over realistic interactions. To build an appropriate evaluation protocol, we combine several open-source evaluation datasets and use real-world interactions with a demo instruction-followin I.M (Alpaca Demo [69]) as guidance for constructing our data combination. Due privacy concerns, do not directly release the demo data and opt to use it to guide how we combine existing open evaluation datasets.
Our final evaluation set consists of 805 instructions, which includes 252 instructions from the self- instruct test set [75], 188 from the Open Assistant (OASST) evtest setaluation, 129 from Authropics' helpful test set [6]. 80 from Vicuna test set [80, 17], and 156 from Koala test set [22]. In ??, we show example instructions from our evaluation set (Table 7) and their root verb distribution(Figure 13), which shows the diverse coverage of our data. We find that aggregating across datasets is important for automatic evaluation to match real-world interactions, as discussed in Section 4.4.
3.4 Reference methods in AlpacaFarm
Finally, AlpacaFarm defines collection of validated LPF methods for instruction following. We describe those methods in detail in Appendix B but provide a brief overview here. In all LPF methods that follow, we begin by first performing an initial fine-tuning step on supervised data of instructions and outputs. We begin by describing two simple baselines that directly operate on pairwise feedback.
Binary FeedME. Binary FeedME [47] continues supervised fine-tuning on the preferred output in each pairwise comparison.
Binary reward conditioning. Reward conditioning [28, 36, 30] is simple scheme that incor- porates learning from negative examples; a token denoting whether the output was preferred is prepended before fine-tuning, and the positive token is used to condition at inference time.
Many LPF methods do not directly operate on pairwise feedback data, but instead first construct a surrogate reward model by fine-tuning classifier from the SFT base using pairwise feedback. The following LPF methods maximize the continuous-valued reward defined by the logits of this classifier.
Best-of-n sampling. Best-of-n (or re-ranking) [67, 5, 23, 9] a simple but effective inference taference-timee method that draws i.i.d. responses from the SFT model and returns the response with the highest surrogate reward. Unless stated otherwise we use n 1024 in our experiments.
Expert iteration. Expert iteration [2, 64, 74] is the natural training-time extension of best-of-n: first generates according to best-of-n on new instructions and then fine-tunes on the best outputs.
Proximal Policy Optimization (PPO). PPO [26, 62) is reinforcement learning algorithm that maximizes surrogate reward, subject to a KI. penalty keeping parameters near SFT initialization.
Quark. We use the top-quantile variant of Quark [42] which bins sequences by reward and trains the best bin, along with adding KL and entropy regularization.
5

---

The ranking of methods trained and evaluated in AlpacaFarm matches that of methods trained and evaluated in the buman-basee pipeline. Each blue point represents one method . (e.g. PPO). The x-axis shows the simulated evaluation (win-rates measured by poral) on methods trained in simulation Maim The y-axis shows human eval- uation (win-rates measured by Phuman) on methods trained with human feedback M Mhmman- Gray points show models that we did not train, so their x and y values only differ in the evaluation (simulated vs human). Without those points, we have R2 0.83 and Spearman Correlation of 0.94.
Figure 2:
0.8
Spearman Correlation: 0.98
Human Win-rate
0.7
R²
0.87
0.6
0.5
0.4
0.3
0.2
0.1
0.0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
Simulated Win-rate
Validating the AlpacaFarm simulator
With the simulator and methods defined, now evaluate AlpacaFarm. As main result, in Section 4.2 we analyze the correlation between the final rankings of methods in both the simulated LPF workflow and human-based LPF, Afterward, we will analyze more specifically whether our pairwise feedback accurately mimics human pairwise feedback (Section 4.3) and whether rankings on evaluation data match rankings the Alpaca Demo data (Section 4.4).
4.1 Experimental details
Models. As baseline and starting point for LPF methods, we fine-tuned LLaMA 7B on the 10k SFT split. We take SFT 10k as starting point for all LPF methods and collect the simulated preference yam and human preference Phuman from SFT 10k's outputs (with temp=1. 0) the 10k instruction PREF split. Then, for each of the six reference LPF methods M:
We trained and tuned M on simulated preferences pam evaluating the resulting model Msin against the Davinci003 reference with the simulated evaluator paual
We trained few models M human preferences across hyperparameter ranges identified in simulation, evaluating the resulting model Mhuman against Davinci003 with humans Phuman.
In addition to the six methods, also evaluate existing instruction-folloving and base models: GPT-4 (gpt-4-0314) 14), ChatGPT (gpt-3.5 -.5--ttbbtb--0101) Davinci001 (tcxt-davinci-001) LLaMA 7B [72]. and Alpaca 7B [69]. Alpaca 7B LLaMA 7B model finetuned all data splits, denoted SFT 52k. For these models, we measure both the simulated win-rate puml and human win-rate Phoman-
At inference time, for all systems except best-of-n, we sample with temp=0. 7 and set the maximum number of tokens to be 300. For best-of-n sampling, we found higher temperature to be helpful in encouraging output diversity, and so we rerank samples from SFT 10k with temp= 1 .. We provide more thorough experimental details and hyperparameters for all methods in Appendix C.
Human annotation. We collected reference human annotation by showing crowd-workers two potential outputs yo or y1 for given instruction æ and asked them to select the index = E {0. 1} of their preferred output. Annotators test of 25 questions. Out of an initial pool of 34 annotators, we selected the 16 whose agreement rate was higher than 70% with the author's annotations. We paid the annotators median hourly rate of $21, leading to a one-time $3000 cost of annotating our PREF split and recurring $242 cost for evaluating single model the 805 evaluation instructions. See Appendix E for additional details.
recruited from Amazon Mechanical Turk using a qualification
4.2 End-to-end validation of AlpacaFarm
We now analyze the correlation between rankings in simulation and on human data. Figure 2 shows the win-rate of methods AlpacaFarm (x-axis) with the win-rate from the human-based pipeline (y-axis). We see that the rankings have Spearman correlation of 0.98, which suggests that AlpacaFarm faith- fully captures the rankings among different LPF methods. This enables researchers develop models at low-cost in AlpacaFarm and transfer insights train models on real-world human interactions.

---

Annotator:
Human Pret
Trainer Pim
Evaluator pam
GPT4 pam
Model:
Human Pref
Simulated Psim
GPT4
ChatGPT
Davinci003
Figure 3: Our simulated annotators are cheap and have a high agreement with human annotators. We show price (X-axis) vs agreement (y-axis) of each annotator with the majority vote among 3 human annotations. Grey points the simulated annotators in the pool, the green shows the resulting pool of annotators (used for evaluation), the orange O shows the same pool with dom noise added during training. This does not change the implied reward function from but makes the learning problem more challenging. The blue shows the average of human annotators, and the red shows a single low variance GPT-4 annotator analyzed below.
Human mode agreement
0.66
0.63
0.60
0.57
100
10
102
$/1000 examples
Eval win-rate Phumis
0.55
Eval win-ratt powl
win-rate pzme
0.50
0.45
0.50
0.45
0.40
0.45
Expert Beration
Expert Iteration Best-of-n
Expert Iteration Best.af-n
0.35
PPO
PPO
PPO
0.30
0.5
1.0
1.
-0.55
0.00
0.25
0.50
1.5
2.5
3.0
3.5
Proxy reward, trained on
Proxy reward, trained on pmm
Proxy reward, trained on pam
(a) Human preferences
(b) AlpacaFarm
(c) Single-prompt GPT-4
Figure 4: Human and AlpacaFarm preferences result in over-optimization, while a naive single-prompe prompt GPT-4 simulator does not. Left: training and evaluation with human preferences M. Middle: training and evaluation with AlpacaFarm preferences Right: training and evaluation with single promppt GPT-4 preferences
The x-axis measures the average surrogate rewards on the eval set.
Inspecting these results more closely, we point out the two rank mismatches. The first comparison is SFT10k against SFT52k, where human annotators preferred SFT10k (44.3% 40.7%) while the simulator had the opposite preference (36.7% vs 39.2%, Table 1). The other mismatch is ChatGPT against PPO, where human annotators preferred PPO (55. % vs 52.9%) unlike the simulator (46.8% vs 61.4%) In both cases, these are not major mistakes, as we do not expect SFT52k to be much worse than SFT10k or for a LLaMA model substantially outperform ChatGPT.
4.3 Validating the pairwise preferences component
Having demonstrated that AlpacaFarm succeeds at the end-to-end validation of methods rankings, we now take closer look our pairwise preferences, showing that they agree with human annotators and replicate important qualitative features of model training. For additional details see appendix D.
Simulated annotators match human agreement. We begin by computing agreement levels between our simulated annotator and a majority vote 3 human annotators, comparing this the agreement level of held-out human annotator, shown in Figure 3. We find that our evaluator poval has 65% agreement rate with the human majority vote, which is similar to the held-out human agreement rate 66% At the same time, eval is 25x cheaper ($300 + $12 per 1000 examples). The training time annotator pim has lower agreement due to label flip noise but this does not mean that pinn is less faithful to human annotations, since this noise is unbiased and both annotators (pano poal) represent the same underlying preference function.
Figure also shows that we identified single prompts performing even better than poval with one prompts achieving 68% agreement. While this high agreement level is impressive, we do not use single
7

---

0.8
Simulated Win-rate (Demo)
0.7
Speaman Correlation: 0.90
0.6
R²
0.97
0.5
Figure 5: Correlation plot of simulated win- rates computed on AlpacaFarm's 's evaluation versus that on real-world interactions with the Alpaca Demo
0.4
0.3
0.2
0.1
0.1
0.2
0.3
0.4
0.5
0.6 0.7 0.8
Simulated Win-rate (AlpacaFarm)
prompts for AlpacalFarm, as single prompts do not replicate the inter-annotator variability important for simulator. Instead, randomizing over different annotators and injecting additional noise is needed match the distributional features and learning dynamics of human data, which we discuss next.
Simulated annotators replicate overoptimization. We now show that modeling annotator vari- ability in the simulated annotator (pmm) is necessary capture important qualitative features of LPF model training. To do so, we compare the behavior of three the best-performing models trained under Pam with those trained using the single GPT-4 prompt P GPT4 which has higher human agreement but little annotator variability.
Figure 4 shows the learning dynamics of these models for pairwise feedback by Phuman (left), Pam (middle), and P (right) as the PPO iteration count and rerank sample counts for best-of-n and expert iteration are increased. For both the human and AlpacaFarm preferences, models that more effectively optimize the surrogate reward (X-axis) improve in win-rate (y-axis) up until a point, where overoptimization of the reward occurs and win-rates decrease. In contrast, simple GPT-4 feedback shows no overoptimization, leading to the false conclusion that the LPF methods can be optimized for much longer than is actually the case. For example, Figure right shows best-of- 1024 be much better than PPO, which disagrees strongly with the results on human data.
Noted in prior work [21], overoptimization is the result of the proxy reward model R. being an imperfect estimate of the true reward R. We hypothesize that the strong overoptimization seen for human annotators is partly due to the (inter- and intra-) variability of their annotations, which degrades the quality of the reward model. To test this hypothesis, we measured the variance of each the three annotators by calculating the average error of held-out annotation to the majority vote over 3 random draws. At 0.26 for porl and 0.43 for pam AlpacaFarm is close the human variance of 0.35; in con- trast, the GPT-4 annotator has much lower variance at 0. Finally, in Appendix F.1 we ablate the Al- pacaFarm design more finely and find that added label noise provides the majority of the benefit for inducing overoptimization. Appendix D contains further analyses of bias and variability of annotators.
4.4 Validating the evaluation protocol
Finally, test evaluation data that combines existing open-source evaluation datasets. While have observed that this dataset is diverse (see Figure 13), it is unclear whether it evaluates performance for any type of real-world human usage. To resolve this, we measure method-level correlations against a set of real user interactions recorded on the Alpaca Demo [69]. We manually went through the interactions and identified 200 instructions that do not contain any personal identifying information, toxic or unsafe questions, and those that refer to the chatbot directly (c.g. "who are you developed by?"). The terms of use for the demo do allow us publicly release this data, but we use this data to evaluate the proposed evaluation set.
We use the 11 systems displayed in Figure 2, with the LPF methods trained in simulation, and evaluate them using pmm Figure 5 plots the simulated win-rates on the Demo instructions against those on the Al AlpacaFarm evaluation data. The two win-rates are strongly correlated 2n 0.97), indicating that Al AlpacaFarm evaluation data can serve proxy for evaluating methods on simple demo interactions.

---

Table AlpacaFarm evaluation results on baseline and LHF methods. For methods without * the left column shows win-rates when we train and evaluate in simulation, while the right column shows win-rates when we train and evaluate with human feedback, For models with *, those are not trained by us so the left and right columns respectively show simulated and human evaluation. Win-rates are computed against Davinci003. We did not evaluate Quark and Binary Reward Conditioning for human evaluation because they underperformed in development. ChatGPT and GPT-4 were prompted for responses shorter than 1000 characters to have length comparable to other methods.
Method
Simulated Win-rate (%)
Human Win-rate (%)
GPT-447
79.0 + 1.4
69.8 = 1.6
ChatGPT
61.4 + 1.7
52.9 + 1.7
PPO
46.8 1 1.8
55.1 +: 1.7
Best-of-1024
45.0 + 1.7
50.7 1 1.8
Expert Iteration
41.9 1 1.7
45.7 1 1.7
SFT 52k
39.2 1.7
40.7 1.7
SFT 10k
1.7
44.3 = 1.7
Binary FoedME
36,6 10.6117 1.7
37.9 = 1.7
Quark
35.6 + 1.7
Binary Reward Conditioning
32.4 + 1.6
Davinci001 *
24.4 + 1.5
32.5 1.6
I.LaMA 7B*
11.3 + 1.1
6.5 + 0.9
5
Benchmarking reference methods on the AlpacaFarm
We now study the performance reference methods in AlpacaFarm. Table contains the details of the main evaluation results (presented in Figure 2). In the rest of this section, we discuss our findings from these results, demonstrating that the conclusions we reach using human feedback could have been derived using AlpacaFarm at substantially lower cost. See Appendix C for experimental details.
5.1 Comparing LPF methods
Supervised fine-tuning is highly effective. Table shows that the SFT is effective and provides the majority of gains. SFT brings the base LLaMA model up from a simulator win-rate of 1 11% to 37% and human win-rate of 7% 44%. However, observe gain from SFT 10k SFT 52k.
PPO tops the LPF leaderboard. Among the LPF methods we study, PPO performs the best in both the simulator (47%) and human data (55%). Notably, with win-rate of 55%, the PPO trained with human feedback was preferred to ChatGPT by our annotators for single-turn instruction wwwectioonoooloiingg..mm This surprising result is likely because ChatGPT was prompted for responses shorter than 1000 charac- ters and both crowd-workers and simulators annotators preferred longer responses (see appendix D.2). For example, in AlapcaEval [35] (slightly different simulated annotator) show that the uncon- strained ChatGPT achieves 89% win-rate, much higher than the 44% win-rate for the PPO model.
Best-of-n is simple and competitive. Best-of-n sampling outperforms all LPF methods besides PPO. The performance best-of-n sampling points to the useful learning signals captured by the reward model and helps us understand how training-time LPF methods benefit from a reward model.
Expert Iteration and Quark lag behind. Even though expert iteration trains with best-of-n outputs, it trails best-of-n with 3-6% win-rate gap in both simulated and human feedback workflows. This result suggests that acquiring the improvement from best-of-n sampling is not straightforware and points why more complex learning algorithms like PPO might be useful. For Quark, although the rewards of samples improve during training, the resulting models did not outperform the SFT baseline.
Methods that directly learn from pairwise feedback do not perform well. We observe that binary reward conditioning and binary FeedME do not improve over the SFT 10k baseline. This result suggests that learning surrogate reward model may be important ingredient to LPF.
In appendix F.3 we analyze the outputs different models. We find major difference that L.PF methods produce longer outputs in both human and simulated feedback.

---

5.2 Using Alrm to train models directly for human deployment
A natural question is whether models trained in AlpacaFarm can directly perform well on human evaluation Table 2 shows that AlpacaFarm can be repurposed for this goal by using simulated annotators that maximize agreement rather than match human variability, i.e., using preferences from the single low-variance GPT-4 annotator Pau rather than our noisy annotators sim
Table 2: Model transfer results.
Table compares the best model PPO trained on pim with a similar model PPO trained on the GPT-4 annotator. We find that PPO only achieves 43% win-rate, while PPO achieves 50% win-rate. As context, we show win-rates of the initial SFT, the human PPO PPOnuman. and the best non-PPO human method 1est-of-16numan Overall, training in AlpacaFarm can provide good models for deployment, but still suffers a 5% performance gap relative to real human annotations.
sun
Method
Human Win-rate (%)
PPO numan
55%
Best-of- 16 1umman
51%
PPO sim SFT 10k PPO sim
50%
44%
43%
Limitations
Validation. Section 4 validates the use of AlpacaFarm, but there are nevertheless some limitations with the validation setting. First, the instructions we considered (even those from the real world-demo in section 4.4) are relatively simple and single-turn. Second, all models fine-tuned use LLaMA 7B as starting point. Finally, Human validation is based on feedback from 13 crowd-workers, which may not reflect broader human preferences and seem to have biases such preferences for longer outputs as highlighted in appendix D.2. Although we do not expect the previous limitations to significantly affect the usefulness of AlpacaFarm, we encourage users to be vigilant when using any simulator and hope to further validate AlpacaFarm as more datasets and base models become available.
Assumption of an oracle LLM. AlpacaFarm assumes that we can access an oracle LLM, much more powerful than the investigated models, that can be used to approximate human feedback. While this may be true in research settings, it is not always the case in practice. We believe that investigating the use of the same model for the simulator and for learning is promising direction for future work.
AlpacaFarm for fine-grained development. Section 4 shows that AlpacaFarm is effective for model selection and can replicate important behaviors such as over-optimization, We have neverthe- less found that suitable hyperparameters for learning algorithms can be different for training with sim- ulated feedback compared to human feedback. For example, due to changes in the scale of values of the surrogate reward model, the range of suitable KL regularization coefficients for RLHF is different.
Simulated annotators. A key component of AlpacaFarm is the simulated annotators. In section 4.3 and appendix we showed that our annotators have high agreement with human annotators and exhibit similar biases, such preferences for longer outputs and lists. There are nevertheless some biases that are specific to simulated annotators. For example, we found that simulated annotators prefer the first outputs shown in the prompt and outputs that come from the same model as the simulator. Although we controlled for those biases by randomizing the order of output in the prompt, and using the same training data for all considered methods, there may be other biases that we have not identified. For better automatic annotators and further analysis refer to AlpacaEval [35].
7 Related works and outlook
The idea of building a simulator is closely related to recent efforts in automatic model evaluations [16, 55, 15, 53, 37] and constitutional AI [8]. However, our work crucially differs in our goal of building a simulator, which requires us to mimic distributional properties such as inter-annotator disagreement. We discuss these differences and the broader context of our work in Appendix A.
We showed that AlpacaFarm substantially lowers the cost and iteration time for developing methods that learn with pairwise feedback. AlpacaFarm provides a blueprint for constructing simulators for Al research that requires human supervision, and we view as an exciting opportunity to expand this simulation approach to data from other domains and alternative forms of human feedback.
10

---

Acknowledgments and Disclosure of Funding
We thank the Stanford Center for Research on Foundation Models (CRFM), Stanford HAI and Stability AI for compute support. Data collection efforts were supported through the Tianqiao & Chrissy Chen Institute and an Open Philanthropy grant. XL is supported by Stanford Graduate Fellowship. RT is supported by the NSF GRFP under Grant No. DGE 1656518. YD is supported by a Knights-Hennessy Scholarship.
References
[1] Gati Aher, Rosa Arriaga, and Adam Tauman Kalai. Using large language models simulate multiple humans. arXiv preprint arXiv:2208. 10264, 2022.
[2] Thomas Anthony, Zheng Tian, and David Barber. Thinking fast and slow with deep learning and tree search. Advances in neural information processing systems, 30, 2017.
[3] Lisa P Argyle, Ethan C Busby, Nancy Fulda, Joshua Gubler, Christopher Rytting, and David Wingate. Out of one, many: Using language models to simulate human samples. arXiv preprint MXXi:2200900899 2022.
[4] Vamsi Aribandi, Yi Tay, Tal Schuster, Jinfeng Rao, Huaixiu Steven Zheng, Sanket Vaibhav Mchta, Honglei Zhuang, Vinh Q Tran, Dara Bahri, Jianmo Ni, et al. Ext5: Towards extreme multi-task scaling for transfer learning. arXiv preprint arXiv:2111. 109552, 2021.
[5] Amanda Askell, Yuntao Bai, Anna Chen, Dawn Drain, Deep Ganguli, Tom Henighan, Andy Jones, Nicholas Joseph, Ben Mann, Nova DasSarma, Nelson Elhage, Zac Hatfield-Dodds. Danny Hernandez, Jackson Kernion, Kamal Ndousse, Catherine Olsson, Dario Amodei, Tom Brown, Jack Clark, Sam McCandlish, Chris Olah, and Jared Kaplan. A general language assistant as a laboratory for alignment, 2021.
[6] Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, Nicholas Joseph, Saurav Kadavath, Jackson Kernion, Tom Conerly, Sheer El-Showk, Nelson Elhage, Zac Hatfield-Dodde Danny Hernandez, Tristan Hume, Scott Johnston, Shauna Kravec, Liane Lovitt, Neel Nanda, Catherine Olsson, Dario Amodei, Tom Brown, Jack Clark, Sam McCandlish, Chris Olah, Ben Mann, and Jared Kaplan. Training helpful and harmless assistant with reinforcement learning from human feedback, 2022.
[7] Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al. Constitutional ai: Harmlessness from ai feedback. arXiv preprint arXiv:2212.08077. 2022.
[8] Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al. Constitutional ai: Harmlessness from ai feedback. arXiv preprint arXiv:2212.08077 2022.
[9] Michiel Bakker. Martin Chadwick, Hannah Sheahan, Michael Tessler, Lucy Campbell- Gillingham, Jan Balaguer, Nat McAleese, Amelia Glaese, John Aslanides, Matt Botvinick, et al. Fine-tuning language models to find agreement among humans with diverse preferences. Advances Neural Information Processing Systems, 35:38176-3881 2022.
[10] R. Bommasani al. On the opportunities and risks of foundation models. arXiv preprint arXiv:2108.07258. 2021.
[11] Ralph Allan Bradley and Milton E Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39(3/4):324-344 1952.
[12] Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym. arXiv preprint
2016.
11

---

[13] T. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan, Shyam, G. Sastry, A. Askell, Agarwal, A. Herbert- Voss, G. Krueger. T. Henighan, R. Child, A. Ramesh, D. M. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. Chess, Clark, C. Berner, S. McCandlish, A. Radford, I. Sutskever, and D. Amodei. Lan- guage models are few-shot learners. In Advances in Neural Information Processing Systems (NeurIPS), 2020.
[14] Angelica Chen, Jérémy Scheurer, Tomasz Korbak, Jon Ander Campos. Jun Shern Chan, Samuel R Bowman, Kyunghyun Cho, and Ethan Perez. Improving code generation by training with natural language feedback. arXiv preprint 11X4::2303.16749, 2023.
[15] Cheng-Han Chiang and Hung-yi yi Lee. Can large language models be an alternative to human evaluations? arXiv preprint arXiv:2305.01937, 2023.
[16] Cheng-Han Chiang and Hung-yi yi Lee. Can large language models be an alternative to human evaluations? arXiv preprint arXiv:2305.01937, 2023.
[17] Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. Vieuna: An open-source chatbot impressing gpt-4 with 90% chatgpt quality, March 2023.
[18] Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. Advances in neural information processing systems, 30, 2017.
[19] Linxi Fan, Yuke Zhu, Jiren Zhu, Zihua Liu, Orien Zeng, Anchit Gupta, Joan Creus-Coste Silvio Savarese, and Li Fei-Fei. Surreal: Open-source reinforcement learning framework and robot manipulation benchmark. In Conference on Robot Learning, pages 767-782. PMLR, 2018.
[20] C Daniel Freeman, Erik Frey, Anton Raichuk, Sertan Girgin, Igor Mordatch, and Olivier Bachem. Brax-a differentiable physics engine for large scale rigid body simulation. arXiv preprint arXiv:2106. 13281, 2021.
[21] Leo Gao, John Schulman, and Jacob Hilton. Scaling laws for reward model overoptimization. arXiv preprini arXiv:2210.10760) 2022.
[22] Xinyang Geng, Arnay Gudibande, Hao Liu, Eric Wallace, Pieter Abbeel, Sergey Levine, and Dawn Song. Koala: A dialogue model for academic research, March 2023.
[23] Amelia Glaese Nat McAleese, Maja Trębacz, John Aslanides, Vlad Firoiu, Timo Ewalds, Maribeth Rauh, Laura Weidinger, Martin Chadwick, Phoebe Thacker, et al. Improving alignment of dialogue agents via targeted human judgements. arXiv preprint arXiv:2209.14375. 2022.
[24] Braden Hancock, Antoine Bordes, Pierre-Emmanus Mazare, and Jason Weston. Learning from dialogue after deployment: Feed yourself, chatbot! arXiv preprint 1rXiv:1901.054155 2019.
[25] Arthur Juliani, Vincent-Piere Berges, Ervin Teng, Andrew Cohen, Jonathan Harper, Chris Elion, Chris Goy, Yuan Gao, Hunter Henry, Marwan Mattar, et al. Unity: A general platform for intelligent agents. arXiv preprint arXiv: (00.xv:88000000 0209002627, 2018
[26] S. Kakade, Y. W. Teh, and S. Roweis. An alternate objective function for Markovian fields. In International Conference on Machine Learning (ICMI.,, 2002.
[27] Saketh Reddy Karra, Son Nguyen, and Theja Tulabandhula. Ai personification: Estimating the personality of language models. arXiv preprint arXiv:2204. 0 /2000, 2022.
[28] N. S. Keskar, B. McCann, L. R. Varshney, C. Xiong, and R. Socher. CTRL: A Conditional Transformer Language Model for Controllable Generation. arXiv preprint arXiv: 1909.99999.0 05858, 2019.
[29] Samuel Kiegeland and Julia Kreutzer. Revisiting the weaknesses of reinforcement learning for neural machine translation. arXiy preprint arXiv:2106. 08940,, 2021.
12

---

[30] Tomasz Korbak, Kejian Shi, Angelica Chen, Rasika Bhalerao, Christopher L Buckley, Ja- son Phang, Samuel R Bowman, and Ethan Perez. Pretraining language models with human preferences. arXiv preprint arXiv:2302. 08582, 2023.
[31] Julia Kreutzer, Shahram Khadivi, Evgeny Matusov, and Stefan Riezler. Can neural machine translation be improved with user feedback? arXiv preprint arXiv:1804.0595) 00958, 2018.
[32] Tsz Kin Lam, Julia Kreutzer, and Stefan Riezler. A reinforcement learning approach to interactive-prediitive neural machine translation. arXiv preprini 1rXiv:/805.01555. 2018.
[33] Kimin Lee, Hao Liu, Moonkyung Ryu, Olivia Watkins, Yuqing Du, Craig Boutilier, Pieter Abbeel, Mohammad Ghavamzadeh, and Shixiang Shane Gu. Aligning text-to-image models using human feedback. arXiv preprint arXiv:2302. 2023.
[34] Jiwei Li, Alexander H Miller, Sumit Chopra, Mare' Aurelio Ranzato, and Jason Weston. Dia- logue learning with human-in-the-loop. arXiv preprini arXiv: 16/ 1.09823, 2016.
[35] Xuechen Li, Tianyi Zhang, Yann Dubois, Rohan Taori, Ishaan Gulrajani, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Alpacaeval: An automatic evaluator of instruction-followin models. https://gat htt:://gittbb thub. com/tatsu- lab/ alpaca_evall 2023.
[36] H Liu, C Sferrazza, and P Abbeel. Chain of hindsight aligns language models with feedback. arXiv preprint Sttii::302.0026772 2023.
[37] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. arXiv preprint 08X44 2023.
[38] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee, Visual instruction tuning, arXiv preprint arXiv:2304.08885, 2023.
[39] Yang Liu, Dan Iter, Xu Yichong, Wang Shuohang, Xu Ruochen, and Chenguang Zhu. G-eval: Nlg evaluation using gpt-4 with better human alignmentg. arXiv preprint arXiv:2303. 16634, 2023.
[40] Shayne Longpre, Le Hou, Tu Vu, Albert Webson, Hyung Won Chung, Yi Tay, Denny Zhou, Quoc V Le, Barret Zoph, Jason Wei, et al. The flan collection: Designing data and methods for effective instruction tuning. arXiv preprint arXiv:2301.13688 2023.
[41] X. Lu, S. Welleck, J. Hessel, L. Jiang, L. Qin, P. West, P. Ammanabrolu, and Y. Choi. Quark: Controllable text generation with reinforced unlearning. In Advances in Neural Information Processing Systems, 2022.
[42] Ximing Lu, Sean Welleck, Jack Hessel, Liwei Jiang, Lianhui Qin, Peter West, Prithviraj Ammanabrolu, and Yejin Choi. Quark: Controllable text generation with reinforced unlearning. Advances in neural information processing systems, 35:27591-27609, 2022.
[43] Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao. Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, et al. Self-refine: Iterative refinement with self-feedback. arXiv preprint arXiv:2303.17655 2023.
[44] Swaroop Mishra, Daniel Khashabi, Chitta Baral, and Hannaneh Hajishirzi. Cross-task gener- alization via natural language crowdsourcing instructions. arXiv preprint arXiv:2104.08773 2021.
[45] Khanh Nguyen, Hal Daumé III, and Jordan Boyd-Graber. Reinforcement learning for bandit neural machine translation with simulated human feedback. arXiv preprint arXiv: (eoo::777000000 2017.
[46] OpenAl. Introducing chatgpt.
[47] OpenAl. Model index for researchers.
[48] OpenAl. Gpt-4 technical report, 2023,
13

---

[49] Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang. Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback, 2022.
[50] Joon Sung Park, Joseph C O''Brrien Carrie J Cai, Meredith Ringel Morris, Percy Liang, and Michael $ Bernstein. Generative agents: Interactive simulacra of human behavior. arXiv preprint WrXiv:2334.034440 2023.
[51] Joon Sung Park, Lindsay Popowski, Carrie Cai, Meredith Ringel Morris, Percy Liang, and Michael S Bernstein. Social simulacra: Creating populated prototypes for social computing systems. In Proceedings of the 35th Annual ACM Symposium on User Interface Software and Technology, pages 1-18, 2022.
[52] Romain Paulus, Caiming Xiong, and Richard Socher. A deep reinforced model for abstractive summarization. arXiv preprint 11Xiv:1705.04304. 2017.
[53] Baolin Peng, Chunyuan Li, Pengcheng He, Michel Galley, and Jianfeng Gao. Instruction tuning with gpt-4. arXiv preprint arXiv:2304.03277, 2023.
[54] Baolin Peng, Chunyuan Li, Pengcheng He, Michel Galley, and Jianfeng Gao. Instruction tuning with gpt-4, arXiv preprini arXivv2304.03277) 2023.
[55] Ethan Perez, Sam Ringer, Kamilé Lukošiūte, Karina Nguyen, Edwin Chen, Scott Heiner, Craig Pettit, Catherine Olsson, Sandipan Kundu, Saurav Kadavath, et al. Discovering language model behaviors with model-written evaluations. arXiv preprint arXiv:2212.09251, 2022.
[56] Ethan Perez, Sam Ringer, Kamilé Lukošiūté, Karina Nguyen, Edwin Chen, Scott Heiner, Craig Pettit, Catherine Olsson, Sandipan Kundu, Saurav Kadavath, et al. Discovering language model behaviors with model-written evaluations. arXiv preprint arXiv:22/2.09255 2022.
[57] Rajkumar Ramamurthy, Prithviraj Ammanabrolu, Kianté Brantley, Jack Hessel, Rafet Sifa, Christian Bauckhage, Hannaneh Hajishirzi, and Yejin Choi. Is reinforcement learning (not) for natural language processing?: Benchmarks, baselines, and building blocks for natural language policy optimization. arXiv preprint arXiv:2210.01241, 2022.
[58] Victor Sanh, Albert Webson, Colin Raffel, Stephen H Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Teven Le Scao, Arun Raja, al. Multitask prompted training enables zero-shot task generalization. arXiv preprint lrXiv:2110.082000 2021.
[59] William Saunders, Catherine Yeh, Jeff Wu, Steven Bills, Long Ouyang, Jonathan Ward, and Jan Leike. Self-critiquing models for assisting human evaluators. arXiv preprint arXiv:2206.05802, 2022.
[60] Jérémy Scheurer, Jon Ander Campos, Tomasz Korbak, Jun Shern Chan, Angelica Chen, Kyunghyun Cho, and Ethan Perez. Training language models with language feedback at scale. arXiv preprint arXiv:2303. 16755, 2023.
[61] John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. High- dimensional continuous control using generalized advantage estimation. arXiv preprint W.0vv:550.024:3 2015.
[62] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms, 2017.
[63] Weiyan Shi, Emily Dinan, Kurt Shuster, Jason Weston, and Jing Xu. When life gives you lemons, make cherryade: Converting feedback from bad responses into good labels. arXiv preprint arXiv:2210. 15893, 2022.
[64] D. Silver, J. Schrittwieser, K. Simonyan, I. Antonoglou, A. Huang, A. Guez, T. Hubert, L., M. Lai, A. Bolton, et al. Mastering the game of go without human knowledge. Nature, 55017676):354-55,, 2017.
14

---

[65] Charlie Snell, Ilya Kostrikov, Yi Su, Mengjiao Yang, and Sergey Levine. Offline rl for natural language generation with implicit language q learning. arXiv preprint arXiv:2206. 18771, 2022.
[66] Artem Sokolov, Stefan Riezler, and Tanguy Urvoy. Bandit structured prediction for learning from partial feedback in statistical machine translation. arXiv preprint arXiv: 2016.
[67] Nisan Stiennon, Long Ouyang, Jeff Wu, Daniel M. Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul Christiano. Learning to summarize from human feedback, 2020.
[68] Colin Summers, Kendall Lowrey, Aravind Rajeswaran, Siddhartha Srinivasa, and Emanuel Todorov. Lyceum: An efficient and scalable ecosystem for robot learning. In Learning for Dynamics and Control, pages 793-803. PMLR, 2020.
[69] Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Alpaca: A strong, replicable instruction-follovin followiig modely, March 2023.
[70] Yuval Tassa, Yotam Doron, Alistair Muldal, Tom Erez, Yazhe Li, Diego de Las Casas, David Budden, Abbas Abdolmaleki. Josh Merel, Andrew Lefrancq, et al. Deepmind control suite. arXiv preprint arXiv: 1001:0100000000 006900, 2018
[71] Emanuel Todorov, Tom Erez, and Yuval Tassa Mujoco: A physics engine for model-based control. In 2012 TEEE/RS/ international conference on intelligent robots and systems, pages 5026-5033. IEEE, 2012.
[72] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timo- thée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302. 13971, 2023.
[73] Adaku Uchendu, Zeyu Ma, Thai Le, Rui Zhang, and Dongwon Lee. Turingbench: A benchmark environment for turing test in the age of neural text generation. arXiv preprint arXiv:2109.1 13296, 2021.
[74] Jonathan Uesato, Nate Kushman, Ramana Kumar, Francis Song, Noah Siegel, Lisa Wang, Antonia Creswell, Geoffrey Irving, and Irina Higgins. Solving math word problems with process- and outcome-based feedback, 2022.
[75] Yizhong Wang. Yeganch Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith, Daniel Khashabi, and Hannaneh Hajishirzi. Self-instruct: Aligning language model with self generated instruc- tions, 2022.
[76] Yizhong Wang, Swaroop Mishra, Pegah Alipoormolabashi, Yeganeh Kordi, Amirreza Mirzaei, Atharva Naik, Arjun Ashok, Arut Selvan Dhanasekaran, Anjana Arunkumar, David Stap, et al. Super-naturaiistrrctions Generalization via declarative instructions on 1600+ nlp tasks. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 5085-5109, 2022.
[77] Jason Wei, Maarten Bosma, Vincent Y Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoc V Le. Finetuned language models zero-shot learners. arXiv preprint arXiv:2109.01653 2021.
[78] J. E. Weston. baslog-based language learning. In Advances in Neural Information Processing Systems (NeurIPS), pages 829-837, 2016.
[79] Y. Wu, M. Schuster, Z. Chen, Q. V. Le, M. Norouzi, W. Macherey, M. Krikun, Y. Cao, Q Gao, K. Macherey, et al. Google's 's neural machine translation system: Bridging the gap between human and machine translation. arXiv preprint
1609.08//10010 2016.
[80] Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin. Zhuohan Li. Dacheng Li. Eric. P Xing, Hao Zhang, Joseph E. Gonzalez, and lon Stoica. Judging auddde---udge with mt- bench and chatbot arena, 2023.
15

---

A Related work
Instruction following. There have been an number of works studying instruction following as cross-task generalization across pool of NLP tasks [44, 77, 58, 4, 76]. Our work focuses on recent trend in instruction following methods which increasingly focus on real world human interaction patterns [49, 6], rather than collections of existing NLP benchmarks. For example, InstructGPT was developed on user instructions submitted to OpenAl API [49]. Our work builds upon these works by attempting to bridge the gap between the ease of development and evaluation of traditional academic benchmarks and the more complex algorithms and real- weal-world settings of recent works on instruction following.
Simulating human feedback. Constitutional AI [7] simulates human feedback with AI feedback for model development to improve harmlessness and helpfulness. Alpacalaarm, on the other hand, simulates human feedback with API LLMs so that simulated experiments reflect the outcomes of experiments performed with real human feedback. Due to the difference in goals, the construction and usage of the feedback simulator are different in the two settings. For example, AlpacaFarm's s simulator perturbs LLM preferences with bit-flip label noise to mimic the noisiness of human annotation, whereas Constitutional Al's simulator doesn't 't inject extra noise.
The evaluation aspects of our work are related to growing line of work on simulating human annotation for evaluation [15, 56, 15, 54, 38, 39]. Our core evaluation and feedback mechanism makes use of the same underlying ideas, but our work is distinguished by a focus using pairwise feedback for training, as well as careful validation beyond per-example agreement metrics. AlpacaFarm shows that LLM feedback can capture method-level correlations well as qualitatively important features of human annotation for phenomena such as overoptimization.
Our goal of emulating human annotators also connects to work on simulating humans with LMs based on personas [51, 50, 3], well works that simulate human behavior in the context of cognitive science, social science, and economics [73, 27]. Our work complements these works by showing that simulated L.LM annotators can replicate many of the qualitative features of training on pairwise human feedback.
More broadly, building a simulator environment to enable low-cost experimentation is common in the field of reinforcement learning underlying motivations, but instead of simulating physical systems, AlpacaFarm simulates human preference feedback.
and robotics [12, 71, 70, 68, 19, 25, 20). Our work shares the same
Methods for learning from feedback. To hold annotation cost constant across learning methods, we have focused only on methods that learn from pairwise feedback in this work. However, there exist methods in the literature other than those explored in AlpacaFarm that can incorporate alternative sources of feedback such as natural language [78, 34, 24, 63, 59, 14, 60, 43], numeric ratings [47, 33], or execution traces [14]. We view extensions of AlpacaFarm to these settings as exciting future work.
We have included a set of RI. algorithms in our study that optimize the surrogate reward, but this set is by means comprehensive. RL research applied to NLP has long history [79, 66, 29, 52, 45, 32, 31, 57, 65), and we expect future work in this direction to benefit from the ideas and artifacts in AlpacaFarm.
16

---

B Reference LPF methods on AlpacaFarm
We provide more thorough description of the methods in AlpacaFarm along with the custom modifications we make. For more on hyper-parameter tuning, experiment details, and further ablations, please see Appendix C.
Our methods fall into two categories based on whether they fit a surrogate reward model as part of the learning process. In addition, to those methods which are trained in AlpacaFarm, also include well-known baseline methods from the OpenAI API.
APT baseline methods. In both the human and the simulated feedback workflows, we evaluate the following methods from the OpenAl API: GPT-4, ChatGPT, Davinci001, and implicitly Davinci003, which is the baseline model we compare every model with. Outputs from all these models are sampled at temperature 0.7 and with top-p 1.0. For the text models, we use the same prompt as all other reference methods in AlpacaFarm. For both chat models (ChatGPT and Davinci003), we had to change prompts for two reasons. First, those models require prompts in a chat format, which is different from the text format. Second, we found that those models generated sequences that were much longer than the rest of our models, which were trained to output sequences of less than 300 tokens. We thus ask in the system prompt for response that shorter than 1000 and 500 characters respectively for ChatGPT and GPT-4, which we found to give shorter answers while working similarly to raw prompt that does not mention the length.
B.1 Methods that directly learn from pairwise feedback
Binary FeedME. FeedME a method proposed by OpenAl [47] that incorporates human feedback with supervised fine-tuning on model generations that are rated 7/7 by human labelers. We adapt this approach to the pairwise feedback setting and call this baseline binary FeedME. This approach fine-tunes the SFT model the chosen response in each preference pair with supervised learning.
Binary Reward Conditioning. Motivated by controllable generation through conditioning [28, 36, 30, 22], we propose binary reward conditioning, baseline method that fine-tunes the SFT model with the feedback data paiirwis by conditioning instances with either a positive or negative control token. Specifically, for each instance (Jc, yo, z) E 'D pairwises the string concatenation of instruction r and response 1/s denoted as [ar, V/ prepended with the positive token and used in supervised fine-tuning (similarly [z,, 3/1=> is prepended with the negative token). This process creates modified demonstration dataset that is double the size of D pairwise. At test time, we draw samples from the fine-tuned model conditioned on the positive token.
B.2 Methods that optimize a surrogate reward function
We now describe methods that incorporate feedback by first building a surrogate reward model with pairwise feedback data. To start, we describe the step of training the surrogate reward model.
To train a parameterized surrogate Rig, one can maximize the log-likelihood of the preferences e under the Bradley-Terry model [11]
exp( (2( ,)(")) (xp(Rx(x)1) ("))) + exp( R (ar(), .y ()))
() maximizes Elog log P(26) (1) .y0 .3 .
(1) (1) Elog log
1
Once the surrogate reward model is trained, both training and inference algorithms can optimize against the reward model rather than query pairwise feedback. While this can be a powerful approach, we will see that it can also lead to over-optinization [21] where models learn to exploit the reward model rather than achieve high true reward. We now describe 4 methods that leverage the surrogate reward model.
Best-of-n Sampling. Best-of-n sampling (or re-ranking) [67, 5, 23, 9] is common inference-time method that aims to improve the generation quality. Given an input the method returns the response with the highest surrogate reward value among n i.i.d. responses drawn from the SFT model. While simple to implement and useful as a baseline, this approach incurs high inference costs.
17

---

Expert Iteration. Expert iteration [2, 64, 74] is a technique that has recently been used to train language models. We adapt this approach in AlpacaFarm as a two-step method. In the first step, we perform best-of-n sampling and store the generated samples. In the second step, we fine-tune PSFT on these samples with supervised learning. While prior work applies expert iteration for multiple rounds by performing best-of-n sampling again for intermediate models, we focus performing a single round. In Appendix C, we include our preliminary study multi-round expert iteration.
Proximal Policy Optimization. Proximal Policy Optimization [PPO; 26, 62] is popular RL algorithm that has been recently used to develop InstructGPT [49] and CharGPT [46]. When applied to fine-tune LMs with RLHF, PPO maximizes the following KL-regularized objective w.r.t model parameters
Expe((x), pmpp(()lz) Ro( - Blg
pe (y x)
PSET (y x))
where p(x)) is an unlabeled instruction distribution, pa (y x) is fine-tuned from the psFT model, and IR is regularization coefficient. Each step of PPO alternates between drawing samples from the current policy and performing gradient updates based on the pool of samples with importance sampling and clipping.
Quark. Quark is inspired by reward conditioning and has been shown to be effective for controllable generation tasks. Like binary reward conditioning, Quark on train sequences with prepended control tokens. Unlike binary reward conditioning, Quark bins model samples into multiple groups based on the reward value, adds KL and entropy regularization, and repeats the entire process across multiple rounds.
In our preliminary analysis, we find the top-quantile variant reported in [41], i.e., only training on the best reward group, to perform the better than the all all-quaatiles variant which trains on all groups
18

---

C
Details on methods implementation and hyperparameters
For all hyperparameters and code see https: http:///gggchb..cc//aataaaaaaa/
C.1
PPO
We follow an existing PPO implementation for fine-tuning language models, but also introduce modifications. First, off-the-shelf PPO implementations for language model fine-tuning tend to normalize the estimated advantage for each minibatch, We found this led to training instabilities for small minibatch sizes and instead normalize the advantage across the entire batch of rollouts obtained for each PPO step. Second, we initialize the value model from the reward model as opposed to the SFT model, following more recent documented practice [49] (the authors did not release code). Our preliminary experiments showed that initializing from reward worked much better than initializing from SFT for maximizing the surrogate reward.
We tuned hyperparameters to improve training stability and reduce convergence time so that experi- ments can reliably finish with relatively tight compute budgets. In the end, we settled on batch size of 512 for each PPO step, which consisted of 2 epochs of gradient steps each performed with a batch of 256 rollouts. We used peak learning rate of 10-5 which decayed 0 throughout training. We clipped the gradient by Euclidean norm with a threshold of 1. We trained for 10 full passes over the unlabeled set, which amounts 390 PPO steps. Performance typically peaked very early on during training (see Figure 4). We set À and y both to I for generalized advantage estimation [61]. We used a fixed KL regularizer coefficient as opposed an adaptive one. We tuned the coefficient value for both simulated and human PPO, and settled with 0.02 for human PPO, and 0.002 for simulated PPO. We note that suitable values for the KL regularizer coefficient depend on the early stopping criteria and the scale of surrogate reward values.
C.2 Quark
We re-implement Quark for our needs and make several modifications. First, the original Quark formulation accumulates rollouts during training and stores them in a pool that consistently grows. We found this led to overhead that increased during training (since after each rollout batch is generated, the pool is expanded and rollouts in the pool are re-sorted by their reward values). To operate under reasonable compute budget, we discard previous rollouts once a new batch of rollouts generated. In other words, the pool is reset once rollout is performed. This modification made the compute cost constant throughout training and thus more predictable overall. Second, we found that training on rollouts of more bins led to worse efficiency for reward optimization, and thus opted to train only on rollouts of the top-scoring bin (best-quantile variant in the original paper [42]). Preliminary ablations on simple sentiment task showed that any potential loss in perplexity for the best-quantile variant can be compensated by turning up the KL regularizer. Lastly, we found the entropy penalty used in the original Quark formulation to give no benefit for working with instruction following. Small entropy penalty terms were enough to cause big degradations in text generation quality in terms of fluency.
For the official run with reported results, we used a KL regularizer coefficient of 0.05, a peak learning rate of 3 x 10-6 which decayed to 0 throughout training. Each Quark step had batch size 512 for rollout, and epochs of gradients updates each with batch size 256. We clipped the gradient by Euclidean norm with threshold of 1. We trained for 10 full passes over the unlabeled set, which amounts to 390 Quark steps.
http:://gghhuu..coo//pea/////aa/ human- www.--hannnppeeeeeeeeee
19

---

D Pairwise preference simulation
D.I Details about simulated annotators
For all our simulated annotators we used OpenAl API to generate outputs. We first discuss below the overall design choices for all our simulators below, and then discuss annotator pool below in more detail. For all the actual prompts we used refer to https: //::://ghhtt com/tatsu lab/ alpaca_ farm.
Randomized order. For each annotator, we randomize the ordering between the two outputs to annotate, i.c., we randomly choose which output is the first and which is the second. We found randomization to be important given that the first output often preferred by simulated annotators.
Prompts with and without inputs. Following the Alpaca dataset [69] and self-instruct framework [75] some instructions have associated inputs, while others do not. For each annotator, we thus write two corresponding prompts, one for instructions with inputs and one for instructions without inputs. Both prompts are essentially the same but in-context examples differ in the presence of the input.
Batching for GPT4. When adding in-context examples, prompts can become relatively long, which leads to high-cost and waiting time when using GPT-4 as simulator. To decrease cost and increase annotation speed, we amortize the cost of in-context examples by providing a batch of instruction- output pairs to annotate at once by GPT-4. For our simulated annotator we use maximum batch size of but found during development that we could fit batch size up to 20 the context window without significantly decreasing performance. To improve performance when using batching, we found it useful to provide few in-context examples in a batched format and to index every component of an annotation (instruction, input, output, ...).
Improving parsing for ChatGPT. Overall we found ChatGPT to be much more sensitive and harder to use as a simulator. In particular, we found it to be more sensitive to the prompt format and to often fail to generate annotations that could be parsed, e.g., by responding "Neither is better, this depends on personal preferences" despite being explicitly instructed to choose preference. We found two tricks to be effective to make ChatGPT's more parsable. First, we add negative bias to tokens such as "Neither" and "Both" and positive bias to the tokens that we hoped to match. We found the aforementioned biasing of tokens work well but it can be problematic when using Chain of Thought reasoning. A second trick that we found to be effective is to ask ChatGPT to generate JSON object that contains a string field with short explanation (Chain of Thought) and boolean field that indicates whether the first output was preferred.
Now that we have discussed the overall design choices for simulated annotators, we discuss in more detail the prompts and parameters for each of our annotators.
AlpacaFarm's evaluation annotators p ovall To try to match the bias and variance of human annotators, we use a pool of 13 simulated annotators that were developed at different stages of the project. In particular, use the following sources of variations:
Models. Five of the annotators are powered by GPT-4, four by ChatGPT, and four by Davinci003. The difference between different annotators for the same model is mostly the prompt.
In-context examples. Prompts for the same models use different numbers of in-context examples.
Prompt format. We use different prompt formats between and for the same model. For example different batch sizes and different formats of outputs (JSON vs raw text).
Preferences. Two of the GPT4 annotators are explicitly prompted to prefer sequences that are respectively long and short.
Sampling. For each annotator in the pool, we use a sampling temperature of 1.0 with top p also 1.0. The high temperature means that we have variability that arises from sampling.
20

---

Annotator:
Human Pret
Trainer Pim
Evaluator pam
GPT4 pam
Model:
Human Pref
Simulated Psim
GPT4
ChatGPT
Davinci003
Figure 6: Our simulated annotators achieve relatively low bias with human annotators and match human vari- ance. The y-axis shows the estimated bias, i.e., the error between the majority vote of simulated annotators and the majority vote of 4 human annotators. The x-axis shows the estimated variance, i.c., the error between a held-out annotation and the majority vote of the other three annotators. The bias of humans by definition 0, and variance is shown with blue line. Grey points are all the annotators in our simulated pool, the green point shows the resulting pool of annotators (which we use for evaluation), the orange point shows the same simulated pool with additional noise (which we use for training), the blue point the average human annotator, and the red point shows single low variance GPT-4 annotator we analyze.
0.44
0,40
0.36
0.32
0.0
0.1
0.2
0.3
0.4
Variance
AlpacaFarm's training annotators Psim Our simulated annotators for training are the same as the evaluation annotators Psi except that we flip the output with 0.25 probability. We implement this by taking mixture between paval and independent Bernoulli random variable with probability 0.5. This means that we only need to label half of the outputs for training, which makes it 2x faster and cheaper.
GPT4. For the GPT4 annotator Psi one of the prompts from our simulated pool of annotators. For GPTT4 temperature 0, i.e., deterministic annotations.
prompt with batch size five that corresponds to
D.2
Additional results
We now provide additional results for understanding pairwise annotators. For more results and code for generating plots see https://github.. cttp:/touotttttuutoaoeeeooo lain-llabaaaae hlbb/aaaca_evaa..
Our pool of annotators has low bias and matches human variance. Figure 6 shows the estimated bias (y-axis) and variance (X-axis) of simulated evaluators. We see that single evaluators have smaller variance (less than 0.2) than humans (blue line, 0.34). This lack of variability makes emulating it with a proxy reward very easy and leads to unrealistic over-optimization properties in the simulator, as seen in Figure 4. Using a pool of annotators (green point) for evaluation and additionally adding noise (orange) during training gives an estimated variance significantly closer to humans (blue line 0.35). We hypothesize that this is necessary for the simulator to show similar over-optimization behavior as humans. Concerning the bias, we find that our simulated annotators for evaluation peval and training prain both have low bias values (0.38 and 0.35) par with one of our best GPT-4 annotators (0.33).
Variability pool of annotators mostly comes from the underlying model. Figure 7 show the pairwise agreement between all annotators in our pool and all other annotators including the majority vote of humans (first column) and single humans (second column). The desired high variance corresponds to low values on the diagonal (annotators disagree with themselves) and low bias corresponds to high values in the first column (high agreement with the mode of humans). As in Figure 6, we see that our pool of annotators oooal has low bias and high variance. Figure 7 also shows that the largest source of variability between annotators comes from the underlying model, as illustrated by the clusters that arise from GPT4, ChatGPT and Davinci003 annotators.
21

---

Turkers
0.62
Alpazafiam's Noisy Armotators
Alpacafarm's Annotators
Bd
01
0163
Greedy GFT4
6:5
3.3
GPT4 #2
667
0.63
6162
9:12
0.66
6:64
0.67
GPT4
GPT4
8/6
0:
0:92
0.04
5:84
0:83
6:92
6:96
GPT4
3.69
0:77
6.79
ChatGPT
0:00
3.73
0:03
0.70
8.70
ChatGPT #2
0160
0666
1-11
ChatGPT
ChaRGPT #4
661
0.00
0.00
0.12
0.87
C:06
15
C:05
0
0.64
D.72
2.7
0.53
8:53
Davincit03
Davinc1003 #2
0.20
5:74
0.00
C:2
0:02
D:64
6:63
Davinci003 #3
9.50
01
RD
D:02
0.81
Davinci003
Figure 7: The largest source of variability between annotators comes from the underlying model. Every cell of the heatmap shows the agreement between two annotators (x- and y- axis).
30
50
60
70
Preference for lists (%)
Preference for longer outputs (%)
Figure Humans and our simulated annotators prefer outputs that are longer and have lists.
Humans and simulated annotators prefer longer outputs that contain lists. One natural question is whether simulated and human annotators have biases towards different type of outputs, which would cause models in both frameworks to be qualitatively different. We identify two stylistic features, the length and the presence of lists, for which humans have strong preference and analyze whether simulated annotators match those preferences. We found that humans prefer longer outputs 62% of the time, while our simulated annotators prefer those 64% of the time. Similarly, humans prefer outputs with lists 69% of the time, while our simulated annotators prefer those 63% of time. This shows that simulated annotators match well the stylistic preferences of humans, which suggests that models trained in our sandbox are optimizing similar preferences as those trained with human feedback and they will likely exhibit similar behaviors.
22

---

E Details on human data collection
Qualification. We conducted the qualification of our annotators based 25 qualification examples. The qualification examples were generated but an OPT 6B model that was studied in the earlier development phase of this project. The five student authors of this paper annotated shared set of pairwise preferences. From the shared set, we selected 25 questions where the majority of the authors reached an agreement on the correct annotation. We then use these questions as a qualification test and selected the top 16 annotators whose agreement is the highest with the authors. We paid the annotators the same price for the qualification round as we did the main qualification.
During the annotation process, we also compare each annotator's preference to that of GPT-4. We identified one annotator whose agreement around 50% with GPT-4, which clear outlier from other annotators. Therefore, we discontinued working with this annotator during the annotation project and removed their annotation.
Annotation guideline. We display our annotation guideline in Figure 9 and annotation interface in Figure 10. In our annotation process, we find that there are pairs that only differ punctations or have minimal edit distance and we instruct the annotators to select response as slightly better/ worse the difference between the pairs marginal. As result, around 18% of the collected preference selected the slightly better options. In our LPF experiments, we binarize the preference and treated the slightly better options the same as the normal preference labels. However, we release the more fine-grained labels resources and leave the study to future work.
23

---

Hi We are group of researchers working Artificial Intelligence (AO. In this task, we will ask you to help us rate an Al model's responses instructions In the area below, you will first read:
1. An intruction we give the system.
2. An that is provided along with instruction. This optional and all instructions will have inputs. responses
system
response is
that you along.
the questions:
recipe healthy 2. the response language natural? For example, Al responses are often repetitie, which is not natural.
response helpful? exampla, estruction
response is rocipe, then
helpful.
3. the response accuual/accratee? Al responses often make up new information. For example, the response claime that Donald T Trump is the current U.S. president, you should consider
on your aesthetics, which do you 5. And so on ultimtteyy you should decide which response is better besed on your judgment and besed on your own preference.
example, you might
poem over another poem.
Thore are four options for you choose from:
1. Response is better: you think response has an advantage, then choose this option.
2. Response ls alightly botter: Response is very marginally better than response and the difforence small Response slightly better: Response very marginally better than Response
and the difference small.
response
advantage,
option.
There are cases you to choose those options as few as possible.
difference between the two responses not case, you can choose
option. However, in general, we ask
Annotation Examples
To help you understand the annotation lask, we provide two examples below.
Example
Instruction: Paraphrase the following question: "What is the copital of France?" Input: Response The capital of France Paris, Response Can you tell the name the city that serves as the capital of Frence?
In this example, response 2 paraphrases the question as ssked by the instraction. In contresl, resporie does not follow instruction as enswers the question instead. Ther efore, should choose Response is better.
Example 2
Instruction: Change the first verb to eynorym Input: love surfing Response 1: hate sur fing Response 2: and em planning on going surfing fomorrow momiigg.
this example, ther output correct. Output (a) tries to follow more closely the instruction as changes the first verb of the sentence. Therefore, we should choose Response better.
Example 3
Instruction Change the first b to eynorym Input: love surfing Response 1 hate sur fing Response 2: don't like surfing
In this example, neither output is correct and the responses are very similar. In such case, can choose either Response is elightly better Response 2 choose those options few possible.
elightly better. However,
Payment structure
estiiaate usually you minute compare responses on average. Therefore, we currently pey 30 per pair resporiees
general pay work 8115/hour veraage
Qualification
you guidelline, which we pay
pairs responses pay part. you total, we pay qualification
Main
spread over qualification round, you pairs
responses. We pay $3. each
Figure 9: Our annotation guideline.

---

Hi tes:22. Not MTurk id? Logout
Your annotation progress for this HIT:
0/6 completed.
Sample of
Lets you skip forward (all samples still need markedy: Next sample.
Returns you back to the next un-marked sample to mark: Go nost un-marked sample.
Annotation Guidelines
Instruction
driving
Input
AI Response
selt driving care are vehicles that are cagable of detecting their murroundings and navigating without any ingut frem himan driver. They use variety of sensers, cuch as caneras, radar, and lidar, to detect objects in their environmant and advanowd algorithua to sake decisions about low then. Self driving In variety applications, delivery
being becoming increseingly popolar
ride-baling
AI Response
software, allows that equipped
self driving
and
their withoot
for driver.
able driving
technologies
such as CPS, radar, lidar, computer visice dotect their enviromeest avoid obstacles. they are also able update their sapa jujust their routes to acoount for variables each as tratfic and constraction.
The rain berafit of self driving cars is thair ability to reduce prasbes and save Lives, Autononous vehicles have the potential to save to 30,000 lives annually: ly. They are also nore energy ofticient than traditional vehicles, they do not requize driver who has saintain ninizun speed. Ausoncnous vebiclas alsa create new opgortanition irrrttiiiiin for incrwased robility, allowing thows unable drive
around.
Rating
Response is better.
Response only slighlly better. (only this it's truly close)
Response only slightly better. (only pick it's truly close)
Response better.
Optional explanation (never required): for where rating random.
Figure 10: Our annotation interface.
25

---

Additional results
Label noise ablations for simulated annotators
GPT4 for blue, eval for orange)
Train distribution for reward model
Eval win-rate
Paim
Pkim
p w/
25% label flips
0.40
PSlm
pann eval w/ 25% label flips
0.35
(Pain
0.30
2
2
2
23
24
27
2
2' 2
Number reranked samples (best-of-n)
Figure 11: Label noise is the most crucial ingredient for inducing overoptimization.
In this section, we ablate the different components of pmm that add variability along two axes: ran- domizing across different simulated annotators, and adding label noise. To ablate the randomization across different annotators, we compare the simple GPT-4 prompt Pun GPT- with added label noise. To ablate the label noise, we compare poual which pam without the label noise. We train reward models on these preference distributions and compare the performance of best-of-n sampling.
Figure 11 shows the results of the ablation, demonstrating clearly that added label noise provides the majority of the overoptimization effect. In particular, the two options that do not add label noise, Psim 4PT4 and peral keep increasing win-rates with more samples. This result suggests that modeling intra-annotator variability via label noise may be an important component to understanding learning from human preference data. The exact ratio of 25% label noise was chosen to be the smallest considered ratio that induced overoptimization out of the ones we considered 11%%.,5% as shown in fig. 12.
0.48
flips for RM training
0.46
0%
17.5%
Eval win-rate
0.44
25%
35%
0.38
0.36
0.34
16 32
64
128 256 512 1024
Number reranked samples (best-of-n)
Figure 12: 25% label flipping is the smallest ratio we tested that induced over-optimization. The y-axis shows the simulated win-rate. The z-axis shows the number of samples used in best-of-n sampling, which is proportional to the output's average reward. Each curve shows a reward model trained on samples with different label flipping ratios (0%, 17.5%, 25%, 35%). We that over- optimization seems to be approximately monotonic with respect to the label-flippig ratio.
26

---

Table Comparing system output before and after training with PPO, we observe that outputs often become longer and contain more lists.
Instruction: What the five most common causes of stress among people?
SFT1OK: The five most common causes of stress among peo- ple are work, money, relationships, health, and family.
PPO: The five most common causes of stress among people are: 1. Pres- sure at work: Whether it is due to tight deadlines, high expectations, overwhelming workload, pressure at work can be major source of stress. 2. Money problems: Financial issues can be significant source of stress, from not being able to make ends meet to struggling with debt. 3. Un- healthy lifestyle: Poor diet, lack of exercise, and sleep deprivation can all contribute to stress. 4. Unhappiness at home: Problems in relationships, conflicts with family members, and other domestic issues can be a major source of stress. 5. Uncertainty: Excessive worry and anxiety about the future, health issues, or other unknowns can be a major cause of stress.
F.2 Understanding computational cost
While we have focused primarily on the performance of the final model, the computational cost of these methods is an important consideration. We provide time estimates for training on our specific implementation and compute environment (a single 8x A 100 machine). While these timings are specific to our experiments, we believe these insights may be useful in understanding the cost of learning from pairwise feedback.
To begin with, supervised fine-tuning and methods that directly adapt supervised fine-tuning like Binary Reward Conditioning and Binary FeedME are generally fast, taking less than an hour for 10k instructions. Best-of-n sampling incurs no training cost but instead suffers a substantial inference time cost. The optimal for best-of-n was around 16, which (in principle) translates into an increased cost of 16x for batched inference.
In our experiments, Expert Iteration works best when training on the outputs of best-of-16 sampling, which involves substantial amounts of compute to perform best-of-16 sampling on the unlabeled split. Concretely, this sampling process takes around 4 hours on the 20k unlabeled split and the subsequent fine-tuning takes less than an hour. We observe that PPO performance peaks on both human data and in simulation after 40-80 steps, which translates to less than 2 hours of compute time on the full machine.
F.3 Analysis of model outputs
PPO and best-of-n reranking both demonstrate substantial gains with both AlpacaFarm's 's simulated feedback and human feedback. While the changes to the win-rate are clear, less clear what changes to the model lead to these improvements - are the gains driven by improvements to factuality or reasoning, or are they driven by stylistic differences? We now focus on an analysis of the outputs of these systems to understand what factors contribute to the improvement in their win-rates.
We find major difference in the length distributions of the outputs, with outputs becoming increas- ingly longer after applying LPF methods in both human and simulated feedback. For example, the average length of SFT 10k outputs is 278 characters. Best-of-16 increases the average length 570 characters and applying PPO increases it to 637 tokens. In Table 3, we show a qualitative example of how PPO training changes the output for evaluation instruction. This is not surprising given that humans and simulated annotators prefer longer outputs as discussed in appendix D.2.
In Table 6, we futher show three example outputs from SFT 10k, PPO and PPO In Table 4 and Table 5, we show the average length of model outputs of various systems in the number of characters, which qualitatively illustrates this phenomenon. While longer responses indeed lead to better human preference, we continue to see hallucinations in the output (e.g. in the example question about Berlin) and it is not clear to whether our LPF models improve the more critical aspects such as faithfulness, knowledge, etc.
27

---

Table 4: Average length of the model outputs of baselines and LPF models trained with human preferences. We observe that LPF training gen- erally leads to longer outputs.
Table 3: Average length of the model outputs of baselines and LPF models trained with simu- lated preferences. We observe that LPF training generally leads to longer outputs
Model
Number of characters
Model
Number of characters
GPT-4
504.4
ChatGPT
333.4
GPT-4
504.4
Davinci001
286.3
ChatGPT
333.4
SFT 52K
383.2
Davinci001
286.3
SFT 10K
277.5
SFT 52K
383.2
LLaMA 7B
950.5
SFT 10K
277.5
PPO 200 steps
495.6
LLaMA 7B
950.5
PPO 80 steps
623.7
PPO 80 steps
863.4
PPO 40 steps
683.1
PPO 20 steps
637.7
Best-of-128
680.0
Best-of-
704.7
Best-of-16
565.2
Best-of- 16
570.5
Best-of-4
478.7
Best-of-4
483.3
Explter-122 128
524.7
Explter-12 128
527.5
Explter-16
458.3
Explter-16
458.3
Explter-4
422.1
Explter-4
407.4
FeedMe
371.4
28

---

Table 6: Comparing system oulput before and after training with PPO, we observe that outputs often become longer. Example showing oulput from PPO-tuned models trained with the human preference Phuman and with the simulated preference Psim-
What do you think about ChatGPT?
SFT 10k: I think ChatGPT is promising technology that has the potential to revolutionize the way we interact with computers. It has the ability to generate natural language text quickly and accurately, which can be used for many different applications. think it's an exciting technology that we Ill continue see more and more uses for in the future.
PPO I think ChatGPT is a fascinating development in natural language processing. It is an artificial intelligence system that is able to generate human- like text, allowing it to converse with humans on a variety of topics. It is able to learn from large datasets and improve its performance over time, allowing it to produce text with increasing accuracy and sophistication. think it has lot of potential applications, from natural language processing and machine translation, to automated customer service and natural language processing. It is an incredible step forward in AI technology, and look forward to seeing what new developments come out of it in the future.
PPO smm think ChatGPT is really cool technology that can generate natural language text based on an input. It allows for users to quickly generate text responses without needing to write code or create custom natural language processor. It also is very good at understanding context and can generate text based on context and prompts. I think it has a lot of potential applications, such as automated customer service, natural language processing, and text generation. I think it's pretty amazing how quickly the technology is advancing and it's really cool to see what's possible with Al and machine learning. I'm really excited to see what the future holds for ChatGPT and other AI technologies. Who made Berlin?
SFT 10k:
The city of Berlin was made by the German people.
PPOpuman:
The city of Berlin was made by various people over the course of its history. It was first established as a medieval trading post in the 12th century, and subsequently built up by variety of different rulers. The city was heavily damaged during the Thirty Years' War, but was rebuilt and expanded by the Hohenzollern dynasty in the 17th and 18th centuries. During the 19th century, Berlin became the capital of the German Empire, and was significantly expanded and modernized under the leadership of Mayor Otto von Bismarck. The city was also divided into East and West sectors following World War II, with each sector being ruled by separate Cold War-era governments.
pPO The city of Berlin was made by various people over the centuries. The earliest known inhabitants of the area were the Tubal and Jabal tribes, who arrived in the region around 2300 BC. These tribes were later replaced by Celtic tribes, who founded the city of Berlin in the 1st century BC. The city was then conquered by the Romans in the 1st century AD and was renamed as 'Berlinium'. During the late Middle Ages, Berlin became an important city in the Holy Roman Empire and was ruled by various princes and electors. In the I 17th century, Berlin gained its current shape under the rule of Prince-Elector Frederick William I, who established the city's defensive ring of bastions and set up the first permanent settlements outside the city walls. In the 18th and 19th centuries, Berlin experienced a cultural and economic boom, becoming the capital of the Kingdom of Prussia in 1701 and the seat of the German Empire in 1871. The city was also the site of many historical events, including the foundation of the Weimar Republic in 1919 and the fall of the Berlin Wall in 1989, Today, Berlin is bustling city with population of over 3.7 million people, making it the third-largest city in Germany.
How many black holes are known to exist at the time?
SET 10k: Nearly 100 million black holes are known to exist at the time.
PPOnuman: At the time, there are believed to be several million black holes in the universe. This number is constantly increasing as astronomers discover more and more black holes through observation and study. Most black holes are found at the centers of many different galaxies, and are thought to be formed when the most massive stars in the galaxy run out of nuclear fuel and explode in a supernova. The resulting core of the star is densely packed that nothing, not even light, can escape its powerful gravitational pull, giving it the name "black hole".
PPO At the time of this writing, there are over 1000 known black holes in the universe.
29

---

Discuss the causes of Great Depression
Make list of desirable Skills for software engineers to add to LinkedIn.
Are there any free SAST tools out there?
I'm trying to teach myself have nicer handwriting. Can you help?
Bug
What Turing had not cracked the Enigma code during World War II?
plrds
Take MLK speech had a dream" but turn it into top 100 rap song What are some toys can buy my kids for imaginative play?
depeenox
Grapp
Hi, have question about MFCC (mel frequency cepstral coefficients).
they the same thing as MEL-Spectogram,
there difference?
Figure 13: Root verb dis- tribution of the eval instruc- tions.
Table 7: Example instructions in AlpacaFarm's evaluation data.
p program
Esample
list
questioo
sentence
title
story
story
sentence
essay
sentence
description
type
rewrite
sentence
write
list
text
tam
classify
generate
sentence
edit
convert
text
explain
identify
name
equation
design
concept
summart m
calculate
type
meaning
give
provide
describe
6669
construct
take
waig
ifferenne
mpllicate mplication
find
type
example
algorithi
eteraca
aigo
coo
article
area
example
text
content
Aiany Lanb
omant
solusion
process
feature
purpose
Mord
Figure 14: Breakdowns of the 52k Alpaca training instructions.
G Additional Analysis on Training and Evaluation Instructions
We show in table examples from the Alpaca evaluation data, showcasing their diversity. We plot in Figure 14 and Figure 15 the breakdowns of the Alpaca training instruction distribution and the Al AaaaaFarrm evaluation instruction distribution respectively. distribution of the instructions and in the outer wheel, we plot the direct subject distribution. We find that both the training distribution and the evaluation distribution cover a diverse range instructions and the distributions match at a high level.
the inner wheel, we plot the root verb
30

---

example
Bica
list
description
caption
tip
table
elittratton dharacter
story
basic
script
process
death
make
give
dffference
create
review
iist
explain
name
write
poem
solution
provide
summary
build
answer
question
implement
Use
cabin
po
rewrite
suggest
take
process
algorithm
emojis
exoression
tinction
tell
choose
describe
uassap
Find
have
jist
conversation ei
cash
eye
program
balloon
care
text
paragrash sentence
paper
name
bundle
song
story
book
benent
cak
mistory
aystem
title
book
nane
process
Ecenanio
icene
aurratt
gra
Figure 15: Breakdowns of 805 AlpacaFarm evaluation instructions.
31


Reviews:
[{"review_id": "i8uNBowEuU", "reviewer": "Reviewer_goj1", "summary": "The paper identifies three major challenges in training models with human feedback: (a) the cost of *preference* data collection, (b) the lack of trustworthy eval, and © the absence of implementations for reference methods. I completely agree with the fact that the process of training LLMs with human feedback is less understood due to the lack of published information and tools on them. The paper does a great job at addressing these dimensions in great detail with solid experimental results and thought-provoking findings. The paper establishes AlpacaFarm as a framework whose feedback and evaluation is synergetic with training on human feedback and human evaluation. Overall, the paper is well-written and clear to follow.\n\n\n*Cost of preference data collection*\n\n- The paper’s contribution in creating prompts for API LLMs that follow high human agreement and replicate human variability is very novel! \n- I do not fully understand why the authors want the practitioners to collect high quality human feedback (post-AlpacaFarm simulations) if they already have AlpacaFarm API LLM annotators. It makes it difficult for me to digest how Alpaca Farm is reducing the cost of collecting preference data if eventually we do need to collect actual human feedback. \n- I will be interested in understanding the gap between the performance of the methods trained with API LLM feedback vs human feedback by fixing the evaluator as Alpacafarm eval in the first case and humans in the other. Currently, the presented results focus on the gap between training with the simulated feedback + simulated evaluation and human feedback + human evaluation.\n\n*Trustworthy eval*\n\n- The experiments establish the high correlation between the simulated win-rates and human win-rates. It was interesting to observe that the rankings of the methods match well under the simulated environment and the real-world environment.\n- I feel the paper lacks a discussion on evaluating the LLMs on the existing NLP datasets such as MMLU, SuperGLUE and benchmarks such as BigBench. I understand that 805 instructions may be a set of instructions that humans care about but I still feel that the prior works on creating datasets for model eval deserve credit in the main text.\n\n*Reference Implementations*\n\n- I agree that there are not a lot of implementations for reference methods, and really appreciate the authors for providing them. \n- Given, PPO is known to be finicky and hard to stabilize, it would have been good to get some more details about the hyperparameter search in the main/supplementary material. \n- Minor comment: It would be good to mention that Best-of-n policy uses n = 1024 more frequently in the plots.\n", "questions": "More details on the reference method implementations in the main/appendix would be great.", "limitations": "No major issues.", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "Mentioned in the main comment.", "weaknesses": "No major issues with the contribution and experiments.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their encouraging and thorough review. We will clarify and emphasize the answers to all their questions in the updated manuscripts.\n\n## Training in simulating vs with humans\n\n> I will be interested in understanding the gap between the performance of the methods trained with API LLM feedback vs human feedback by fixing the evaluator as Alpacafarm eval in the first case and humans in the other. \n\nThe main validation of the paper involves training and evaluation with the oracle LLM versus training and evaluation with humans. As the reviewer suggests, understanding the setting of training in simulation and evaluating with humans is also useful. We investigated this setting in Appendix B.2, where we show how to modify AlpacaFarm’s simulated annotators if one wants to use them as a source of supervision rather than as simulator.\n\nIn light of the reviewer's feedback, we have now incorporated a summary of those results into the main text. In essence, the PPO model trained on AlpacaFarm's simulator achieved a 43% human-evaluated win-rate, which is significantly worse than the 55% win-rate of the PPO model trained on human feedback. However, we show that the PPO model trained using feedback from a low-variance GPT4 annotator achieves a 50% win-rate.\n\n--- \n\n> I do not fully understand why the authors want the practitioners to collect high-quality human feedback (post-AlpacaFarm simulations) if they already have AlpacaFarm API LLM annotators. [...] how Alpaca Farm is reducing the cost of collecting preference data [...]\n\nWe thank the reviewer for pointing this out and will make the distinction clearer in the manuscript. The primary cost-saving benefit of introducing a simulator before human engagement is to facilitate an affordable method development, e.g., when developing a new RLHF algorithm. By refining the model pipeline in simulation (often entailing multiple rounds of training), we need only run the final training with human feedback.\n\nAs discussed above, Appendix B.2 shows how to repurpose AlpacaFarm as a source of supervision that does not require a final human round, further decreasing the cost. However, we find that PPO trained with this automated source of supervision still lag behind human PPO. We see narrowing this gap as a promising direction for future work.\n\n## Other\n\n> I feel the paper lacks a discussion on evaluating the LLMs on the existing NLP datasets such as MMLU, SuperGLUE and benchmarks such as BigBench.\n\n >get some more details about the hyperparameter search in the main/supplementary material.\n\n> mention that Best-of-n policy uses n = 1024 more frequently.\n\nWe thank the reviewer for their feedback, which we have incorporated in the updated manuscript.\n"}, {"review_id": "yRISrS8k8E", "reviewer": "Reviewer_pSDY", "summary": "The authors provide a simulator for experiments with LLMs that aim to learn from human-feedback, in particular, human binary comparisons. This allows researchers to run exploratory experiments with, e.g., RLHF, quickly and cheaply, without having to collect human data. The main contribution is the open source library that includes various components needed for these type of experiments. In the paper, the authors evaluate their simulator; using end-to-end validation, validating subparts of the library, and showcasing how it can be used to produce findings match those found when using human feedback.", "questions": "Why does Figure 2 have 10 data points? What are these data points? Comparing to Table 1, it seems like you included GPT-4 on Figure 2; this seems misleading, as this only measures the evaluation part of your simulator, and not the training part. You should show a Figure where you only include models that were actually trained on the simulated preferences.\n\nThere has been recent work that shows that MTurk crowdworkers use AI a lot. https://arxiv.org/abs/2306.07899 . So, it's not wonder that MTurk evaluations agree with the evaluations from you GPT evaluator models. How do you expect this to affect your results?\n", "limitations": "The paper could explore the limitations of Alpaca Farm more. I don't actually believe that Llama-7B with some some instruction fine-tuning and a bit of PPO is comparable to GPT-3.5; you probably don't believe this either. However, Table 1 shows a winrate of 55% of \"PPO\". This is probably because your evaluation instructions are relatively easy, only single turn, you don't factor in adversarial inputs, and so on. This is fine, but it should be discussed more in the paper. In general, the paper would be better if it elucidated the limitations of your simulator a bit more, as there clearly are some. Try to break it, show not only where it works, but where it stops working.", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "Learning from human feedback is an important topic that has long been hard to study for any but the most well-resourced labs. Alpaca Farm will enable more research on this topic\n\nOpen source software that enables other researchers is always great.\n\nThe paper is well written.\n\nThorough validation.\n\nThe validation results, especially Figure 2, are impressive.\n\nThe finding in Figure 3 (reward model over-optimisation only happens with strong inter-rater disagreement) is intriguing (seems worth further study)\n", "weaknesses": "From more to less relevant concerns. I will raise my score if these are addressed sufficiently in the rebuttal.\n\nThe paper could explore the limitations of Alpaca Farm more. I don't actually believe that Llama-7B with some some instruction fine-tuning and a bit of PPO is comparable to GPT-3.5; you probably don't believe this either. However, Table 1 shows a winrate of 55% of \"PPO\". This is probably because your evaluation instructions are relatively easy, only single turn, you don't factor in adversarial inputs, and so on. This is fine, but it should be discussed more in the paper. In general, the paper would be better if it elucidated the limitations of your simulator a bit more, as there clearly are some. Try to break it, show not only where it works, but where it stops working.\n\nThere has been recent work that shows that MTurk crowdworkers use AI a lot. https://arxiv.org/abs/2306.07899 . So, it's not wonder that MTurk evaluations agree with the evaluations from you GPT evaluator models. How do you expect this to affect your results?\n\nFigure 4: this experiment would be better if you used humans to evaluate the winrate on the demo. The x-axis is the evaluation in Alpaca Farm. The y-axis should be the \"realistic evaluation\", which means realistic data AND human evaluation.\n\nWhy does Figure 2 have 10 data points? What are these data points? Comparing to Table 1, it seems like you included GPT-4 on Figure 2; this seems misleading, as this only measures the evaluation part of your simulator, and not the training part. You should show a Figure where you only include models that were actually trained on the simulated preferences.\n\nI have some worries thinking about that the training signal (pairwise comparisons) and the evaluators (pairwise comparisons) come from the same API-models. This might lead to overfitting to these particular evaluators. However, in practicse, this doesn't seem to be an issue (see Figure 2). Of course, if you work with humans, you may also use the same humans for training signal and eval (although best practice would probably be to use different humans)\n\nPaper could be shorter, a bit much repetition.\n\n\n-------\nEDIT: some of these points have been convincingly addressed in the rebuttal, so I raise my score\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their review and insightful feedback, which we incorporated in the updated manuscript.\n\n## Limitations \n\n>*The paper could explore the limitations of Alpaca Farm more.*\n\nWe agree with the reviewer’s suggestion and have incorporated the feedback. Please see the [general rebuttal](https://openreview.net/forum?id=4hturzLcKX&referrer=%5BAuthor%20Console%5D(%2Fgroup%3Fid%3DNeurIPS.cc%2F2023%2FConference%2FAuthors%23your-submissions))\n\n---\n\n> *I don't actually believe that Llama-7B with some instruction fine-tuning and a bit of PPO is comparable to GPT-3.5. [...] your evaluation instructions are relatively easy, only single turn, you don't factor in adversarial inputs, and so on*\n\nWe agree with the reviewer’s concerns and have added all the necessary disclaimers for that result in the revised manuscript. Furthermore, we have detailed the mentioned limitations in a new section (refer to [general rebuttal](https://openreview.net/forum?id=4hturzLcKX&noteId=Hq0kpzSHSw)).\n\nEvaluation of instruction following models is an active area of research. Our team is actively working on that domain, and we intend to incorporate recent and upcoming evaluation improvements into AlpacaFarm.\n\n---\n\n> *There has been recent work that shows that MTurk crowdworkers use AI a lot. https://arxiv.org/abs/2306.07899 [...]’How do you expect this to affect your results?*\n\nThis is a significant potential concern that we were aware of. We undertook two measures to address this during the project:\n1. We manually annotated over 200 examples without any language model and assessed our agreement with the auto annotator. The agreement was 64%, akin to the agreement between crowd-workers and the auto annotator and among different authors.\n2. We continuously monitored the agreement between crowd-workers and GPT4, as well as the time taken for annotations. This monitoring led us to exclude 3 crowd-workers and their annotations, who were annotating particularly fast with a questionable agreement with GPT4.\n\nThough there might still be instances of LLM use we didn't detect, given these precautions, we're confident it hasn't majorly affected our results. We did not include a discussion of the paper since it was published after our submission, but we will include it in the updated manuscript.\n\n---\n\n>*I have some worries thinking about that the training signal (pairwise comparisons) and the evaluators (pairwise comparisons) come from the same API-models. This might lead to overfitting to these particular evaluators. However, in practice, this doesn't seem to be an issue (see Figure 2).*\n\nWe agree with everything that the reviewer said.  We now address this as a potential limitation, but as the reviewer pointed out, it doesn't appear to be a major concern in practice. This potential issue might be partly alleviated given that we utilize a diverse pool of annotators (simulated or crow-dworkers).\n\n## Other\n\n> *You should show a Figure where you only include models that were actually trained on the simulated preferences.*\n\nWe agree and have updated the figure in the main paper. We also clarified in Table 1 which models were only evaluated (and not trained) in both human and simulated settings.\n"}, {"review_id": "7RaiN3jYgV", "reviewer": "Reviewer_JtTg", "summary": "This paper introduces AlpacaFarm, a simulator that enables faster and cheaper research and development of fine-tuning LLMs with human feedback. The authors propose to use an LLM to simulate human feedback, which is 45x cheaper than using crowdworkers and displays high agreement with humans. They also identify an evaluation dataset which is representative of real-world instructions from interactions between humans and chatbots or other LLMs. The authors also plan to open-source reference implementations for multiple methods which are widely used in the research and applications such as PPO, Best-of-N (BoN), and Expert Iteration (EI). The authors validate their simulator and evaluation protocols by showing  that rankings of models trained in AlpacaFarm match those of models trained on human data. The results indicate that leveraging a reward model can improve performance over supervised fine-tuning, which is in line with prior work.  \n", "questions": "1. In section 3.2, you claim that the agreement between your model and human annotators is high and quote 65%, which seems rather low to me. Can you provide more details on how this number was computed and what it means in practice? How does it compare with the agreement across humans? \n\n2. Also in section 3.2, you mention that to emulate intra-annotator variability you inject random noise flipping the simulated preference 25% of the time. How did you come up with this number, did you run experiments to find out the intra-human variability? How did you validate that the resulting models match human variability?\n", "limitations": "The authors mention some of the assumptions and limitations of the paper but I strongly suggest having a separate section that discusses these in greater depth. Given the potential impact of this simulator and relevance to real-world applications, it is essential for readers to understand that this is just a simulator which may be inaccurate so it should only be used for research and preliminary experiments; human evaluations and potentially feedback should always be used before deployment or to make stronger claims about a model's capabilities. ", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "## Clarity \nThe paper is well-written and clear overall. I also appreciated that the authors didn't only consider the agreement between LLMs and humans but also the variability in the answers which is an important aspect of human evaluation and feedback. \n\n## Impact\nThe paper addresses a timely topic of great importance, namely understanding how to best fine-tune LLMs with human feedback. Although it is more of an engineering / datasets / benchmarks paper, I believe it fills a gap in the literature and could lower the entry barrier for doing research on RLHF and related methods which are rather complex and expensive to train well. \n\n## Novelty\nAlthough the paper does not propose a novel approach, it open-sources reliable implementations of popular approaches for fine-tuning LLMs with human feedback and proposes an automatic (fast and cheap) way of evaluating such methods. I believe these could be quite impactful and valuable for the community. I expect this can enable research and development of new methods, as well as a platform for fairly comparing them in order to make faster progress on these problems. ", "weaknesses": "## Generality \nOne of the main weaknesses of this paper is the fact that only LLaMA-based models are considered. However, the LLM simulator is advertised as a general platform for researching, developing, and evaluating human feedback fine-tuning of LLMs. In particular, one open question is whether the models you use for evaluation and simulating human feedback transfer to other base models such as OPT, Pythia etc. \n\n1) Thus, it would be useful to include experiments validating this hypothesis to demonstrate your protocol is robust to the base model used. \n\n2) In addition, it would also be good to include different models for evaluating and even for simulating human feedback to see how much variance there is and increase the robustness of the results obtained with this platform. In particular, I expect that the results won't perfectly match human evaluation and feedback (particularly for the absolute performance), but adding different ways of simulating human feedback and evaluating LLMs can at least decrease the variance in the final results.\n\n3) It would also be good if the authors can include human evaluations in the paper. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their questions and will address their concerns in the updated manuscript.\n\n## Limitations\n> *The authors mention some of the assumptions and limitations of the paper but I strongly suggest having a separate section that discusses these in greater depth.* \n\nWe agree with the reviewer’s suggestion and have incorporated the feedback. We refer the reviewer to the [general rebuttal](https://openreview.net/forum?id=4hturzLcKX&noteId=Hq0kpzSHSw).\n\n---\n\n> *one open question is whether the models you use for evaluation and simulating human feedback transfer to other base models such as OPT, Pythia etc.*\n\nThis is an interesting and important question, which we now discuss in the new limitation section. During earlier stages of development, we experimented with OPT and Flan-T5 but found that outputs from these models were often completely wrong, making it challenging to obtain enough signal from human feedback. Since submitting our paper, many strong strong base models have been released (e.g. MPT, Falcon, LLaMA-2) and we think that it’s an important avenue for future work to consider the impact of the choice base model.\n\n\n## Simulated annotators\n\n>  *you inject random noise flipping the simulated preference 25% of the time. How did you come up with this number, did you run experiments to find out the intra-human variability? How did you validate that the resulting models match human variability?*\n\nWe thank the reviewer of their important questions, which we now expanded on in the paper. We refer the reviewer to the [general rebuttal](https://openreview.net/forum?id=4hturzLcKX&noteId=Hq0kpzSHSw).\n\n---\n\n> *In section 3.2, you claim that the agreement between your model and human annotators is high and quote 65%, which seems rather low to me. Can you provide more details on how this number was computed and what it means in practice? How does it compare with the agreement across humans*\n\nAs elaborated in section 4.3, the 65% is comparable to the human-human agreement rate at 66%. Quote from line 259:\n\n>> We begin computing agreement levels between our simulated annotator and a majority vote of 3 human annotators, comparing this to the agreement level of a held-out human annotator. We find that our evaluator $p_{eval}^{sim}$ has a 65% agreement rate with the human majority vote, which is similar to the held-out human agreement rate (66%) [...] \n\nWe agree that this number seems low and see two reasons for that. First, samples from the same SFT model are often quite similar, making it hard and often very subjective to select a preferred sample. Second, we use a pool of human annotators that may have different preferences.  We emphasize that the disparity is not merely due to the crowd-workers; even when we (the paper's authors) annotated >200 examples, we observed just a 64% agreement rate.\n\nThe agreement here estimates the frequency with which an annotator aligns with the majority of humans. Concretely, we collected 650 examples each annotated by 4 different crowdworkers (i.e., 2600 in total). For human-human agreement, each example was evaluated by one human and then cross-checked against the majority preference of the remaining three. The resulting agreement was then averaged over all four humans and 650 examples. For model-human agreement, we performed the same computation: comparing the model's response to the majority agreement of the 3 humans, then averaging over samples and human participants. \n\nWe thank the reviewer for those questions, which we now expanded on in the paper.\n\n----\n\n> *it would also be good to include different models for evaluating and even for simulating human feedback to see how much variance there is and increase the robustness of the results obtained with this platform.*\n\nWe agree that understanding the robustness of the simulator to the oracle model is important. Figures 9, 10, and 11 in the appendices investigate the use of other models and prompts for simulation. Here is a table with a more in-depth analysis of the use of different simulated evaluators, their agreement with humans, their variance, and their biases. Our annotators are bolded. The last 4 columns in the following table refer to the metrics from figures 9  (bias, variance) and figure 11 (preference for list and for longer outputs) in appendices E.3. of the submitted manuscript.\n\n|  | Human agreement [%] | Price [$/1000 examples] | Time [seconds/1000 examples] | Bias | Variance | Proba. prefer longer | Proba. prefer lists |\n|---|---|---|---|---|---|---|---|\n| alpaca_eval_gpt4_fn | 71.0 | 14.5 | 5046 | 27.6 | 11.1 | 0.75 | 0.63 |\n| alpaca_farm_greedy_gpt4 | 66.4 | 15.3 | 878 | 30.2 | 19.3 | 0.60 | 0.59 |\n| humans | 65.7 | 300.0 | 36800 | 0.0 | 34.3 | 0.64 | 0.61 |\n| claude | 65.5 | 11.1 | 173 | 31.9 | 18.0 | 0.62 | 0.58 |\n| text_davinci_003 | 64.1 | 8.7 | 121 | 33.8 | 22.7 | 0.70 | 0.64 |\n| lmsys_gpt4 | 63.2 | 13.9 | 17982 | 34.7 | 16.1 | 0.74 | 0.64 |\n| alpaca_farm | 60.0 | 11.5 | 820 |  |  | 0.60 | 0.63 |\n| chatgpt_fn | 60.0 | 1.0 | 530 | 36.9 | 27.7 | 0.62 | 0.65 |\n| chatgpt | 57.2 | 0.8 | 285 | 39.4 | 34.1 | 0.59 | 0.56 |\n| cohere | 53.4 | 3.5 | 217 |  |  | 0.50 | 0.51 |\n\n## Other\n\n> *It would also be good if the authors can include human evaluations in the paper.*\n\nAll validation of our pipeline is done using human evaluations. Is the reviewer asking about releasing those annotations? If so, we have released them as part of our code release but have not linked to them due to the anonymity guidelines.\n"}, {"review_id": "p7f18cKQhM", "reviewer": "Reviewer_NPeD", "summary": "This paper presents a simulation framework, AlpacaFarm, for developing LLMs with human feedback.\nAlpacaFarm adopts LLMs (e.g., GPT-4) to generate feedback (i.e., the ranking of candidate responses given the query), and evaluate the performance by calculating the win-rate against the baseline. \nThis framework can obtain synthetic feedback data cheaply compared with human annotators.\n\nThe major contribution is the system design and validation of the AlpacaFarm framework.\n\nSystem Design: \n- i) the framework constructs a relatively comprehensive evaluation dataset.  \n- ii) multiple LLMs are adopted as annotators proxy (GPT-4, ChatGPT and Davinci003) \n- iii) random noise is injected to mimic the variability of human annotators.\n\nValidation: Simulated evaluation  (i.e., the model performance ranking ) results of learning with the synthetic feedback dataset match that obtained by training models with human feedback and evaluated by real annotators. Besides, the pairwise ranking and variability also correlate with the human results well.\n\nFinally, the paper benchmarks the reference methods with the framework of LLAMA-7B, suggesting that SFT plays a significant role and highlights the superiority of PPO.\n\n", "questions": "- The paper mentioned that 25\\%  of the simulated preferences are randomly flipped. Any explanation on the ratio? Besides, while the results are consistent with the human results, I think this randomness may not faithfully reflect the inherent variance as human variability depends on the background and geography of the annotators.\n\n- What results would be when the base LLM is close to the annotator LLMs? Will this framework still effective when we are  adopting a powerful LLM itself as annotators?\n\n- What prompt is used to generate feedback and evaluate the candidate responses? Did you do some stabilization techniques such as self-consistency in [1] to stable the evaluation results? \n\n- Typo:  Captions Table 4 and Table 5 of the Appendix: longer outputs.s ->  longer outputs. \n\n[1] Large Language Models are not Fair Evaluators\n\n\n\n\n\n", "limitations": "N/A", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Clarity: The overall presentation is good, clearly conveying the key ideas of this paper.\n\n- Significance: This framework would potentially become a useful resource for building, and evaluating the LLMs with feedback datasets when budgets are limited.\n\n- Originality: The novelty of the framework is somewhat limited as generating synthetic feedback datasets & evaluating with powerful LLMs have already been explored recently. Yet, it is still worthwhile to explore combing these dots and justify its effectiveness.\n\n", "weaknesses": "- Some hyper-parameter choices are unjustified (such as the ratio in Question 1).\n- The effect of this framework is under-explored when the baseline model is close to the annotators LLMs.\n\n---\nAfter Rebuttal: \n\nThanks for the response which well-addressed my questions well. I have raised my score to Accept.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their useful review, which we have incorporated to improve our paper. \n\n# Simulated annotators\n\n> *The paper mentioned that 25% of the simulated preferences are randomly flipped. Any explanation on the ratio? \n\nWe have expanded our explanation in Appendix B.1 about the 25% ratio. We refer the reviewer to the [general rebuttal](https://openreview.net/forum?id=4hturzLcKX&noteId=Hq0kpzSHSw).*\n\n---- \n>*randomly flipped [...] while the results are consistent with the human results, I think this randomness may not faithfully reflect the inherent variance as human variability depends on the background and geography of the annotators.*\n\nAs discussed in section 3.2 (text quoted below from line 153), there are two important sources of variability for human annotators: intra- and inter-annotator. \n\n>> To more completely emulate human annotators, we first emulate inter-annotator variability into the simulated pairwise preference by mimicking a pool of annotators. We design different annotators by querying different API LLMs and varying the prompts with different formats, batch sizes, and in-context examples.[...]. To emulate intra-annotator variability, we directly inject random noise and flip the simulated preference 25% of the time.\n\nThe random flipping concerns intra-annotator variability, while using different API LLMs with varying in-context examples addresses the inter-annotator variability that the reviewer seems to be referencing. That said, we completely agree with the reviewer that our approach to emulating inter-annotator variability does not capture complex human variability, such as that due to background and geography. Our work is only a first step in this direction, and this important and challenging problem will require much future work to tackle. We have added this to our new limitations section. We refer the reviewer to the [general rebuttal](https://openreview.net/forum?id=4hturzLcKX&noteId=Hq0kpzSHSw).\n\n--- \n\n> *What prompt is used to generate feedback and evaluate the candidate responses? Did you do some stabilization techniques such as self-consistency in [1] to stable the evaluation results?*\n\nWe thank the reviewer for their insightful comments. We detail our prompts in appendix G.1 and are releasing them as part of our code release. With respect to paper [1], we concur that the order of examples is crucial, as emphasized in the following excerpt from Appendix G.1:\n\n>>For each annotator, we randomize the ordering between the two outputs to annotate, i.e., we randomly choose which output is the first and which is the second. We found randomization to be important given that the first output is often preferred by simulated annotators\nGiven this randomization we do not require the stabilization techniques form [1]. \n"}]

Author rebuttal:
# General

We thank the reviewers for their insightful and constructive feedback.

We are pleased that the reviewers found our paper well-written [goj1, pSDY, JtTg, NPeD], thorough [goj1, pSDY], and believe that it may be an impactful and valuable contribution to the community [goj1, pSDY, JtTg, NPeD].

There are two questions/feedback shared by different reviewers: the need for a limitation section [pSDY, JtTg, NPeD] and how we came up with 25% label noise [JtTg, NPeD]. We address general feedback here and will answer questions specific to each reviewer separately.

We will upload the updated manuscript as soon as OpenReview allows us to. We appreciate the reviewers' feedback, which we believe has improved the quality of our work.

## Limitation section

We agree with the reviewers' suggestions about highlighting our limitations [pSDY, JtTg, NPeD]. We have taken advantage of the additional page in the updated manuscript to include a limitation section, which consolidates and elaborates on the limitations discussed throughout the paper and appendices. We briefly highlight some of these here:

- **Validation**: Although we provided strong validation results of AlpacaFarm in Section 4, there are limitations in how we perform such validation. First, instructions are relatively simple and are single-turn (even those from the real world-demo). Second, we only consider LLaMA 7B as the base model as this was the only model powerful enough for learning from human feedback at the time of submission. Finally, human validation is based on feedback from 13 crowd-workers, which may not reflect broader human preferences.
- **Assumption**: We assume access to an “oracle” LLM, which is more powerful than the ones we are training. While this may be true in research settings, it's not always the case in practice.
- **Generalization of Hyperparameters**: Specific hyperparameters, like the KL regularization coefficient, might not translate seamlessly from simulated environments to training with human preferences. The AlpacaFarm simulator is thus likely more useful for method development than hyperparameter tuning (and we only validated it in that setting).

## 25% noise

In response to reviewers [JtTg, NPeD]'s question about how we arrived at a 25% label flipping rate, we have highlighted and expanded our discussion about it in the updated manuscript.

In summary, we selected 25% based on two factors: estimated human (intra- and inter-annotator) variability (\~0.35) and overoptimization. Injecting a 25% label flip brings our annotator variability closer to the human one (\~0.43), and it was also the point at which we began to observe overoptimization (see rebuttal figure).

For a more in-depth discussion about annotator variability, including the variability of standard simulated annotators (<0.1), please refer to Appendix B.1 of the original manuscript.

Meta-review: This paper starts from a simple but compelling premise (make a simulator to replace humans in RLHF). There is massive potential for impact since RLHF research needs a cheap way to test-drive algorithms before they are deployed on real (and expensive) humans.  Ethical concerns seem adequately addressed. 

There are some inevitable modelling simplifications (like the constant rate of flipping label), but no significant faults.
Decision: Accept (spotlight)