SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Figure 3: Our simulated annotators are cheap and have a high agreement with human annotators. We show price (x-axis) vs agreement (y-axis) of each annotator with the majority vote among 3 human annotations. Grey points are the simulated annotators in the pool, the green  $\diamond$  shows the resulting pool of annotators (used for evaluation), the orange  $\diamond$  shows the same pool with random noise added during training. This does not change the implied reward function from  $\diamond$ , but makes the learning problem more challenging. The blue  $\square$  shows the average of human annotators, and the red  $\bullet$  shows a single low variance GPT-4 annotator analyzed below.
(a) Human preferences
(b) AlpacaFarm
Figure 4: Human and AlpacaFarm preferences result in over-optimization, while a naive single-prompt GPT-4 simulator does not. Left: training and evaluation with human preferences  $\blacksquare$ . Middle: training and evaluation with AlpacaFarm preferences  $\diamond$ ,  $\diamond$ . Right: training and evaluation with single-prompt GPT-4 preferences  $\bullet$ . The x-axis measures the average surrogate rewards on the eval set.
(c) Single-prompt GPT-4
Inspecting these results more closely, we point out the two rank mismatches. The first comparison is SFT10k against SFT52k, where human annotators preferred SFT10k (44.3% vs 40.7%) while the simulator had the opposite preference (36.7% vs 39.2%, Table 1). The other mismatch is ChatGPT against PPO, where human annotators preferred PPO (55.1% vs 52.9%) unlike the simulator (46.8% vs 61.4%). In both cases, these are not major mistakes, as we do not expect SFT52k to be much worse than SFT10k or for a 7B LLaMA model to substantially outperform ChatGPT.
4.3 Validating the pairwise preferences component
Having demonstrated that AlpacaFarm succeeds at the end-to-end validation of methods rankings, we now take a closer look at our pairwise preferences, showing that they agree with human annotators and replicate important qualitative features of model training. For additional details see appendix D.
Simulated annotators match human agreement. We begin by computing agreement levels between our simulated annotator and a majority vote of 3 human annotators, comparing this to the agreement level of a held-out human annotator, as shown in Figure 3. We find that our evaluator  $p_{\mathrm{sim}}^{\mathrm{eval}}$  ( $\diamond$ ) has a  $65\%$  agreement rate with the human majority vote, which is similar to the held-out human agreement rate at  $66\%$  ( $\square$ ). At the same time,  $p_{\mathrm{sim}}^{\mathrm{eval}}$  is  $25 \times$  cheaper ( $300 \rightarrow$ 12 per 1000 examples). The training time annotator  $p_{\mathrm{sim}}^{\mathrm{ann}}$  ( $\diamond$ ) has lower agreement due to label flip noise but this does not mean that  $p_{\mathrm{sim}}^{\mathrm{ann}}$  is less faithful to human annotations, since this noise is unbiased and both annotators ( $p_{\mathrm{sim}}^{\mathrm{ann}}, p_{\mathrm{sim}}^{\mathrm{eval}}$ ) represent the same underlying preference function.
Figure 3 also shows that we identified single prompts performing even better than  $p_{\mathrm{sim}}^{\mathrm{eval}}$ , with one prompts achieving  $68\%$  agreement. While this high agreement level is impressive, we do not use single
Figure 5: Correlation plot of simulated winrates computed on AlpacaFarm's evaluation versus that on real-world interactions with the Alpaca Demo.
prompts for AlpacaFarm, as single prompts do not replicate the inter-annotator variability important for a simulator. Instead, randomizing over different annotators and injecting additional noise is needed to match the distributional features and learning dynamics of human data, which we discuss next.
Simulated annotators replicate overoptimization. We now show that modeling annotator variability in the simulated annotator  $(p_{\mathrm{sim}}^{\mathrm{ann}})$  is necessary to capture important qualitative features of LPF model training. To do so, we compare the behavior of three of the best-performing models trained under  $p_{\mathrm{sim}}^{\mathrm{ann}}$  with those trained using the single GPT-4 prompt  $p_{\mathrm{sim}}^{\mathrm{GPT-4}}$ , which has higher human agreement but little annotator variability.
Figure 4 shows the learning dynamics of these models for pairwise feedback by  $p_{\mathrm{human}}$  (left),  $p_{\mathrm{sim}}^{\mathrm{ann}}$  (middle), and  $p_{\mathrm{sim}}^{\mathrm{GPT-4}}$  (right) as the PPO iteration count and rerank sample counts for best-of-  $n$  and expert iteration are increased. For both the human and AlpacaFarm preferences, models that more effectively optimize the surrogate reward (x-axis) improve in win-rate (y-axis) up until a point, where overoptimization of the reward occurs and win-rates decrease. In contrast, simple GPT-4 feedback shows no overoptimization, leading to the false conclusion that the LPF methods can be optimized for much longer than is actually the case. For example, Figure 4 right shows best-of-1024 to be much better than PPO, which disagrees strongly with the results on human data.
Noted in prior work [21], overoptimization is the result of the proxy reward model  $\hat{R}_{\phi}$  being an imperfect estimate of the true reward  $R$ . We hypothesize that the strong overoptimization seen for human annotators is partly due to the (inter- and intra-) variability of their annotations, which degrades the quality of the reward model. To test this hypothesis, we measured the variance of each of the three annotators by calculating the average error of a held-out annotation to the majority vote over 3 random draws. At 0.26 for  $p_{\mathrm{sim}}^{\mathrm{eval}}$  and 0.43 for  $p_{\mathrm{sim}}^{\mathrm{ann}}$ , AlpacaFarm is close to the human variance of 0.35; in contrast, the GPT-4 annotator has a much lower variance at 0.1. Finally, in Appendix F.1 we ablate the AlpacaFarm design more finely and find that the added label noise provides the majority of the benefit for inducing overoptimization. Appendix D contains further analyses of bias and variability of annotators.
4.4 Validating the evaluation protocol
Finally, we test our evaluation data that combines existing open-source evaluation datasets. While we have observed that this dataset is diverse (see Figure 13), it is unclear whether it evaluates performance for any type of real-world human usage. To resolve this, we measure method-level correlations against a set of real user interactions recorded on the Alpaca Demo [69]. We manually went through the interactions and identified 200 instructions that do not contain any personal identifying information, toxic or unsafe questions, and those that refer to the chatbot directly (e.g. "who are you developed by?"). The terms of use for the demo do not allow us to publicly release this data, but we use this data to evaluate the proposed evaluation set.
We use the 11 systems displayed in Figure 2, with the LPF methods trained in simulation, and evaluate them using  $p_{\mathrm{sim}}^{\mathrm{eval}}$ . Figure 5 plots the simulated win-rates on the Demo instructions against those on the AlpacaFarm evaluation data. The two win-rates are strongly correlated ( $r^2 = 0.97$ ), indicating that AlpacaFarm evaluation data can serve as a proxy for evaluating methods on simple demo interactions.
Table 1: AlpacaFarm evaluation results on baseline and LHF methods. For methods without a *, the left column shows win-rates when we train and evaluate in simulation, while the right column shows win-rates when we train and evaluate with human feedback. For models with a *, those are not trained by us so the left and right columns respectively show simulated and human evaluation. Win-rates are computed against Davinci003. We did not evaluate Quark and Binary Reward Conditioning for human evaluation because they underperformed in development. † ChatGPT and GPT-4 were prompted for responses shorter than 1000 characters to have length comparable to other methods.
Method	Simulated Win-rate (%)	Human Win-rate (%)
GPT-4*†	79.0 ± 1.4	69.8 ± 1.6
ChatGPT*†	61.4 ± 1.7	52.9 ± 1.7
PPO	46.8 ± 1.8	55.1 ± 1.7
Best-of-1024	45.0 ± 1.7	50.7 ± 1.8
Expert Iteration	41.9 ± 1.7	45.7 ± 1.7
SFT 52k	39.2 ± 1.7	40.7 ± 1.7
SFT 10k	36.7 ± 1.7	44.3 ± 1.7
Binary FeedME	36.6 ± 1.7	37.9 ± 1.7
Quark	35.6 ± 1.7	-
Binary Reward Conditioning	32.4 ± 1.6	-
Davinci001*	24.4 ± 1.5	32.5 ± 1.6
LLaMA 7B*	11.3 ± 1.1	6.5 ± 0.9
5 Benchmarking reference methods on the AlpacaFarm
We now study the performance of reference methods in AlpacaFarm. Table 1 contains the details of the main evaluation results (presented in Figure 2). In the rest of this section, we discuss our findings from these results, demonstrating that the conclusions we reach using human feedback could have been derived using AlpacaFarm at a substantially lower cost. See Appendix C for experimental details.
5.1 Comparing LPF methods
Supervised fine-tuning is highly effective. Table 1 shows that the SFT is effective and provides the majority of gains. SFT brings the base LLaMA model up from a simulator win-rate of  $11\%$  to  $37\%$  and a human win-rate of  $7\%$  to  $44\%$ . However, we observe no gain from SFT 10k to SFT 52k.
PPO tops the LPF leaderboard. Among the LPF methods we study, PPO performs the best in both the simulator (47%) and on human data (55%). Notably, with a win-rate of 55%, the PPO trained with human feedback was preferred to ChatGPT by our annotators for single-turn instruction-following. This surprising result is likely because ChatGPT was prompted for responses shorter than 1000 characters and both crowd-workers and simulators annotators preferred longer responses (see appendix D.2). For example, in AlapcaEval [35] (slightly different simulated annotator) we show that the unconstrained ChatGPT achieves 89% win-rate, much higher than the 44% win-rate for the PPO model.
Best-of- $n$  is simple and competitive. Best-of- $n$  sampling outperforms all LPF methods besides PPO. The performance best-of- $n$  sampling points to the useful learning signals captured by the reward model and helps us understand how training-time LPF methods benefit from a reward model.
Expert Iteration and Quark lag behind. Even though expert iteration trains with best-of-  $n$  outputs, it trails best-of-  $n$  with a  $3 - 6\%$  win-rate gap in both simulated and human feedback workflows. This result suggests that acquiring the improvement from best-of-  $n$  sampling is not straightforward and points to why more complex learning algorithms like PPO might be useful. For Quark, although the rewards of samples improve during training, the resulting models did not outperform the SFT baseline.
Methods that directly learn from pairwise feedback do not perform well. We observe that binary reward conditioning and binary FeedME do not improve over the SFT 10k baseline. This result suggests that learning a surrogate reward model may be an important ingredient to LPF.
In appendix F.3 we analyze the outputs of different models. We find a major difference is that LPF methods produce longer outputs in both human and simulated feedback.
5.2 Using AlpacaFarm to train models directly for human deployment
A natural question is whether models trained in AlpacaFarm can directly perform well on human evaluation Table 2 shows that AlpacaFarm can be repurposed for this goal by using simulated annotators that maximize agreement rather than match human variability, i.e., using preferences from the single low-variance GPT-4 annotator  $p_{\mathrm{sim}}^{\mathrm{GPT - 4}}$  rather than our noisy annotators  $p_{\mathrm{sim}}^{\mathrm{ann}}$ .
Table 2: Model transfer results.
Method	Human Win-rate (%)
PPOhuman	55%
Best-of-16human	51%
PPOGPT-4sim	50%
SFT 10k	44%
PPOannsim	43%
Table 2 compares the best model  $\mathrm{PPO}_{\mathrm{sim}}^{\mathrm{ann}}$  trained on  $p_{\mathrm{sim}}^{\mathrm{ann}}$  with a similar model  $\mathrm{PPO}_{\mathrm{sim}}^{\mathrm{GPT-4}}$  trained on the GPT-4 annotator. We find that  $\mathrm{PPO}_{\mathrm{sim}}^{\mathrm{ann}}$  only achieves  $43\%$  win-rate, while  $\mathrm{PPO}_{\mathrm{sim}}^{\mathrm{GPT-4}}$  achieves  $50\%$  win-rate. As context, we show win-rates of the initial SFT, the human PPO  $\mathrm{PPO}_{\mathrm{human}}$ , and the best non-PPO human method Best-of-16human. Overall, training in AlpacaFarm can provide good models for deployment, but still suffers a  $5\%$  performance gap relative to real human annotations.
Validation. Section 4 validates the use of AlpacaFarm, but there are nevertheless some limitations with the validation setting. First, the instructions we considered (even those from the real world-demo in section 4.4) are relatively simple and single-turn. Second, all models we fine-tuned use a LLaMA 7B as starting point. Finally, Human validation is based on feedback from 13 crowd-workers, which may not reflect broader human preferences and seem to have biases such as preferences for longer outputs as highlighted in appendix D.2. Although we do not expect the previous limitations to significantly affect the usefulness of AlpacaFarm, we encourage users to be vigilant when using any simulator and hope to further validate AlpacaFarm as more datasets and base models become available.
Assumption of an oracle LLM. AlpacaFarm assumes that we can access an oracle LLM, much more powerful than the investigated models, that can be used to approximate human feedback. While this may be true in research settings, it is not always the case in practice. We believe that investigating the use of the same model for the simulator and for learning is a promising direction for future work.
AlpacaFarm for fine-grained development. Section 4 shows that AlpacaFarm is effective for model selection and can replicate important behaviors such as over-optimization. We have nevertheless found that suitable hyperparameters for learning algorithms can be different for training with simulated feedback compared to human feedback. For example, due to changes in the scale of values of the surrogate reward model, the range of suitable KL regularization coefficients for RLHF is different.
Simulated annotators. A key component of AlpacaFarm is the simulated annotators. In section 4.3 and appendix D we showed that our annotators have high agreement with human annotators and exhibit similar biases, such as preferences for longer outputs and lists. There are nevertheless some biases that are specific to simulated annotators. For example, we found that simulated annotators prefer the first outputs shown in the prompt and outputs that come from the same model as the simulator. Although we controlled for those biases by randomizing the order of output in the prompt, and using the same training data for all considered methods, there may be other biases that we have not identified. For better automatic annotators and further analysis refer to AlpacaEval [35].
7 Related works and outlook
The idea of building a simulator is closely related to recent efforts in automatic model evaluations [16, 55, 15, 53, 37] and constitutional AI [8]. However, our work crucially differs in our goal of building a simulator, which requires us to mimic distributional properties such as inter-annotator disagreement. We discuss these differences and the broader context of our work in Appendix A.
We showed that AlpacaFarm substantially lowers the cost and iteration time for developing methods that learn with pairwise feedback. AlpacaFarm provides a blueprint for constructing simulators for AI research that requires human supervision, and we view it as an exciting opportunity to expand this simulation approach to data from other domains and alternative forms of human feedback.
We thank the Stanford Center for Research on Foundation Models (CRFM), Stanford HAI, and Stability AI for compute support. Data collection efforts were supported through the Tianqiao & Chrissy Chen Institute and an Open Philanthropy grant. XL is supported by a Stanford Graduate Fellowship. RT is supported by the NSF GRFP under Grant No. DGE 1656518. YD is supported by a Knights-Hennessy Scholarship.
[1] Gati Aher, Rosa I Arriaga, and Adam Tauman Kalai. Using large language models to simulate multiple humans. arXiv preprint arXiv:2208.10264, 2022.
[2] Thomas Anthony, Zheng Tian, and David Barber. Thinking fast and slow with deep learning and tree search. Advances in neural information processing systems, 30, 2017.
[3] Lisa P Argyle, Ethan C Busby, Nancy Fulda, Joshua Gubler, Christopher Ryting, and David Wingate. Out of one, many: Using language models to simulate human samples. arXiv preprint arXiv:2209.06899, 2022.
[4] Vamsi Aribandi, Yi Tay, Tal Schuster, Jinfeng Rao, Huaixiu Steven Zheng, Sanket Vaibhav Mehta, Honglei Zhuang, Vinh Q Tran, Dara Bahri, Jianmo Ni, et al. Ext5: Towards extreme multi-task scaling for transfer learning. arXiv preprint arXiv:2111.10952, 2021.
[5] Amanda Askell, Yuntao Bai, Anna Chen, Dawn Drain, Deep Ganguli, Tom Henighan, Andy Jones, Nicholas Joseph, Ben Mann, Nova DasSarma, Nelson Elhage, Zac Hatfield-Dodds, Danny Hernandez, Jackson Kernion, Kamal Ndousse, Catherine Olsson, Dario Amodei, Tom Brown, Jack Clark, Sam McCandlish, Chris Olah, and Jared Kaplan. A general language assistant as a laboratory for alignment, 2021.
[6] Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, Nicholas Joseph, Saurav Kadavath, Jackson Kernion, Tom Conerly, Sheer El-Showk, Nelson Elhage, Zac Hatfield-Dodds, Danny Hernandez, Tristan Hume, Scott Johnston, Shauna Kravec, Liane Lovitt, Neel Nanda, Catherine Olsson, Dario Amodei, Tom Brown, Jack Clark, Sam McCandlish, Chris Olah, Ben Mann, and Jared Kaplan. Training a helpful and harmless assistant with reinforcement learning from human feedback, 2022.
[7] Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al. Constitutional ai: Harmlessness from ai feedback. arXiv preprint arXiv:2212.08073, 2022.
[8] Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al. Constitutional ai: Harmlessness from ai feedback. arXiv preprint arXiv:2212.08073, 2022.
[9] Michiel Bakker, Martin Chadwick, Hannah Sheahan, Michael Tessler, Lucy Campbell-Gillingham, Jan Balaguer, Nat McAleese, Amelia Glaese, John Aslanides, Matt Botvinick, et al. Fine-tuning language models to find agreement among humans with diverse preferences. Advances in Neural Information Processing Systems, 35:38176-38189, 2022.
[10] R. Bommasani et al. On the opportunities and risks of foundation models. arXiv preprint arXiv:2108.07258, 2021.
[11] Ralph Allan Bradley and Milton E Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39(3/4):324-345, 1952.
[12] Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym. arXiv preprint arXiv:1606.01540, 2016.
A Related work
Instruction following. There have been an number of works studying instruction following as a cross-task generalization across a pool of NLP tasks [44, 77, 58, 4, 76]. Our work focuses on a recent trend in instruction following methods which increasingly focus on real world human interaction patterns [49, 6], rather than collections of existing NLP benchmarks. For example, InstructGPT was developed on user instructions submitted to OpenAI API [49]. Our work builds upon these works by attempting to bridge the gap between the ease of development and evaluation of traditional academic benchmarks and the more complex algorithms and real-world settings of recent works on instruction following.
Simulating human feedback. Constitutional AI [7] simulates human feedback with AI feedback for model development to improve harmlessness and helpfulness. AlpacaFarm, on the other hand, simulates human feedback with API LLMs so that simulated experiments reflect the outcomes of experiments performed with real human feedback. Due to the difference in goals, the construction and usage of the feedback simulator are different in the two settings. For example, AlpacaFarm's simulator perturbs LLM preferences with bit-flip label noise to mimic the noisiness of human annotation, whereas Constitutional AI's simulator doesn't inject extra noise.
The evaluation aspects of our work are related to a growing line of work on simulating human annotation for evaluation [15, 56, 15, 54, 38, 39]. Our core evaluation and feedback mechanism makes use of the same underlying ideas, but our work is distinguished by a focus on using pairwise feedback for training, as well as careful validation beyond per-example agreement metrics. AlpacaFarm shows that LLM feedback can capture method-level correlations as well as qualitatively important features of human annotation for phenomena such as overoptimization.
Our goal of emulating human annotators also connects to work on simulating humans with LMs based on personas [51, 50, 1, 3], as well as works that simulate human behavior in the context of cognitive science, social science, and economics [73, 27]. Our work complements these works by showing that simulated LLM annotators can replicate many of the qualitative features of training on pairwise human feedback.
More broadly, building a simulator environment to enable low-cost experimentation is common in the field of reinforcement learning and robotics [12, 71, 70, 68, 19, 25, 20]. Our work shares the same underlying motivations, but instead of simulating physical systems, AlpacaFarm simulates human preference feedback.
Methods for learning from feedback. To hold annotation cost constant across learning methods, we have focused only on methods that learn from pairwise feedback in this work. However, there exist methods in the literature other than those explored in AlpacaFarm that can incorporate alternative sources of feedback such as natural language [78, 34, 24, 63, 59, 14, 60, 43], numeric ratings [47, 33], or execution traces [14]. We view extensions of AlpacaFarm to these settings as exciting future work.
We have included a set of RL algorithms in our study that optimize the surrogate reward, but this set is by no means comprehensive. RL research applied to NLP has a long history [79, 66, 29, 52, 45, 32, 31, 57, 65], and we expect future work in this direction to benefit from the ideas and artifacts in AlpacaFarm.
B Reference LPF methods on AlpacaFarm
We provide a more thorough description of the methods in AlpacaFarm along with the custom modifications we make. For more on hyper-parameter tuning, experiment details, and further ablations, please see Appendix C.
Our methods fall into two categories based on whether they fit a surrogate reward model as part of the learning process. In addition, to those methods which are trained in AlpacaFarm, we also include well-known baseline methods from the OpenAI API.
API baseline methods. In both the human and the simulated feedback workflows, we evaluate the following methods from the OpenAI API: GPT-4, ChatGPT, Davinci001, and implicitly Davinci003, which is the baseline model we compare every model with. Outputs from all these models are sampled at temperature 0.7 and with top- $p$  1.0. For the text models, we use the same prompt as all other reference methods in AlpacaFarm. For both chat models (ChatGPT and Davinci003), we had to change prompts for two reasons. First, those models require prompts in a chat format, which is different from the text format. Second, we found that those models generated sequences that were much longer than the rest of our models, which were trained to output sequences of less than 300 tokens. We thus ask in the system prompt for a response that is shorter than 1000 and 500 characters respectively for ChatGPT and GPT-4, which we found to give shorter answers while working similarly to a raw prompt that does not mention the length.
B.1 Methods that directly learn from pairwise feedback