SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Yann Dubois*
Xuechen Li*
Rohan Taori*
Tianyi Zhang*
Ishaan Gulrajani
Jimmy Ba
Carlos Guestrin
Percy Liang
Tatsunori B. Hashimoto
Large language models (LLMs) such as ChatGPT have seen widespread adoption due to their ability to follow user instructions well. Developing these LLMs involves a complex yet poorly understood workflow requiring training with human feedback. Replicating and understanding this instruction-following process faces three major challenges: the high cost of data collection, the lack of trustworthy evaluation, and the absence of reference method implementations. We address these challenges with AlpacaFarm, a simulator that enables research and development for learning from feedback at a low cost. First, we design LLM prompts to simulate human feedback that are  $45\mathrm{x}$  cheaper than crowd-workers and display high agreement with humans. Second, we propose an automatic evaluation and validate it against human instructions obtained on real-world interactions. Third, we contribute reference implementations for several methods (PPO, best-of- $n$ , expert iteration, and more) that learn from pairwise feedback. Finally, as an end-to-end validation of AlpacaFarm, we train and evaluate eleven models on 10k pairs of real human feedback and show that the rankings of models trained in AlpacaFarm match the rankings of models trained on human data. As a demonstration of the research possible in AlpacaFarm, we find that methods that use a reward model can substantially improve over supervised fine-tuning and that our reference PPO implementation leads to a  $+10\%$  improvement in win-rate against Davinci003. We release AlpacaFarm at https://github.com/tatsu-lab/alpaca_farm.
Large language models (LLMs) [10, 13, 48] have demonstrated unprecedented capabilities in following diverse and open-ended instructions [49, 5, 40]. These achievements have often been attributed to the fine-tuning of pretrained LLMs using human feedback, but this process remains poorly understood due to the lack of published information on the training methods from LLM vendors. For example, it was recently revealed that only the Davinci003 model in the instruct series of OpenAI models used reinforcement learning (RL) with the PPO algorithm [47], leading some to question the importance of RL in the training process. Understanding and improving these methods requires open and transparent replications of the training process, but this remains challenging due to the cost and complexity associated with methods that learn from human feedback.
Our goal is to facilitate research and development on instruction following models and methods that learn from human feedback. We identify three main challenges: the high cost of data annotation, the lack of automated evaluation for model development, and the absence of validated implementations
*Equal contribution; random order. Contact {yanndubs, lxuechen, rtaori, tz58}@stanford.edu.
Figure 1: AlpacaFarm is a simulation sandbox that enables fast and cheap experimentation on methods that learn from human feedback. It simulates human feedback with API LLMs, provides a validated evaluation protocol, and offers a suite of reference method implementations. Researchers can iterate on model development and transfer their methods to training on human data to maximize performance.
of existing methods. To address these three challenges, we introduce AlpacaFarm (Figure 1), a simulation sandbox that enables experimentation at a low cost. Using AlpacaFarm, researchers can rapidly iterate on method development in simulation and transfer these insights to build high-performance systems with actual human feedback.
For the first challenge of data annotation costs, AlpacaFarm simulates human annotators with API LLMs that are faster and lower cost. To collect simulated feedback data, we design prompts for API LLMs (e.g. GPT-4) that enable us to simulate human pairwise comparisons at a cost that is  $45\mathrm{x}$  cheaper than crowd-workers, and tune these prompts to faithfully capture many aspects of human annotators, such as their quality judgments, inter-annotator variability, and stylistic preferences.
For the second challenge of automated evaluation, we design an automatic evaluation protocol that aims to quantify system performance on simple but realistic real-world human instructions. Improving evaluations of open-ended instruction following has been challenging due to the cost and non-replicability of human evaluation, the lack of real human interaction data, and the diversity of natural human instructions. To address this, we use instructions from user interactions (Alpaca Demo [69]) as a reference for simple but real human interactions and show that we can combine existing evaluation datasets to mimic this evaluation. Quantitative evaluations of system rankings on our evaluation data show a high correlation with system rankings on the Alpaca Demo instructions.
For the third challenge of missing reference implementations, we implement and test several popular learning algorithms including PPO [62], expert iteration [2], and Quark [42], and release reference implementations. We show that, among the methods we studied, PPO with a surrogate reward model is the most effective training-time method in our leaderboard, improving the win-rate against Davinci003 of an instruction fine-tuned LLaMA 7B model from  $44\%$  to  $55\%$ . Other baselines that have been validated on simpler tasks fall short in comparison, highlighting the importance of testing these algorithms in a real instruction-following setting.
As an end-to-end evaluation of the AlpacaFarm, we compare eleven methods trained and evaluated in AlpacaFarm with the same methods trained and evaluated on actual human feedback. We show that the method rankings obtained from developing on AlpacaFarm closely agree with the method rankings obtained from training on actual human data (Spearman correlation of 0.98) and that the best method in AlpacaFarm leads to substantial gains with human feedback. Finally, we find that AlpacaFarm can replicate qualitative behaviors of human feedback such as over-optimization of the reward model, suggesting that AlpacaFarm serves as an effective way for researchers to rapidly study and develop methods that learn from human feedback.
2 Background & problem statement
To begin, we introduce the instruction following task and the pairwise-comparison-based human feedback setting that we study. With this background, we formally define the goals of developing a low-cost simulator for studying instruction following and learning from human feedback.
2.1 Learning to follow instructions
In the instruction following task [49, 6, 76, 40], we are presented with user instructions  $x \in \mathcal{X}$  (e.g. "Tell me something about Alpacas"), and our goal is to develop a model  $p_{\theta}$  that generates high-quality responses  $y \sim p_{\theta}(y \mid x)$  as judged by an unobserved human reward function  $R: \mathcal{X} \times \mathcal{Y} \rightarrow \mathbb{R}$ .
While there are a rich set of methods that learn from human feedback to directly optimize  $R$  (see Appendix A), in this work we focus on the setting of learning from pairwise feedback (LPF) due to its central role in recent instruction-following LLMs [49]. The starting point of this process is a model that is fine-tuned on instruction-following demonstrations  $(x, y)$ , which we denote as  $p_{\theta}^{\mathrm{SFT}}(y \mid x)$ . The LPF process then involves taking pairs of samples from  $p_{\theta}^{\mathrm{SFT}}$ , querying humans for which sample within each pair is better, and learning from this pairwise feedback. Since all methods start from the SFT base, we use  $p_{\theta}$  for notational simplicity.
Learning from pairwise feedback (LPF). More formally, we define the pairwise feedback dataset as  $\mathcal{D}_{\mathrm{pairwise}} = \{(x^{(j)},y_0^{(j)},y_1^{(j)},z^{(j)})\} _j$ . In this notation, a human annotator rates two candidate responses  $y_{0},y_{1}\sim p_{\theta}(y\mid x)$  for the instruction  $x$ . These binary ratings  $z\in \{0,1\}$  are assumed to be generated according to their unobserved reward  $R$  and  $z$  indicates a (potentially stochastic) comparison for the better response  $y_{z}$ , where  $R(x,y_z) > R(x,y_{1 - z})$ .
Many algorithms have been proposed to learn on  $\mathcal{D}_{\mathrm{pairwise}}$ . Some algorithms like RLHF [18, 49] learn a surrogate reward function as the learning signal and some operate more directly on  $\mathcal{D}_{\mathrm{pairwise}}$ . We defer the discussion of different learning algorithms to Section 3.4.
Pairwise evaluation. Once instruction-following models are trained, researchers need to evaluate these models. One common approach for evaluating models is pairwise model evaluation [15, 22, 7], which performs pairwise comparisons on outputs generated by the model  $p_{\theta}$  and a reference model  $p_{\mathrm{ref}}$ . Concretely, we collect pairwise preference for two models  $\left(\left\{\left(x^{(j)},y_{\theta}^{(j)},y_{\mathrm{ref}}^{(j)},z^{(j)}\right)\right\}_{j}\right)$ , which is aggregated by computing the average win-rate – the percentage of times  $p_{\theta}$  is preferred to  $p_{\mathrm{ref}}$ . Researchers can then compare LPF models by their win-rates against the same reference model.
2.2 Problem statement
The goal of AlpacaFarm is to provide three key components that enable rapid research and development of instruction following models: low-cost pairwise feedback generators, automated evaluations for methods development, and reference implementations for comparison and modification. With these three components, researchers can develop new methods in simulation and transfer these insights to build high-performance systems on actual human feedback.
For pairwise feedback, we substitute human preference judgements  $z_{\text{human}} \sim p_{\text{human}}(z \mid x, y_0, y_1)$  with a simulated preference  $z_{\text{sim}} \sim p_{\text{sim}}(z \mid x, y_0, y_1)$  using API LLMs. Our goal is to construct a  $p_{\text{sim}}$  that is both low-cost and faithfully captures different aspects of human preference feedback, such as quality judgments, inter-annotator agreement rates, and stylistic preferences.
For evaluations, we evaluate system outputs using the pairwise preference simulator and identify evaluation datasets that reflect natural human-LLM interactions. The goal for our evaluations is to ensure that system rankings on the new evaluation dataset closely match both human rankings and rankings on instructions from real usage of the Alpaca Demo.
For reference methods, we develop and evaluate six LPF methods. Our goal will be to provide simple and working implementations that provide substantial improvements on both simulated and human feedback data. This will allow researchers to build upon and compare to competitive baselines in a complex instruction-following environment.
AlpacaFarm combines these three components into a simulation framework for learning from pairwise feedback. We evaluate the complete system by an end-to-end workflow of developing methods in simulation and transferring the insights to the real world.
Concretely, we will run each method  $M$  on the simulated preferences (called  $M_{\mathrm{sim}}$ ) and evaluate with simulated rankings  $p_{\mathrm{sim}}$ . In parallel, we will run  $M$  on human preferences (called  $M_{\mathrm{human}}$ ) and evaluate with human rankings  $p_{\mathrm{human}}$ . We consider AlpacaFarm to be successful if the simulated method rankings correlate well with the human method rankings. The rest of this work will present the details of the pairwise feedback and evaluation design (Section 3), evaluate these designs (Section 4), and analyze the different reference methods we implemented in the AlpacaFarm (Section 5).
3 Constructing the AlpacaFarm
In this section, we detail how we construct the AlpacaFarm. In Section 4, we then validate our design choices by comparing the LPF workflow with human feedback and evaluation.
3.1 Instruction following data
Before defining the details of how we simulate pairwise feedback, we must first specify a large and diverse set of instructions  $x$  upon which we can build the rest of AlpacaFarm. We opt to use the Alpaca data [69] as a starting point due to its large size (52k ( $x, y$ ) examples) and the non-trivial instruction following capabilities of models trained on this data.
We repurpose the Alpaca data into splits suitable for learning from human feedback by following a similar splitting ratio as [49]. We created four splits (42k in total), leaving 10k for the future:
- Supervised finetuning (SFT) split: 10k data for fine-tuning the base instruction-following LLM used in subsequent steps.
- Pairwise preference (PREF) split: 10k instructions on which we collect pairwise feedback data.
- Unlabeled split: 20k unlabeled instructions used in algorithms such as PPO.
- Validation split: 2k data for development and tuning.
3.2 Designing simulated pairwise preference  $p_{\mathrm{sim}}$
We now describe the design of our simulated annotator for pairwise preferences. Our core proposal is to design the simulator  $p_{\mathrm{sim}}(z \mid x, y_0, y_1)$  by prompting OpenAI API LLMs. While using LLMs as a proxy for annotators has become increasingly popular [15, 39], using LLMs as part of a simulation environment poses major additional challenges. Our simulated preferences  $p_{\mathrm{sim}}$  must not only have a high agreement with human preferences  $p_{\mathrm{human}}$ , it must also capture other qualitative aspects of human feedback such as inter- and intra-annotator inconsistencies. Intuitively, the noise and variability in pairwise feedback are key parts of the challenge in the LPF problem and we find that ignoring these factors leads to a simulator that diverges substantially from real-world behavior (Section 4.3).
Basic GPT-4 prompt design. To start with, we design prompts by providing an annotation guideline, feeding in-context examples, and leveraging batch generation to save on costs. As a first baseline, we query GPT-4 with a single prompt (denoted as  $p_{\mathrm{sim}}^{\mathrm{GPT-4}}$ ) and we find  $p_{\mathrm{sim}}^{\mathrm{GPT-4}}$  has a high agreement rate with human annotators (65%; see results in Section 4.3). However, we find that this simple baseline of  $p_{\mathrm{sim}}^{\mathrm{GPT-4}}$  fails to capture the variability in human annotation and can lead to qualitatively different results for method development, especially for reward over-optimization (Section 4.3).
Simulating human variability. To more completely emulate human annotators, we modify the basic simulated annotator design to capture annotator variability in two ways. First, we emulate inter-annotator variability in the simulated pairwise preference  $p_{\mathrm{sim}}$  by mimicking a pool of annotators. We design different annotators by querying different API LLMs and varying the prompts with different formats, batch sizes, and in-context examples. In the end, we created 13 simulated annotators which we describe fully in Appendix D. Second, we emulate intra-annotator variability by directly injecting random noise and flipping the simulated preference  $25\%$  of the time.
With these ingredients, we come up with a simulated preference  $p_{\mathrm{sim}}^{\mathrm{ann}}$  that meets our requirement of agreement and variability. Overall, annotating 1000 outputs using simulated preference only costs $6,
which is  $50\mathrm{x}$  cheaper than human annotation. In Section 4, we collect actual human preference and quantitatively verify the agreement and variability of our simulated preference.
3.3 Designing an automatic evaluation
For researchers to develop LPF methods in the AlpacaFarm, we want to support them with an automatic evaluation so they can quickly iterate while reliably comparing methods. To replace the usual human evaluation, there are two challenges. First, how do we quantify the quality of different models? Second, what instructions can we use that are representative of human interactions?
Evaluation protocol. To quantify the quality of an LLM  $p_{\theta}$ , we measure the win-rate of that LLM against a reference model, i.e., the expected number of times that the output from  $p_{\theta}$  is preferred to the output of a reference model  $p_{\mathrm{ref}}$  on the same instruction  $x$ . The benefits of using simulated win-rates are that it provides a metric that is easy to understand, is comparable across methods, and can reuse the routine we built for pairwise feedback. We use the 13 simulated annotators described in Section 3.2 without injecting the additional noise (as adding uniform noise does not change model rankings) and denote this preference simulator as  $p_{\mathrm{sim}}^{\mathrm{eval}}$ . For the reference model, we use Davinci003 as it is a well-studied system that performs similarly to the models we fine-tune.
Evaluation data. Instruction following requires diverse coverage over realistic interactions. To build an appropriate evaluation protocol, we combine several open-source evaluation datasets and use real-world interactions with a demo instruction-following LM (Alpaca Demo [69]) as guidance for constructing our data combination. Due to privacy concerns, we do not directly release the demo data and opt to use it to guide how we combine existing open evaluation datasets.
Our final evaluation set consists of 805 instructions, which includes 252 instructions from the self-instruct test set [75], 188 from the Open Assistant (OASST) evtest setaluation, 129 from Anthropics' helpful test set [6], 80 from Vicuna test set [80, 17], and 156 from Koala test set [22]. In ??, we show example instructions from our evaluation set (Table 7) and their root verb distribution(Figure 13), which shows the diverse coverage of our data. We find that aggregating across datasets is important for automatic evaluation to match real-world interactions, as discussed in Section 4.4.
3.4 Reference methods in AlpacaFarm
Finally, AlpacaFarm defines a collection of validated LPF methods for instruction following. We describe those methods in detail in Appendix B but provide a brief overview here. In all LPF methods that follow, we begin by first performing an initial fine-tuning step on supervised data of instructions and outputs. We begin by describing two simple baselines that directly operate on pairwise feedback.
- Binary FeedME. Binary FeedME [47] continues supervised fine-tuning on the preferred output in each pairwise comparison.
- Binary reward conditioning. Reward conditioning [28, 36, 30] is a simple scheme that incorporates learning from negative examples; a token denoting whether the output was preferred is pretended before fine-tuning, and the positive token is used to condition at inference time.
Many LPF methods do not directly operate on pairwise feedback data, but instead first construct a surrogate reward model by fine-tuning a classifier from the SFT base using pairwise feedback. The following LPF methods maximize the continuous-valued reward defined by the logits of this classifier.
- Best-of- $n$  sampling. Best-of- $n$  (or re-ranking) [67, 5, 23, 9] is a simple but effective inference-time method that draws  $n$  i.i.d. responses from the SFT model and returns the response with the highest surrogate reward. Unless stated otherwise we use  $n = 1024$  in our experiments.
- Expert iteration. Expert iteration [2, 64, 74] is the natural training-time extension of best-of- $n$ : it first generates according to best-of- $n$  on new instructions and then fine-tunes on the best outputs.
- Proximal Policy Optimization (PPO). PPO [26, 62] is a reinforcement learning algorithm that maximizes surrogate reward, subject to a KL penalty keeping parameters near SFT initialization.
- Quark. We use the top-quantile variant of Quark [42] which bins sequences by reward and trains on the best bin, along with adding KL and entropy regularization.
Figure 2: The ranking of methods trained and evaluated in AlpacaFarm matches that of methods trained and evaluated in the human-based pipeline. Each blue point represents one method  $M$  (e.g. PPO). The x-axis shows the simulated evaluation (win-rates measured by  $p_{\mathrm{sim}}^{\mathrm{eval}}$ ) on methods trained in simulation  $M_{\mathrm{sim}}$ . The y-axis shows human evaluation (win-rates measured by  $p_{\mathrm{human}}$ ) on methods trained with human feedback  $M_{\mathrm{human}}$ . Gray points show models that we did not train, so their  $x$  and  $y$  values only differ in the evaluation (simulated vs human). Without those points, we have  $R^2 = 0.83$  and a Spearman Correlation of 0.94.
4 Validating the AlpacaFarm simulator
With the simulator and methods defined, we now evaluate AlpacaFarm. As our main result, in Section 4.2 we analyze the correlation between the final rankings of methods in both the simulated LPF workflow and human-based LPF. Afterward, we will analyze more specifically whether our pairwise feedback accurately mimics human pairwise feedback (Section 4.3) and whether rankings on our evaluation data match rankings on the Alpaca Demo data (Section 4.4).
4.1 Experimental details
Models. As baseline and starting point for LPF methods, we fine-tuned LLaMA 7B on the 10k SFT split. We take SFT 10k as starting point for all LPF methods and collect the simulated preference  $p_{\mathrm{sim}}^{\mathrm{ann}}$  and human preference  $p_{\mathrm{human}}$  from SFT 10k's outputs (with temp=1.0) on the 10k instruction PREF split. Then, for each of the six reference LPF methods  $M$ :
- We trained and tuned  $M$  on simulated preferences  $p_{\mathrm{sim}}^{\mathrm{ann}}$ , evaluating the resulting model  $M_{\mathrm{sim}}$  against the Davinci003 reference with the simulated evaluator  $p_{\mathrm{sim}}^{\mathrm{eval}}$ .
- We trained a few models  $M$  on human preferences across hyperparameter ranges identified in simulation, evaluating the resulting model  $M_{\text{human}}$  against Davinci003 with humans  $p_{\text{human}}$ .
In addition to the six methods, we also evaluate existing instruction-following and base models: GPT-4 (gpt-4-0314), ChatGPT (gpt-3.5-turbo-0301), Davinci001 (text-davinci-001), LLaMA 7B [72], and Alpaca 7B [69]. Alpaca 7B is a LLaMA 7B model finetuned on all data splits, denoted SFT 52k. For these models, we measure both the simulated win-rate  $p_{\mathrm{sim}}^{\mathrm{eval}}$  and human win-rate  $p_{\mathrm{human}}$ .
At inference time, for all systems except best-of-  $n$ , we sample with  $\text{temp} = 0.7$  and set the maximum number of tokens to be 300. For best-of-  $n$  sampling, we found a higher temperature to be helpful in encouraging output diversity, and so we rerank samples from SFT 10k with  $\text{temp} = 1.0$ . We provide more thorough experimental details and hyperparameters for all methods in Appendix C.
Human annotation. We collected reference human annotation by showing crowd-workers two potential outputs  $y_{0}$  or  $y_{1}$  for a given instruction  $x$  and asked them to select the index  $z \in \{0, 1\}$  of their preferred output. Annotators are recruited from Amazon Mechanical Turk using a qualification test of 25 questions. Out of an initial pool of 34 annotators, we selected the 16 whose agreement rate was higher than  $70\%$  with the author's annotations. We paid the annotators a median hourly rate of $21, leading to a one-time $3000 cost of annotating our PREF split and a recurring $242 cost for evaluating a single model on the 805 evaluation instructions. See Appendix E for additional details.
4.2 End-to-end validation of AlpacaFarm
We now analyze the correlation between rankings in simulation and on human data. Figure 2 shows the win-rate of methods in AlpacaFarm (x-axis) with the win-rate from the human-based pipeline (y-axis). We see that the rankings have a Spearman correlation of 0.98, which suggests that AlpacaFarm faithfully captures the rankings among different LPF methods. This enables researchers to develop models at low-cost in AlpacaFarm and transfer insights to train models on real-world human interactions.