SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Binary FeedME. FeedME is a method proposed by OpenAI [47] that incorporates human feedback with supervised fine-tuning on model generations that are rated 7/7 by human labelers. We adapt this approach to the pairwise feedback setting and call this baseline binary FeedME. This approach fine-tunes the SFT model on the chosen response in each preference pair with supervised learning.
Binary Reward Conditioning. Motivated by controllable generation through conditioning [28, 36, 30, 22], we propose binary reward conditioning, a baseline method that fine-tunes the SFT model with the feedback data  $\mathcal{D}_{\mathrm{pairwise}}$  by conditioning instances with either a positive or negative control token. Specifically, for each instance  $(x,y_0,y_1,z)\in \mathcal{D}_{\mathrm{pairwise}}$ , the string concatenation of instruction  $x$  and response  $y_{z}$  denoted as  $[x,y_z]$  is prepended with the positive token and used in supervised fine-tuning (similarly  $[x,y_{1 - z}]$  is prepended with the negative token). This process creates a modified demonstration dataset that is double the size of  $\mathcal{D}_{\mathrm{pairwise}}$ . At test time, we draw samples from the fine-tuned model conditioned on the positive token.
B.2 Methods that optimize a surrogate reward function
We now describe methods that incorporate feedback by first building a surrogate reward model with pairwise feedback data. To start, we describe the step of training the surrogate reward model.
To train a parameterized surrogate  $\hat{R}_{\phi}$ , one can maximize the log-likelihood of the preferences  $z$  under the Bradley-Terry model [11]
$$
\operatorname {m a x i m i z e} _ {\phi} \sum_ {j} \log P \left(z ^ {(j)} \mid x ^ {(j)}, y _ {0} ^ {(j)}, y _ {1} ^ {(j)}\right) = \sum_ {j} \log \frac {\exp \left(\hat {R} _ {\phi} \left(x ^ {(j)} , y _ {z} ^ {(j)}\right)\right)}{\exp \left(\hat {R} _ {\phi} \left(x ^ {(j)} , y _ {0} ^ {(j)}\right)\right) + \exp \left(\hat {R} _ {\phi} \left(x ^ {(j)} , y _ {1} ^ {(j)}\right)\right)}. \tag {1}
$$
Once the surrogate reward model is trained, both training and inference algorithms can optimize against the reward model rather than query pairwise feedback. While this can be a powerful approach, we will see that it can also lead to over-optimization [21] where models learn to exploit the reward model rather than achieve high true reward. We now describe 4 methods that leverage the surrogate reward model.
Best-of- $n$  Sampling. Best-of- $n$  sampling (or re-ranking) [67, 5, 23, 9] is a common inference-time method that aims to improve the generation quality. Given an input  $x$ , the method returns the response with the highest surrogate reward value among  $n$  i.i.d. responses drawn from the SFT model. While simple to implement and useful as a baseline, this approach incurs high inference costs.
Expert Iteration. Expert iteration [2, 64, 74] is a technique that has recently been used to train language models. We adapt this approach in AlpacaFarm as a two-step method. In the first step, we perform best-of-  $n$  sampling and store the generated samples. In the second step, we fine-tune  $p_{\mathrm{SFT}}$  on these samples with supervised learning. While prior work applies expert iteration for multiple rounds by performing best-of-  $n$  sampling again for intermediate models, we focus on performing a single round. In Appendix C, we include our preliminary study of multi-round expert iteration.
Proximal Policy Optimization. Proximal Policy Optimization [PPO; 26, 62] is a popular RL algorithm that has been recently used to develop InstructGPT [49] and ChatGPT [46]. When applied to fine-tune LMs with RLHF, PPO maximizes the following KL-regularized objective w.r.t. model parameters  $\theta$
$$
\mathbb {E} _ {x \sim p (x), y \sim p _ {\theta} (y | x)} \left[ \hat {R} _ {\phi} (x, y) - \beta \log \frac {p _ {\theta} (y \mid x)}{p _ {\mathrm {S F T}} (y \mid x)} \right], \tag {2}
$$
where  $p(x)$  is an unlabeled instruction distribution,  $p_{\theta}(y \mid x)$  is fine-tuned from the  $p_{\mathrm{SFT}}$  model, and  $\beta \in \mathbb{R}$  is a regularization coefficient. Each step of PPO alternates between drawing samples from the current policy and performing gradient updates based on the pool of samples with importance sampling and clipping.
Quark. Quark is inspired by reward conditioning and has been shown to be effective for controllable generation tasks. Like binary reward conditioning, Quark on train sequences with prepended control tokens. Unlike binary reward conditioning, Quark bins model samples into multiple groups based on the reward value, adds KL and entropy regularization, and repeats the entire process across multiple rounds.
In our preliminary analysis, we find the top-quantile variant reported in [41], i.e., only training on the best reward group, to perform the better than the all-quantiles variant which trains on all groups.
C Details on methods implementation and hyperparameters
For all hyperparameters and code see https://github.com/tatsu-lab/alpaca_farm
C.1 PPO
We follow an existing PPO implementation for fine-tuning language models, $^{2}$  but also introduce modifications. First, off-the-shelf PPO implementations for language model fine-tuning tend to normalize the estimated advantage for each minibatch. We found this led to training instabilities for small minibatch sizes and instead normalize the advantage across the entire batch of rollouts obtained for each PPO step. Second, we initialize the value model from the reward model as opposed to the SFT model, following more recent documented practice [49] (the authors did not release code). Our preliminary experiments showed that initializing from reward worked much better than initializing from SFT for maximizing the surrogate reward.
We tuned hyperparameters to improve training stability and reduce convergence time so that experiments can reliably finish with relatively tight compute budgets. In the end, we settled on a batch size of 512 for each PPO step, which consisted of 2 epochs of gradient steps each performed with a batch of 256 rollouts. We used a peak learning rate of  $10^{-5}$  which decayed to 0 throughout training. We clipped the gradient by Euclidean norm with a threshold of 1. We trained for 10 full passes over the unlabeled set, which amounts to 390 PPO steps. Performance typically peaked very early on during training (see Figure 4). We set  $\lambda$  and  $\gamma$  both to 1 for generalized advantage estimation [61]. We used a fixed KL regularizer coefficient as opposed to an adaptive one. We tuned the coefficient value for both simulated and human PPO, and settled with 0.02 for human PPO, and 0.002 for simulated PPO. We note that suitable values for the KL regularizer coefficient depend on the early stopping criteria and the scale of surrogate reward values.
C.2 Quark
We re-implement Quark for our needs and make several modifications. First, the original Quark formulation accumulates rollouts during training and stores them in a pool that consistently grows. We found this led to overhead that increased during training (since after each rollout batch is generated, the pool is expanded and rollouts in the pool are re-sorted by their reward values). To operate under a reasonable compute budget, we discard previous rollouts once a new batch of rollouts is generated. In other words, the pool is reset once rollout is performed. This modification made the compute cost constant throughout training and thus more predictable overall. Second, we found that training on rollouts of more bins led to worse efficiency for reward optimization, and thus opted to train only on rollouts of the top-scoring bin (best-quantile variant in the original paper [42]). Preliminary ablations on a simple sentiment task showed that any potential loss in perplexity for the best-quantile variant can be compensated by turning up the KL regularizer. Lastly, we found the entropy penalty used in the original Quark formulation to give no benefit for working with instruction following. Small entropy penalty terms were enough to cause big degradations in text generation quality in terms of fluency.
For the official run with reported results, we used a KL regularizer coefficient of 0.05, a peak learning rate of  $3 \times 10^{-6}$  which decayed to 0 throughout training. Each Quark step had batch size 512 for rollout, and 2 epochs of gradients updates each with batch size 256. We clipped the gradient by Euclidean norm with a threshold of 1. We trained for 10 full passes over the unlabeled set, which amounts to 390 Quark steps.
$^{2}$ https://github.com/openai/lm-human/preferences
D Pairwise preference simulation
D.1 Details about simulated annotators
For all our simulated annotators we used OpenAI API to generate outputs. We first discuss below the overall design choices for all our simulators below, and then discuss our annotator pool below in more detail. For all the actual prompts we used refer to https://github.com/tatsu-lab/alpaca_farm.
Randomized order. For each annotator, we randomize the ordering between the two outputs to annotate, i.e., we randomly choose which output is the first and which is the second. We found randomization to be important given that the first output is often preferred by simulated annotators.
Prompts with and without inputs. Following the Alpaca dataset [69] and self-instruct framework [75] some instructions have associated inputs, while others do not. For each annotator, we thus write two corresponding prompts, one for instructions with inputs and one for instructions without inputs. Both prompts are essentially the same but in-context examples differ in the presence of the input.
Batching for GPT4. When adding in-context examples, prompts can become relatively long, which leads to high-cost and waiting time when using GPT-4 as a simulator. To decrease cost and increase annotation speed, we amortize the cost of in-context examples by providing a batch of instruction-output pairs to annotate at once by GPT-4. For our simulated annotator we use a maximum batch size of 5 but found during development that we could fit batch size up to 20 in the context window without significantly decreasing performance. To improve performance when using batching, we found it useful to provide a few in-context examples in a batched format and to index every component of an annotation (instruction, input, output, ...).
Improving parsing for ChatGPT. Overall we found ChatGPT to be much more sensitive and harder to use as a simulator. In particular, we found it to be more sensitive to the prompt format and to often fail to generate annotations that could be parsed, e.g., by responding "Neither is better, this depends on personal preferences" despite being explicitly instructed to choose a preference. We found two tricks to be effective to make ChatGPT's more parsable. First, we add a negative bias to tokens such as "Neither" and "Both" and a positive bias to the tokens that we hoped to match. We found the aforementioned biasing of tokens to work well but it can be problematic when using Chain of Thought reasoning. A second trick that we found to be effective is to ask ChatGPT to generate a JSON object that contains a string field with a short explanation (Chain of Thought) and a boolean field that indicates whether the first output was preferred.
Now that we have discussed the overall design choices for our simulated annotators, we discuss in more detail the prompts and parameters for each of our annotators.
AlpacaFarm's evaluation annotators  $p_{\mathrm{sim}}^{\mathrm{eval}}$ . To try to match the bias and variance of human annotators, we use a pool of 13 simulated annotators that were developed at different stages of the project. In particular, we use the following sources of variations:
- Models. Five of the annotators are powered by GPT-4, four by ChatGPT, and four by Davinci003. The difference between different annotators for the same model is mostly the prompt.
- In-context examples. Prompts for the same models use different numbers of in-context examples.
- Prompt format. We use different prompt formats between and for the same model. For example different batch sizes and different formats of outputs (JSON vs raw text).
- Preferences. Two of the GPT4 annotators are explicitly prompted to prefer sequences that are respectively long and short.
- Sampling. For each annotator in the pool, we use a sampling temperature of 1.0 with top  $p$  also 1.0. The high temperature means that we have variability that arises from sampling.
Figure 6: Our simulated annotators achieve relatively low bias with human annotators and match human variance. The y-axis shows the estimated bias, i.e., the error between the majority vote of 4 simulated annotators and the majority vote of 4 human annotators. The x-axis shows the estimated variance, i.e., the error between a held-out annotation and the majority vote of the other three annotators. The bias of humans is by definition 0, and variance is shown with a blue line. Grey points are all the annotators in our simulated pool, the green point shows the resulting pool of annotators (which we use for evaluation), the orange point shows the same simulated pool with additional noise (which we use for training), the blue point the average human annotator, and the red point shows a single low variance GPT-4 annotator we analyze.
AlpacaFarm's training annotators  $p_{\mathrm{sim}}^{\mathrm{ann}}$ . Our simulated annotators for training are the same as the evaluation annotators  $p_{\mathrm{sim}}^{\mathrm{eval}}$  except that we flip the output with 0.25 probability. We implement this by taking a mixture between  $p_{\mathrm{sim}}^{\mathrm{eval}}$  and an independent Bernoulli random variable with probability 0.5. This means that we only need to label half of the outputs for training, which makes it  $2 \times$  faster and cheaper.
GPT4. For the GPT4 annotator  $p_{\mathrm{sim}}^{\mathrm{GPT-4}}$  we use a prompt with batch size five that corresponds to one of the prompts from our simulated pool of annotators. For  $p_{\mathrm{sim}}^{\mathrm{GPT-4}}$  we use temperature 0, i.e., deterministic annotations.
D.2 Additional results
We now provide additional results for understanding our pairwise annotators. For more results and code for generating plots see https://github.com/tatsu-lab/alpaca_eval.
Our pool of annotators has low bias and matches human variance. Figure 6 shows the estimated bias (y-axis) and variance (x-axis) of simulated evaluators. We see that single evaluators have a smaller variance (less than 0.2) than humans (blue line, 0.34). This lack of variability makes emulating it with a proxy reward very easy and leads to unrealistic over-optimization properties in the simulator, as seen in Figure 4. Using a pool of annotators (green point) for evaluation and additionally adding noise (orange) during training gives an estimated variance significantly closer to humans (blue line 0.35). We hypothesize that this is necessary for the simulator to show a similar over-optimization behavior as humans. Concerning the bias, we find that our simulated annotators for evaluation  $p_{\mathrm{sim}}^{\mathrm{eval}}$  and training  $p_{\mathrm{sim}}^{\mathrm{train}}$  both have low bias values (0.38 and 0.35) on par with one of our best GPT-4 annotators (0.33).
Variability in a pool of annotators mostly comes from the underlying model. In Figure 7 we show the pairwise agreement between all annotators in our pool and all other annotators including the majority vote of humans (first column) and single humans (second column). The desired high variance corresponds to low values on the diagonal (annotators disagree with themselves) and low bias corresponds to high values in the first column (high agreement with the mode of humans). As in Figure 6, we see that our pool of annotators  $p_{\mathrm{sim}}^{\mathrm{eval}}$  has low bias and high variance. Figure 7 also shows that the largest source of variability between annotators comes from the underlying model, as illustrated by the clusters that arise from GPT4, ChatGPT and Davinci003 annotators.
Figure 7: The largest source of variability between annotators comes from the underlying model. Every cell of the heatmap shows the agreement between two annotators (x- and y-axis).
Figure 8: Humans and our simulated annotators prefer outputs that are longer and have lists.
Humans and simulated annotators prefer longer outputs that contain lists. One natural question is whether simulated and human annotators have biases towards different type of outputs, which would cause models in both frameworks to be qualitatively different. We identify two stylistic features, the length and the presence of lists, for which humans have a strong preference and analyze whether simulated annotators match those preferences. We found that humans prefer longer outputs  $62\%$  of the time, while our simulated annotators prefer those  $64\%$  of the time. Similarly, humans prefer outputs with lists  $69\%$  of the time, while our simulated annotators prefer those  $63\%$  of the time. This shows that our simulated annotators match well the stylistic preferences of humans, which suggests that models trained in our sandbox are optimizing similar preferences as those trained with human feedback and they will likely exhibit similar behaviors.
E Details on human data collection
Qualification. We conducted the qualification of our annotators based on 25 qualification examples. The qualification examples were generated but an OPT 6B model that was studied in the earlier development phase of this project. The five student authors of this paper annotated a shared set of pairwise preferences. From the shared set, we selected 25 questions where the majority of the authors reached an agreement on the correct annotation. We then use these questions as a qualification test and selected the top 16 annotators whose agreement is the highest with the authors. We paid the annotators the same price for the qualification round as we did for the main qualification.
During the annotation process, we also compare each annotator's preference to that of GPT-4. We identified one annotator whose agreement is around  $50\%$  with GPT-4, which is a clear outlier from other annotators. Therefore, we discontinued working with this annotator during the annotation project and removed their annotation.
**Annotation guideline.** We display our annotation guideline in Figure 9 and annotation interface in Figure 10. In our annotation process, we find that there are pairs that only differ in punctations or have minimal edit distance and we instruct the annotators to select a response as slightly better/worse if the difference between the pairs is marginal. As a result, around  $18\%$  of the collected preference selected the slightly better options. In our LPF experiments, we binarize the preference and treated the slightly better options the same as the normal preference labels. However, we release the more fine-grained labels as resources and leave the study to future work.
Hi! We are a group of researchers working on Artificial Intelligence (AI). In this task, we will ask you to help us rate an AI model's responses to instructions.
In the area below, you will first read:
1. An instruction we give to the AI system.
2. An input that is provided along with the instruction. This is an optional input and not all instructions will have inputs.
3. Two responses from the AI system
Your task is to decide which response is better. There are several dimensions that you can think along. Consider the following questions:
1. Is the response helpful? For example, if the instruction asked for a recipe for healthy food, and the response is a useful recipe, then we can consider it helpful.
2. Is the response language natural? For example, AI responses are often repetitive, which is not natural.
3. Is the response factual/accurate? AI responses often make up new information. For example, if the response claims that Donald Trump is the current U.S.
president, then you should consider it inaccurate.
4. Based on your aesthetics, which one do you prefer? For example, you might prefer one poem over another poem.
5. And so on ... ultimately, you should decide which response is better based on your judgment and based on your own preference.
There are four options for you to choose from:
1. Response 1 is better: If you think response 1 has an advantage, then choose this option.
2. Response 1 is slightly better: Response 1 is very marginally better than response 2 and the difference is small.
3. Response 2 is slightly better: Response 2 is very marginally better than response 1 and the difference is small.
4. Response 2 is better: If you think response 2 has an advantage, then choose this option.
There are cases where the difference between the two responses is not clear. In this case, you can choose the second or the third option. However, in general, we ask you to choose those options as few as possible.
Annotation Examples
To help you understand the annotation task, we provide two examples below.
Instruction: Paraphrase the following question: "What is the capital of France?" Input: Response 1: The capital of France is Paris. Response 2: Can you tell me the name of the city that serves as the capital of France?
In this example, response 2 paraphrases the question as asked by the instruction. In contrast, response 1 does not follow instruction as it answers the question instead. Therefore, we should choose Response 2 is better.
Instruction: Change the first verb to a synonym Input: I love surfing Response 1: I hate surfing Response 2: and am planning on going surfing tomorrow morning.
In this example, Neither output is correct. Output (a) tries to follow more closely the instruction as it changes the first verb of the sentence. Therefore, we should choose Response 1 is better.
Instruction: Change the first verb to a synonym Input: I love surfing Response 1: I hate surfing Response 2: I don't like surfing
In this example, neither output is correct and the responses are very similar. In such a case, you can choose either Response 1 is slightly better or Response 2 is slightly better. However, in general we ask you to choose those options as few as possible.
Payment structure
In general, we aim to pay crowd workers $15/hour on average. Our current estimate is that it will usually not take you more than a minute to compare each pair of responses on average. Therefore, we currently pay $0.25 per pair of responses.
In the qualification round, you will need to compare 24 pairs of responses and we pay  $6 for this part. We also allocate 2 minutes for you to read the annotation guideline, for which we pay$ 0.50. In total, we pay $6.50 for the qualification round.
Main HIT
In total, we have around 20K pairs to annotate and we plan to spread the annotation over 2 weeks. In each HIT after the qualification round, you can work on 15 pairs of responses. We pay $3.75 for each HIT.
Figure 9: Our annotation guideline.
Hi test22. Not your MTurk id?Logout
Your annotation progress for this HIT: 0/6 completed.
Sample 1 of 6.
Lets you skip forward (all samples still need to be marked):  $\rightarrow$  Next sample.
Returns you back to the next un-marked sample to mark: Go to next un-marked sample.
$\triangleright$  Annotation Guidelines
Tell me about self driving cars
AI Response 1
Self driving cars are vehicles that are capable of detecting their surroundings and navigating without any input from a human driver. They use a variety of sensors, such as cameras, radar, and lidar, to detect objects in their environment and use advanced algorithms to make decisions about how to maneuver around them. Self driving cars are becoming increasingly popular and are being used in a variety of applications, such as ride-hailing services and autonomous delivery services.
AI Response 2