SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Self driving cars are cars that are equipped with sensors, cameras, and software, which allows them to navigate without the need for a driver. Self driving cars are able to monitor their environment and navigate without the assistance of a driver. Self driving cars use a combination of technologies such as GPS, radar, lidar, and computer vision to detect their environment and avoid obstacles. They are also able to update their maps and adjust their routes to account for variables such as traffic and construction.
The main benefit of self driving cars is their ability to reduce crashes and save lives. Autonomous vehicles have the potential to save up to 30,000 lives annually. They are also more energy efficient than traditional vehicles, as they do not require a driver who has to maintain a minimum speed. Autonomous vehicles also create new opportunities for increased mobility, allowing those who are unable to drive to get around.
$\bigcirc$  Response 1 is better.
$\bigcirc$  Response 1 is only slightly better. (only pick this if it's truly close)
$\bigcirc$  Response 2 is only slightly better. (only pick this if it's truly close)
$\bigcirc$  Response 2 is better.
Optional explanation (never required): for cases where rating is near random.
Figure 10: Our annotation interface.
F Additional results
F.1 Label noise ablations for simulated annotators
Figure 11: Label noise is the most crucial ingredient for inducing overoptimization.
In this section, we ablate the different components of  $p_{\mathrm{sim}}^{\mathrm{ann}}$  that add variability along two axes: randomizing across different simulated annotators, and adding label noise. To ablate the randomization across different annotators, we compare to the simple GPT-4 prompt  $p_{\mathrm{sim}}^{\mathrm{GPT-4}}$  with added label noise. To ablate the label noise, we compare to  $p_{\mathrm{sim}}^{\mathrm{eval}}$ , which is  $p_{\mathrm{sim}}^{\mathrm{ann}}$  without the label noise. We train reward models on these preference distributions and compare the performance of best-of-  $n$  sampling.
Figure 11 shows the results of the ablation, demonstrating clearly that added label noise provides the majority of the overoptimization effect. In particular, the two options that do not add label noise,  $p_{\mathrm{sim}}^{\mathrm{GPT-4}}$  and  $p_{\mathrm{sim}}^{\mathrm{eval}}$ , keep increasing win-rates with more samples. This result suggests that modeling intra-annotator variability via label noise may be an important component to understanding learning from human preference data. The exact ratio of  $25\%$  label noise was chosen to be the smallest considered ratio that induced overoptimization out of the ones we considered (0,17%,25%,35%), as shown in fig. 12.
Figure 12:  $25\%$  label flipping is the smallest ratio we tested that induced over-optimization. The  $y$ -axis shows the simulated win-rate. The  $x$ -axis shows the number  $n$  of samples used in best-of- $n$  sampling, which is proportional to the output's average reward. Each curve shows a reward model trained on samples with different label flipping ratios ( $0\%$ ,  $17.5\%$ ,  $25\%$ ,  $35\%$ ). We see that over-optimization seems to be approximately monotonic with respect to the label-flipping ratio.
Table 3: Comparing system output before and after training with PPO, we observe that outputs often become longer and contain more lists.
Instruction: What are the five most common causes of stress among people?
SFT10K: The five most common causes of stress among people are work, money, relationships, health, and family.	PPO: The five most common causes of stress among people are: 1. Pres- sure at work: Whether it is due to tight deadlines, high expectations, or an overwhelming workload, pressure at work can be a major source of stress.
2. Money problems: Financial issues can be a significant source of stress, from not being able to make ends meet to struggling with debt. 3. Un-healthy lifestyle: Poor diet, lack of exercise, and sleep deprivation can all contribute to stress. 4. Unhappiness at home: Problems in relationships, conflicts with family members, and other domestic issues can be a major source of stress. 5. Uncertainty: Excessive worry and anxiety about the future, health issues, or other unknowns can be a major cause of stress.
F.2 Understanding computational cost
While we have focused primarily on the performance of the final model, the computational cost of these methods is an important consideration. We provide time estimates for training on our specific implementation and compute environment (a single  $8 \times \mathrm{A}100$  machine). While these timings are specific to our experiments, we believe these insights may be useful in understanding the cost of learning from pairwise feedback.
To begin with, supervised fine-tuning and methods that directly adapt supervised fine-tuning like Binary Reward Conditioning and Binary FeedME are generally fast, taking less than an hour for 10k instructions. Best-of-  $n$  sampling incurs no training cost but instead suffers a substantial inference time cost. The optimal  $n$  for best-of-  $n$  was around 16, which (in principle) translates into an increased cost of 16x for batched inference.
In our experiments, Expert Iteration works best when training on the outputs of best-of-16 sampling, which involves substantial amounts of compute to perform best-of-16 sampling on the unlabeled split. Concretely, this sampling process takes around 4 hours on the  $20\mathrm{k}$  unlabeled split and the subsequent fine-tuning takes less than an hour. We observe that PPO performance peaks on both human data and in simulation after 40-80 steps, which translates to less than 2 hours of compute time on the full machine.
F.3 Analysis of model outputs
PPO and best-of- $n$  reranking both demonstrate substantial gains with both AlpacaFarm's simulated feedback and human feedback. While the changes to the win-rate are clear, it is less clear what changes to the model lead to these improvements – are the gains driven by improvements to factuality or reasoning, or are they driven by stylistic differences? We now focus on an analysis of the outputs of these systems to understand what factors contribute to the improvement in their win-rates.
We find a major difference in the length distributions of the outputs, with outputs becoming increasingly longer after applying LPF methods in both human and simulated feedback. For example, the average length of SFT 10k outputs is 278 characters. Best-of-16 increases the average length to 570 characters and applying PPO increases it to 637 tokens. In Table 3, we show a qualitative example of how PPO training changes the output for an evaluation instruction. This is not surprising given that humans and simulated annotators prefer longer outputs as discussed in appendix D.2.
In Table 6, we further show three example outputs from SFT  $10\mathrm{k}$ ,  $\mathrm{PPO}_{\mathrm{human}}$ , and  $\mathrm{PPO}_{\mathrm{sim}}$ . In Table 4 and Table 5, we show the average length of model outputs of various systems in the number of characters, which qualitatively illustrates this phenomenon. While longer responses indeed lead to better human preference, we continue to see hallucinations in the output (e.g. in the example question about Berlin) and it is not clear to us whether our LPF models improve on the more critical aspects such as faithfulness, knowledge, etc.
Table 4: Average length of the model outputs of baselines and LPF models trained with human preferences. We observe that LPF training generally leads to longer outputs.
Model	Number of characters
GPT-4	504.4
ChatGPT	333.4
Davinci001	286.3
SFT 52K	383.2
SFT 10K	277.5
LLaMA 7B	950.5
PPO 200 steps	495.6
PPO 80 steps	623.7
PPO 40 steps	683.1
Best-of-128	680.0
Best-of-16	565.2
Best-of-4	478.7
ExpIter-128	524.7
ExpIter-16	458.3
ExpIter-4	422.1
FeedMe	371.4
Table 5: Average length of the model outputs of baselines and LPF models trained with simulated preferences. We observe that LPF training generally leads to longer outputs
Model	Number of characters
GPT-4	504.4
ChatGPT	333.4
Davinci001	286.3
SFT 52K	383.2
SFT 10K	277.5
LLaMA 7B	950.5
PPO 80 steps	863.4
PPO 20 steps	637.7
Best-of-128	704.7
Best-of-16	570.5
Best-of-4	483.3
ExpIter-128	527.5
ExpIter-16	458.3
ExpIter-4	407.4
Discuss the causes of the Great Depression
Make a list of desirable Skills for software engineers to add to LinkedIn. Are there any free SAST tools out there?
I'm trying to teach myself to have nicer handwriting. Can you help? What if Turing had not cracked the Enigma code during World War II?
Take MLK speech "I had a dream" but turn it into a top 100 rap song. What are some toys I can buy my kids for imaginative play?
Hi, I have a question about MFCC (mel frequency cepstral coefficients).
Are they the same thing as a MEL-spectogram, or is there a difference?
Table 7: Example instructions in AlpacaFarm's evaluation data.
Figure 13: Root verb distribution of the eval instructions.
Figure 14: Breakdowns of the 52k Alpaca training instructions.
G Additional Analysis on Training and Evaluation Instructions
We show in table 7 examples from the Alpaca evaluation data, showcasing their diversity. We plot in Figure 14 and Figure 15 the breakdowns of the Alpaca training instruction distribution and the AlpacaFarm evaluation instruction distribution respectively. In the inner wheel, we plot the root verb distribution of the instructions and in the outer wheel, we plot the direct subject distribution. We find that both the training distribution and the evaluation distribution cover a diverse range of instructions and the distributions match at a high level.
Figure 15: Breakdowns of 805 AlpacaFarm evaluation instructions.