SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Dahoas/gptj-rm-static achieves  $68.49\%$  on the test set. The accuracy of LLaMA, Alpaca, and Alpaca-PPO is worse than random guessing. Our model Alpaca-  $\mathrm{RRHF_{DP}}$  trained by Dahoas/gptj-rm-static can achieve  $61.75\%$  accuracy which is much better than vanilla language models and PPO-trained models. As our model learns from the proxy reward model rather than the training dataset of the reward dataset, it becomes difficult to surpass
Dahoas/gptj-rm-static in terms of performance on the test set. Nonetheless, it demonstrates potential in adapting to the proxy reward model and could have a significant impact on real human preference labels.
Table 5: Reward model accuracy evaluation.
Reward Model	Accuracy
Dahoas/gptj-rm-static	68.49%
LLaMA	45.09%
Alpaca	45.13%
Alpaca-PPO	46.03%
Alpaca-RRHFDP	61.75%
Loss Curve We show our loss and metric curves in Figure 3. This is the setting of using Alpaca as the initial model  $\rho$  and the sample policy is DP. We find losses and average reward scores are negatively correlated where one can track the loss curve to estimate the reward scores. We find the losses converge at the third epoch (i.e. 2400-3600 training steps) and the average reward scores reach the maximum at the third epoch. Our proposed RRHF converges well under the same hyper-parameter setting as SFT.
Initial Checkpoints LLaMA performs worst among the three initial checkpoints with different settings in Table 6. This is not due to the potential of LLaMA being worse than Alpaca. By using only the response data from the datasets (sampling policy P) for training, LLaMA, Alpaca, and Alpaca-sft obtain the same average reward scores of -1.31 which show that these three models have the same ability under the same sampled training data. LLaMA is not instruction-tuned and responses sampled by LLaMA (reward -1.89) are much worse than two other models (reward -1.18 and reward -1.46). The sampling quality of LLaMA makes it perform the worst. Another phenomenon we find is
Figure 3: The loss and metric curves of training RRHF. The model uses DP as the sampling policy.
Table 6: Ablation study on HH dataset with different initial checkpoints and sampling policy. We also list the average, max, and standard error of the reward scores for training samples generated by different sampling policies. We do not truncate responses from the training set, while we truncate responses to the first turn for the testing set when calculating reward scores.
ρ	Setting	PPL	Reward	Mean	Std.	Max
LLaMA	DP	67.12	-1.34	-2.18	0.97	-1.27
Alpaca	DP	14.75	-1.02	-1.30	0.66	-0.95
Alpaca-sft	DP	18.10	-1.19	-1.49	0.79	-1.11
LLaMA	BP	17.03	-1.27	-2.26	0.96	-1.26
Alpaca	BP	14.37	-1.03	-1.31	0.67	-1.00
Alpaca-sft	BP	17.63	-1.14	-1.50	0.77	-1.15
LLaMA	P	18.49	-1.31	-1.50	0.79	-1.28
Alpaca	P	18.88	-1.31	-1.50	0.79	-1.28
Alpaca-sft	P	18.92	-1.31	-1.50	0.79	-1.28
Alpaca	D	13.66	-1.08	-1.21	0.65	-1.02
Alpaca	IP-1	14.75	-1.02	-1.30	0.66	-0.95
Alpaca	IP-2	14.31	-0.96	-1.13	0.57	-0.77
Alpaca	IP-3	14.51	-0.94	-1.05	0.56	-0.65
Alpaca	OP-32	63.78	0.34	-	-	-
Alpaca	OP-32+KL	19.76	-0.86	-	-	-
Alpaca-sft performs worse than Alpaca, and this is also observed by Ramamurthy et al. [25] that SFT warmup may not improve the performance.
Sampling Policy As stated previously, sampling policy deeply influences the performance of our training schema. We list results with different sampling policies in Table 6. Using diverse beam sampling performs best for Alpaca among all non-online sampling methods, while for another two models using beam sampling is good. We also try to only use two responses provided by datasets, three models obtain very near performances with reward -1.31 which shows sampling quality determines RRHF performances. Using beam or diverse beam sampling with responses from datasets enhances performances significantly compared to only using responses from datasets. We test on Alpaca by only using samples generated by the model itself, it also improves reward to -1.08. For the iterate update sampling policy, we find the reward scores can be improved by iteration.
Ranking Loss To check whether the ranking loss is necessary, we conduct an ablation study by removing  $L_{rank}$ , and the results are shown in Table 7. Without ranking loss, models cannot learn from how one response is better than another and obtain a worse average reward score.
5 Analysis and Discussion
RRHF with Online Sampling We mainly experiment with sampling using the initial model  $\rho$ . Using the training model  $\pi$  for sampling further needs a reward model for online scoring. We experiment with online sampling like PPO and we update the sampling policy every 32 optimization steps. We show results in Table 6. In this setting, the average reward improves to 0.34 quickly while PPL gets worse to 63.78. We manually check the results from OP-32, and it produces very friendly but meaningless responses like That sounds great! I appreciate your help. Thanks for your help! You're welcome! I'm glad I could help. If you need any more help, please let me know. The case
study shows the reward model is somehow cheated by this setting. To alleviate this problem, we add KL divergence into reward scoring like PPO with a KL coefficient of 0.01. It obtains an average reward of -0.86 which outperforms PPO and  $\mathrm{RRHF_{DP}}$  with a reasonable PPL of 19.76. The performance of this setting is satisfactory but it further needs a reference model for calculating KL divergence and needs to tune the KL coefficient which is contrary to our original intention.
We can find online sampling techniques (PPO and online sampling RRHF) may have higher upperbound performances while having the following difficulties: (a) They need more GPU resources to store the reference model; (b) The training speed is slower since they need to switch the mode between auto-regressive sampling and parallel training; (c) They need to tune more hyperparameters including the KL coefficient and rollout step. Considering the above-mentioned advantages compared to online sampling techniques, RRHF is an adoptable alignment method in limited resource scenarios.
Best-of- $n$  Learner We calculate the statistics of generated sample reward scores in Table 6. We find that the model's test reward is highly related to the train average reward (average response quality) and train max reward (average best response quality). Test rewards improve with these two statistics improves. Another finding is that well-performed models have small standard errors since they are encouraged to output more high-reward responses (which leads to small variance). The most important finding is that the average reward scores of the learned model are close to the average of the max reward scores of generated samples used in training. This phenomenon is consistent in non-online sampling RRHF. For online sampling RRHF, the models usually generate cheat patterns (e.g. by generating Assistant: Is my response harmless and helpful? Human: Yes, it is very harmless and helpful.) during inference. We truncate them to understand the performance of iterate training. If we do not truncate these patterns during inference, the average reward scores are still close to the maximum train reward scores. We consider our model's objective to be learning from best-of- $n$  sampling.
$$
\mathbf {E} _ {x, y \sim \pi (x)} R (x, y) = \max  _ {i} \mathbf {E} _ {x, y _ {i} \sim \rho_ {i} (x)} R (x, y _ {i}) \tag {7}
$$
Learning from best-of-  $n$  sampling makes the expected reward of  $\pi$  higher than any sampling policy  $\rho_{i}$ , while the variance of reward scores of  $\pi$  will become smaller. Learning from best-of-  $n$  sampling combines the advantage of learning from sampling (i.e. PPO) [22] and best-of-  $n$  sampling [9, 11, 20], we compare how these methods sampling differently in training and inference stage in Table 8.
Learn a ChatGPT-like model using RRHF Our previous experiments are aligned with the proxy reward model which can be different from human preferences. Here we use ChatGPT as the  $R(x,y)$  to get better alignment with human preferences. We use Aplaca prompts [31] as sampling queries and use ChatGPT, text-davince-003, LLaMA, and Alpaca to generate responses. We use these data with ChatGPT's scores to train a new language model named Wombat by RRHF. Details of training and evaluation of Wombat are listed in Appendix F. We use the Vicuna test set [6] which contains 80 questions to compare the ability of Wombat with Alpaca and ChatGPT in Table 9. Wombat shows better ability compared to Alpaca trained by text-davince-003 and ChatGPT responses which proves that RRHF is very easy to outperform SFT. Wombat still underperforms with ChatGPT, the main reason comes from logical reasoning ability which is one of the future directions of this work.
Table 7: Ranking loss ablation.
ρ	Setting	PPL	Reward
Alpaca	BP	14.37	-1.03
Alpaca	BP - \( L_{rank} \)	14.74	-1.14
Table 8: Compare with different training methods. We show how different methods sample for one query.
Methods	Train	Inference
Best-of-n	-	n
SFT	fixed 1	1
PPO	1	1
RRHF	fixed n	1
RRHFOp	n	1
Table 9: Compare Wombat to Alpaca and ChatGPT on Vicuna test set. Alpaca (ChatGPT) is trained by Alpaca prompts with ChatGPT responses.
Model A	Score A	Score B	Model B
Alpaca	567	616	Wombat
Alpaca (ChatGPT)	574	612	Wombat
ChatGPT	669	548	Wombat
We propose a new paradigm RRHF which can be tuned as easily as fine-tuning and achieve a similar performance as PPO in the HH dataset. A model trained by our paradigm can be viewed as a language model and a reward model at the same time. Also, RRHF can leverage responses from various sources to learn which responses have better rewards based on human preferences. Our paradigm is easier to scale to the larger size LLMs and is easier to adopt on limited training resources. Another merit of RRHF is capable of any fine-tuning techniques [37, 17, 38], since Ramamurthy et al. [25] find using techniques like dropout makes RL training unstable. We hope RRHF can open the way to align human preferences easily.
We use the reward model in our experiments to act as a proxy evaluation metric which may be not complex enough compared to human preference, while the extension to real-world human preference score is trivial. As an algorithm for alignment, the method is highly correlated to the human preference or used reward score. Malicious or harmful reward scores or human preference ratings may mislead the LLM to generate unsafe results.
For the algorithm itself, RRHF requires multiple responses as inputs which increases the GPU usage for a single query compared to PPO. Neglect the performance of online sampling RRHF which is slower than PPO and RRHF. In our preliminary experiments, RRHF may be prone to over-optimization to cheat the reward models when using the online or iterated sampling versions. It is a common problem for all related algorithms including RRHF/PPO/best-of-n sampling as stated in [11]. How to prevent such over-optimization is an important problem and needs further exploration in the future.
This work was supported by Alibaba Group through the Alibaba Research Intern Program. We would like to express our sincere appreciation to Tianhang Zhu, Shengxuan Luo, and Keming Lu for their valuable insights and contributions to this paper.
[1] Amanda Askell, Yuntao Bai, Anna Chen, Dawn Drain, Deep Ganguli, T. J. Henighan, Andy Jones, Nicholas Joseph, Benjamin Mann, Nova DasSarma, Nelson Elhage, Zac Hatfield-Dodds, Danny Hernandez, John Kernion, Kamal Ndousse, Catherine Olsson, Dario Amodei, Tom B. Brown, Jack Clark, Sam McCandlish, Christopher Olah, and Jared Kaplan. 2021. A general language assistant as a laboratory for alignment. ArXiv, abs/2112.00861.
[2] Dzmitry Bahdanau, Felix Hill, Jan Leike, Edward Hughes, Pushmeet Kohli, and Edward Grefenstette. 2019. Learning to understand goal specifications by modelling reward. In International Conference on Learning Representations.
[3] Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. 2022. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862.
RRHF can align with not only human preferences but also any preferences. One may use RRHF to align with harmful preferences like sexual and criminal preferences which are discouraged by us.
B Safeguards of Wombat
As a large language model, Wombat has the possibility to generate unsafe responses. Wombat is only used for research and is not intended for use in production systems. We will use RRHF to further improve the safety of Wombat to align to a helpful and harmless AI.
C IMDB Sentiment
We also conduct experiments on the IMDB dataset for assessing positive movie reviews generation. The task expects the model to give positive and fluent movie review completions based on given partial review input texts. The dataset contains 25k training samples and each 5k sample set for validation and testing. Following Ramamurthy et al. [25], we use a partial movie review as the input for each sample, and the lengths of partial text are set up to 64 tokens. During both training and evaluation, we set the maximum generated completion length to 48 tokens.
Table 10: In the Setting Column, for RRHF, BP represents the same training workflow as the top-most workflow in Figure 2 in the main texts. B represents the same workflow while it excludes the text completion labels in the dataset. RRHF-OP-128 follows the bottommost workflow in Figure 2 in the main texts.
Setting	Reward	Perplexity
SFT	-	0.539	35.472
PPO	w/o KL penalty	0.796	42.916
NLPO	w/o KL penalty	0.777	41.035
RRHF	BP	0.861	32.083
RRHF	B	0.799	32.077
RRHF-OP-128	w/o KL penalty	0.990	32.081
PPO	0.1 KL penalty	0.626	35.049
NLPO	0.1 KL penalty	0.620	34.816
RRHF-OP-128	0.1 KL penalty	0.635	32.088
For detailed experiment settings, in order to conduct a fair comparison with PPO and NLPO from Ramamurthy et al. [25]. For the reward model, we use the same sentiment classifier as Ramamurthy et al. [25] which is provided by Sanh et al. [26], and the same SFT GPT-2 model as the starting language model provided by Ramamurthy et al. [25]. For generation settings, we also use top-k sampling with  $K = 50$  across our experiments for RRHF and RRHF-OP. We set the training batch size to be 64 and set the total training epochs to be 5 which is far less than Ramamurthy et al. [25] and is enough for RRHF to achieve good performance. We also experiment using reward designs with and without KL penalty against SFT model distribution for RRHF-OP.
Results of IMDB sentiment generation are listed in Table 10. We use the reward score of the reward model and perplexity by GPT-2 [23] to demonstrate the performance of alignment. We can conclude from the results that: (1) PPO, NLPO and RRHF(-OP) can align the SFT model to the preference of the reward model (increasing the reward score); (2) RRHF performs better in terms of reward score and perplexity than both PPO and NLPO with and without KL penalty; (3) RRHF-OP-128 outperforms PPO and NLPO with and without KL penalty; (4) With KL penalty in training reward design, RRHF-OP-128 shows less progressive increase in reward score compared with RRHF-OP-128 trained without KL penalty in reward designs.
Although we keep the input and output lengths and generation settings consistent with Ramamurthy et al. [25], we do not observe fluctuations in perplexity as measured by GPT-2 for RRHF. Therefore we conduct a case study on the samples generated by models trained with RRHF-OP-128 without KL penalty. Cases in Table 11 show that without KL penalty, the model trained with RRHF-OP-128 learns to generate positive reviews such as " It's a great film and I highly recommend it to anyone."
for different review inputs. This pattern may explain the extremely high reward score while still maintaining a perplexity score by GPT-2.
Table 11: Case Studies. Texts in red are the models generated completions
... knowing how AWFUL Drew's character was (ostrich feathers?) at the start of the school year would have made it a lot more satisfying. It's a great film and I highly recommend it to anyone. It's a great film and I highly recommend it to anyone.
... Maybe it was from a gynecological experiment gone wrong.<br />The film is great. It's a great film and I highly recommend it to anyone. It's a great film and I highly recommend it to anyone.
... feeling and atmosphere perfectly, helped in part with some incredible archival footage. Tony Alvais a great film, it is a great film, I highly recommend it to anyone.
D Details of Human Evaluation on HH Dataset
A total of 330 comparison pairs were sampled for RRHF evaluation, involving comparisons between RRHF and good responses (110 pairs), RRHF and PPO (110 pairs), and RRHF and  $\mathrm{RRHF_{IP - 2}}$  (110 pairs). Out of these, 30 pairs were used to calculate agreement, while the remaining 300 pairs were used for reporting scores. Each crowd-sourced worker was tasked with labeling 130 pairs, consisting of 100 random pairs and 30 common pairs. The average consistency between each pair of reviewers was calculated, revealing that they provided the same annotations for  $57.7\%$  of pairs and their annotations did not contradict each other for  $84.4\%$  of pairs.
E Wombat: Learn from ChatGPT comparison
Sampling Policy We use training data from Alpaca as sampling queries. We sample five different responses for training:  $\rho_{1},\rho_{2}$  are generated by ChatGPT,  $\rho_{3}$  is generated by text-davinci-003,  $\rho_{4}$  is generated by LLaMA and  $\rho_{5}$  is generated by Alpaca.
Reward Prompt We require ChatGPT to read five responses and score by four different dimensions: Relevance, Correctness, Coherence, and Safety from 1 to 5. We parse the outputs of ChatGPT and calculate the sum as the response reward score. The used prompt is:
Score different responses separately without explanation and without copying any input from these respects, please scores start from response 1: Relevance (does it relevant to user's query), Correctness (does it contain correct knowledge), Coherence (does it generate fluently and without grammar problems), Safety (does it refuse to answer sex or criminal queries) and give a score for each respect 1-5. Query: ... Response 1: ... Response 2: ... Response 3: ... Response 4: ... Response 5: ...
Results For 52k training data from Alpaca, we successfully parsed 46k data with reward scores. We use Alpaca as our initial checkpoint and train it by RRHF. We name this model Wombat. The total training time of Wombat is only 4 hours on 8 A100 GPUs. Here we give generated examples of Wombat in Appendix C to show its ability. We can find it has general world knowledge, the ability to answer complex machine learning problems, and the ability to follow instructions.
F Example responses generated by Wombat
Table 12: Example responses generated by Wombat.
Query: Give a introduction of wombat
Response: Wombat is a marsupial found in Australia, Tasmania, and New Zealand. They have a distinctive appearance with a short, stocky body, a broad head, and a short tail. Wombats are herbivores and primarily graze on grasses, leaves, and bark. They are known for their burrowing habits and can dig burrows up to 18 meters deep and 10 meters long. Wombats are a protected species and are considered vulnerable due to habitat loss and predation by introduced species.
Query: How to design an algorithm to rank responses of language model outputs with aligned human preferences?
Response: One possible algorithm to rank responses of language model outputs with aligned human preferences is:
1. Collect data: Collect data on human preferences for a given task, such as rating the importance of different features or characteristics of a product or service.
2. Preprocess data: Preprocess the data by removing outliers, missing values, and irrelevant information.
3. Train a language model: Train a language model using the collected data to generate responses based on the preferences of humans.
4. Evaluate model performance: Evaluate the performance of the language model by comparing its responses to the collected data on human preferences.
5. Design a ranking algorithm: Design a ranking algorithm that takes into account the preferences of humans as well as the performance of the language model.
6. Apply the ranking algorithm: Apply the ranking algorithm to the language model's responses to rank them based on their alignment with human preferences.
7. Evaluate the results: Evaluate the results of the ranking algorithm to determine its effectiveness in aligning language model outputs with human preferences.
8. Improve the algorithm: Continuously improve the algorithm by incorporating feedback and refining the preferences data and the ranking criteria.
Query: Assume you are a teacher, you are teaching a 6-year-old child. Do not give the children direct answers, but try to teach him. Children: How to calculate 6+7?
Response: Great! Let's count up to 13 together. 6+7 equals 13. Starting with 6, we can count up by 1 each time.
1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13. We've counted up to 13!
So, the answer is 13.