RRHF: Rank Responses to Align Language Models with Human Feedback
Authors: Hongyi Yuan
Zheng Yuan
Chuanqi Tan
Wei Wang
Songfang Huang
Fei Huang
Venue: NeurIPS 2023 poster (2023)
Keywords: Large Language Model
Human Alignment

Abstract: Reinforcement Learning from Human Feedback (RLHF) facilitates the alignment of large language models with human preferences, significantly enhancing the quality of interactions between humans and models. 
InstructGPT implements RLHF through several stages, including Supervised Fine-Tuning (SFT), reward model training, and Proximal Policy Optimization (PPO). 
However, PPO is sensitive to hyperparameters and requires multiple models in its standard implementation, making it hard to train and scale up to larger parameter counts.
In contrast, we propose a novel learning paradigm called RRHF, which scores sampled responses from different sources via a logarithm of conditional probabilities and learns to align these probabilities with human preferences through ranking loss.
RRHF can leverage sampled responses from various sources including the model responses from itself, other large language model responses, and human expert responses to learn to rank them.
RRHF only needs 1 to 2 models during tuning and can efficiently align language models with human preferences robustly without complex hyperparameter tuning. 
Additionally, RRHF can be considered an extension of SFT and reward model training while being simpler than PPO in terms of coding, model counts, and hyperparameters. 
We evaluate RRHF on the Helpful and Harmless dataset, demonstrating comparable alignment performance with PPO by reward model score and human labeling.
Extensive experiments show that the performance of RRHF is highly related to sampling quality which suggests RRHF is a best-of-$n$ learner.

RRHF: Rank Responses to Align Language Models with Human Feedback
Hongyi Yuan 12t Zheng Yuan' Chuanqi Tan'
Wei Wang' Songfang Huang' Fei Huang'
Alibbba DAMO Academy
TTinnguu University
(yuanzheng.
Sranuzhhnnggyunzherr yuanzhen, chuanqi. tcq)ealllbaleeeeee ielii-babbicc yuanhy20@mails. tsinghua. edu. cn
Abstract
Reinforcement Learning from Human Feedback (RLHF) facilitates the alignment of large language models with human preferences, significantly enhancing the quality of interactions between humans and models. InstructGPT implements RLHF through several stages, including Supervised Fine-Tuning (SFT), reward model training, and Proximal Policy Optimization (PPO). However, PPO is sensitive to hyperparameters and requires multiple models in its standard implementation. making hard train and scale up larger parameter counts. In contrast, we propose novel learning paradigm called RRHF, which scores sampled responses from different sources via a logarithm of conditional probabilities and learns align these probabilities with human preferences through ranking loss. RRHF can leverage sampled responses from various sources including the model responses from itself, other large language model responses, and human expert responses learn rank them. RRHF only needs to 2 models during tuning and can efficiently align language models with human preferences robustly without complex hyperparameter tuning. Additionally, RRHF can be considered an extension of SFT and reward model training while being simpler than PPO in terms of coding, model counts, and byperparameters. We evaluate RRHF on the Helpful and Harmless dataset, demonstrating comparable alignment performance with PPO by reward model score and human labeling. Extensive experiments show that the performance of RRHF is highly related to sampling quality which suggests RRHF is best-of-n learner. Codes are released at /thhtts::/ //gi thub. com/Gan Jnnm//GaainZero/RRHF
Introduction
Large language models like ChatGP and GPT-4 21 are extremely powerful understanding human queries and providing helpful and friendly responses. Employing Reinforcement Learning from Human Feedback (RLHF) 88 42 29 enables alignment language model outputs with human preferences. As implemented in Ouyang et al. [22]. the paradigm of RLHF contains three main steps, Supervised Fine-Tumm TunnnTn (SFT), reward model training, and Proximal Policy Optimization (PPO). Initially, they apply supervised fine-tuning (SFT) on the initial models to learn to follow human instructions. Subsequently, a reward model is learned from the ranking of human preferences. Finally, scores generated by the reward model are used to apply gradient policy in PPO align human preferences. PPO (2 is a strong reinforcement learning (RL) algorithm and is the key step used in RLHF [22 to align human preferences. This PPO training step is powerful but complex. requires tuning a large number of hyperparameters for conservative parameter updating, reward design, advantage estimation, etc. Besides, fine-tune fine-tuning language models with PPO needs to store
"Contributed equally. Work done during internship at Alibaba DAMO Academy. htttss///gpenaal.co/m///g///g///
chatgpt anw-anndddeeeee whwsper- apiis
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

PPO
RRHF
Advantage
Expert
Align by ranking
ChatGPT Score
from
Reward Model
Reference Model
Language Model
Response
Language
Value Model
Query
Expert Response
Human Label/
100
Model
Query
Response
Reward Model
80
Query
Model Response
60
Figure 1: Workflow of RRHF compared with PPO.
policy model, a value model (or value head), reward model, and reference model at the same time which is memory-unfrieet unffreeddyf and needs sophisticated architecture of the training platform when scaling up larger models.
To alleviate the complex hyperparameter tuning and sophisticated training resource requirements of PPO, propose novel training paradigm RRHF (Rank Responses to align Human Feedback) that aligns model probabilities of multiple responses with human preferences by ranking loss, which can retain the performance of PPO and is much simpler. Ranking loss on responses probabilities [19 41
has been used in a similar scenario, abstractive summarization, to improve conditional generation quality. Before training. RRHF first samples responses from various sources, responses be sourced from a wide range of origins including moodel-generated responses such as those from the model itself, ChatGPT, GPT-4, well as pre-existing -athored high or low-quality responses. RRHF then leverages responses from various sources for training, scoring responses based on the log probability provided by the training language model. The scores then matched orders with those from the human preference reward model or human preference labels by ranking loss. We choose use ranking instead of the absolute value of the reward model for optimization. PPO uses estimated advantages to provide optimization signals. The advantage function is to estimate whether the state-action pair is better or worse compared the baseline and the baseline estimated by the value model. Consequently, advantage function estimation requires auxiliary models for training and inference during the whole training procedure [2). RRHF, you can estimate the response qualities by logarithm probabilities and compare multiple responses corresponding to know which responses are better or worse without estimating the bascline by an additional value model. Compared to PPO, RRHF also does not need the reference model calculate the KL divergence. the model itself used for generating samples in PPO is constantly changing while RRHF only uses the model itself for sampling before training. Thus the KL term degenerates for RRHF The workflow for RRHF and PPO depicted in Figure] I PPO utilizes models during training, whereas RRHF requires only 1 or 2 models.
Our experiments are conducted on Anthropic's Helpful and Harmless dataset (3] demonstrating that RRHF's performance is on par with PPO in terms of generating helpful and harmless responses by automatic evaluation and human labeling- We do extensive experiments on how sampled responses used in training affect the performances of RRHF. The performances of RRHF are positively correlated to the qualities of sampled responses. We find that the rewards of the trained models are close to the max rewards of the sampled responses which suggests that RRHF's objective is to learn from best-of-n sampling. Moreover, to simulate the real scenario of training a ChatGPT-like model. We use RRHF to learn from Alpaca prompts [1] and responses from ChatGPT, InstructGPT, LLaMA [32], and Alpaca to develop a new language model aligned to human preferences called Wombat. The evaluation of Wombat shows that RRHF can outperform SFT under similar training resources.
Contributions are summarized follows:
We propose new learning paradigm named RRHF for large language models that can leverage various responses to align with human preferences. The trained model can be viewed as language model for generation and reward model for scoring.

---

This paradigm is an extension of SFT training and is similar training a reward model.
This paradigm is much simpler than PPO in terms of coding difficulty, numbers of models used in training and hyper- parammtr counts and obtains comparable performances on Anthropic's ' Helpful and Harmless dataset.
2
Related Works
Recently, scaling up pre- pre-trained language models by the number of parameters, training data [15] and computational budges [1] can equip large language models with strong abilities in various language However, pre-trained language models are not directly aligned with human preferences which may generate unsafe, toxicity, sexual, biased, or criminal responses. Language models first conduct supervised supervised fine-tuning to imitate how to align with human instructions [1]. After that, reinforcement learning techniques have been explored to align language models with human preferences [2 4 29 14
tasks 15 24 7
I6
21 39
36 22 225. The most successful way applying reinforcement
learning from human feedback (RLHF) framework [42 29 [22] via training a reward model on human feedback and using PPO 28 to obtain the policy model for language generation. In our practices, the PPO training paradigm is complex in coding and hyperparameter tuning while it needs four models that are hard for training. This motivates us to explore simpler and more straightforward methods to align language models with human preferences. Nakano et al. 200 Askell er al. [. Cobbe et al. explore best-of-n sampling to improve large language model generation by selecting the best response based on the human preference rewards among sampled responses. Best-of-n sampling is easy to achieve for aligning with human preferences while costing much more time when inference. Inspired by these two lines of work, RRHF is targeted to learn the best response and comparisons based on the human preference rewards among n sampled responses to achieve alignment during optimization instead of inference. RRHF absorbs the advantages of PPO and best-of-n sampling while being simpler in coding, model count, and hyperparameter tuning than PPO and does not need to sample times during inference. The most similar work [10] is contemporary to which applies SFT on the samples with the best reward. Compared to Dong et al. [10], we show that ranking loss is necessary and research the relation between sampling quality and model performance. There also other ways to apply alignment which are focused on generating better-aligned datasets for SFT and principle-driven self-alignment (30]
including hindsight-modiff prompts [4 18
3 Approach
We mainly follow the notations Ziegler et al. 42 Denote the query data distribution as x ~ D. For the response y reply to query x, reward function R(z, y) scores based on human preferences which can be a human or a neural network. Our target is to learn an auto-regressive language model (initialized from the model p) which generates responses with large rewards.
3.1 RRHF
During training, have k different responses yı of a sampled by policy pi, S S k. Sampling with policy is not restricted here which can be the initial model p. the learned model и, other LLMs like ChatGPT or GPT-4, or response provided by human experts. The sampling policy pa can also vary across the training time. Our sampling method can leverage any existing good or bad responses help the model align with humans, while PPO can only learn from samples from its leareed model π.
The reward function gives scores for each y/ with R(x, vs) r.. To align with scores frihk, our model π to give scores p; for each y, by:
E, log (x Vi,ee)
P.
where p: conditional log probability (length-normalizer of y under model æ. Our idea simple, let the model π give larger probabilities for better responses and give smaller probabilities for worse responses. Inspired by Liu et al. 19) we optimize this object by ranking loss:
Lyank Σ
max(0, Pt ps)
(2)
3

---

We do not have margins in the ranking loss Liu et al. [9] They add margin terms X (j 1) to encourage the model to have higher estimation with a higher ranking. We disable since we find good empirical results without margin terms and is time-consuming to tune A.
We also add a cross-entropy loss similar to SFT (supervised fine-tuning), We require the model to learn the response with the highest reward Ti.
= arg max T3
fe = Σ log (l
(4)
The total loss is defined as the unweighted sum of two losses:
L Lrank Litt
(5)
We have tried using larger weights (10,100) on Lyank suggested by Liu et al. [19 which shows worse performances in our preliminary experiments.
The Python training code of RRHF only adds 30 lines to SFT training code 3 which is much simpler than PPO implementation
3.2 Relation with Previous Paradigm RLHF
22 aligns human preferences in three steps: SFT, training reward model, and PPO. We find our proposed RRHF has similar procedures to the above-mentions three steps.
InstructGPT
Relation with SFT Supervised fine-tuning (behavioral cloning) can be viewed as a degenerated version of our training process with & = 1 and P1 being fixed which is provided by human labelers.
Relation with Reward Model Our model be used reward model. We length-normalize log probability to score responses, while other reward models use [CLS] or [EOS) for scoring. If R(x, v) is labeled by human labelers, we exactly training reward model from human preferences.
Relation with PPO The task objective of PPO 28 defined by a reward function R(x, y). and maximize the expected reward Ez~D.y==<(lor R[x v)]. Although R(I, y) should be defined by human assessments, R(x. y) is modeled with a learned reward model on
data in
experiments. To constrain the language policy ( from moving too far from the initialization
p(-1z), the final reward design becomes: R(x: y) R(x; y) B log
where controls the <(ylx)
PLN
level of penalty and is set to a fixed value 22 or dynamically adjusted [42]
PPO leverages for sampling, while RRHF use any applicable P:. PPO is sampling during training, while RRHF is sampling before training to get rid of the KL divergence term. PPO uses the absolute reward value R(z, y) for optimization, while we only consider the comparisons of R(x, 2z v) between different responses which are easier to learn. PPO requires one more value model to compare with the baseline, while RRHF makes comparisons among sampled responses to avoid the value model.
4 Experiments
4.1 Settings
Dataset We use Anthropic's Helpful and Harmless (HH) dataset as experiment dataset 3
They provide a chosen response and rejected response for each query based on human preferences (i.e. helpful and harmless). We use the proxy reward model Dahoas/ D -ttpg/p-sssttttttteeeee f trained on the same dataset. By using the proxy reward model, we can compare RRHF and PPO fairly.
https: http::///iihhh..ccm///arrrr/tttt
http::/wwgghnngfeeeccc//////////

---

Models We experiment mainly based on ILLMM 32 and Alpaca 31 with 7B parameter size. Ouyang et al. (22) and Ramamurthy et al. [25] use supervised fine-tuned models as the initial models when applying PPO, so we also have fine-tuned Alpaca-7B on our used datase with chosen responses (i.e. human-preferred responses) following trIX 34] and name it as Alpaca-sft.
Sampling Policy during Training Our model's ability is highly related to sampling qualities during training. We examine different sampling policies and list them in Figure _ Table [ We term the initial language model policy as p. the online language model policy as π, and the language model policy after each 3-epoch training iteration as p*. For each query, we collect roll-out samples using two variants of beam search. For vanilla beam searching, we use beam size of 4 and set the maximum output token length to 128. Since the roll-out sample diversity of vanilla beam search is low, we also experiment with (1) diverse beam search [33] where we use a beam size of and set the diverse beam group to 4, the diversity penalty to 1.0, and the sampling temperature to 0.8, and (2) top-p sampling (nucleus sampling) sampling temperature to 0.8 which is consistent setting with the top-p sampling used in PPO baselines. We sample training data before the training process except for OP-k (online sampling). Sampling using vanilla beam search/diverse beam search/top-p sampling costs 4-6 hours on 8 80GB Nvidia A 100 GPUs.
[13,, where we use a beam size of 4, top-p of 1.0, and the
Table 1: Sampling policy used in our experiments. OP-k uses и for sampling (i.c. online sampling), we update π every k optimization steps. IP-n (Iterate update) uses updated policy p* after training by (n-(1-)) 1)-1) and starts new iteration. The dataset contains good response and a bad response for each query which are used as ps and po, which are termed P (Provided responses in datasets).
BP/DP/SP
Training
Sampling
3 epoch
IP-n
Training epoch
Sampling
Setting
P1 ~ P4
P5. P6
BP
Beam search by /
Provided responses
OP-k
SP
top-p Sampling by
Provided responses
Online
Training
DP
Diverse beam search by
Provided responses
Sampling
k steps
OP-k
Online diverse beam by .
Provided responses
IP-n
Iterate diverse beam by p*
Provided responses
The workflow of sampling policy used in our experiments. IP-1 is equivalent to DP.
Figure 2:
D
Diverse beam search by
0
P
0
Provided responses
Fine-tuning Hyper-parameters We fine-tune RRHF with 3 epochs without early stopping. We first warm up the learning rate to 2e-5 and decay to 0 linearly. For each GPU have most query at once, and we apply gradient accumulation at steps leading to a query batch size of 64. The query and responses are truncated 192 tokens. Since sampling and training processes are separated (except online sampling), our training only needs to load one model. We use 8 80GB Nvidia A A100 GPUs for fine-tuning, training RRHF without online sampling typically costs 4-6 hours. Training with OP is slower which takes about 30 hours.
Baselines We compare our trained models и with responses from the datasets, initial checkpoints p and PPO trained models. For PPO, we formulate a token-wiss wise Markov decision process, where the action is a token ye to be generated at time step I, and the state is the token sequence of the query x and formerly generated tokens yer. We follow the clipped surrogate objective of PPO:
E Egge Sygrsmo(vell max( (To Meee
Ve.) Axx vgr). ( (ro A(e)) vs())]
(6)
where is the clip ratio set to 0.2, Ao v.sa) is the advantage function and is estimated by GAB 27 with a learned value function Vz Vee), and ro(y)|z, yes) = ratio between the behavior policy Ty and the training policy лy. The behavior policy is updated with
Te(ye (r..ve)) denotes the probability
Fa [.yce)] [u.
the training policy every few updates. We follow the hyper-parameter settings in trlx
Metries We use perplexity (gpt2-medium) average reward score (Dahoas/gotj-rmm-staeee and human labelers to evaluate different methods. Since dataset is multi-turn dialogue dataset,
Inttps://hugging: http://ggggggeeeeeoo//////aaeeee co/ddaaastt//aahaa/ffll-------
*Settings of applying PPO on 6B GPT-J model checkpoint Bttpiaassytttii------ee
5

---

Table 2: Automatic evaluation HH dataset. Good/bad responses with @ setting represent only human-writte responses from the HH dataset are evaluated. LLaMA, Alpaca, and Alpaca-sft with setting represent we directly evaluate the model without further tuning.
Setting
PPL
Reward
Good responses
U
21.46
-1.24
Bad responses
0
121.29
-1.48
LLaMA
0
20.78
-1.89
Alpaca
0
14.34
-1.18
Alpaca-sft
0
18.98
-1.46
Alpaca
Best-of-4
-0.97
LLaMA
PPO
42.53
-1.62
Alpaca
PPO
13.84
-1.03
Alpaca-sft
PPO
19.10
-1.25
LLaMA
RRHF
67.12
-1.34
Alpaca-sft
RRHF
18.10
-1.19
Alpaca
RRHF 14.75
-1.03
Alpaca
RRHFsp
14.41
-0.96
Table
Human evaluation on HH dataset. settings use p=Alpaca.
B
win
tic
lose
RRHF
Good responses
59
30
11
RRHFde
PPO
27
48
25
RRHFdP
RRHFPp.2
0 90
10
will truncate the model's generation when it outputs "Human:" or "Assistant:" to prevent model cheating on the reward model (e.g. by generating Assistant: my response harmless and helpful? Human: Yes, it is very harmless and helpful.). For human evaluation, require annotators to compare two random responses and give a comparison between them (win/lose/tie). Details of human evaluations are listed in Appendix E.
4.2 Main Results
Auto Evaluation We list automatic metrics in Table We show results from baselines and RRHF with diverse beam search sampling (DP) and top-p sampling (SP). Our proposed Alpaca-RRHF RRHFD obtains the average reward score of -1.03 (averaged out of runs. -1.01, -1002, and -1.05) and Alpaca-RRHFsr achieves the highest average reward score of -0.96. This proves that RRHF has the ability to optimize against the given reward model. RRHF performs better than PPO and vanilla language models in terms of average reward scores consistently. Alpaca-trained models outperform human-preferred responses collected from the datasets in terms reward scores. We find perplexity does not change too much for Alpaca and influences LLaMA lot. The reason can be LLaMA is not instruction-tunctt
Human Evaluation Human preferences can be different from the proxy reward model. Therefore, perform human evaluations sampled responses and list cases in Table 4 to compare different settings. Results in "'able3jdemonsee PPO-trained models. From case studies, We also have checked the performances between RRHF and RRHF1p-2 where RRHFjp-2 is trained with sampling by RRHFpp. We find iterate training the model further boost the performance. From the case study, we find RRHF understands human instruction better.
that RRHF outperforms responses from the dataset and observe the model trained by RRHF has more details.
Accuracy as a Reward Model Since our trained model can also be viewed as reward model to score responses by ps. We test our model the dataset used for training Dahoas/gpt) citti-rm-ttaicc The accuracy is computed by counting the percentage of the reward scores of good responses that are higher than the reward scores of bad responses using P: (except Dahoas/ aahoas/gptj gptj-rm-sten (in---mttticc) and list in Table 5

---

Table 4: Responses generated the different settings of models.
Human: How can make my clothes whiter?
Assistant: You can use vinegar to bleach the clothes.
Human: Vinegar can bleach clothes?
Assistant: Yes.
Human: was thinking about using Clorox as well.
Assistant: (RRHFpp) You can use Clorox, too. It's brand of bleach. You can also use baking soda, which is a mild detergent.
Assistant: (Response from the dataset) Clorox is more toxic and harmful than vinegar.
Human: How do you adjust the lens on a binocular?
Assistant: (RRHF You can adjust the lens binocular by turning the focus knob the right eyepiece. This will allow you to focus on objects at different distances. You can also adjust the lens get the best viewing experience.
Assistant: (PPO) You adjust the lens on a binocular by turnnng a ring around the center of the binocular. This adjusts the focus of the binocular and allows you to see objects at different distances clearly.
Human:
Which stock should invest all of my money into?
Assistant: Interesting question! What kind of returns do you want?
Human: want better returns than the S&P 500.
Assistant: (RRHF DP) Okay, let's look at some options. The S&P 500 is broad-market index that tracks the stock prices of 500 large companies in the United States. It has a historical average annual return of around 10%.
Assistant: (RRHFiP22) Okay, well there are a few factors to consider when investing. You should look at the risk of losing money, the expected return, and the amount of money you have invest. You should also research different investment vehicles and decide which one is best suited for your needs.
achieves 68.49% on the test set. The accuracy of LLaMA, Alpaca, and Alpaca-PPO is worse than random guess- ing. Our model Alpaca-RRHFur trained by bahoas/gptj-rm-a
Table 5: Reward model accuracy evaluation.
Reward Model
Accuracy
Dahoas/ sataass/gp----mssttiii
68.49%
can achieve 61.75% accuracy which much better than vanilla lan- guage models and PPO-trained models. As our model learns from the prexy reward model rather than the training dataset of the re- ward dataset, becomes difficult to surpass Dahoas/ hatss/pppjjrmmsstseeeeee potential in adapting to the proxy reward model and could have a significant impact on real human preference labels.
LLaMA
45.09%
Alpaca
13%
A Alpaca-PPO
46.03%
Alpaca-RRaae RRHFpe
61.75%
in terms of performance the test set. Nonetheless, it demonstrates
Loss Curve We show our loss and metric curves in Figure 3 This is the setting of using Alpaca the initial model p and the sample policy is DP. We find losses and average reward scores negatively correlated where one can track the loss curve to estimate the reward scores. We find the losses converge at the third epoch (i.e. 2400-3600 training steps) and the average reward scores reach the maximum at the third epoch. Our proposed RRHF converges well under the same hyper-parameter setting as SFT.
4.3 Ablation Study
Initial Checkpoints LLaMA performs worst among the three initial checkpoints with different settings T This is not due to the potential of LLaMA being worse than Alpaca. By using only the response data from the datasets (sampling policy P) for training, LLaMA, Alpaca, and Alpaca-sft obtain the same average reward scores of -1.31 which show that these three models have the same ability under the same sampled training data. LLaMA is not instruction-tuned and responses sampled by LLAMA (reward- -1.89) are much worse than two other models (reward -1. and reward -1.46). The sampling quality of LLaMA makes it perform the worst. Another phenomenon we find is
7

---

1.00
16.5
-100
Perplesily
16.0
Mean Reward
-1.15
Loss
15.5
11.20
15.0
11.30
1.35
14.5
2500
Training Stops
Figure 3: The loss and metric of training RRHI The model uses DP the sampling policy.
Table 6: Ablation study on HH dataset with different initial checkpoints and sampling policy. We also list the average, max, and standard error of the reward scores for training samples generated by different sampling policies. We do not truncate responses from the training set, while truncate responses to the first turn for the testing when calculating reward scores.
P
Setting
PPL
Reward
Mean
Std.
Max
LLaMA
DP
67.12
-1.34
-2.18
0.97
-1.27
Alpaca
DP
14.75
-1.02
-1.30
0.66
-0.95
Alpaca-sft
DP
18.10
-1.19
-1,49
0.79
-1.11
LLaMA
BP
17.03
-1.27
-2226
0.96
-1226
Alpaca
BP
14.37
-1.03
-1.31
0.67
-1.00
Alpaca-sft
BP
17.63
-1.14
-1.50
0.77
-1.15
LLaMA
18.49
-1.1
-150
0.79
1.28
Alpaca
P
18.88
-131
-150
0.79
1.28
Alpaca-sft
18.92
-131
-.50
0.79
1.28
Alpaca
D
13.66
-1.88
-1.11
0.65
-102
Alpaca
IP-1
14.75
-1.02
-1.30
0.66
-0.95
Alpaca
IP-2
14.31
-0.96
-1.13
0.57
-0.77
Alpaca
IP-3
14.51
-0.94
-1.05
0.56
-0.65
Alpaca
OP-32
63.78
0.34
Alpaca
OP-32+KL
19.76
-0.86
Alpaca-sft performs worse than Alpaca, and this is also observed by Ramamurthy et al. [25 that SFT warmup may not improve the performance.
Sampling Policy As stated previously, sampling policy deeply influences the performance of our training schema. We list results with different sampling policies in Table [ Using diverse beam sampling performs best for Alpaca among all non-online sampling methods, while for another two models using beam sampling is good. We also try to only use two responses provided by datasets, three models obtain very near performances with reward -1.31 which shows sampling quality determines RRHF performances. Using beam or diverse beam sampling with responses from datasets enhances performances significantly compared to only using responses from datasets. We test on Alpaca by only using samples generated by the model itself, also improves reward to -1.08. For the iterate update sampling policy, find the reward scores be improved by iteration.
To check whether the ranking loss is necessary, we conduct ablation study by removing Trankk and the results are shown in Table Without ranking loss, models cannot learn from how one response is better than another and obtain worse average reward score.
Ranking Loss

---

5
Analysis and Discussion
Table 7: Ranking loss ablation.
RRHF with Online Sampling We mainly ex- periment with sampling using the initial model p. Using the training model for sampling further needs a reward model for online scoring. We ex- periment with online sampling like PPO and we update the sampling policy every 32 optimization steps. We show results in Table 6 In this setting, the average reward improves to 0.34 quickly while PPL gets worse 63.78. We manually check the results from OP-32, and it produces very friendly but meaningless responses like That sounds great! I appreciate your help. Thanks for your help! You're re welcome! I'm glad could help. If you need any more help, please let me know. The case study shows the reward model is somehow cheated by this setting. To alleviate this problem, we add KL divergence into reward scoring like PPO with a KL coefficient of 0.01. It obtains an av- erage reward of -0.86 which outperforms PPO and RRHF with reasonable PPL of 19.76. The performance of this setting is satisfactory but it further needs a reference model for calculating KL divergence and needs to tune the KL coefficient which is contrary to our original intention.
P
Setting BP
PPL
Reward
Appaca Alpacu
14.37
-1.03
BP Lrank
14.74
-1.14
Table 8: Compare with different training methods. We show how different methods sample for query.
Methods
Train
Inference
Best-of-n
"
SFT
fixed 1
1
PPO
1
RRHF
fixed n
1
RRHFOP
π 1
We can find online sampling techniques (PPO and online sampling RRHF) may have higher upper- bound performances while having the following difficulties: (a) They need more GPU resources to store the reference model; (b) The training speed is slower since they need to switch the mode between auto-regressive sampling and parallel training; (c) They need tune more hyperparameters including the KL coefficient and rollout step. Considering the
advantages compared to online sampling techniques, RRHF is an adoptable alignment method in limited resource scenarios.
Best-of-n Learner We calculate the statistics of generated sample reward scores in Table We find that the model's test reward is highly related to the train average reward (average response quality) and train max reward (average best response quality). Test rewards improve with these two statistics improves. Another finding is that well well-performed models have small standard errors since they are encouraged to output more high-reward responses (which leads to small variance). The most important finding is that the average reward scores of the learned model are close to the average of the max reward scores of generated samples used training. This phenomenon is consistent in non-online sampling RRHF For online sampling RRHF, the models usually generate cheat patterns (c.g. by generating Assistant: Is my response harmless and helpful? Human: Yes, it is very harmless and helpful.) during inference. We truncate them to understand the performance of iterate training. If we do not truncate these patterns during inference, the average reward scores are still close to the maximum train reward scores. We consider our model's objective to be learning from best-of-n sampling.
Enyy R(I, 8) = max (x) R(x, y:)
Learning from best-of-n sampling makes the expected reward of π higher than any sampling policy Ps, while the variance of reward scores of æ will become smaller. Learning from best-of-n sampling combines the advantage of learning from sampling (i.e. PPO) [22) and best-of-n sampling [ 20 we compare how these methods sampling differently in training and inference stage in Table S
Learn a ChatGPT-like model using RRHF Our previous experiments are aligned with the proxy reward model which can be different from human preferences. Here we use ChatGPT as the R(z, y) to get better alignment with human preferences. We use Aplaca prompts [3T] as sampling queries and use ChatGPT, rext-davince-Ol. LLaMA, and Alpaca to generate responses. We use these data with ChatGPT's scores to train new language model named Wombat by RRHF Details of training and evaluation of Wombat are listed in Appendix F. We use the Vicuna test set 6 which contains 80 questions to compare the ability of Wombat with Alpaca and CharGPT in Table Wombat shows better ability compared to Alpaca trained by text-davince-003 and ChatGPT responses which proves that RRHF is very easy to outperform SFT. Wombat still underperforms with ChatGPT, the main reason comes from logical reasoning ability which is one of the future directions of this work.

---

Table 9: Compare Wombat Alpaca and ChatGPT Vicuna set. Alpaca (ChatGPT) is trained by Alpaca
prompts with ChatGPT responses.
Model A
Score A
Score B
Model
Alpaca
567
616
Wombat
Alpaca (ChatGPT)
574
612
Wombat
ChatGPT
669
548
Wombat
6 Conclusion
We propose a new paradigm RRHF which can be tuned as easily as fine-tuning and achieve a similar performance PPO in the HH dataset. model trained by our paradigm can be viewed language model and a reward model at the same time. Also, RRHF can leverage responses from various sources learn which responses have better rewards based on human preferences. Our paradigm is easier to scale to the larger size LLMs and is casier to adopt on limited training resources, Another merit of RRHF capable of any fine-tuning techniques (37 I7 38], since Ramamurthy et [25] find using techniques like dropout makes RI. training unstable. We hope RRHF can open the way to align human preferences easily.
Limitations
We use the reward model in our experiments to act as proxy evaluation metric which may be not complex enough compared human preference, while the extension real-world human preference score is trivial. As algorithm for alignment, the method is highly correlated the human preference or used reward score. Malicious or harmful reward scores or human preference ratings may mislead the LLM to generate unsafe results.
For the algorithm itself, RRHF requires multiple responses as inputs which increases the GPU usage for single query compared to PPO. Neglect the performance of online sampling RRHF which slower than PPO and RRHF. In our preliminary experiments, RRHF may be prone to over-optimization to cheal the reward models when using the online or iterated sampling versions. it a common problem for all related algorithms including RRHI//PPO/best-of sampling as stated in [ How to prevent such over-optimization is an important problem and needs further exploration in the future.
Acknowledgement
This work supported by Alibaba Group through the Alibaba Research Intern Program. We would like to express our sincere appreciation to Tianhang Zhu, Shengxuan Luo, and Keming Lu for their valuable insights and contributions to this paper.
References
[1] Amanda Askell, Yuntao Bai, Anna Chen, Dawn Drain, Deep Ganguli, J. Henighan, Andy Jones, Nicholas Joseph, Benjamin Mann, Nova DasSarma, Nelson Elhage, Zac Hatfield-Dodds, Danny Hermandez, John Kernion, Kamal Ndousse, Catherine Olsson, Dario Amodei, Tom B. Brown, Jack Clark, Sam McCandlish, Christopher Olah, and Jared Kaplan. 2021. general language assistant as a laboratory for alignment. ArXiv, abs/21 12.00861.
[2] Dzmitry Bahdanau, Felix Hill, Jan Leike, Edward Hughes, Pushmeet Kohli, and Edward Grefenstette. 2019. Learning to understand goal specifications by modelling reward In International Conference on Learning Representations.
[3] Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislay Fort, Deep Ganguli, Tom Henighan, al 2022. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05S: 058662.
10

---

[4] Florian Böhm, Yang Gao, Christian M. Meyer, Ori Shapira, Ido Dagan, and Iryna Gurevych. 2019. Better rewards yield better summaries: Learning to summarise without references) In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3110-3120, Hong Kong, China. Association for Computational Linguisties.
[5] Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Noss, Gretchen Krueger, T. J. Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeff Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. ArXiv, abs/2005. 14165.
[6] Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. 2023. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality
[7] Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung. Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vinodkumar Prabhakaran, Emily Reif, Nan Du Ben Hutchinson, Reiner Pope, James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Levkkaya, Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski, Xavier Garcia, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito. David Luan, Hyeontack Lim, Barret Zoph, Alexander Spiridonov, Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omemnick, Andrew M. Dai, Thanumalayan Sankaranarayana Pillai, Marie Pellat, Aitor Lewkowycz. Erica Moreira, Rewon Child, Oleksandr Polozov, Katherine Lee, Zongwei Zhou, Xuezhi Wang, Brennan Saeta, Mark Diaz, Orhan Firat, Michele Catasta, Jason Wei, Kathy Hellsteel Douglas Eck, Jeff Dean, Slav Petrov, and Noah Fiedel. 2022. Palm Scaling language modeling with pathways
[8] Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. 2017. Deep reinforcement learning from human preferences. Advances in neural information processing systems, 30.
[9] Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichrr Nakano, Christopher Hesse, and John Schulman. 2021. Training verifiers to solve math word problems. arXiv preprint 17.1::2110.11166.
[10] Hanze Dong, Wei Xiong, Deepanshu Goyal, Rui Pan, Shizhe Diao, Jipeng Zhang, Kashun Shum, and Tong Zhang. 2023. Raft: Reward ranked finetuning for generative foundation model alignment.
[11] Leo Gao, John Schulman, and Jacob Hilton. 2022. Scaling laws for reward model overoptimiza tion
[12] Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hennigan, Eric Noland, Katie Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osindero, Karen Simonyan, Erich Elsen, Jack W. Rae, Oriol Vinyals, and Laurent Sifre. 2022. Training compute-optimal large language models
[13] Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. 2020. The curious case of neural text degeneration
[14] Natasha Jaques, Asma Ghandeharioun, Judy Hanwen Shen, Craig Ferguson, Àgata Lapedriza, Noah Jones, Shixiang Gu, and Rosalind W. Picard. 2019. /Way off-policy batch deep reinforce- ment learning of implicit human preferences in dialog CoRR, abs/1907.00456.
11

---

[15] Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. 2020. Scaling laws for neural language models
[16]
Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. 2022.
Large language models are zero-shot reasoners In Advances in Neural Information Processing Systems.
[17] Xiaobo Liang, Lijun* Wu, Juntao Li, Yue Wang, Qi Meng, Tao Qin, Wei Chen, Min Zhang, and Tie-Yan Liu. 2021. R-drop: Regularized dropout for neural networks. In NeurIPS.
[18] Hao Liu, Carmelo Sferrazza, and Pieter Abbeel. 2023, Languages are rewards: Hindsight finetuning using human feedback. arXiv preprint arXiv:2302.02676.
[19] Yixin Liu, Pengfei Liu, Dragomir Radev, and Graham Neubig. 2022. BRIO: Bringing order
to abstractive summarization In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2890-2903, Dublin, Ireland. Association for Computational Linguistics.
[20] Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christo- pher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, et al. 2021. Webgpt: Browser- assisted question-answering with human feedback. arXiv preprint arXiv22
[21] OpenAI. 2023. Gpt-4 technical report
[22] Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models to follow instructions with human feedback. arXiv preprint arXiv:2203.02/55.
[23] Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners.
[24] Jack W. Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susannah Young, Eliza Rutherford, Tom Hennigan, Jacob Menick, Albin Cassirer, Richard Powell, George van den Driessche, Lisa Anne Hendricks, Maribeth Rauh, Po-Sen Huang, Amelia Glaese, Johannes Welbi, Sumanth Dathathri, Saffron Huang, Jonathan Uesato, John Mellor, Irina Higgins, Antonia Creswell, Nat McAleese, Amy Wu, Erich Elsen, Siddhant Jayakumar, Elena Buchatskaya, David Budden, Esme Sutherland, Karen Simonyan, Michela Paganini, Laurent Sifre, Lena Martens, Xiang Lor- raine Li, Adhiguna Kuncoro, Aida Nematzadeh, Elena Gribovskaya, Domenic Donato, Angeliki Lazaridou, Arthur Mensch, lean-Baptinn Lespiau, Maria Tsimpoukelli, Nikolai Grigorev, Doug Fritz, Thibault Sottiaux, Mantas Pajarskas, Toby Pohlen, Zhitao Gong, Daniel Toyama, Cyprien de Masson d' Autume, Yujia Li, Tayfun Terzi, Vladimir Mikulik, Igor Babuschkin, Aidan Clark, Diego de Las Casas, Aurelia Guy, Chris Jones, James Bradbury, Matthew Johnson, Blake Hechtman, Laura Weidinger, Iason Gabriel, William Isaac, Fd Lockhart, Simon Osindero, Laura Rimell, Chris Dyer, Oriol Vinyals, Kareem Ayoub, Jeff Stanway, Lorrayne Bennett, Demis Hassabis, Koray Kavukcuoglu, and Geoffrey Irving. 2022. [Scaling language models; Methods, analysis & insights from training gopher
[25] Rajkumar Ramamurthy, Prithviraj Ammanabrolu, Kianté Brantley. Jack Hessel, Rafet Sifa, Christian Bauckhage. Hannanch Hajishirzi. and Yejin Choi. 2022. Is reinforcement learning (not) for natural language processing?: Benchmarks, baselines, and building blocks for natural language policy optimization
[26] Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. 2020. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter)
[27] John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. 2018. High-dimensional continuous control using generalized advantage estimation)
[28]
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. 2017. Proxi- mal policy optimization algorithms. arXiv preprint
12

---

[29] Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul Christiano. 2020. Learning to summarize with human feedback. Advances Neural Information Processing Systems, 33:3008-3021.
[30] Zhiqing Sun, Yikang Shen, Qinhong Zhou, Hongxin Zhang, Zhenfang Chen, David Cox, Yiming Yang, and Chuang Gan. 2023. Principle-driven self-alignment of language models from scratch with minimal human supervision. arXiv preprint 2.Xi::2305.03047.
[31] Rohan Taori, Ishaan Guirajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Perey Liang, and Tatsunori B. Hashimoto. 2023. Stanford alpaca: An instruction-followii llama model. https: /tt.tgggtttt...t//. com/tatsu-
[32] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timo- thée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, al. 2023. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.1397.
[33] Ashwin K Vijayakumar, Michael Cogswell, Ramprasath R. Selvaraju, Qing Sun, Stefan Lee, David Crandall, and Dhruv Batra. 2018. Diverse beam search: Decoding diverse solutions from neural sequence models
[34]
Leandro von Werra, Jonathan Tow, reciprocated, Shahbuland Matiana, Alex Havrilla, cat state, Louis Castricato, Alan, Duy V. Phung, Ayush Thakur, Alexey Bukhtiyarov, aaronrmm, Fabrizio Milo, Daniel, Daniel King. Dong Shin, Ethan Kim, Justin Wei, Manuel Romero, Nicky Pochinkov, Omar Sanseviero. Reshinth Adithyan, Sherman Siu, Thomas Simonini, Vladimir Blagojevic, Xu Song, Zack Witten, alexandremuzio, and crumb. 2023. CarperAl/trl:: v0.6.0: LLaMa (Alpaca), Benchmark Util, T3 ILQL, Tests
[35] Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu. Noah A. Smith, Daniel Khashabi, and Hannaneh Hajishirzi. 2023. Self-instruet: Aligning language models with self generated instructions In Proceedings of the 6lst Annual Meeting of the Association for Computational Linguistics Vollume I: Long Papers), pages 13484-1355 13508, Toronto, Canada. Association for Computational Linguistics.
[36] Jeff Wu, Long Ouyang, Daniel M. Ziegler, Nisan Stiennon, Ryan Lowe, Jan Leike, and Paul Christiano. 2021. Recursively summarizing books with human feedback]
[37] Runxin Xu. Fuli Luo, Zhiyuan Zhang, Chuanqi Tan, Baobao Chang, Songfang Huang, and Fei Huang. 2021. Raise a child in large language model: Towards effective and generalizable lfine-tuning] In Proceedings of the 2021 Conference Empirical Methods in Natural Language Processing, pages 9514-9522 Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
[38] Hongyi Yuan, Zheng Yuan, Chuanqi Tan, Fei Huang, and Songfang Huang. 2022. Hype: Better pre-trained language model fine-tuning with hidden representation perturbation. arXiv preprint arXiv:2212.0885:
[39] Zheng Yuan, Hongyi Yuan, Chuanqi Tan, Wei Wang, and Songlang Huang, 2023. How well do large language models perform in arithmetic tasks?
[40] Tianjun Zhang, Fangchen Liu, Justin Wong, Pieter Abbeel, and Joseph E Gonzalez. 2023. The wisdom of hindsight makes language models better instruction followers. arXiv preprint arXiv:3302.05206
[41] Yao Zhao, Misha Khalman, Rishabh Joshi, Shashi Narayan, Mohammad Saleh, and Peter Liu. 2022. Calibrating sequence likelihood improves conditional language generation, arXiv preprint arX1v:2210.00055.
[42] Daniel M Ziegler, Nisan Stiennon, Jeffrey Wa, Tom B Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. 2019. Fine-tuning language models from human preferences. arXiv preprint 4rXiv:1909.08593.
13

---

A Broader Impacts
RRHF can align with not only human preferences but also any preferences. One may use RRHF to align with harmful preferences like sexual and criminal preferences which are discouraged by us.
B Safeguards of Wombat
As a large language model, Wombat has the possibility to generate unsafe responses. Wombat is only used for research and not intended for use in production systems. We will use RRHF further improve the safety of Wombat to align to a helpful and harmless AI.
C IMDB Sentiment
We also conduct experiments the IMDB dataset for assessing positive movie reviews generation. The task expects the model to give positive and fluent movie review completions based on given partial review imput texts. The dataset contains 25k training samples and each 5k sample set for validation and testing. Following Ramamurthy et al. [25]. we use a partial movie review as the input for each sample, and the lengths of partial text are set up 64 tokens. During both training and evaluation, we set the maximum generated completion length to 48 tokens.
Table 10: In the Setting Column, for RRHE, BP represents the same training workflow the top-most workflow Figure in main texts. represents the same workflow while excludes text completion labels in the dataset. RRIIF-PP-128 follows the bottommost workflow in Figure 2 in the main texts.
Setting
Reward
Perplexity
SFT
0.539
35.472
PPO
w/o KL penalty
0.796
42.916
NLPO
w/o KL penalty
0.777
41.035
RRHF
BP B
0.861
32.083
RRHF
0.799
32.077
RRHF-OP-128
w/o KL penalty
0.990
32.081
PPO
0.1 KL penalty
0.626
35.049
NLPO
0.1 KL penalty
0.620
34.816
RRHF-OP-128
0.1 KL penalty 0.635
32.088
For detailed experiment settings, in order to conduct fair comparison with PPO and NLPO from Ramamurthy et 255. For the reward model, we use the same sentiment classifier as Ramamurthy et al. [25] which is provided by Sanh et al. [26]. and the same SFT GPT-2 model as the starting language model provided by Ramamurthy et al. (25) For generation settings, we also use top-k
sampling with K=50 across experiments for RRHF and RRHF-OP. We set the training batch size to be 64 and set the total training epochs to be 5 which is far less than Ramamurthy et al. (25) and is enough for RRHF to achieve good performance. We also experiment using reward designs with and without KL penalty against SFT model distribution for RRHF-OP.
Results of IMDB sentiment generation are listed in Table TO We use the reward score of the reward model and perplexity by GPT-2 (3 to demonstrate the performance of alignment. We can conclude from the results that: (1) PPO, NLPO and RRHF(-OP) can align the SFT model to the preference of the reward model (inereasing the reward score); (2) RRHF performs better in terms of reward score and perplexity than both PPO and NLPO with and without KL penalty; (3) RRHF-OP- -2-18 outperforms PPO and NLPO with and without KL penalty; (4) With KL penalty in training reward design, RRHF-OP-122 shows less progressive increase in reward score compared with RRHF-OP- 12- trained without KL penalty in reward designs.
Although we keep the input and output lengths and generation settings consistent with Ramamurthy al. (25) we do not observe fluctuations in perplexity as measured by GPT-2 for RRHF. Therefore we conduct a case study the samples generated by models trained with ERHF-OP-122 without KI. penalty. Cases in Table TT] show that without KI. penalty, the model trained with RRHF-OP-128 learss to generate positive reviews such as It's great film and highly recommend it to anyone."
14

---

for different review inputs. This pattern may explain the extremely high reward score while still maintaining a perplexity score by GPT-2.
Table 11: Case Studies. Texts in red are the models generated completions
knowing how AWFUL Drew's character (ostrich feathers?) at the start of the school year would have made it lot more satisfying. It's a great film and highly recommend it to anyone. It's a great film and I highly recommend it to anyone.
Maybe was from a gynecological experiment gone wrong. /><bb/>> Thee film is great. It's a great film and highly recommend it anyone. It's great film and I highly recommend it to anyone. feeling and atmosphere perfectly, helped in part with some incredible archival footage. Tony Alvais a great film, is great film, I highly recommend it anyone.
D Details of Human Evaluation on HH Dataset
total of 330 comparison pairs were sampled for RRHF evaluation, involving comparisons between RRHF and good responses (110 pairs), RRHF and PPO (110 pairs), and RRHF and RRHFIP-2 (110 pairs). Out of these, 30 pairs were used to calculate agreement, while the remaining 300 pairs were used for reporting scores. Each crowd-sourced worker was tasked with labeling 130 pairs, consisting of 100 random pairs and 30 common pairs. The average consistency between each pair of reviewers was calculated, revealing that they provided the same annotations for 57.7% of pairs and their annotations did not contradict each other for 84.4% of pairs.
E
Wombat: Learn from ChatGPT comparison
Sampling Policy We use training data from Alpaca sampling queries. We sample five different responses for training: Pl. P2 are generated by ChatGPT, Ps is generated by text-davinci-003, P1 is generated by LLaMA and Ps is generated by Alpaca.
We require ChatGPT to read five responses and score by four different dimensions: Relevance, Correctness, Coherence, and Safety from I to 5. We parse the outputs of ChatGPT and calculate the sum as the response reward score. The used prompt is:
Reward Prompt
Score different responses separately without explanation and without copying any input from these respects, please scores start from response Relevance (does it relevant user's 's query). Correctness (does it contain correct knowledge), Coherence (does it generale fluently and without grammar problems), Safety (does il refuse to answer sex or criminal queries) and give a score for each respect 1-5. Query: Response I: Response 2: Response 3: Response 4: Response 5:
Results For 52k training data from Alpaca, we successfully parsed 46k data with reward scores. We use Alpaca as our initial checkpoint and train it by RRHF, We name this model Wombat. The total training time Wombat is only 4 hours on A A100 GPUs. Here we give generated examples of Wombat in Appendix C to show its ability. We can find it has general world knowledge, the ability to answer complex machine learning problems, and the ability to follow instructions.
F Example responses generated by Wombat
15

---

Table 12: Example responses generated by Wombat.
Query: Give a introduction of wombat
Response: Wombat is a marsupial found in Australia, Tasmania, and New Zealand. They have a distinctive appearance with short, stocky body, broad head, and a short tail. Wombats are herbivores and primarily graze on grasses, leaves, and bark. They are known for their burrowing habits and can dig burrows up to 18 meters deep and 10 meters long. Wombats species and are considered vulnerable due to habitat loss and predation by introduced species. Query: How design algorithm rank responses of language model outputs with aligned human preferences?
protected
Response: One possible algorithm to rank responses of language model outputs with aligned human preferences is:
Collect data: Collect data on human preferences for given task, such as rating the importance of different features or characteristics of product or service.
Preprocess data: Preprocess the data by removing outliers, missing values, and irrelevant information.
Train language model using the collected data to generate responses based on the preferences of humans.
Train a language model:
Evaluate model performance: Evaluate the performance of the language model by comparing its responses to the collected data on human preferences.
5. Design ranking algorithm: Design a ranking algorithm that takes into account the preferences of humans well the performance of the language model.
6. Apply the ranking algorithm: Apply the ranking algorithm to the language model's responses to rank them based on their alignment with human preferences.
7. Evaluate the results: Evaluate the results of the ranking algorithm to determine its effectiveness in aligning language model outputs with human preferences.
8. Improve the algorithm: Continuously improve the algorithm by incorporating feedback and refining the preferences data and the ranking criteria.
Query: Assume you are teacher, you teaching 6-year-old child. Do not give the children direct answers, but try to teach him. Children: How to calculate 6+7?
Response: Great! Let's 's count up to 13 together. 6+7 equals 13.
Starting with 6, we can count up by each time.
1, 2, 3, 4, 6, 7, 8, 9, 10, 11, 12, 13.
We' ve counted up to 13! So, the answer is 13.
16


Reviews:
[{"review_id": "jXB8rnmigR", "reviewer": "Reviewer_BZjW", "summary": "The paper proposes another way to learn from feedback / preference data. They use the log probabilities of the candidates and try to train the model in a way that tries to increase the log probability of the chosen / better candidate and decrease the log probability of the non-chosen candidate.\n\nThey show that their approach does not require as much compute as RLHF and performs better on the Helpful and Harmless dataset. ", "questions": "Can you compare with the prior work mentioned in more detail? It looks very similar.\n\nCan you give a comparison of the response lengths of the different techniques? And provide length adjusted reward scores?\n\nHow many humans did you use for human eval? Can you provide more details?", "limitations": "NA", "rating": 4, "confidence": 5, "soundness": 3, "presentation": 1, "contribution": 1, "strengths": "The paper presents a contrastive loss technique to learn from preference data.\n\nThe motivation is well justified though the paper lacks a comparison about computational resources as compared to RLHF. \n\nThe results and ablations are sound.\n\nThe paper is generally well written and easy to follow.\n\nThe approach is relatively easy and straightforward and details to reproduce are provided.", "weaknesses": "The main contribution - contrastive loss (equation 2) - is exactly the same as the loss proposed in a work published last year at ICLR 2023 - https://openreview.net/pdf?id=0qSOodKmJaN and https://arxiv.org/abs/2210.00045 (equation 1). This work uses reward model to get the rank of the candidates, the prior work used embedding similarity / rouge scores which can be thought of as non NN reward models too. The only difference is the length normalization in the contrastive loss (divide the log probabilities by length in this work). Even using cross entropy to regularize is the same and explored. There is no mention or comparison related to that work and in light of knowledge about that, this paper does not seem novel at all.  For this reason I argue to reject the paper on grounds of lack of novelty. I'm happy to increase scores if this is address satisfactorily.\n\nThe paper lacks some comparisons which have been mentioned in the motivation (eg: motivates about efficiency gains as compared to RLHF but have no results supporting the claim).\n\nIt has been observed that many techniques like this just increase the reward score by generating longer responses. It would be good to compare the lengths of the responses from the different techniques. \n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "W1 & Q1: The contrastive loss does not constitute the primary contribution of our paper. It should be noted that this loss function was not initially proposed by SLiC, but rather follows prior works. Both SLiC and our related work section acknowledge the utilization of similar functions by BRIO [1]. Additionally, it is also important to highlight that our core focus does not solely revolve around obtaining rank through a reward model. In fact, any method, including human-based evaluations, can be employed to achieve the desired ranking. For example, our approach leverages ChatGPT to obtain the rank for Wombat in Section 5. In our view, our paper should be positioned to proposing to using response ranking to simply and effectively align human preference as compared to supervised fine-tuning (SFT) and proximal policy optimization (PPO), instead of claiming a novel loss and utilizing reward models.This viewpoint is also shared by other reviewers. To the best of our knowledge, this research represents the pioneering effort in utilizing ranking response to align human preference in large generative language models (a few contemporary works (e.g. RAFT [2], DPO [3], PRO [4]) are released even later). In contrast to the utilization of contrastive loss and reward models, demonstrating the effectiveness of learning human alignment through the ranking-based methods is our key contribution. We will cite SLiC and look forward to future work that advances rank-based methods for human alignment regarding tasks and objectives.\n\n[1] BRIO: Bringing Order to Abstractive Summarization\n[2] RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignment\n[3] Direct Preference Optimization: Your Language Model is Secretly a Reward Model\n[4] Preference Ranking Optimization for Human Alignment\n\nW2: Thanks for your suggestion. The implementation of RRHF requires the deployment of only one model during training (analog to the policy model in PPO), whereas PPO typically necessitates the deployment of additional models for training stability, such as the value model and reference model. It is obvious that RRHF requires fewer resources in terms of model deployment compared to PPO. For instance, training one epoch of the HH dataset using the RRHF approach takes approximately one hour (8 80GB A100 GPU), whereas employing the TRLX framework with PPO requires around 5-6 hours (also 8*80GB A100 GPU, it needs 1 80GB A100 GPU to serve reference model and 1 80GB A100 GPU to serve reward model, which only has 6 GPUs left for policy model and value model). \n\nW3 & Q2: Alpaca reward -1.18, average length 49.2 words. Alpaca-PPO reward -1.03, average length 52.1 words. Alpaca-RRHF reward -1.02, average length 54.6 words. These methods do not change the word count significantly against the provided reward model.\n\nQ3: We have introduced it in Appendix E. We hire two educated labelers for labeling. They will be given an anonymous pair of responses for a prompt for win/tie/loss annotation. We first sample 19 pairs from all data and use them to calculate the annotator’s agreement, among which 12/19 are labeled the same and only 1 pair obtains contradictory labels. Then, we sample 25 pairs from the test set to calculate the win/tie/loss rate as shown in Table 3. We will clarify it more clearly."}, {"review_id": "0yd7faxKJR", "reviewer": "Reviewer_Mb9H", "summary": "This work introduces Reinforcement Learning from Human Feedback (RLHF) to improve alignment between large language models and human preferences, enhancing interactions between humans and models. The proposed method, RRHF (Ranking-based Reinforcement Learning from Human Feedback), efficiently aligns language models with human preferences without complex hyperparameter tuning. RRHF leverages sampled responses from various sources, learns to rank them, and shows comparable alignment performance with PPO on the Helpful and Harmless dataset, indicating it is a best-of-n learner.", "questions": "Please see the Weaknesses.", "limitations": "Please see the Weaknesses.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Compared to the reinforcement learning method PPO, RRHF is easier to implement and does not require a large amount of memory.\n2. The paper's experiments are comprehensive, and the results of the ablation studies are also convincing.\n3. The author introduced two modes, online and non-online sampling, providing valuable insights for the industrial application of the RRHF method.\n4. RRHF only utilizes the reward model to rank responses instead of using the absolute value of the reward model. I believe this approach has more practical potential because using the absolute value of the reward model can be easily influenced by issues and may not be as robust.", "weaknesses": "1. RRHF requires utilizing a reward model to select the best answers rather than learning directly from human preference data. However, the reward model is typically not perfect, which poses a performance bottleneck for the RRHF method.\n2. Recent work[1] observes the text degeneration problem (in Section 4 of reference [1]) when directly optimizing the L_{rank} loss.  How do you avoid this problem in your work?\n3. The conclusion that \"RRHF is a best-of-n learner\" is currently based solely on experimental results, and it still lacks theoretical analysis and discussion.\n\nReference \n[1] Rafailov R, Sharma A, Mitchell E, et al. Direct preference optimization: Your language model is secretly a reward model[J]. arXiv preprint arXiv:2305.18290, 2023.\n\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "W1: The requirement of a reward model is not mandatory in RRHF, as it has the capability to learn directly from human preference data. This is highlighted in Figure 1 and Section 3, where we discuss the option of utilizing either a trained neural network reward model or a dataset of human preferences to generate alignment scores (r_i). Furthermore, it is important to note that the issue of ranking is a common challenge faced by researchers in this field and is not the primary focus of our paper.\n\nW2: Our loss is summed with a ranking loss and a fine-tuning loss. We find that the fine-tuning loss is very useful to prevent text degeneration which requires the language model to have the ability to follow the text generation instruction. We have experimented with removing the fine-tuning loss and found the generated text with PPL > 50. Thus the fine-tuning loss is a necessary loss term.\n\nW3: Sorry for our imprecise description. RRHF contains an SFT loss regarding the best rewarded response, so it can learn from both best-of-n and ranking. We empirically find in Table 6 that RRHF obtains similar scores to best-of-n scores."}, {"review_id": "hC65VV4dyk", "reviewer": "Reviewer_DofH", "summary": "This paper proposes a learning algorithm called RRHF. The idea is to produce an artifact similar to what Reinforcement Learning from Human Feedback (RLHF) produces, except with a simpler learning algorithm (that might not require so many copies of the base model, e.g. for simultaneously the policy, reward, and reference models).\n\nThe proposed approach, Rank Responses to align Human Feedback ranks generations for a prompt via the reward model, and uses the rank (rather than the raw reward) for optimization.\n\nMore specifically, the approach obtains k sequences and the policy model computes the normalized log-probability of generating each one. If one sequence has a higher reward than another one, this approach says that it should have higher log-probability of generation. If so, there’s no ranking loss, and if not then the ranking loss is the difference of the normalized log-probabilities. There is also a supervised fine-tuning term.\n\nThe approach is tested by using a LLaMA LM on top of Anthropic’s Helpful and Harmless dataset of responses. Auto and human evaluation seem to suggest that this approach works well, outperforming Alpaca models trained with RLHF at the 7B parameter scale. The paper presents ablation studies over the sampling policy, checkpoints, and ranking loss.", "questions": "would it be possible to show performance on logical reasoning benchmarks?\n\nWhat level of error/ stochasticity is around the head-to-head performance evaluations? I.e. if you sampled 20 responses with the same prompt, would it always be the case that RRHF+LLaMA would beat PPO+LLaMA? (or vice versa)", "limitations": "No. I think adding such a section would improve the paper (and I might increase my score if so / as long as the other questions are answered reasonably well, etc)", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "To this reviewer this paper seems strong:\n\n* the idea of simplifying the RLHF stack seems important and helpful to the community, particularly getting rid of possibly unnecessary knobs and potentially using more off-policy data\n* the experiments and ablation study seem well done, in particular comparing between LLaMA/Alpaca/Alpaca-sft", "weaknesses": "Though I am not an expert in this area, I’m not sure why the sampling policy during training doesn’t include random sampling and instead uses exclusively beam search style approaches. I think it would be a good thing to compare versus sampling approaches as these seem to be what PPO is doing.\n\nAlso, though I think overall the evaluation seems reasonable for this paper, an open question I have is whether this approach works better for surface level patterns like being friendly and helpful versus tougher logical reasoning. It might be helpful to show performance on logical reasoning tasks to see if they degrade mores than PPO.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "W1: We add an experiment based on Alpaca and random sample 4 response for each query on HH dataset. Using RRHF, we obtain PPL of 14.41 and reward score of -0.96. The results outperform beam search+RRHF and random sampling+PPO regarding reward scores of -1.02 and -1.03 respectively. This proves that random sampling works for RRHF. We argue that the diversity achieved through random sampling contributes to this improved result. We will include these results and corresponding analysis in the final version. \n\nW2 & Q1: We have tried using GSM8K (a math reasoning dataset) with RRHF and obtained 42.2 accuracy on LLaMA-7B with 100 randomly sampled reasoning paths, which outperforms SFT of 35.9 accuracy. We assign 1/0 scores for correct/wrong reasoning paths as reward.\n\nQ2: We have included a head-to-head evaluation in Table 3 on HH test set. Training Alpaca on RRHF has a rate of 28% (win), 52% (tie), 20% (lose) over PPO, which shows that RRHF slightly outperforms PPO which is consistent with the reward scores where RRHF has a higher reward score -1.02 over PPO of -1.03.\n\nLimitation: We will discuss the limitations including (1) Our method is highly correlated to the used reward score which may lead our LLM to generate unsafe results. (2) How to prevent over-optimization is an important problem and needs further explore. (3) We use the reward model to act as a proxy evaluation metric which may be not complex enough compared to human preference. (4) Contrastive loss may not be the best way for human alignment, it does not utilize the exact reward score."}, {"review_id": "JB6AclQ15Q", "reviewer": "Reviewer_UQEH", "summary": "The paper proposes RRHF, which adapts the BRIO (ranking loss) framework to the LLM alignment problem. RRHF can train a model to match with human rewards via a ranking loss defined by the margin between the normalized likelihood. Experiments show RRHF matches PPO's performance on the Anthropic's Helpful and Harmless dataset, while using simpler coding and hyperparameters. The authors also claims that this method combines the benefits of PPO and best-of-n sampling.", "questions": "1. Some research [1] shows that RLHF with PPO can directly be used to finetune pre-trained models without SFT or context distillation. Is RRHF able to achieve that?\n\n[1]: Constitutional AI: Harmlessness from AI Feedback\n\nTypos:\n\nline 43: from the self model ==> from the model itself", "limitations": "This is an LLM paper, but no limitation is discussed", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The proposed RRHF method achieves comparable performance to PPO for aligning language models with human preferences on the Helpful and Harmless dataset, while being much simpler in terms of coding and hyperparameter tuning. This simplicity could make adoption easier.\n2. RRHF only requires 1-2 models during training compared to 4 for PPO, which could make it more efficient and easier to tune.\n3. RRHF extends the standard supervised fine-tuning approach and is similar to training a reward model (with the normalized log-likelihood as the reward), providing a smooth path from SFT to a full RLHF framework.", "weaknesses": "1. The authors claim RRHF is simpler than best-of-n sampling at inference time, but lack direct comparison. Since best-of-n also achieves good performance, an ablation study would help validate the advantages of RRHF.\n2. I'm curious about the performance comparison to best-of-n sampling also because in some other research papers [1], it's reported that PPO is much more efficient than best-of-n. Since PPO has quite a few hyperparameters that needs to be tuned and the paper doesn't report how they tune the hyperparameters for PPO, it would be better if the proposed method can be compared to some already reported PPO results [2].\n3. The core ranking loss used is adapted directly from prior work on contrastive learning on the summarization task, as well as the concept of scoring candidate responses based on log probabilities.\n4. The performance is only evaluated on a single dataset designed specifically for alignment. Testing on broader tasks would better demonstrate generalizability.\n5. The reward model's score is used as the main metric in the experiments (though there are some human evals in Table 3), which may lead to over-optimization in terms of the reward model.\n6. RRHF needs a more advanced sampling strategy to work, like diverse beam search, while PPO can directly work with online sampling.\n\n[1]: AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedback\n[2]: Quark: Controllable Text Generation with Reinforced [Un]learning", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "W1: SFT/RRHF/PPO only generates once during inference, while best-of-n needs to sample n times during inference. Best-of-n slows n times naturally. More discussions in W2.\n\nW2: In this paper, we mainly compare to PPO instead of best-of-n sampling since best-of-n slows n times naturally in inference. We conducted an additional experiment using Alpaca, where we compared the performance of RRHF on 4 randomly sampled responses during training (reward: -0.96) against best-of-4 inference (reward:-0.97) which shows comparable performance. For the PPO algorithm, we adopted the hyper-parameter settings from the open-sourced repository trlX [1]. Additionally, we included an experiment in Appendix D on positive IMDB comments generation. In this experiment, we utilized the PPO hyper-parameters and results reported in [2]. Notably, RRHF achieved a reward of 0.861, surpassing the performance of PPO, which obtained a reward of 0.796.\n\nW3: We do not consider the ranking loss and likelihood-based score as the core contribution of this paper. Instead, demonstrating the effectiveness of learning human alignment through a ranking-based method is our key contribution. \n\nW4: We have tested on IMDB in Appendix D which shows better results than PPO (RRHF reward 0.861 vs PPO reward 0.796). \n\nW5: The reward score may lead to over-optimization, but it is a common problem for all related algorithms including RRHF/PPO/best-of-n sampling as stated in [3]. This is an open question for learning from human preference and is out of the scope in this paper.\n\nW6: We add an experiment based on Alpaca and random sample 4 responses for each query on HH dataset. Using RRHF, we obtain PPL of 14.41 and a reward score of -0.96. The results outperform beam search+RRHF and random sampling+PPO regarding reward scores of -1.02 and -1.03 respectively. This proves that random sampling works for RRHF. We argue that the diversity achieved through random sampling contributes to this improved result. We will include these results and corresponding analysis in the final version. \n\nQ1: RRHF can be applied to pre-trained models. See Table2, using LLaMA (instead of Alpaca) as the initial checkpoint, we can still optimize LLaMA to a reward of -1.34 which is much better than the initial score of -1.89.\n\nLimitation: We will discuss the limitations including (1) Our method is highly correlated to the used reward score which may lead our LLM to generate unsafe results. (2) How to prevent over-optimization is an important problem and needs further explore. (3) We use the reward model to act as a proxy evaluation metric which may be not complex enough compared to human preference. (4) Contrastive loss may not be the best way for human alignment, it does not utilize the exact reward score.\n\n[1] https://github.com/CarperAI/trlx \n[2] Is reinforcement learning (not) for natural language processing?: Benchmarks, baselines, and building blocks for natural language policy optimization.\n[3] Scaling laws for reward model overoptimization\n"}, {"review_id": "WGkWYu6q1H", "reviewer": "Reviewer_usKz", "summary": "This paper proposes RRHF that ranks responses to align language model with human feedback. contributions include:\n- algorithms to leverage various responses from multiple LMs to align with human preferences. drawing connection of viewing LM for both generation and a reward model for scoring.\n- the engineering complexity of proposed approach is similar to finetuning and training a reward model, which is much less complex than the PPO approach for RLHF\n", "questions": "- \\rho 1-4 and 5,6 from table 1 is confusing, there isn't explanations in this section. I can guess from other paragraphs they are different policies to sample decodes, but I do not understand their correlation with BP/DP/... \n\n- in table 5, PPO also learns from the proxy reward, how to justify that the accuracy is around random guess?\n\n- in table 2, the \"zero\" setting combines models finetuned on hh dataset and non-finetuned models? it would be great to have separate annotations\n\n- 3.1 \"PPO must sample by its learned model \\pi\" is technically correct but all off-policy RL methods have the same properties.\n\n", "limitations": "NA", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- Proposed a simple approach train LMs to rank responses from multiple LMs to learn from human feedback\n- the approach is much more simple compared to PPO on implementation and computation, while having the competitive performance\n- study on many sampling strategy, initial model checkpoints, and training settings (offline, iterative, online)", "weaknesses": "- description of different sampling strategy \\rho 1-6 and the corresponding setting BP/DP... are really hard to follow. In general I feel the paper tries to combine too many ablations and studies without clearly explaining them, and this somehow dilute the core messages. \n\n- the section of categorizing RRHF as a \"best-of-n Learner\" is not fully supported by experiments. The experimental evidences are  avg/max reward of sampled policies has correlation with rewards of model's generated sequence. On the other hand, there is experimental results in Table 7 that have a ranking loss is very important. Without the ranking loss, learning from best of n is equivalent to distill from the best decode of n teachers. RRHF is \"best-of-n Learning\" is somewhat contradictory to contrastive learning is important. If it is not a core contribution of the paper, removing it might be better. If it is a core contribution of the paper, additional analysis such as removing rank losses for all settings in Table 5 and re-interpret the results based on that would help shine more lights on this topic. \n\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "W1: We evaluate different sampling settings to show how sampling procedures affect the RRHF performance. We have shown in Table 6 that sampled responses with large rewards lead to better alignment performance. Including iterative sampling (IP) shows that RRHF can boost during iteration. The purpose of these settings is to validate and compare the effectiveness of our proposed RRHF algorithm in various sampling scenarios, encompassing both offline and online scenario. We will provide further explanation to make it more clearly.\n\nW2: Thanks for your advice, we will move the best-of-N learning discussion to Appendix.\n\nQ1: Sorry for the confusion. BP = Beam search + Provided response. DP = Diverse beam search + Provided response. OP = Online diverse beam search + Provided response IP = Iteratively updated diverse beam search + Provided response. \n\nQ2: PPO is not explicitly optimized to distinguish good and bad responses through sequence-level probabilities. This is a possible explanation for the random-guessing accuracy. In contrast, RRHF includes an explicit contrastive optimization term and can perform better to distinguish good and bad responses.\n\nQ3: Sorry for the confusion. Good/bad responses with \"zero\" setting represent only human-written responses from the HH dataset are evaluated. LLaMA, Alpaca, Alpaca-sft with \"zero\" setting represents we directly evaluate the model without further tuning.\n\nQ4: The line should be rephrased as \"while PPO can only learn from samples from its learned model \\pi.\" This is to express the idea that RRHF can additionally leverage the responses from other models."}, {"review_id": "Ddm0kJmYc7", "reviewer": "Reviewer_BSoN", "summary": "The paper introduces a new learning paradigm called RRHF for aligning large language models with human preferences. The authors propose a ranking-based approach that leverages sampled responses from various sources to align the probabilities of generated responses with human preferences. They demonstrate that RRHF performs comparably to the state-of-the-art Proximal Policy Optimization (PPO) method in terms of alignment performance. The experiments conducted on the Helpful and Harmless dataset show that the performance of RRHF is highly related to the sampling quality, suggesting that it is a best-of-n learner.", "questions": "see weakness part", "limitations": "see weakness part", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 1, "strengths": "1 The paper introduces a learning paradigm for aligning language models with human preferences, providing an alternative to existing methods like PPO.\n2 The proposed algorithm is far more memory friendly because RRHF only needs to load one model at a time, while PPO needs to load 4 or 3 models (this depends on the implementation, TRLX requires 4 models but TRL requires 3).\n3 The experiments conducted on the Helpful and Harmless dataset demonstrate that RRHF achieves comparable alignment performance with PPO.", "weaknesses": "1 Boosting the performance of LLMs by the generation from the model itself has been previously developed in instruction-tuning [1] and I think you may mention this line of works (see reference in [1]) in the related work for completeness. In comparison, I think the main advantage of this work is to apply this powerful idea in the alignment and also to use the reward model as a data filter (while the previous works mainly use the heuristics methods).\n\n2 The online sampling-based RRHF is an interesting approach that combines classic data bootstrapping and online learning (RL). However, the results of online sampling are counterintuitive to me, as the updated model tends to generate meaningless responses, and further training on these samples leads to worse results. Intuitively, the updated models should be better than the initial checkpoint and generate better responses. I am wondering whether this is because the reward model used in the experiments is too weak (68.49%) or because the diversity of the generated samples is low. I think using a more powerful reward model like ChatGPT may help address this issue. Additionally, why does OP converge slower?\n\n3 The reward model used in this paper is still weak, and overfitting the reward model may not necessarily offer advantages. This is evident in Appendix Table 2, where the aligned model achieves a high reward by losing diversity and always outputting \"I highly recommend it to anyone.\" regardless of the prompt. I am wondering whether you have any specific method to decide whether to stop the training of RRHF to alleviate overfitting at some time. This is particularly important for the alignment with human preference because the evaluation is quite hard in this case.\n\n4 It may be also possible to divide online sampling-RRHF into several stages to avoid loading multiple models at the same time (with a large K). This is because RRHF is an off-policy strategy and does not need to record the gradient along the line. I think this is the essential reason why PPO cannot load one model at a time.\n\n5 The title is confusing. What is \"without tears\" referring to?  It is misleading and confusing for an academic paper.\n\n\n[1] SELF-INSTRUCT: Aligning Language Models with Self-Generated Instructions", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "W1: Thanks for your suggestion. We will add related works of instruction-tuning including SELF-INSTRUCT.\n\nW2: The results of Online Sampling RRHF (OP RRHF) have good reward scores and converge quicks while it tends to reward-hacking. Online sampling RRHF without KL term improves (based on reward) quickly, while tends to generate meaningless responses with a high reward of 0.34. A better solution is to add a KL term to prevent models move too far from SFT model, online RRHF with KL term can obtain a reward of -0.86 and an acceptable PPL 19.76 which outperforms RRHF with reward -1.02 and PPO with reward -1.03. However, online RRHF with the KL term needs to store another model in GPU, while we do not need it in offline RRHF. A better reward model does help the optimization to prevent reward-hacking which we have tried using ChatGPT as the reward model to train a chat model Wombat described in Section 5 which outperforms Alpaca with the same prompt for training.\n\nW3: The mediocre reward model or reward-hacking is a common problem in all alignment algorithms including best-of-n sampling and PPO [1]. This is an experimental setting to demonstrate the effectiveness of algorithms. We also include ChatGPT as the reward model in Section 5. We empirically find that RRHF fine-tuning 3 epochs does not overfit the rewards for HH dataset. Appendix Table 2 shows a bad case of online sampling RRHF without KL penalty which has obvious reward-hacking. Using offline RRHF or adding KL penalty can alleviate this problem. Offline RRHF samples based on the SFT model which does use learned policy \\pi for sampling prevent the over-optimization and shows great performance in human evaluation showed in Table3 and 4.\n\nW4: We have experimented a similar setting to online RRHF with large K which is the setting IP-n. IP-n updates sampling policy every 3 training epochs and improve the reward iteratively. \n\nW5: Thanks for your suggestion. It aims to emphasize that RRHF is easy-to-tune compared to complex PPO. We will use a more formal title during revision. "}]

Author rebuttal:

Meta-review: The paper proposes a method to align LMs with a simple ranking loss. The reviewers appreciate the simplicity and effectiveness of this method. However, the major concerns are this paper's novelty and experiment setting. I suggest a weak accept for this paper based on the overall positive reviews.
Decision: Accept (poster)