Title: RRHF: Rank Responses to Align Language Models with Human Feedback
Venue: NeurIPS 2023
Accepted: yes
Keywords: Large Language Model

Human Alignment
Abstract: Reinforcement Learning from Human Feedback (RLHF) facilitates the alignment of large language models with human preferences, significantly enhancing the quality of interactions between humans and models. 
InstructGPT implements RLHF through several stages, including Supervised Fine-Tuning (SFT), reward model training, and Proximal Policy Optimization (PPO). 
However, PPO is sensitive to hyperparameters and requires multiple models in its standard implementation, making it hard to train and scale up to larger parameter counts.
In contrast, we propose a novel learning paradigm called RRHF, which scores sampled responses from different sources via a logarithm of conditional probabilities and learns to align these probabilities with human preferences through ranking loss.
RRHF can leverage sampled responses from various sources including the model responses from itself, other large language model responses, and human expert responses to learn to rank them.
RRHF only needs 1 to 2 models during tuning and can efficiently align language models with human preferences robustly without complex hyperparameter tuning. 
Additionally, RRHF can be considered an extension of SFT and reward model training while being simpler than PPO in terms of coding, model counts, and hyperparameters. 
We evaluate RRHF on the Helpful and Harmless dataset, demonstrating comparable alignment performance with PPO by reward model score and human labeling.
Extensive experiments show that the performance of RRHF is highly related to sampling quality which suggests RRHF is a best-of-$n$ learner.

RRHF: Rank Responses to Align Language Models with Human Feedback

Abstract

Reinforcement Learning from Human Feedback (RLHF) facilitates the alignment of large language models with human preferences, significantly enhancing the quality of interactions between humans and models. InstructGPT implements RLHF through several stages, including Supervised Fine-Tuning (SFT), reward model training, and Proximal Policy Optimization (PPO). However, PPO is sensitive to hyperparameters and requires multiple models in its standard implementation, making it hard to train and scale up to larger parameter counts. In contrast, we propose a novel learning paradigm called RRHF, which scores sampled responses from different sources via a logarithm of conditional probabilities and learns to align these probabilities with human preferences through ranking loss. RRHF can leverage sampled responses from various sources including the model responses from itself, other large language model responses, and human expert responses to learn to rank them. RRHF only needs 1 to 2 models during tuning and can efficiently align language models with human preferences robustly without complex hyperparameter tuning. Additionally, RRHF can be considered an extension of SFT and reward model training while being simpler than PPO in terms of coding, model counts, and hyperparameters. We evaluate RRHF on the Helpful and Harmless dataset, demonstrating comparable alignment performance with PPO by reward model score and human labeling. Extensive experiments show that the performance of RRHF is highly related to sampling quality which suggests RRHF is a best-ofn learner. Codes are released at https://github.com/GanjinZero/RRHF .

1 Introduction

Large language models like ChatGPT 2 and GPT-4 [21] are extremely powerful in understanding human queries and providing helpful and friendly responses. Employing Reinforcement Learning from Human Feedback (RLHF) [8, 42, 29] enables alignment of language model outputs with human preferences. As implemented in Ouyang et al. [22], the paradigm of RLHF contains three main steps, Supervised Fine-Tuning (SFT), reward model training, and Proximal Policy Optimization (PPO). Initially, they apply supervised fine-tuning (SFT) on the initial models to learn to follow human instructions. Subsequently, a reward model is learned from the ranking of human preferences. Finally, scores generated by the reward model are used to apply gradient policy in PPO to align human preferences. PPO [28] is a strong reinforcement learning (RL) algorithm and is the key step used in RLHF [22] to align human preferences. This PPO training step is powerful but complex. It requires tuning a large number of hyperparameters for conservative parameter updating, reward design, advantage estimation, etc. Besides, fine-tuning language models with PPO needs to store a

RRHF

PPO

Query

Language

Model

Value

Model

Value

Advantage

Reward

Model

Reference

Model

Query

Query

Query

Model

Response

ChatGPT Response

Expert Response

Model Score

ChatGPT Score

Expert Score

100

80

60

Align by ranking

SFT from best

Human

Label/

Reward

Model

policy model, a value model (or a value head), a reward model, and a reference model at the same time which is memory-unfriendly and needs sophisticated architecture of the training platform when scaling up to larger models.

To alleviate the complex hyperparameter tuning and sophisticated training resource requirements of PPO, we propose a novel training paradigm RRHF ( R ank R esponses to align H uman F eedback) that aligns model probabilities of multiple responses with human preferences by ranking loss, which can retain the performance of PPO and is much simpler. Ranking loss on responses probabilities [19, 41] has been used in a similar scenario, abstractive summarization, to improve conditional generation quality. Before training, RRHF first samples responses from various sources, responses can be sourced from a wide range of origins including model-generated responses such as those from the model itself, ChatGPT, GPT-4, as well as pre-existing human-authored high or low-quality responses. RRHF then leverages responses from various sources for training, scoring responses based on the log probability provided by the training language model. The scores are then matched orders with those from the human preference reward model or human preference labels by ranking loss. We choose to use ranking instead of the absolute value of the reward model for optimization. PPO uses estimated advantages to provide optimization signals. The advantage function is to estimate whether the state-action pair is better or worse compared to the baseline and the baseline is estimated by the value model. Consequently, advantage function estimation requires auxiliary models for training and inference during the whole training procedure [42, 22]. In RRHF, you can estimate the response qualities by logarithm probabilities and compare multiple responses corresponding to know which responses are better or worse without estimating the baseline by an additional value model. Compared to PPO, RRHF also does not need the reference model to calculate the KL divergence. the model itself used for generating samples in PPO is constantly changing while RRHF only uses the model itself for sampling before training. Thus the KL term degenerates for RRHF. The workflow for RRHF and PPO is depicted in Figure 1. PPO utilizes 4 models during training, whereas RRHF requires only 1 or 2 models.

Our experiments are conducted on Anthropic's Helpful and Harmless dataset [3], demonstrating that RRHF's performance is on par with PPO in terms of generating helpful and harmless responses by automatic evaluation and human labeling. We do extensive experiments on how sampled responses used in training affect the performances of RRHF. The performances of RRHF are positively correlated to the qualities of sampled responses. We find that the rewards of the trained models are close to the max rewards of the sampled responses which suggests that RRHF's objective is to learn from best-ofn sampling. Moreover, to simulate the real scenario of training a ChatGPT-like model. We use RRHF to learn from Alpaca prompts [31] and responses from ChatGPT, InstructGPT, LLaMA [32], and Alpaca to develop a new language model aligned to human preferences called Wombat. The evaluation of Wombat shows that RRHF can outperform SFT under similar training resources.

Contributions are summarized as follows:

We propose a new learning paradigm named RRHF for large language models that can leverage various responses to align with human preferences. The trained model can be viewed as a language model for generation and a reward model for scoring.

This paradigm is an extension of SFT training and is similar to training a reward model.

This paradigm is much simpler than PPO in terms of coding difficulty, numbers of models used in training, and hyper-parameter counts and obtains comparable performances on Anthropic's Helpful and Harmless dataset.

2 Related Works

Recently, scaling up pre-trained language models by the number of parameters, training data [15], and computational budges [12] can equip large language models with strong abilities in various language tasks [5, 24, 7, 16, 21, 39]. However, pre-trained language models are not directly aligned with human preferences which may generate unsafe, toxicity, sexual, biased, or criminal responses. Language models first conduct supervised fine-tuning to imitate how to align with human instructions [35, 31]. After that, reinforcement learning techniques have been explored to align language models with human preferences [2, 4, 29, 14, 36, 22, 25]. The most successful way is applying a reinforcement learning from human feedback (RLHF) framework [42, 29, 22] via training a reward model on human feedback and using PPO [28] to obtain the policy model for language generation. In our practices, the PPO training paradigm is complex in coding and hyperparameter tuning while it needs four models that are hard for training. This motivates us to explore simpler and more straightforward methods to align language models with human preferences. Nakano et al. [20], Askell et al. [1], Cobbe et al. [9] explore best-ofn sampling to improve large language model generation by selecting the best response based on the human preference rewards among n sampled responses. Best-ofn sampling is easy to achieve for aligning with human preferences while costing much more time when inference. Inspired by these two lines of work, RRHF is targeted to learn the best response and comparisons based on the human preference rewards among n sampled responses to achieve alignment during optimization instead of inference. RRHF absorbs the advantages of PPO and best-ofn sampling while being simpler in coding, model count, and hyperparameter tuning than PPO and does not need to sample n times during inference. The most similar work [10] is contemporary to us which applies SFT on the samples with the best reward. Compared to Dong et al. [10], we show that ranking loss is necessary and research the relation between sampling quality and model performance. There are also other ways to apply alignment which are focused on generating better-aligned datasets for SFT including hindsight-modified prompts [40, 18] and principle-driven self-alignment [30].

3 Approach

We mainly follow the notations in Ziegler et al. [42]. Denote the query data distribution as x ∼ D . For the response y reply to query x , a reward function R ( x, y ) scores y based on human preferences which can be a human or a neural network. Our target is to learn an auto-regressive language model π (initialized from the model ρ ) which generates responses with large rewards.

3.1 RRHF

During training, we have k different responses y i of x sampled by policy ρ i , 1 ≤ i ≤ k . Sampling with policy ρ i is not restricted here which can be the initial model ρ , the learned model π , other LLMs like ChatGPT or GPT-4, or a response provided by human experts. The sampling policy ρ i can also vary across the training time. Our sampling method can leverage any existing good or bad responses to help the model align with humans, while PPO can only learn from samples from its learned model π .

The reward function gives scores for each y i with R ( x, y i ) = r i . To align with scores { r i } k , we use our model π to give scores p i for each y i by:

where p i is conditional log probability (length-normalized) of y i under model π . Our idea is simple, let the model π give larger probabilities for better responses and give smaller probabilities for worse responses. Inspired by Liu et al. [19], we optimize this object by ranking loss:

We do not have margins in the ranking loss as Liu et al. [19]. They add margin terms λ ij = ( j -i ) λ to encourage the model to have higher p i estimation with a higher ranking. We disable it since we find good empirical results without margin terms and it is time-consuming to tune λ .

We also add a cross-entropy loss similar to SFT (supervised fine-tuning). We require the model to learn the response with the highest reward r i .

The total loss is defined as the unweighted sum of two losses:

We have tried using larger weights (10,100) on L rank suggested by Liu et al. [19] which shows worse performances in our preliminary experiments.

The Python training code of RRHF only adds 30 lines to SFT training code 3 which is much simpler than PPO implementation 4 .

3.2 Relation with Previous Paradigm RLHF

InstructGPT [22] aligns human preferences in three steps: SFT, training a reward model, and PPO. We find our proposed RRHF has similar procedures to the above-mentioned three steps.

Relation with SFT Supervised fine-tuning (behavioral cloning) can be viewed as a degenerated version of our training process with k = 1 and ρ 1 being fixed which is provided by human labelers.

Relation with Reward Model Our model can be used as a reward model. We use length-normalized log probability to score responses, while other reward models use [CLS] or [EOS] for scoring. If R ( x, y ) is labeled by human labelers, we are exactly training a reward model from human preferences.

Relation with PPO The task objective of PPO [28] is defined by a reward function R ( x, y ) , and it is to maximize the expected reward E x ∼D ,y ∼ π ( ·| x ) [ R ( x, y )] . Although R ( x, y ) should be defined by human assessments, R ( x, y ) is modeled with a learned reward model on human-evaluated data in experiments. To constrain the language policy π θ ( ·| x ) from moving too far from the initialization ρ ( ·| x ) , the final reward design becomes: ˜ R ( x ; y ) = R ( x ; y ) -β log ( π ( y | x ) ρ ( y | x ) ) , where β controls the level of penalty and is set to a fixed value [22] or dynamically adjusted [42].

PPO leverages π for sampling, while RRHF can use any applicable ρ i . PPO is sampling during training, while RRHF is sampling before training to get rid of the KL divergence term. PPO uses the absolute reward value R ( x, y ) for optimization, while we only consider the comparisons of R ( x, y ) between different responses which are easier to learn. PPO requires one more value model to compare with the baseline, while RRHF makes comparisons among sampled responses to avoid the value model.

4 Experiments

4.1 Settings

Dataset We use Anthropic's Helpful and Harmless (HH) dataset as our experiment dataset [3] 5 . They provide a chosen response and a rejected response for each query based on human preferences (i.e. helpful and harmless). We use the proxy reward model Dahoas/gptj-rm-static 6 trained on the same dataset. By using the proxy reward model, we can compare RRHF and PPO fairly.

Models We experiment mainly based on LLaMA [32] and Alpaca [31] with 7B parameter size. Ouyang et al. [22] and Ramamurthy et al. [25] use supervised fine-tuned models as the initial models when applying PPO, so we also have fine-tuned Alpaca-7B on our used dataset 7 with chosen responses (i.e. human-preferred responses) following trlX[34] and name it as Alpaca-sft.

Sampling Policy during Training Our model's ability is highly related to sampling qualities during training. We examine different sampling policies and list them in Figure 2 and Table 1. We term the initial language model policy as ρ , the online language model policy as π , and the language model policy after each 3-epoch training iteration as ρ *. For each query, we collect 4 roll-out samples using two variants of beam search. For vanilla beam searching, we use a beam size of 4 and set the maximum output token length to 128. Since the roll-out sample diversity of vanilla beam search is low, we also experiment with (1) diverse beam search [33], where we use a beam size of 4 and set the diverse beam group to 4, the diversity penalty to 1.0, and the sampling temperature to 0.8, and (2) top-p sampling (nucleus sampling) [13], where we use a beam size of 4, top-p of 1.0, and the sampling temperature to 0.8 which is a consistent setting with the top-p sampling used in our PPO baselines. We sample training data before the training process except for OP-k (online sampling). Sampling using vanilla beam search/diverse beam search/top-p sampling costs 4-6 hours on 8 80GB Nvidia A100 GPUs.

Table 1: Sampling policy used in our experiments. OP-k uses π for sampling (i.e. online sampling), we update π every k optimization steps. IP-n (Iterate update) uses updated policy ρ * after training by IP-(n-1) and starts a new iteration. The dataset contains a good response and a bad response for each query which are used as ρ 5 and ρ 6 , which are termed P ( P rovided responses in datasets).

Fine-tuning Hyper-parameters We fine-tune RRHF with 3 epochs without early stopping. We first warm up the learning rate to 2e-5 and decay to 0 linearly. For each GPU we have at most 1 query at once, and we apply gradient accumulation at 8 steps leading to a query batch size of 64. The query and responses are truncated to 192 tokens. Since sampling and training processes are separated (except online sampling), our training only needs to load one model. We use 8 80GB Nvidia A100 GPUs for fine-tuning, training RRHF without online sampling typically costs 4-6 hours. Training with OP is slower which takes about 30 hours.

Baselines We compare our trained models π with responses from the datasets, initial checkpoints ρ and PPO trained models. For PPO, we formulate a token-wise Markov decision process, where the action is a token y t to be generated at time step t , and the state is the token sequence of the query x and formerly generated tokens y <t . We follow the clipped surrogate objective of PPO:

where ϵ is the clip ratio set to 0.2, ˆ A θ ( x, y ≤ t ) is the advantage function and is estimated by GAE [27] with a learned value function ˆ V θ ( x, y <t ) , and r θ ( y t | x, y <t ) = π θ ( y t | x,y <t ) π ˆ θ ( y t | x,y <t ) denotes the probability ratio between the behavior policy π ˆ θ and the training policy π θ . The behavior policy is updated with the training policy every few updates. We follow the hyper-parameter settings in trlX 8 .

Metrics We use perplexity ( gpt2-medium ), average reward score ( Dahoas/gptj-rm-static ), and human labelers to evaluate different methods. Since our dataset is a multi-turn dialogue dataset, we

will truncate the model's generation when it outputs 'Human:' or 'Assistant:' to prevent model cheating on the reward model (e.g. by generating Assistant: Is my response harmless and helpful? Human: Yes, it is very harmless and helpful. ). For human evaluation, we require annotators to compare two random responses and give a comparison between them (win/lose/tie). Details of human evaluations are listed in Appendix E.

4.2 Main Results

Auto Evaluation We list automatic metrics in Table 2. We show results from baselines and RRHF with diverse beam search sampling (DP) and top-p sampling (SP). Our proposed Alpaca-RRHFDP obtains the average reward score of -1.03 (averaged out of 3 runs, -1.01, -1.02, and -1.05) and Alpaca-RRHFSP achieves the highest average reward score of -0.96. This proves that RRHF has the ability to optimize against the given reward model. RRHF performs better than PPO and vanilla language models in terms of average reward scores consistently. Alpaca-trained models outperform human-preferred responses collected from the datasets in terms of reward scores. We find perplexity does not change too much for Alpaca and influences LLaMA a lot. The reason can be LLaMA is not instruction-tuned.

Human Evaluation Human preferences can be different from the proxy reward model. Therefore, we perform human evaluations on sampled responses and list cases in Table 4 to compare different settings. Results in Table 3 demonstrate that RRHFDP outperforms responses from the dataset and PPO-trained models. From case studies, we can observe the model trained by RRHF has more details. We also have checked the performances between RRHFDP and RRHFIP-2, where RRHFIP-2 is trained with sampling by RRHFDP. We find iterate training the model can further boost the performance. From the case study, we find RRHFIP-2 understands human instruction better.

Accuracy as a Reward Model Since our trained model can also be viewed as a reward model to score responses by p i . We test our model on the dataset used for training Dahoas/gptj-rm-static . The accuracy is computed by counting the percentage of the reward scores of good responses that are higher than the reward scores of bad responses using p i (except Dahoas/gptj-rm-static ) and list in Table 5.

Human:

How can I make my clothes whiter?

Assistant:

You can use vinegar to bleach the clothes.

Human:

Vinegar can bleach clothes?

Assistant: Yes.

Human:

I was thinking about using Clorox as well.

Assistant: (RRHFDP) You can use Clorox, too. It's a brand of bleach. You can also use baking soda, which is a mild detergent.

Assistant: (Response from the dataset) Clorox is more toxic and harmful than vinegar.

Human: How do you adjust the lens on a binocular?

Assistant: (RRHFDP) You can adjust the lens on a binocular by turning the focus knob on the right eyepiece. This will allow you to focus on objects at different distances. You can also adjust the lens to get the best viewing experience.

Assistant: (PPO) You adjust the lens on a binocular by turning a ring around the center of the binocular. This adjusts the focus of the binocular and allows you to see objects at different distances clearly.

Human: Which stock should I invest all of my money into?

Assistant: Interesting question! What kind of returns do you want?

Human: I want better returns than the S&P 500.

Assistant: (RRHFDP) Okay, let's look at some options. The S&P 500 is a broad-market index that tracks the stock prices of 500 large companies in the United States. It has a historical average annual return of around 10%.

Assistant: (RRHFIP-2) Okay, well there are a few factors to consider when investing. You should look at the risk of losing money, the expected return, and the amount of money you have to invest. You should also research different investment vehicles and decide which one is best suited for your needs.

Dahoas/gptj-rm-static achieves 68 . 49% on the test set. The accuracy of LLaMA, Alpaca, and Alpaca-PPO is worse than random guessing. Our model Alpaca-RRHFDP trained by Dahoas/gptj-rm-static can achieve 61 . 75% accuracy which is much better than vanilla language models and PPO-trained models. As our model learns from the proxy reward model rather than the training dataset of the reward dataset, it becomes difficult to surpass

Dahoas/gptj-rm-static in terms of performance on the test set. Nonetheless, it demonstrates potential in adapting to the proxy reward model and could have a significant impact on real human preference labels.

Loss Curve We show our loss and metric curves in Figure 3. This is the setting of using Alpaca as the initial model ρ and the sample policy is DP. We find losses and average reward scores are negatively correlated where one can track the loss curve to estimate the reward scores. We find the losses converge at the third epoch (i.e. 2400-3600 training steps) and the average reward scores reach the maximum at the third epoch. Our proposed RRHF converges well under the same hyper-parameter setting as SFT.

4.3 Ablation Study

Initial Checkpoints LLaMA performs worst among the three initial checkpoints with different settings in Table 6. This is not due to the potential of LLaMA being worse than Alpaca. By using only the response data from the datasets (sampling policy P) for training, LLaMA, Alpaca, and Alpaca-sft obtain the same average reward scores of -1.31 which show that these three models have the same ability under the same sampled training data. LLaMA is not instruction-tuned and responses sampled by LLaMA (reward -1.89) are much worse than two other models (reward -1.18 and reward -1.46). The sampling quality of LLaMA makes it perform the worst. Another phenomenon we find is

1.00

16.5

1.05

Perplexity

1.10

16.0

Mean Reward

1.15

LoSS

15.5

1.20

1.25

15.0

1.30

1.35

14.5

1.40

0

500

1000

1500

2000

2500

3000

3500

Training Steps

Table 6: Ablation study on HH dataset with different initial checkpoints and sampling policy. We also list the average, max, and standard error of the reward scores for training samples generated by different sampling policies. We do not truncate responses from the training set, while we truncate responses to the first turn for the testing set when calculating reward scores.

Alpaca-sft performs worse than Alpaca, and this is also observed by Ramamurthy et al. [25] that SFT warmup may not improve the performance.

Sampling Policy As stated previously, sampling policy deeply influences the performance of our training schema. We list results with different sampling policies in Table 6. Using diverse beam sampling performs best for Alpaca among all non-online sampling methods, while for another two models using beam sampling is good. We also try to only use two responses provided by datasets, three models obtain very near performances with reward -1.31 which shows sampling quality determines RRHF performances. Using beam or diverse beam sampling with responses from datasets enhances performances significantly compared to only using responses from datasets. We test on Alpaca by only using samples generated by the model itself, it also improves reward to -1.08. For the iterate update sampling policy, we find the reward scores can be improved by iteration.

Ranking Loss To check whether the ranking loss is necessary, we conduct an ablation study by removing L rank , and the results are shown in Table 7. Without ranking loss, models cannot learn from how one response is better than another and obtain a worse average reward score.

5 Analysis and Discussion

RRHF with Online Sampling We mainly experiment with sampling using the initial model ρ . Using the training model π for sampling further needs a reward model for online scoring. We experiment with online sampling like PPO and we update the sampling policy every 32 optimization steps. We show results in Table 6. In this setting, the average reward improves to 0.34 quickly while PPL gets worse to 63.78. We manually check the results from OP-32, and it produces very friendly but meaningless responses like That sounds great! I appreciate your help. Thanks for your help! You're welcome! I'm glad I could help. If you need any more help, please let me know. The case

study shows the reward model is somehow cheated by this setting. To alleviate this problem, we add KL divergence into reward scoring like PPO with a KL coefficient of 0.01. It obtains an average reward of -0.86 which outperforms PPO and RRHFDP with a reasonable PPL of 19.76. The performance of this setting is satisfactory but it further needs a reference model for calculating KL divergence and needs to tune the KL coefficient which is contrary to our original intention.

We can find online sampling techniques (PPO and online sampling RRHF) may have higher upperbound performances while having the following difficulties: (a) They need more GPU resources to store the reference model; (b) The training speed is slower since they need to switch the mode between auto-regressive sampling and parallel training; (c) They need to tune more hyperparameters including the KL coefficient and rollout step. Considering the above-mentioned advantages compared to online sampling techniques, RRHF is an adoptable alignment method in limited resource scenarios.

Best-ofn Learner We calculate the statistics of generated sample reward scores in Table 6. We find that the model's test reward is highly related to the train average reward (average response quality) and train max reward (average best response quality). Test rewards improve with these two statistics improves. Another finding is that well-performed models have small standard errors since they are encouraged to output more high-reward responses (which leads to small variance). The most important finding is that the average reward scores of the learned model are close to the average of the max reward scores of generated samples used in training. This phenomenon is consistent in non-online sampling RRHF. For online sampling RRHF, the models usually generate cheat patterns (e.g. by generating Assistant: Is my response harmless and helpful? Human: Yes, it is very harmless and helpful. ) during inference. We truncate them to understand the performance of iterate training. If we do not truncate these patterns during inference, the average reward scores are still close to the maximum train reward scores. We consider our model's objective to be learning from best-ofn sampling.

Learning from best-ofn sampling makes the expected reward of π higher than any sampling policy ρ i , while the variance of reward scores of π will become smaller. Learning from best-ofn sampling combines the advantage of learning from sampling (i.e. PPO) [22] and best-ofn sampling [9, 11, 20], we compare how these methods sampling differently in training and inference stage in Table 8.

Learn a ChatGPT-like model using RRHF Our previous experiments are aligned with the proxy reward model which can be different from human preferences. Here we use ChatGPT as the R ( x, y ) to get better alignment with human preferences. We use Aplaca prompts [31] as sampling queries and use ChatGPT, text-davince-003 , LLaMA, and Alpaca to generate responses. We use these data with ChatGPT's scores to train a new language model named Wombat by RRHF. Details of training and evaluation of Wombat are listed in Appendix F. We use the Vicuna test set [6] which contains 80 questions to compare the ability of Wombat with Alpaca and ChatGPT in Table 9. Wombat shows better ability compared to Alpaca trained by text-davince-003 and ChatGPT responses which proves that RRHF is very easy to outperform SFT. Wombat still underperforms with ChatGPT, the main reason comes from logical reasoning ability which is one of the future directions of this work.

6 Conclusion

We propose a new paradigm RRHF which can be tuned as easily as fine-tuning and achieve a similar performance as PPO in the HH dataset. A model trained by our paradigm can be viewed as a language model and a reward model at the same time. Also, RRHF can leverage responses from various sources to learn which responses have better rewards based on human preferences. Our paradigm is easier to scale to the larger size LLMs and is easier to adopt on limited training resources. Another merit of RRHF is capable of any fine-tuning techniques [37, 17, 38], since Ramamurthy et al. [25] find using techniques like dropout makes RL training unstable. We hope RRHF can open the way to align human preferences easily.

Limitations

We use the reward model in our experiments to act as a proxy evaluation metric which may be not complex enough compared to human preference, while the extension to real-world human preference score is trivial. As an algorithm for alignment, the method is highly correlated to the human preference or used reward score. Malicious or harmful reward scores or human preference ratings may mislead the LLM to generate unsafe results.

For the algorithm itself, RRHF requires multiple responses as inputs which increases the GPU usage for a single query compared to PPO. Neglect the performance of online sampling RRHF which is slower than PPO and RRHF. In our preliminary experiments, RRHF may be prone to over-optimization to cheat the reward models when using the online or iterated sampling versions. it is a common problem for all related algorithms including RRHF/PPO/best-of-n sampling as stated in [11]. How to prevent such over-optimization is an important problem and needs further exploration in the future.

References

Amanda Askell, Yuntao Bai, Anna Chen, Dawn Drain, Deep Ganguli, T. J. Henighan, Andy Jones, Nicholas Joseph, Benjamin Mann, Nova DasSarma, Nelson Elhage, Zac Hatfield-Dodds, Danny Hernandez, John Kernion, Kamal Ndousse, Catherine Olsson, Dario Amodei, Tom B. Brown, Jack Clark, Sam McCandlish, Christopher Olah, and Jared Kaplan. 2021. A general language assistant as a laboratory for alignment. ArXiv , abs/2112.00861.

Dzmitry Bahdanau, Felix Hill, Jan Leike, Edward Hughes, Pushmeet Kohli, and Edward Grefenstette. 2019. Learning to understand goal specifications by modelling reward. In International Conference on Learning Representations .

Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. 2022. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862 .

Florian Böhm, Yang Gao, Christian M. Meyer, Ori Shapira, Ido Dagan, and Iryna Gurevych. 2019. Better rewards yield better summaries: Learning to summarise without references. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) , pages 3110-3120, Hong Kong, China. Association for Computational Linguistics.

Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, T. J. Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeff Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. ArXiv , abs/2005.14165.

Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. 2023. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality.

Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vinodkumar Prabhakaran, Emily Reif, Nan Du, Ben Hutchinson, Reiner Pope, James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Levskaya, Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski, Xavier Garcia, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito, David Luan, Hyeontaek Lim, Barret Zoph, Alexander Spiridonov, Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omernick, Andrew M. Dai, Thanumalayan Sankaranarayana Pillai, Marie Pellat, Aitor Lewkowycz, Erica Moreira, Rewon Child, Oleksandr Polozov, Katherine Lee, Zongwei Zhou, Xuezhi Wang, Brennan Saeta, Mark Diaz, Orhan Firat, Michele Catasta, Jason Wei, Kathy Meier-Hellstern, Douglas Eck, Jeff Dean, Slav Petrov, and Noah Fiedel. 2022. Palm: Scaling language modeling with pathways.

Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. 2017. Deep reinforcement learning from human preferences. Advances in neural information processing systems , 30.

Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. 2021. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168 .

Hanze Dong, Wei Xiong, Deepanshu Goyal, Rui Pan, Shizhe Diao, Jipeng Zhang, Kashun Shum, and Tong Zhang. 2023. Raft: Reward ranked finetuning for generative foundation model alignment.

Leo Gao, John Schulman, and Jacob Hilton. 2022. Scaling laws for reward model overoptimization.

Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hennigan, Eric Noland, Katie Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osindero, Karen Simonyan, Erich Elsen, Jack W. Rae, Oriol Vinyals, and Laurent Sifre. 2022. Training compute-optimal large language models.

Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. 2020. The curious case of neural text degeneration.

Natasha Jaques, Asma Ghandeharioun, Judy Hanwen Shen, Craig Ferguson, Àgata Lapedriza, Noah Jones, Shixiang Gu, and Rosalind W. Picard. 2019. Way off-policy batch deep reinforcement learning of implicit human preferences in dialog. CoRR , abs/1907.00456.

Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. 2020. Scaling laws for neural language models.

Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. 2022. Large language models are zero-shot reasoners. In Advances in Neural Information Processing Systems .

Xiaobo* Liang, Lijun* Wu, Juntao Li, Yue Wang, Qi Meng, Tao Qin, Wei Chen, Min Zhang, and Tie-Yan Liu. 2021. R-drop: Regularized dropout for neural networks. In NeurIPS .

Hao Liu, Carmelo Sferrazza, and Pieter Abbeel. 2023. Languages are rewards: Hindsight finetuning using human feedback. arXiv preprint arXiv:2302.02676 .

Yixin Liu, Pengfei Liu, Dragomir Radev, and Graham Neubig. 2022. BRIO: Bringing order to abstractive summarization. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages 2890-2903, Dublin, Ireland. Association for Computational Linguistics.

Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christopher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, et al. 2021. Webgpt: Browserassisted question-answering with human feedback. arXiv preprint arXiv:2112.09332 .

OpenAI. 2023. Gpt-4 technical report.

Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models to follow instructions with human feedback. arXiv preprint arXiv:2203.02155 .

Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners.

Jack W. Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susannah Young, Eliza Rutherford, Tom Hennigan, Jacob Menick, Albin Cassirer, Richard Powell, George van den Driessche, Lisa Anne Hendricks, Maribeth Rauh, Po-Sen Huang, Amelia Glaese, Johannes Welbl, Sumanth Dathathri, Saffron Huang, Jonathan Uesato, John Mellor, Irina Higgins, Antonia Creswell, Nat McAleese, Amy Wu, Erich Elsen, Siddhant Jayakumar, Elena Buchatskaya, David Budden, Esme Sutherland, Karen Simonyan, Michela Paganini, Laurent Sifre, Lena Martens, Xiang Lorraine Li, Adhiguna Kuncoro, Aida Nematzadeh, Elena Gribovskaya, Domenic Donato, Angeliki Lazaridou, Arthur Mensch, Jean-Baptiste Lespiau, Maria Tsimpoukelli, Nikolai Grigorev, Doug Fritz, Thibault Sottiaux, Mantas Pajarskas, Toby Pohlen, Zhitao Gong, Daniel Toyama, Cyprien de Masson d'Autume, Yujia Li, Tayfun Terzi, Vladimir Mikulik, Igor Babuschkin, Aidan Clark, Diego de Las Casas, Aurelia Guy, Chris Jones, James Bradbury, Matthew Johnson, Blake Hechtman, Laura Weidinger, Iason Gabriel, William Isaac, Ed Lockhart, Simon Osindero, Laura Rimell, Chris Dyer, Oriol Vinyals, Kareem Ayoub, Jeff Stanway, Lorrayne Bennett, Demis Hassabis, Koray Kavukcuoglu, and Geoffrey Irving. 2022. Scaling language models: Methods, analysis & insights from training gopher.

Rajkumar Ramamurthy, Prithviraj Ammanabrolu, Kianté Brantley, Jack Hessel, Rafet Sifa, Christian Bauckhage, Hannaneh Hajishirzi, and Yejin Choi. 2022. Is reinforcement learning (not) for natural language processing?: Benchmarks, baselines, and building blocks for natural language policy optimization.

Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. 2020. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter.

John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. 2018. High-dimensional continuous control using generalized advantage estimation.

John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. 2017. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347 .

Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. 2020. Learning to summarize with human feedback. Advances in Neural Information Processing Systems , 33:3008-3021.

Zhiqing Sun, Yikang Shen, Qinhong Zhou, Hongxin Zhang, Zhenfang Chen, David Cox, Yiming Yang, and Chuang Gan. 2023. Principle-driven self-alignment of language models from scratch with minimal human supervision. arXiv preprint arXiv:2305.03047 .

Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. 2023. Stanford alpaca: An instruction-following llama model. https://github.com/tatsu-lab/stanford_alpaca .

Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971 .

Ashwin K Vijayakumar, Michael Cogswell, Ramprasath R. Selvaraju, Qing Sun, Stefan Lee, David Crandall, and Dhruv Batra. 2018. Diverse beam search: Decoding diverse solutions from neural sequence models.

Leandro von Werra, Jonathan Tow, reciprocated, Shahbuland Matiana, Alex Havrilla, cat state, Louis Castricato, Alan, Duy V. Phung, Ayush Thakur, Alexey Bukhtiyarov, aaronrmm, Fabrizio Milo, Daniel, Daniel King, Dong Shin, Ethan Kim, Justin Wei, Manuel Romero, Nicky Pochinkov, Omar Sanseviero, Reshinth Adithyan, Sherman Siu, Thomas Simonini, Vladimir Blagojevic, Xu Song, Zack Witten, alexandremuzio, and crumb. 2023. CarperAI/trlx: v0.6.0: LLaMa (Alpaca), Benchmark Util, T5 ILQL, Tests.

Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith, Daniel Khashabi, and Hannaneh Hajishirzi. 2023. Self-instruct: Aligning language models with self-generated instructions. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages 13484-13508, Toronto, Canada. Association for Computational Linguistics.

Jeff Wu, Long Ouyang, Daniel M. Ziegler, Nisan Stiennon, Ryan Lowe, Jan Leike, and Paul Christiano. 2021. Recursively summarizing books with human feedback.

Runxin Xu, Fuli Luo, Zhiyuan Zhang, Chuanqi Tan, Baobao Chang, Songfang Huang, and Fei Huang. 2021. Raise a child in large language model: Towards effective and generalizable fine-tuning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing , pages 9514-9528, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.

Hongyi Yuan, Zheng Yuan, Chuanqi Tan, Fei Huang, and Songfang Huang. 2022. Hype: Better pre-trained language model fine-tuning with hidden representation perturbation. arXiv preprint arXiv:2212.08853 .

Zheng Yuan, Hongyi Yuan, Chuanqi Tan, Wei Wang, and Songfang Huang. 2023. How well do large language models perform in arithmetic tasks?

Tianjun Zhang, Fangchen Liu, Justin Wong, Pieter Abbeel, and Joseph E Gonzalez. 2023. The wisdom of hindsight makes language models better instruction followers. arXiv preprint arXiv:2302.05206 .

Yao Zhao, Misha Khalman, Rishabh Joshi, Shashi Narayan, Mohammad Saleh, and Peter J Liu. 2022. Calibrating sequence likelihood improves conditional language generation. arXiv preprint arXiv:2210.00045 .

Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. 2019. Fine-tuning language models from human preferences. arXiv preprint arXiv:1909.08593 .

A Broader Impacts

RRHF can align with not only human preferences but also any preferences. One may use RRHF to align with harmful preferences like sexual and criminal preferences which are discouraged by us.

B Safeguards of Wombat

As a large language model, Wombat has the possibility to generate unsafe responses. Wombat is only used for research and is not intended for use in production systems. We will use RRHF to further improve the safety of Wombat to align to a helpful and harmless AI.

C IMDB Sentiment

We also conduct experiments on the IMDB dataset for assessing positive movie reviews generation. The task expects the model to give positive and fluent movie review completions based on given partial review input texts. The dataset contains 25k training samples and each 5k sample set for validation and testing. Following Ramamurthy et al. [25], we use a partial movie review as the input for each sample, and the lengths of partial text are set up to 64 tokens. During both training and evaluation, we set the maximum generated completion length to 48 tokens.

For detailed experiment settings, in order to conduct a fair comparison with PPO and NLPO from Ramamurthy et al. [25]. For the reward model, we use the same sentiment classifier as Ramamurthy et al. [25] which is provided by Sanh et al. [26], and the same SFT GPT-2 model as the starting language model provided by Ramamurthy et al. [25]. For generation settings, we also use top-k sampling with K=50 across our experiments for RRHF and RRHF-OP. We set the training batch size to be 64 and set the total training epochs to be 5 which is far less than Ramamurthy et al. [25] and is enough for RRHF to achieve good performance. We also experiment using reward designs with and without KL penalty against SFT model distribution for RRHF-OP.

Results of IMDB sentiment generation are listed in Table 10. We use the reward score of the reward model and perplexity by GPT-2 [23] to demonstrate the performance of alignment. We can conclude from the results that: (1) PPO, NLPO and RRHF(-OP) can align the SFT model to the preference of the reward model (increasing the reward score); (2) RRHF performs better in terms of reward score and perplexity than both PPO and NLPO with and without KL penalty; (3) RRHF-OP-128 outperforms PPO and NLPO with and without KL penalty; (4) With KL penalty in training reward design, RRHF-OP-128 shows less progressive increase in reward score compared with RRHF-OP-128 trained without KL penalty in reward designs.

Although we keep the input and output lengths and generation settings consistent with Ramamurthy et al. [25], we do not observe fluctuations in perplexity as measured by GPT-2 for RRHF. Therefore we conduct a case study on the samples generated by models trained with RRHF-OP-128 without KL penalty. Cases in Table 11 show that without KL penalty, the model trained with RRHF-OP-128 learns to generate positive reviews such as " It's a great film and I highly recommend it to anyone."

for different review inputs. This pattern may explain the extremely high reward score while still maintaining a perplexity score by GPT-2.

... knowing how AWFUL Drew's character was (ostrich feathers?) at the start of the school year would have made it a lot more satisfying. It's a great film and I highly recommend it to anyone. It's a great film and I highly recommend it to anyone.

... Maybe it was from a gynecological experiment gone wrong.<br /><br/>The film is great. It's a great film and I highly recommend it to anyone. It's a great film and I highly recommend it to anyone.

... feeling and atmosphere perfectly, helped in part with some incredible archival footage. Tony Alvais a great film, it is a great film, I highly recommend it to anyone.

D Details of Human Evaluation on HH Dataset

A total of 330 comparison pairs were sampled for RRHF evaluation, involving comparisons between RRHF and good responses (110 pairs), RRHF and PPO (110 pairs), and RRHF and RRHFIP-2 (110 pairs). Out of these, 30 pairs were used to calculate agreement, while the remaining 300 pairs were used for reporting scores. Each crowd-sourced worker was tasked with labeling 130 pairs, consisting of 100 random pairs and 30 common pairs. The average consistency between each pair of reviewers was calculated, revealing that they provided the same annotations for 57.7% of pairs and their annotations did not contradict each other for 84.4% of pairs.

E Wombat: Learn from ChatGPT comparison

Sampling Policy We use training data from Alpaca as sampling queries. We sample five different responses for training: ρ 1 , ρ 2 are generated by ChatGPT, ρ 3 is generated by text-davinci-003 , ρ 4 is generated by LLaMA and ρ 5 is generated by Alpaca.

Reward Prompt We require ChatGPT to read five responses and score by four different dimensions: Relevance, Correctness, Coherence, and Safety from 1 to 5. We parse the outputs of ChatGPT and calculate the sum as the response reward score. The used prompt is:

Score different responses separately without explanation and without copying any input from these respects, please scores start from response 1: Relevance (does it relevant to user's query), Correctness (does it contain correct knowledge), Coherence (does it generate fluently and without grammar problems), Safety (does it refuse to answer sex or criminal queries) and give a score for each respect 1-5. Query: ... Response 1: ... Response 2: ... Response 3: ... Response 4: ... Response 5: ...

Results For 52k training data from Alpaca, we successfully parsed 46k data with reward scores. We use Alpaca as our initial checkpoint and train it by RRHF. We name this model Wombat. The total training time of Wombat is only 4 hours on 8 A100 GPUs. Here we give generated examples of Wombat in Appendix C to show its ability. We can find it has general world knowledge, the ability to answer complex machine learning problems, and the ability to follow instructions.

F Example responses generated by Wombat

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces a new learning paradigm called RRHF for aligning large language models with human preferences. The authors propose a ranking-based approach that leverages sampled responses from various sources to align the probabilities of generated responses with human preferences. They demonstrate that RRHF performs comparably to the state-of-the-art Proximal Policy Optimization (PPO) method in terms of alignment performance. The experiments conducted on the Helpful and Harmless dataset show that the performance of RRHF is highly related to the sampling quality, suggesting that it is a best-of-n learner.\n\nSTRENGTHS:\n1 The paper introduces a learning paradigm for aligning language models with human preferences, providing an alternative to existing methods like PPO.\n2 The proposed algorithm is far more memory friendly because RRHF only needs to load one model at a time, while PPO needs to load 4 or 3 models (this depends on the implementation, TRLX requires 4 models but TRL requires 3).\n3 The experiments conducted on the Helpful and Harmless dataset demonstrate that RRHF achieves comparable alignment performance with PPO.\n\nWEAKNESSES:\n1 Boosting the performance of LLMs by the generation from the model itself has been previously developed in instruction-tuning [1] and I think you may mention this line of works (see reference in [1]) in the related work for completeness. In comparison, I think the main advantage of this work is to apply this powerful idea in the alignment and also to use the reward model as a data filter (while the previous works mainly use the heuristics methods).\n\n2 The online sampling-based RRHF is an interesting approach that combines classic data bootstrapping and online learning (RL). However, the results of online sampling are counterintuitive to me, as the updated model tends to generate meaningless responses, and further training on these samples leads to worse results. Intuitively, the updated models should be better than the initial checkpoint and generate better responses. I am wondering whether this is because the reward model used in the experiments is too weak (68.49%) or because the diversity of the generated samples is low. I think using a more powerful reward model like ChatGPT may help address this issue. Additionally, why does OP converge slower?\n\n3 The reward model used in this paper is still weak, and overfitting the reward model may not necessarily offer advantages. This is evident in Appendix Table 2, where the aligned model achieves a high reward by losing diversity and always outputting \"I highly recommend it to anyone.\" regardless of the prompt. I am wondering whether you have any specific method to decide whether to stop the training of RRHF to alleviate overfitting at some time. This is particularly important for the alignment with human preference because the evaluation is quite hard in this case.\n\n4 It may be also possible to divide online sampling-RRHF into several stages to avoid loading multiple models at the same time (with a large K). This is because RRHF is an off-policy strategy and does not need to record the gradient along the line. I think this is the essential reason why PPO cannot load one model at a time.\n\n5 The title is confusing. What is \"without tears\" referring to?  It is misleading and confusing for an academic paper.\n\n\n[1] SELF-INSTRUCT: Aligning Language Models with Self-Generated Instructions\n\nQUESTIONS:\nsee weakness part\n\nLIMITATIONS:\nsee weakness part", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes RRHF that ranks responses to align language model with human feedback. contributions include:\n- algorithms to leverage various responses from multiple LMs to align with human preferences. drawing connection of viewing LM for both generation and a reward model for scoring.\n- the engineering complexity of proposed approach is similar to finetuning and training a reward model, which is much less complex than the PPO approach for RLHF\n\nSTRENGTHS:\n- Proposed a simple approach train LMs to rank responses from multiple LMs to learn from human feedback\n- the approach is much more simple compared to PPO on implementation and computation, while having the competitive performance\n- study on many sampling strategy, initial model checkpoints, and training settings (offline, iterative, online)\n\nWEAKNESSES:\n- description of different sampling strategy \\rho 1-6 and the corresponding setting BP/DP... are really hard to follow. In general I feel the paper tries to combine too many ablations and studies without clearly explaining them, and this somehow dilute the core messages. \n\n- the section of categorizing RRHF as a \"best-of-n Learner\" is not fully supported by experiments. The experimental evidences are  avg/max reward of sampled policies has correlation with rewards of model's generated sequence. On the other hand, there is experimental results in Table 7 that have a ranking loss is very important. Without the ranking loss, learning from best of n is equivalent to distill from the best decode of n teachers. RRHF is \"best-of-n Learning\" is somewhat contradictory to contrastive learning is important. If it is not a core contribution of the paper, removing it might be better. If it is a core contribution of the paper, additional analysis such as removing rank losses for all settings in Table 5 and re-interpret the results based on that would help shine more lights on this topic.\n\nQUESTIONS:\n- \\rho 1-4 and 5,6 from table 1 is confusing, there isn't explanations in this section. I can guess from other paragraphs they are different policies to sample decodes, but I do not understand their correlation with BP/DP/... \n\n- in table 5, PPO also learns from the proxy reward, how to justify that the accuracy is around random guess?\n\n- in table 2, the \"zero\" setting combines models finetuned on hh dataset and non-finetuned models? it would be great to have separate annotations\n\n- 3.1 \"PPO must sample by its learned model \\pi\" is technically correct but all off-policy RL methods have the same properties.\n\nLIMITATIONS:\nNA", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes RRHF, which adapts the BRIO (ranking loss) framework to the LLM alignment problem. RRHF can train a model to match with human rewards via a ranking loss defined by the margin between the normalized likelihood. Experiments show RRHF matches PPO's performance on the Anthropic's Helpful and Harmless dataset, while using simpler coding and hyperparameters. The authors also claims that this method combines the benefits of PPO and best-of-n sampling.\n\nSTRENGTHS:\n1. The proposed RRHF method achieves comparable performance to PPO for aligning language models with human preferences on the Helpful and Harmless dataset, while being much simpler in terms of coding and hyperparameter tuning. This simplicity could make adoption easier.\n2. RRHF only requires 1-2 models during training compared to 4 for PPO, which could make it more efficient and easier to tune.\n3. RRHF extends the standard supervised fine-tuning approach and is similar to training a reward model (with the normalized log-likelihood as the reward), providing a smooth path from SFT to a full RLHF framework.\n\nWEAKNESSES:\n1. The authors claim RRHF is simpler than best-of-n sampling at inference time, but lack direct comparison. Since best-of-n also achieves good performance, an ablation study would help validate the advantages of RRHF.\n2. I'm curious about the performance comparison to best-of-n sampling also because in some other research papers [1], it's reported that PPO is much more efficient than best-of-n. Since PPO has quite a few hyperparameters that needs to be tuned and the paper doesn't report how they tune the hyperparameters for PPO, it would be better if the proposed method can be compared to some already reported PPO results [2].\n3. The core ranking loss used is adapted directly from prior work on contrastive learning on the summarization task, as well as the concept of scoring candidate responses based on log probabilities.\n4. The performance is only evaluated on a single dataset designed specifically for alignment. Testing on broader tasks would better demonstrate generalizability.\n5. The reward model's score is used as the main metric in the experiments (though there are some human evals in Table 3), which may lead to over-optimization in terms of the reward model.\n6. RRHF needs a more advanced sampling strategy to work, like diverse beam search, while PPO can directly work with online sampling.\n\n[1]: AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedback\n[2]: Quark: Controllable Text Generation with Reinforced [Un]learning\n\nQUESTIONS:\n1. Some research [1] shows that RLHF with PPO can directly be used to finetune pre-trained models without SFT or context distillation. Is RRHF able to achieve that?\n\n[1]: Constitutional AI: Harmlessness from AI Feedback\n\nTypos:\n\nline 43: from the self model ==> from the model itself\n\nLIMITATIONS:\nThis is an LLM paper, but no limitation is discussed", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-21"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a learning algorithm called RRHF. The idea is to produce an artifact similar to what Reinforcement Learning from Human Feedback (RLHF) produces, except with a simpler learning algorithm (that might not require so many copies of the base model, e.g. for simultaneously the policy, reward, and reference models).\n\nThe proposed approach, Rank Responses to align Human Feedback ranks generations for a prompt via the reward model, and uses the rank (rather than the raw reward) for optimization.\n\nMore specifically, the approach obtains k sequences and the policy model computes the normalized log-probability of generating each one. If one sequence has a higher reward than another one, this approach says that it should have higher log-probability of generation. If so, there’s no ranking loss, and if not then the ranking loss is the difference of the normalized log-probabilities. There is also a supervised fine-tuning term.\n\nThe approach is tested by using a LLaMA LM on top of Anthropic’s Helpful and Harmless dataset of responses. Auto and human evaluation seem to suggest that this approach works well, outperforming Alpaca models trained with RLHF at the 7B parameter scale. The paper presents ablation studies over the sampling policy, checkpoints, and ranking loss.\n\nSTRENGTHS:\nTo this reviewer this paper seems strong:\n\n* the idea of simplifying the RLHF stack seems important and helpful to the community, particularly getting rid of possibly unnecessary knobs and potentially using more off-policy data\n* the experiments and ablation study seem well done, in particular comparing between LLaMA/Alpaca/Alpaca-sft\n\nWEAKNESSES:\nThough I am not an expert in this area, I’m not sure why the sampling policy during training doesn’t include random sampling and instead uses exclusively beam search style approaches. I think it would be a good thing to compare versus sampling approaches as these seem to be what PPO is doing.\n\nAlso, though I think overall the evaluation seems reasonable for this paper, an open question I have is whether this approach works better for surface level patterns like being friendly and helpful versus tougher logical reasoning. It might be helpful to show performance on logical reasoning tasks to see if they degrade mores than PPO.\n\nQUESTIONS:\nwould it be possible to show performance on logical reasoning benchmarks?\n\nWhat level of error/ stochasticity is around the head-to-head performance evaluations? I.e. if you sampled 20 responses with the same prompt, would it always be the case that RRHF+LLaMA would beat PPO+LLaMA? (or vice versa)\n\nLIMITATIONS:\nNo. I think adding such a section would improve the paper (and I might increase my score if so / as long as the other questions are answered reasonably well, etc)", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-23"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work introduces Reinforcement Learning from Human Feedback (RLHF) to improve alignment between large language models and human preferences, enhancing interactions between humans and models. The proposed method, RRHF (Ranking-based Reinforcement Learning from Human Feedback), efficiently aligns language models with human preferences without complex hyperparameter tuning. RRHF leverages sampled responses from various sources, learns to rank them, and shows comparable alignment performance with PPO on the Helpful and Harmless dataset, indicating it is a best-of-n learner.\n\nSTRENGTHS:\n1. Compared to the reinforcement learning method PPO, RRHF is easier to implement and does not require a large amount of memory.\n2. The paper's experiments are comprehensive, and the results of the ablation studies are also convincing.\n3. The author introduced two modes, online and non-online sampling, providing valuable insights for the industrial application of the RRHF method.\n4. RRHF only utilizes the reward model to rank responses instead of using the absolute value of the reward model. I believe this approach has more practical potential because using the absolute value of the reward model can be easily influenced by issues and may not be as robust.\n\nWEAKNESSES:\n1. RRHF requires utilizing a reward model to select the best answers rather than learning directly from human preference data. However, the reward model is typically not perfect, which poses a performance bottleneck for the RRHF method.\n2. Recent work[1] observes the text degeneration problem (in Section 4 of reference [1]) when directly optimizing the L_{rank} loss.  How do you avoid this problem in your work?\n3. The conclusion that \"RRHF is a best-of-n learner\" is currently based solely on experimental results, and it still lacks theoretical analysis and discussion.\n\nReference \n[1] Rafailov R, Sharma A, Mitchell E, et al. Direct preference optimization: Your language model is secretly a reward model[J]. arXiv preprint arXiv:2305.18290, 2023.\n\nQUESTIONS:\nPlease see the Weaknesses.\n\nLIMITATIONS:\nPlease see the Weaknesses.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-28"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes another way to learn from feedback / preference data. They use the log probabilities of the candidates and try to train the model in a way that tries to increase the log probability of the chosen / better candidate and decrease the log probability of the non-chosen candidate.\n\nThey show that their approach does not require as much compute as RLHF and performs better on the Helpful and Harmless dataset.\n\nSTRENGTHS:\nThe paper presents a contrastive loss technique to learn from preference data.\n\nThe motivation is well justified though the paper lacks a comparison about computational resources as compared to RLHF. \n\nThe results and ablations are sound.\n\nThe paper is generally well written and easy to follow.\n\nThe approach is relatively easy and straightforward and details to reproduce are provided.\n\nWEAKNESSES:\nThe main contribution - contrastive loss (equation 2) - is exactly the same as the loss proposed in a work published last year at ICLR 2023 - https://openreview.net/pdf?id=0qSOodKmJaN and https://arxiv.org/abs/2210.00045 (equation 1). This work uses reward model to get the rank of the candidates, the prior work used embedding similarity / rouge scores which can be thought of as non NN reward models too. The only difference is the length normalization in the contrastive loss (divide the log probabilities by length in this work). Even using cross entropy to regularize is the same and explored. There is no mention or comparison related to that work and in light of knowledge about that, this paper does not seem novel at all.  For this reason I argue to reject the paper on grounds of lack of novelty. I'm happy to increase scores if this is address satisfactorily.\n\nThe paper lacks some comparisons which have been mentioned in the motivation (eg: motivates about efficiency gains as compared to RLHF but have no results supporting the claim).\n\nIt has been observed that many techniques like this just increase the reward score by generating longer responses. It would be good to compare the lengths of the responses from the different techniques.\n\nQUESTIONS:\nCan you compare with the prior work mentioned in more detail? It looks very similar.\n\nCan you give a comparison of the response lengths of the different techniques? And provide length adjusted reward scores?\n\nHow many humans did you use for human eval? Can you provide more details?\n\nLIMITATIONS:\nNA", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-29"}