SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Hongyi Yuan $^{12\dagger*}$  Zheng Yuan $^{1*}$  Chuanqi Tan $^{1}$  Wei Wang $^{1}$  Songfang Huang $^{1}$  Fei Huang $^{1}$
$^{1}$ Alibaba DAMO Academy  $^{2}$ Tsinghua University
{yuanzheng.yuanzhen,chuanqi.tcq}@alibaba-inc.com
yuanhy20@mails.tsinghua.edu.cn
Reinforcement Learning from Human Feedback (RLHF) facilitates the alignment of large language models with human preferences, significantly enhancing the quality of interactions between humans and models. InstructGPT implements RLHF through several stages, including Supervised Fine-Tuning (SFT), reward model training, and Proximal Policy Optimization (PPO). However, PPO is sensitive to hyperparameters and requires multiple models in its standard implementation, making it hard to train and scale up to larger parameter counts. In contrast, we propose a novel learning paradigm called RRHF, which scores sampled responses from different sources via a logarithm of conditional probabilities and learns to align these probabilities with human preferences through ranking loss. RRHF can leverage sampled responses from various sources including the model responses from itself, other large language model responses, and human expert responses to learn to rank them. RRHF only needs 1 to 2 models during tuning and can efficiently align language models with human preferences robustly without complex hyperparameter tuning. Additionally, RRHF can be considered an extension of SFT and reward model training while being simpler than PPO in terms of coding, model counts, and hyperparameters. We evaluate RRHF on the Helpful and Harmless dataset, demonstrating comparable alignment performance with PPO by reward model score and human labeling. Extensive experiments show that the performance of RRHF is highly related to sampling quality which suggests RRHF is a best-of- $n$  learner. Codes are released at https://github.com/GanjinZero/RRHF.
Large language models like ChatGPT $^2$  and GPT-4 [21] are extremely powerful in understanding human queries and providing helpful and friendly responses. Employing Reinforcement Learning from Human Feedback (RLHF) [8, 42, 29] enables alignment of language model outputs with human preferences. As implemented in Ouyang et al. [22], the paradigm of RLHF contains three main steps, Supervised Fine-Tuning (SFT), reward model training, and Proximal Policy Optimization (PPO). Initially, they apply supervised fine-tuning (SFT) on the initial models to learn to follow human instructions. Subsequently, a reward model is learned from the ranking of human preferences. Finally, scores generated by the reward model are used to apply gradient policy in PPO to align human preferences. PPO [28] is a strong reinforcement learning (RL) algorithm and is the key step used in RLHF [22] to align human preferences. This PPO training step is powerful but complex. It requires tuning a large number of hyperparameters for conservative parameter updating, reward design, advantage estimation, etc. Besides, fine-tuning language models with PPO needs to store a
*Contributed equally. † Work done during internship at Alibaba DAMO Academy.
$^{2}$ https://openai.com/blog/introducing-chatgpt-and-whisper-apis
Figure 1: Workflow of RRHF compared with PPO.
policy model, a value model (or a value head), a reward model, and a reference model at the same time which is memory-unfriendly and needs sophisticated architecture of the training platform when scaling up to larger models.
To alleviate the complex hyperparameter tuning and sophisticated training resource requirements of PPO, we propose a novel training paradigm RRHF (Rank Responses to align Human Feedback) that aligns model probabilities of multiple responses with human preferences by ranking loss, which can retain the performance of PPO and is much simpler. Ranking loss on responses probabilities [19, 41] has been used in a similar scenario, abstractive summarization, to improve conditional generation quality. Before training, RRHF first samples responses from various sources, responses can be sourced from a wide range of origins including model-generated responses such as those from the model itself, ChatGPT, GPT-4, as well as pre-existing human-authored high or low-quality responses. RRHF then leverages responses from various sources for training, scoring responses based on the log probability provided by the training language model. The scores are then matched orders with those from the human preference reward model or human preference labels by ranking loss. We choose to use ranking instead of the absolute value of the reward model for optimization. PPO uses estimated advantages to provide optimization signals. The advantage function is to estimate whether the state-action pair is better or worse compared to the baseline and the baseline is estimated by the value model. Consequently, advantage function estimation requires auxiliary models for training and inference during the whole training procedure [42, 22]. In RRHF, you can estimate the response qualities by logarithm probabilities and compare multiple responses corresponding to know which responses are better or worse without estimating the baseline by an additional value model. Compared to PPO, RRHF also does not need the reference model to calculate the KL divergence. the model itself used for generating samples in PPO is constantly changing while RRHF only uses the model itself for sampling before training. Thus the KL term degenerates for RRHF. The workflow for RRHF and PPO is depicted in Figure 1. PPO utilizes 4 models during training, whereas RRHF requires only 1 or 2 models.
Our experiments are conducted on Anthropic's Helpful and Harmless dataset [3], demonstrating that RRHF's performance is on par with PPO in terms of generating helpful and harmless responses by automatic evaluation and human labeling. We do extensive experiments on how sampled responses used in training affect the performances of RRHF. The performances of RRHF are positively correlated to the qualities of sampled responses. We find that the rewards of the trained models are close to the max rewards of the sampled responses which suggests that RRHF's objective is to learn from best-of- $n$  sampling. Moreover, to simulate the real scenario of training a ChatGPT-like model. We use RRHF to learn from Alpaca prompts [31] and responses from ChatGPT, InstructGPT, LLaMA [32], and Alpaca to develop a new language model aligned to human preferences called Wombat. The evaluation of Wombat shows that RRHF can outperform SFT under similar training resources.
Contributions are summarized as follows:
- We propose a new learning paradigm named RRHF for large language models that can leverage various responses to align with human preferences. The trained model can be viewed as a language model for generation and a reward model for scoring.
- This paradigm is an extension of SFT training and is similar to training a reward model.
- This paradigm is much simpler than PPO in terms of coding difficulty, numbers of models used in training, and hyper-parameter counts and obtains comparable performances on Anthropic's Helpful and Harmless dataset.
Recently, scaling up pre-trained language models by the number of parameters, training data [15], and computational budgets [12] can equip large language models with strong abilities in various language tasks [5, 24, 7, 16, 21, 39]. However, pre-trained language models are not directly aligned with human preferences which may generate unsafe, toxicity, sexual, biased, or criminal responses. Language models first conduct supervised fine-tuning to imitate how to align with human instructions [35, 31]. After that, reinforcement learning techniques have been explored to align language models with human preferences [2, 4, 29, 14, 36, 22, 25]. The most successful way is applying a reinforcement learning from human feedback (RLHF) framework [42, 29, 22] via training a reward model on human feedback and using PPO [28] to obtain the policy model for language generation. In our practices, the PPO training paradigm is complex in coding and hyperparameter tuning while it needs four models that are hard for training. This motivates us to explore simpler and more straightforward methods to align language models with human preferences. Nakano et al. [20], Askell et al. [1], Cobbe et al. [9] explore best-of- $n$  sampling to improve large language model generation by selecting the best response based on the human preference rewards among  $n$  sampled responses. Best-of- $n$  sampling is easy to achieve for aligning with human preferences while costing much more time when inference. Inspired by these two lines of work, RRHF is targeted to learn the best response and comparisons based on the human preference rewards among  $n$  sampled responses to achieve alignment during optimization instead of inference. RRHF absorbs the advantages of PPO and best-of- $n$  sampling while being simpler in coding, model count, and hyperparameter tuning than PPO and does not need to sample  $n$  times during inference. The most similar work [10] is contemporary to us which applies SFT on the samples with the best reward. Compared to Dong et al. [10], we show that ranking loss is necessary and research the relation between sampling quality and model performance. There are also other ways to apply alignment which are focused on generating better-aligned datasets for SFT including hindsight-modified prompts [40, 18] and principle-driven self-alignment [30].
We mainly follow the notations in Ziegler et al. [42]. Denote the query data distribution as  $x \sim D$ . For the response  $y$  reply to query  $x$ , a reward function  $R(x, y)$  scores  $y$  based on human preferences which can be a human or a neural network. Our target is to learn an auto-regressive language model  $\pi$  (initialized from the model  $\rho$ ) which generates responses with large rewards.
3.1 RRHF
During training, we have  $k$  different responses  $y_{i}$  of  $x$  sampled by policy  $\rho_{i}, 1 \leq i \leq k$ . Sampling with policy  $\rho_{i}$  is not restricted here which can be the initial model  $\rho$ , the learned model  $\pi$ , other LLMs like ChatGPT or GPT-4, or a response provided by human experts. The sampling policy  $\rho_{i}$  can also vary across the training time. Our sampling method can leverage any existing good or bad responses to help the model align with humans, while PPO can only learn from samples from its learned model  $\pi$ .
The reward function gives scores for each  $y_{i}$  with  $R(x,y_{i}) = r_{i}$ . To align with scores  $\{r_i\}_k$ , we use our model  $\pi$  to give scores  $p_i$  for each  $y_{i}$  by:
$$
p _ {i} = \frac {\sum_ {t} \log P _ {\pi} \left(y _ {i , t} \mid x , y _ {i , <   t}\right)}{\| y _ {i} \|}, \tag {1}
$$
where  $p_i$  is conditional log probability (length-normalized) of  $y_i$  under model  $\pi$ . Our idea is simple, let the model  $\pi$  give larger probabilities for better responses and give smaller probabilities for worse responses. Inspired by Liu et al. [19], we optimize this object by ranking loss:
$$
L _ {r a n k} = \sum_ {r _ {i} <   r _ {j}} \max  \left(0, p _ {i} - p _ {j}\right) \tag {2}
$$
We do not have margins in the ranking loss as Liu et al. [19]. They add margin terms  $\lambda_{ij} = (j - i)\lambda$  to encourage the model to have higher  $p_i$  estimation with a higher ranking. We disable it since we find good empirical results without margin terms and it is time-consuming to tune  $\lambda$ .
We also add a cross-entropy loss similar to SFT (supervised fine-tuning). We require the model to learn the response with the highest reward  $r_i$ .
$$
i ^ {\prime} = \arg \max  _ {i} r _ {i} \tag {3}
$$
$$
L _ {f t} = - \sum_ {t} \log P _ {\pi} \left(y _ {i ^ {\prime}, t} \mid x, y _ {i ^ {\prime}, <   t}\right) \tag {4}
$$
The total loss is defined as the unweighted sum of two losses:
$$
L = L _ {\text {r a n k}} + L _ {f t} \tag {5}
$$
We have tried using larger weights (10,100) on  $L_{rank}$  suggested by Liu et al. [19] which shows worse performances in our preliminary experiments.
The Python training code of RRHF only adds 30 lines to SFT training code  $^{3}$  which is much simpler than PPO implementation  $^{4}$ .
3.2 Relation with Previous Paradigm RLHF
InstructGPT [22] aligns human preferences in three steps: SFT, training a reward model, and PPO. We find our proposed RRHF has similar procedures to the above-mentioned three steps.
Relation with SFT Supervised fine-tuning (behavioral cloning) can be viewed as a degenerated version of our training process with  $k = 1$  and  $\rho_{1}$  being fixed which is provided by human labelers.
Relation with Reward Model Our model can be used as a reward model. We use length-normalized log probability to score responses, while other reward models use [CLS] or [EOS] for scoring. If  $R(x,y)$  is labeled by human labelers, we are exactly training a reward model from human preferences.
Relation with PPO The task objective of PPO [28] is defined by a reward function  $R(x,y)$ , and it is to maximize the expected reward  $\mathbf{E}_{x\sim \mathcal{D},y\sim \pi (\cdot |x)}[R(x,y)]$ . Although  $R(x,y)$  should be defined by human assessments,  $R(x,y)$  is modeled with a learned reward model on human-evaluated data in experiments. To constrain the language policy  $\pi_{\theta}(\cdot |x)$  from moving too far from the initialization  $\rho (\cdot |x)$ , the final reward design becomes:  $\tilde{R} (x;y) = R(x;y) - \beta \log \left(\frac{\pi(y|x)}{\rho(y|x)}\right)$ , where  $\beta$  controls the level of penalty and is set to a fixed value [22] or dynamically adjusted [42].
PPO leverages  $\pi$  for sampling, while RRHF can use any applicable  $\rho_{i}$ . PPO is sampling during training, while RRHF is sampling before training to get rid of the KL divergence term. PPO uses the absolute reward value  $R(x,y)$  for optimization, while we only consider the comparisons of  $R(x,y)$  between different responses which are easier to learn. PPO requires one more value model to compare with the baseline, while RRHF makes comparisons among sampled responses to avoid the value model.
Dataset We use Anthropic's Helpful and Harmless (HH) dataset as our experiment dataset [3] $^5$ . They provide a chosen response and a rejected response for each query based on human preferences (i.e. helpful and harmless). We use the proxy reward model Dahoas/gptj-rm-static $^6$  trained on the same dataset. By using the proxy reward model, we can compare RRHF and PPO fairly.
<sup>3</sup>https://github.com/tatsu-lab/stanford_alpaca/blob/main/train.py
4https://github.com/CarperAI/trlx
<sup>5</sup>https://huggingface.co/datasets/Dahoas/rm-static
<sup>6</sup>https://huggingface.co/Dahoas/gptj-rm-static
Models We experiment mainly based on LLaMA [32] and Alpaca [31] with 7B parameter size. Ouyang et al. [22] and Ramamurthy et al. [25] use supervised fine-tuned models as the initial models when applying PPO, so we also have fine-tuned Alpaca-7B on our used dataset<sup>7</sup> with chosen responses (i.e. human-preferred responses) following trlX[34] and name it as Alpaca-sft.
Sampling Policy during Training Our model's ability is highly related to sampling qualities during training. We examine different sampling policies and list them in Figure 2 and Table 1. We term the initial language model policy as  $\rho$ , the online language model policy as  $\pi$ , and the language model policy after each 3-epoch training iteration as  $\rho^{*}$ . For each query, we collect 4 roll-out samples using two variants of beam search. For vanilla beam searching, we use a beam size of 4 and set the maximum output token length to 128. Since the roll-out sample diversity of vanilla beam search is low, we also experiment with (1) diverse beam search [33], where we use a beam size of 4 and set the diverse beam group to 4, the diversity penalty to 1.0, and the sampling temperature to 0.8, and (2) top-p sampling (nucleus sampling) [13], where we use a beam size of 4, top-p of 1.0, and the sampling temperature to 0.8 which is a consistent setting with the top-p sampling used in our PPO baselines. We sample training data before the training process except for OP-k (online sampling). Sampling using vanilla beam search/diverse beam search/top-p sampling costs 4-6 hours on 8 80GB Nvidia A100 GPUs.
Table 1: Sampling policy used in our experiments. OP-k uses  $\pi$  for sampling (i.e. online sampling), we update  $\pi$  every k optimization steps. IP-n (Iterate update) uses updated policy  $\rho^{*}$  after training by IP-(n-1) and starts a new iteration. The dataset contains a good response and a bad response for each query which are used as  $\rho_{5}$  and  $\rho_{6}$ , which are termed P (Provided responses in datasets).
Setting	ρ1 ~ ρ4	ρ5, ρ6
BP	Beam search by ρ	Provided responses
SP	top-p Sampling by ρ	Provided responses
DP	Diverse beam search by ρ	Provided responses
OP-k	Online diverse beam by π†	Provided responses
IP-n	Iterate diverse beam by ρ*	Provided responses
D	Diverse beam search by ρ	∅
P	∅	Provided responses
Figure 2: The workflow of sampling policy used in our experiments. IP-1 is equivalent to DP.
Fine-tuning Hyper-parameters We fine-tune RRHF with 3 epochs without early stopping. We first warm up the learning rate to 2e-5 and decay to 0 linearly. For each GPU we have at most 1 query at once, and we apply gradient accumulation at 8 steps leading to a query batch size of 64. The query and responses are truncated to 192 tokens. Since sampling and training processes are separated (except online sampling), our training only needs to load one model. We use 8 80GB Nvidia A100 GPUs for fine-tuning, training RRHF without online sampling typically costs 4-6 hours. Training with OP is slower which takes about 30 hours.
Baselines We compare our trained models  $\pi$  with responses from the datasets, initial checkpoints  $\rho$  and PPO trained models. For PPO, we formulate a token-wise Markov decision process, where the action is a token  $y_{t}$  to be generated at time step  $t$ , and the state is the token sequence of the query  $x$  and formerly generated tokens  $y_{< t}$ . We follow the clipped surrogate objective of PPO:
$$
\mathbf {E} _ {y \leq t \sim \pi_ {\theta} (y \leq t | x), x \sim \mathcal {D}} \left[ \max  \left(- r _ {\theta} \left(y _ {t} \mid x, y <   t\right) \hat {A} (x, y \leq t), - \operatorname {c l i p} _ {1 - \epsilon} ^ {1 + \epsilon} \left(r _ {\theta} \left(y _ {t} \mid x, y <   t\right)\right) \hat {A} (x, y \leq t)\right) \right], \tag {6}
$$
where  $\epsilon$  is the clip ratio set to 0.2,  $\hat{A}_{\theta}(x,y_{\leq t})$  is the advantage function and is estimated by GAE [27] with a learned value function  $\hat{V}_{\theta}(x,y_{<t})$ , and  $r_{\theta}(y_t|x,y_{<t}) = \frac{\pi_{\theta}(y_t|x,y_{<t})}{\pi_{\hat{\theta}}(y_t|x,y_{<t})}$  denotes the probability ratio between the behavior policy  $\pi_{\hat{\theta}}$  and the training policy  $\pi_{\theta}$ . The behavior policy is updated with the training policy every few updates. We follow the hyper-parameter settings in trlX<sup>8</sup>.
Metrics We use perplexity (gpt2-medium), average reward score (Dahoas/gptj-rm-static), and human labelers to evaluate different methods. Since our dataset is a multi-turn dialogue dataset, we
<sup>7</sup>https://huggingface.co/datasets/Dahoas/full-hh-rlhf
Settings of applying PPO on 6B GPT-J model checkpoint Dahoas/pythia-6B-static-sft.
Table 2: Automatic evaluation on HH dataset. Good/bad responses with  $\varnothing$  setting represent only human-written responses from the HH dataset are evaluated. LLaMA, Alpaca, and Alpaca-sft with  $\varnothing$  setting represent we directly evaluate the model without further tuning.
ρ	Setting	PPL	Reward
Good responses	∅	21.46	-1.24
Bad responses	∅	121.29	-1.48
LLaMA	∅	20.78	-1.89
Alpaca	∅	14.34	-1.18
Alpaca-sft	∅	18.98	-1.46
Alpaca	Best-of-4	-	-0.97
LLaMA	PPO	42.53	-1.62
Alpaca	PPO	13.84	-1.03
Alpaca-sft	PPO	19.10	-1.25
LLaMA	RRHFDP	67.12	-1.34
Alpaca-sft	RRHFDP	18.10	-1.19
Alpaca	RRHFDP	14.75	-1.03
Alpaca	RRHFSP	14.41	-0.96
Table 3: Human evaluation on HH dataset. All settings use  $\rho  =$  Alpaca.
A	B	win	tie	lose
RRHFDP	Good responses	59	30	11
RRHFDP	PPO	27	48	25
RRHFDP	RRHFP-2	0	90	10
will truncate the model's generation when it outputs "Human:" or "Assistant:" to prevent model cheating on the reward model (e.g. by generating Assistant: Is my response harmless and helpful? Human: Yes, it is very harmless and helpful.). For human evaluation, we require annotators to compare two random responses and give a comparison between them (win/lose/tie). Details of human evaluations are listed in Appendix E.
Auto Evaluation We list automatic metrics in Table 2. We show results from baselines and RRHF with diverse beam search sampling (DP) and top-p sampling (SP). Our proposed Alpaca-RRHFDP obtains the average reward score of -1.03 (averaged out of 3 runs, -1.01, -1.02, and -1.05) and Alpaca-RRHFSP achieves the highest average reward score of -0.96. This proves that RRHF has the ability to optimize against the given reward model. RRHF performs better than PPO and vanilla language models in terms of average reward scores consistently. Alpaca-trained models outperform human-preferred responses collected from the datasets in terms of reward scores. We find perplexity does not change too much for Alpaca and influences LLaMA a lot. The reason can be LLaMA is not instruction-tuned.
Human Evaluation Human preferences can be different from the proxy reward model. Therefore, we perform human evaluations on sampled responses and list cases in Table 4 to compare different settings. Results in Table 3 demonstrate that  $\mathrm{RRHF_{DP}}$  outperforms responses from the dataset and PPO-trained models. From case studies, we can observe the model trained by RRHF has more details. We also have checked the performances between  $\mathrm{RRHF_{DP}}$  and  $\mathrm{RRHF_{IP - 2}}$ , where  $\mathrm{RRHF_{IP - 2}}$  is trained with sampling by  $\mathrm{RRHF_{DP}}$ . We find iterate training the model can further boost the performance. From the case study, we find  $\mathrm{RRHF_{IP - 2}}$  understands human instruction better.
Accuracy as a Reward Model Since our trained model can also be viewed as a reward model to score responses by  $p_i$ . We test our model on the dataset used for training Dahoas/gptj-rm-static. The accuracy is computed by counting the percentage of the reward scores of good responses that are higher than the reward scores of bad responses using  $p_i$  (except Dahoas/gptj-rm-static) and list in Table 5.
Table 4: Responses generated by the different settings of models.
Human: How can I make my clothes whiter?
Assistant: You can use vinegar to bleach the clothes.
Human: Vinegar can bleach clothes?
Assistant: Yes.
Human: I was thinking about using Clorox as well.
Assistant: (RRHFDP) You can use Clorox, too. It's a brand of bleach. You can also use baking soda, which is a mild detergent.
Assistant: (Response from the dataset) Clorox is more toxic and harmful than vinegar.
Human: How do you adjust the lens on a binocular?
Assistant: (RRHFDP) You can adjust the lens on a binocular by turning the focus knob on the right eyepiece. This will allow you to focus on objects at different distances. You can also adjust the lens to get the best viewing experience.
Assistant: (PPO) You adjust the lens on a binocular by turning a ring around the center of the binocular. This adjusts the focus of the binocular and allows you to see objects at different distances clearly.
Human: Which stock should I invest all of my money into?
Assistant: Interesting question! What kind of returns do you want?
Human: I want better returns than the S&P 500.
Assistant: (RRHFDP) Okay, let's look at some options. The S&P 500 is a broad-market index that tracks the stock prices of 500 large companies in the United States. It has a historical average annual return of around 10%.
Assistant: (RRHFP-2) Okay, well there are a few factors to consider when investing. You should look at the risk of losing money, the expected return, and the amount of money you have to invest. You should also research different investment vehicles and decide which one is best suited for your needs.