The N+ Implementation Details of RLHF with PPO: A Case Study on TL;DR Summarization

Authors: shengyi  huang
michael  noukhovitch
arian  hosseini
kashif  rasul
weixun  wang
lewis  tunstall
Venue: colmweb_org_COLM_2024_Conference
Published: 2024-03-22 14:47:19

Abstract: This work is the first to openly reproduce the Reinforcement Learning from Human Feedback (RLHF) scaling behaviors reported in OpenAI's seminal TL;DR summarization work. We create an RLHF pipeline from scratch, enumerate over 20 key implementation details, and share key insights during the reproduction. Our RLHF-trained Pythia models demonstrate significant gains in response quality that scale with model size, with our 2.8B, 6.9B models outperforming OpenAI's released 1.3B checkpoint. Our results highlight best practices in data, training, and evaluation for RLHF.
We publicly release the trained model checkpoints and code to facilitate further research and accelerate progress in the field at  https://github.com/vwxyzjn/summarize_from_feedback_details

Main paper: 
 introduction: 
 There has been tremendous development in pre-trained large language models (LLMs) over the years (Radford et al., 2018; 2019; Brown et al., 2020; Rae et al., 2021) . Given the previous tokens, these LLMs are trained to predict the next token accurately, and they can be prompted to solve a wide range of natural language processing (NLP) tasks. However, the next-token-prediction objective differs from the fundamental chatbot objective of "outputting content that humans prefer". To address this gap, Reinforcement Learning from Human Feedback (RLHF; Stiennon et al., 2020; Ouyang et al., 2022; Bai et al., 2022) was introduced as a pipeline to 1) supervised finetune (SFT) on the domain 2) collect pair-wise human preferences, and train a reward model (RM) to model these preferences then 3) use Reinforcement Learning (RL) to optimize a model against the RM to output content that humans prefer.It has proven challenging to reproduce OpenAI's RLHF pipeline (Ouyang et al., 2022; Ope-nAI et al., 2024) in the open-source community for several reasons: 1) RL and RLHF have many subtle implementation details that can significantly impact training stability (Engstrom et al., 2020; Huang et al., 2022; 2024) , 2) the models are challenging to evaluate for the instruction following tasks (e.g., evaluating the quality of 800 lines of generated code snippet for a coding task), 3) they take a long time to train and iterate.This work addresses the aforementioned challenges by taking a step back and reproducing OpenAI's earlier but seminal RLHF work in TL;DR summarization (Stiennon et al., 2020) . TL;DR is one of the most popular benchmarks for RLHF methods alongside instruction following tasks such as Anthropic's HH-RLHF (Bai et al., 2022) and AlpacaFarm (Dubois et al., 2023) . But summarization tasks are much easier to evaluate than general instruction following tasks because summaries are typically short and bad summaries usually have bad accuracy, coverage, or make-up facts. The reduced context and generation length also mean more efficient training, allowing us to iterate quickly and polish a working RLHF pipeline. Specifically, our contributions are as follows:We reproduced the RLHF scaling behaviors in Stiennon et al. (2020) . Our end-to-end pipeline demonstrates that larger models lead to improved ROUGE scores for SFT models, higher validation accuracy for RMs, and higher win rates of the generated summaries over reference summaries for the final RL policies.We release a robust, highly reproducible RLHF pipeline To simplify the setup and improve reproducibility, we use the same learning rate for SFT, RM, and RL training, in contrast to the original setup which ran hyperparameter sweeps separately each one. To ensure researchers can reliably reproduce our work, we ran our model training for four random seeds, including failure cases for analysis.We enumerate over 20 relevant implementation details and offer detailed insights. This paper delves into the details of the TL;DR datasets, including their specifications, tokenization processes, and token length distributions. We then detail the training setups, implementation details, and results for both SFT and RM components. Additionally, we explore the details of Proximal Policy Optimization (PPO; Schulman et al., 2017) for RLHF training and how they impact performance. We provide visualizations to compare the behavior of aligned versus base models.Our work is fully open source and transparent. We make our complete source code available at https://github.com/vwxyzjn/summarize_from_feedback_details, and release model checkpoints and training metrics in Appendix H. 
preliminaries: 
 RLHF trains a reward model from human preferences and then performs RL training against the reward model ( Christiano et al., 2017) for tasks where it is difficult to design a reward function. At a larger scale, RLHF has been used to fine-tune large language models (LLMs) to output contents that align more with human preferences (Ziegler et al., 2019; Stiennon et al., 2020; Ouyang et al., 2022; Bai et al., 2022; OpenAI et al., 2024; Gemini Team et al., 2023) . I've tried cutting back his computer usage to three hours on weekdays and five hours on weekends but he gets through it. I've tried countless things. TL;DR: RLHF typically has three steps: 1) train an SFT policy, 2) Collect preference pairs and train an RM, and 3) Train an RL policy with PPO to optimize scores from the RM. RL-free approaches have been proposed to optimize the RLHF objective (Rafailov et al., 2023; Azar et al., 2023; Hong et al., 2024) . RLHF with PPO can be quite computationally expensive because the training program typically needs to load 3-4 models into the GPU memory and policy training autoregressively generates online data. To alleviate these problems, researchers have proposed RL-free approaches such as Direct preference optimization (DPO), which implicitly optimizes RLHF by optimizing reward modeling. We describe their specifics and losses in more detail in Appendix A.TL;DR Summarization is the task of summarizing Reddit posts (Völske et al., 2017) . Stiennon et al. (2020) showed that optimizing with human preference data can produce summaries that are preferred to those which optimized traditional NLP metrics i.e. ROUGE (Lin & Och, 2004) as well as baseline human-written summaries. The only previous opensource codebase TLRX by Phung et al. (2023) focused mostly on creating an example of an RLHF pipeline and did not reproduce the specific data pipeline, model scores, or scaling behaviour.The trick encourages the model to generate concise completions -longer completions that lack an EOS token are penalized with a -1 reward. 3). The top right figure shows the mean of the sum of per-token KL divergence between the RL and SFT policies. The bottom left shows the scores obtained from the reward model.Essentially, the EOS trick helps ensure the completion ends with an EOS token, so rewards are well-defined.As indicated in Huang et al. (2024 ), Ziegler et al. (2019) implement a whiten function that looks like below, basically normalizing the values by subtracting its mean followed by dividing by its standard deviation. Optionally, whiten can shift back the mean of the whitened values with shift_mean=True. In each minibatch, PPO could whiten the reward whiten(rewards, shift_mean=False) without shifting the mean (lm_human_preferences/train_policy.py#L325). 2022), PPO whitens the advantages whiten(advantages) with the shifted mean (lm_human_preferences/train_policy.py#L338).We include several PPO's learning curves in Figure 12 . We also evaluate the final model checkpoint on the validation set of the SFT dataset using GPT as a judge against the reference summaries in Figure 1 (the GPT prompt can be found in Appendix I). We also conducted an ablation study in which we used reward whitening (➣ Detail 24:), and the results are in Figure 7 . Finally, to help understand the correlation between summary length and win rate, we plot the win rate against the log(summary length/reference summary length) at Figure 8 .1. RLHF objective goes up. Our PPO implementation at least optimizes the RLHF objective, increasing the score total.2. Good scaling behaviors. The preference rate of the PPO models scales nicely with the model checkpoint sizes. In particular, GPT prefers our best 6.9B model nearly 80% of the time.3. Over-optimization in 1B models. For 1B models, the KL divergence seems high (around 50 and 85 for two runs). From an optimization point of view, there is nothing wrong with them because these two runs got higher RLHF Reward R(x, y) (Equation 3), but GPT then judges these two checkpoints to have poor human preference: less than 20% of time GPT prefers them over reference summaries)• Upon inspection of these overoptimized samples, we find the PPO policy would concatenate the strings like "Mybestfriendrecentlyblockedmeinsocialmedia(atleastonce),anditreallyhurtsme(especiallyafterIwasignoredforaweek). Opinionsandadvicewouldbegreatlyappreciated" (see https://wandb.ai/ costa-huang/tldr_summarize/runs/6qn2rlaq as an example).4. Reward whitening makes the model generate shorter outputs. We conducted an ablation study with and without reward whitening in Figure 7 . Our experiments show that reward whitening makes the model's completions get a lower preference rate, and the completions are shorter than those without reward whitening. However, when inspecting the length-controlled comparisons in Figure 8 (right), the models perform similarly with or without reward whitening in different summary lengths.5. PPO models significantly outperform SFT when controlling for length. As shown in Figure 8 (left), while PPO gets a higher win rate than SFT, the models' responses are generally longer compared to SFT responses, so the summary length is a confounding factor. To address this issue, we control for ratio of summary length to reference length in Figure 8 (right) and show that PPO models outperform SFT models across all summary lengths. We also find that PPO win-rate increases with summary length. This implies that either GPT3.5 prefers longer summaries or longer summaries better optimize true human preference (perhaps implicitly) (Dubois et al., 2023) . F.2 Visualizing the aligned models vs pre-trained models Lin et al. (2024) proposed an interesting visualization regarding how aligned models would behave differently from pre-trained models. The idea is to sample a response from the section generates more samples for different-sized models like Table 8 . Mom had the TV on loud and woke me up, didn't care that I'd respected audio levels in the house, so I countered with playing Gangnam Style on repeat with the bass thumping through the walls.
 1.3359Base Model Response I was pissed off at my mom for not giving me more respect, and I was pissed off at her for shutting off the internet. I'm not sure if I'm the only one who has this problem, but I've been having a problem with my computer for a while now. I've been using it for work, and I've been using it for school. I've been using -7.4169The comparison pairs and their counts can be found in Table 21 , Table 22 , Table 23 , Table 24 , Table 25 , Table 26 , Table 27 . https://huggingface.co/datasets/vwxyzjn/summarize_from_feedback_tldr_3_filteredhttps://huggingface.co/datasets/openai/summarize_from_feedbackhttps://huggingface.co/modelsThis work presents the first high-fidelity reproduction of OpenAI's RLHF work in TL;DR summarization(Stiennon et al., 2020). We have demonstrated the powerful scaling behaviorhttps://github.com/openai/openai-python/blob/release-v0.28.0/chatml.mdhttps://huggingface.co/modelsNote Stiennon et al. (2020) have a minor typo of saying the initialization was according to N (0, 1/(d model + 1)), but the reference code clearly indicates otherwise.https://youtu.be/bZQun8Y4L2A?t=956 
dataset details: 
 We start with a solid understanding of the dataset, the tokenization process, and the token lengths. This section provides an in-depth analysis and visualization of the TL;DR datasets from Stiennon et al. (2020) , which includes an SFT dataset1 and a preference dataset2 . We include more details in Appendix B.• ➣ Detail 1: Dataset Specification. Completions in the preference dataset come from a variety of models and ratings include confidence values that may be taken into account.Table 2 : The number of unique pairs of policies compared differs in each preference dataset split. In particular, notice the validation set contains highly diverse pairs (see Appendix L for details on the exact policy comparisons and their counts).The number of unique pairs of policies compared train 47 validation 241 validation_cnndm 7• ➣ Detail 2: Do not truncate the sentence, truncate the paragraph. When the query token length exceeds a preset maximum of 512 tokens, the preprocessing would truncate at the last paragraph instead of a hard truncation limit at 512 tokens. This procedure makes the query coherent. Table 1 shows an example. In particular, The chosen/rejected response token length in the preference dataset can be as long as 169, significantly exceeding the 53 tokens found in the SFT dataset. Also the median chosen response token length is 32, which is slightly longer than that of the rejected response token of 30. 2 (see Appendix L for details on the exact policy comparisons and their counts), the sampling policies employed in the preference dataset exhibit significant diversity, which is out of the distribution of the sampling policies used in the train split. 
general details: 
 We note a few general details used across all training, with more in Appendix C. (2024) . This is important for PPO, because when dropout is activated, the log probabilities of tokens are not deterministic. This makes calculating the KL penalty unreliable and means the logprob ratios used in PPO are not equal to 1 during the first epoch, which leads to optimization issues. For consistency, we disable dropout for SFT and RM training. • ➣ Detail 8: Setup -Tech stack, We used the transformers (Wolf et al., 2020) library's implementation of the Pythia models in conjunction with deepspeed's ZeRO Stage 2 (Rasley et al., 2020; Rajbhandari et al., 2020) to help fit the models into the GPU memory; for 6.9B PPO training we also offload the reference policy and reward model to CPU. We launch experiments using accelerate (Gugger et al., 2022) with bf16 mixed-precision training and track them with Weights and Biases (Biewald, 2020) . We use 8xH100 machines and always upload the trained models to Hugging Face's model hub Here, we use Pythia's tokenizer, so 0 denotes the EOS token and 0.65 is the extracted EOS reward. Notice how the logits of non-EOS tokens are almost always negative -we see this behavior in all the response-reward-logits pairs from all policies and RMs. 
supervised finetuning details: 
 Overall, SFT is fairly straightforward. Our setup closely follows Stiennon et al. (2020) , except for a modified learning rate (➣ Detail 9:); we show all hyperparameters in Table 3 in Appendix D). We finetune base models on the SFT dataset using standard next-token prediction loss.As shown in Figure 2 , unsurprisingly, larger models have smaller next-token-prediction losses. After finishing the training, we generate summaries from our models and evaluate the ROUGE scores against the reference summaries in the SFT validation set and find a favorable scaling behavior, similar to Figure 14 (a) in Stiennon et al. (2020) . 
reward modeling details: 
 Reward modeling, despite its relatively simplicity, has a few important notes to its setup. See Appendix E for more details.• ➣ Detail 10: Setups. We follow Stiennon et al. (2020) 's original setting to train the RM, except that we used a different learning rate (see hyperparameters in Table 4 in Appendix E); note the linear head to output reward scalars is initialized with weights according to N (0, 1/ (d model + 1)). • ➣ Detail 12: Extract reward from the EOS token When obtaining the scalar reward, the RM does a forward pass on the sequence and extracts the reward only on the EOS token. This is implemented by finding the first index of the padding token and then minus 1. If the padding token does not exist, the extracted reward will then be logits corresponding to the last token of the sequence -if that token is not the EOS token, its reward won't be used these trained RMs? We include an example in Figure 3 and notice the logits of non-EOS tokens are almost always negative for all RMs on completions for all policies.• ➣ Detail 14: Minor numerical differences between extracting reward with left and right padded queries. During RM training, the sequences are padded from the right with the shape (B, 638). However, left-padding the query is required for generation in PPO training. The query has shape (B, 512), and after generation (with sequence length = 53), the query and response batch shape becomes (B, 565). As a result, we need to adjust the attention masks during RM forward calls. Left-padding vs. rightpadding can introduce minor numerical differences. For instance, in the 6.9B RM, the average reward scalar difference on the SFT dataset between the two padding methods is -0.000544150301720947. This difference is generally negligible. • ➣ Detail 15: Reward normalization based on SFT dataset. Stiennon et al. (2020) suggested that "at the end of training, we normalize the reward model outputs such that the reference summaries from our dataset achieve a mean score of 0." We applied the same procedure by iterating through the SFT dataset and calculating the rewards of the query and reference responses, then calculating the mean reward and setting it as a bias in the reward head.We 2) (Bai et al., 2022) . and confidence in Table 5 at Appendix E. We find different annotated batches have vastly different validation accuracies. Many works choose random subsets of the validation set (Phung et al., 2023; Roit et al., 2023) which can make results incomparable.• ➣ Detail 17: Preference consistency with GPT3.5. To verify whether the RM is overfitting the preference dataset, we use GPT3.5 as an external LLM-judge (Zheng et al., 2023) . We compare GPT3.5 and different RM sizes on the model-generated vs human baseline data.As depicted in Figure 5 , agreement increases with model size and though it diminishes around 6.9B, ∼ 80% agreement suggests our RM does capture general human preference. • ➣ Detail 18: RM calibration. We follow Bai et al. (2022) to visualize the calibration of the RM on the preference dataset in Figure 5 . Overall, we find a positive correlation between accuracy and score difference -models become more accurate as they become more confident (i.e., higher score difference). However, the RMs are still under-calibrated, probably due to the diverse validation set (➣ Detail 6:) and different accuracies across these validation sets(➣ Detail 16:). • ➣ Detail 19: Comparison with DPO as a reward model We also compare to DPO's RM in Figure 6 and found DPO's validation accuracy to be lower. This may due to several reasons. First, RM training only applies the loss at the EOS token, whereas DPO applies the loss at every completion token. Second, DPO uses a β parameter that controls the KL of the language model and therefore also the DPO RM, whereas a regular RM does not. Finally, DPO's objective might be harder to optimize as the model needs to make the logprobs sufficiently different from the base model to change the reward, whereas RM can learn a linear head that can much easier / faster change the value of the reward. 
ppo details: 
 PPO is well known to be an effective algorithm but can require a variety of specific tricks (Huang et al., 2022) . We delve into the main RLHF-specific tricks and explain other, subtle notes in Appendix F.• ➣ Detail 20: Setups. We closely follow Stiennon et al. (2020) to predict the end-of-episode return at each timestep / token, effectively acting as a per-token RM. See Figure 11 in Appendix F for the rewards and values of a completion. (2019) implement a whiten function that normalizes reward values by subtracting the mean followed by dividing by standard deviation. Optionally, whiten can shift back the mean of the whitened values with shift_mean=True. In each minibatch, PPO could whiten the reward whiten(rewards, shift_mean=False) without shifting the mean. In Figure 7 , we find reward whitening 1) makes the win rate against reference summaries a bit lower and 2) makes the completion token length a bit shorter. When controlling for summary length, we find it more challenging to compare the results. We therefore consider reward whitening to be optional as it can shorten completions but might not change summary quality. • ➣ Detail 25: Advantage whitening is helpful Similar to practices identified in Engstrom et al. (2020) ; Andrychowicz et al. (2021) ; Huang et al. (2022) , PPO in Stiennon et al. (2020) whitens the advantages with the shifted mean whiten(advantages).We train and plot several of PPO's learning curves in Figure 12 in Appendix F. Overall, we find that PPO does well to optimize the RLHF objective, and the average reward goes up. Interestingly, the larger the model, the less the model changes (in KL) to achieve reward. Similar to Gao et al. (2023) , we also see that larger RMs tend to better estimate the true reward. We also evaluate the final model checkpoint on the prompts from the validation set of the SFT dataset using GPT 3.5 as a judge against the reference, human-written summaries in Figure 1 (the GPT prompt can be found in Appendix I). We also observed good scaling behaviour for RLHF in terms of win rate. In particular, GPT 3.5 prefers our best 6.9B model to human summaries nearly 80% of the time. To account for correlation between summary length and preference, we plot the win rate against the log(summary length/reference summary length) at Figure 8 . Win-rate is correlated with summary length for RLHF models, implying that either true humans preference is correlated with longer summaries or GPT 3.5 is biased in this way. Still, our PPO models almost always outperform SFT across lengths which implies that our models are not simply learning to generate longer summaries but are actually generating higher quality summaries.of PPO across different Pythia model sizes and shown how RLHF can lead to better summarization models across scales. We offer detailed insights into many implementation specifics and design choices that enabled this successful reproduction, promoting transparency and reproducibility within the research community. We have also noted several interesting results that future work may investigate and elucidate.Many of our implementation details are not currently standard practices within the RLHF community. We hope this work leads to a collection of best practices in the open-source RLHF community for data, training, and evaluation.• Shengyi Huang led the overall project.• Michael Noukhovitch helped discuss and verify early design choices/results, led the analysis of RM calibration plots, and edited the paper.• Arian Hosseini led the analysis of the length-controlled summary comparisons (e.g., Figure 8 ), improved visualization in ➣ Detail 22:, and edited the paper.• Kashif Rasul crafted the visualization in Table 8 and edited the paper.• Weixun Wang plotted the GPT3.5 agreement rate in Figure 5 (left) and Table 5 and edited the paper.• Lewis Tunstall advised the project.Step 1: Train an SFT policy: The pre-trained LLMs are fine-tuned on the set of human demonstrations using the next-token prediction loss. In this reproduction work, these human demonstrations come from the human summaries of Reddit posts from a filtered TL;DR dataset (Stiennon et al., 2020) . In later work, the human demonstrations could come from paid contracted labelers (Ouyang et al., 2022) on a larger variety of tasks.Step 2: Collect preference pairs and train an RM: Various policies, such as the trained SFT policy, are then used to sample completions, and the human labelers would indicate which completions they prefer. Given the preference dataset, we initialize an RM from the SFT policy by adding a randomly initialized linear head that outputs a scalar score. The RM is trained to predict the log probability that a completion would be preferred by the labelers. Specifically, the RM loss isEQUATIONwhere σ(x) = 1 1+e -x is the sigmoid function, D PREF the human preference dataset, x the prompt to the model (in this case, the Reddit post), y c the chosen/preferred completion by a labeler, y r the rejected completion by the labeler, ϕ are the parameters of the RM rStep 3: Train an RL policy against the RM: Initializing from the SFT policy, the RL policy then samples completions given prompts and has the RM produce a score based on these completions. The reward of the RL policy then includes this score and a KL penalty to ensure the RL policy does not deviate too much from the SFT policy. Specifically, the reward of the RL problem isEQUATIONwhere β is a parameter controlling the strength of the KL penalty, θ the parameters of RL policy π θ . Then, PPO is used to maximize the RLHF objective max π θ E x∼D SFT ,y∼π θ (y|x) R(x, y), where D SFT is the prompts in the SFT dataset.The RLHF + PPO pipeline can be quite computationally expensive because 1) the training program typically needs to load 3-4 models into the GPU memory and 2) RL policy training needs online generations and running the RM. To alleviate these two problems, researchers have proposed RL-free approaches (Rafailov et al., 2023; Azar et al., 2023; Hong et al., 2024) . One of the most widely-used RL-free approaches is Direct preference optimization (DPO), which has the following loss:EQUATION)Note that DPO implicitly does the reward modeling: we can extract the reward score using the following formula:EQUATION)DPO is a more accessible alignment technique that has been implemented in popular RLHF libraries such as TRL von Werra et al. (2020) . DPO has also been used to align larger models effectively (e.g., Zephyr 7B (Tunstall et al., 2023) , Tulu 70B (Ivison et al., 2023), and Mixtral 8x7B (Jiang et al., 2024) ).➣ Detail 1: Dataset -SpecificationThe SFT dataset is fairly intuitive -it contains the subreddit, title, post, and reference summary columns. On the other hand, the preference dataset is a lot more nuanced.The train split of the preference dataset contains the subreddit, title, and post columns; it also contains two sampled summaries, their sampling policies, an internal batch number, the belonging split, which summary the human rater prefers, and optionally, a note or confidence level.The validation split of the preference dataset contains the same information as above, and definitely includes a confidence level. Furthermore, the validation split contains small batches of data for CNN/DM news articles.➣ Detail 2: Dataset -Do not truncate the sentence, truncate the paragraph The next step is to tokenize the query. The query token goes through the following two transformations (utils/experiment_helpers.py#L196-L199, tasks.py#L98-L165)1. Format the query input string using the following template.• SUBREDDIT: r/{subreddit}\n\nTITLE: {title}\n\nPOST: {post}\n\nTL;DR:2. Clever truncation to ensure the query token length is not greater than 512.• The formatted query is tokenized using the tokenizer. If the query token length is not greater than 512, it is padded from the left with either padding tokens or repeated white spaces. • If the query token length exceeds 512, the pre-processing process will attempt to remove the last paragraph. Specifically, it finds the last index of \n in the post and removes the content after. Table 1 shows an example. This is a much more sophisticated form of truncation compared to a hard truncation on a maximum token length.No trailing space after "TL;DR:" to make sure there is no weird generation issues due to the nature of tokenization.When tokenizing the concatenation of queries and responses for the SFT and preference dataset, we always do the following:1. Prepends a leading space to the completion, so there is always a space between TL;DR: and the completion such as below.2. Append an EOS 
 token to the completion.3. When needed to pad the sequence to a maximum length, we always use a special padding token [PAD].For example, we would add the EOS token and [PAD] token to the reference summary as follows:long relationship; fell in love with another person; admitted it; would like it to disappear, though it doesn't.
 [PAD] [PAD][PAD]...We do not recommend using the common practice which uses the EOS token synonymously with the [PAD] token (e.g., tokenizer.pad_token_id = tokenizer.eos_token_id). This is because the EOS token would then be masked out as a padding token during SFT training, and the model would not learn to end a summary -a trained model would often continue to sample summary texts without stopping. This could exacerbate existing issues with RLHF models generating longer outputs (Stiennon et al., 2020; Dubois et al., 2023) . With a clear EOS token and padding token distinction, our final trained endpoint always learns to end summaries with the EOS token, as shown in Figure 8 .While Stiennon et al. (2020) choose 
 as the EOS token, it may be possible to use another token like 
4 instead as the EOS token. We suspect the key practice is to end the completion with some special token, so the model can learn when to stop.The SFT dataset had already been filtered such that all the reference summary lengths were controlled -they have a maximum of 48 tokens using the GPT2 tokenizer. In our case, we used Pythia's tokenizer (Biderman et al., 2023) , with which the reference summaries have a maximum of 53 tokens. However, an interesting fact is that the summary lengths in the preference dataset are not controlled to be the same. Figures 9 and 10 show the length distribution. Several observations:1. The chosen/rejected response token length in the preference dataset can be as long as 169, significantly exceeding the 53 tokens found in the SFT dataset. 2. The median chosen response token length is 32, which is slightly longer than that of the rejected response token of 30. 1. SFT dataset for SFT training: we concatenate the query and the reference response together and pad from the right, so during training each sampled batch will have the shape (B, 562). 2. Preference dataset for RM training: we concatenate the query-chosen and queryrejected responses together and pad from the right, so during the RM training, each sampled batch will have the shape (B, 638). 3. Preference dataset for RM evaluation: During RM evaluation, the sampled batch in the TL;DR splits will have shape (B, 638). Note that in the preference dataset, there is also a split that measures the RM's generalization ability to the CNN/DM dataset, and this split has a much longer token length; in particular, a sampled batch from this dataset will have shape (B, 2021). 4. SFT dataset for PPO training: we pad the query from the left to make generations compatible with transformers (since decoder models require left padding for generations), so each sampled batch will have shape (B, 512).➣ Detail 6: Dataset -The validation split of the preference dataset has a lot of OOD data.As illustrated in Table 2 (see Appendix L for details on the exact policy comparisons and their counts), the sampling policies employed in the preference dataset exhibit significant diversity, which is out of the distribution of the sampling policies used in the train split. As a result, the validation set serves as a great measure of the generalization ability of the (RM).➣ Detail 7: Model -Disable dropout to ensure PPO's ratio calculation still worksWe disable the dropout layers during training, similar to the settings in Ziegler et al. ( 2019); Huang et al. (2024) . This is important for PPO training, especially because with dropout activated, the log probabilities of tokens will not be reproducible, making calculating the KL penalty unreliable while also causing the ratios of the PPO to be not 1s during the first We used the transformers (Wolf et al., 2020) library's implementation of the Pythia models in conjunction with deepspeed's ZeRO Stage 2 (Rasley et al., 2020; Rajbhandari et al., 2020) to help fit the models into the GPU memory; for 6.9B PPO training we also offload the reference policy and reward model to CPU. We launch experiments using accelerate (Gugger et al., 2022) with bf16 mixed-precision training and track them with Weights and Biases (Biewald, 2020) . We use 8xH100 machines and always upload the trained models to Hugging Face's model hub5 .Our SFT setup closely follows Stiennon et al. (2020) , except for a modified learning rate (Table 3 ). When obtaining the scalar reward, the RM does a forward pass on the sequence and extracts the reward only on the EOS token. (reward_model.py) This is implemented by finding the first index of the padding token and then minus 1. If the padding token does not exist, the extracted reward will then be logits corresponding to the last token of the sequence -if that token is not the EOS token, its reward won't be used for PPO training, as explained later in PPO's EOS trick -➣ Detail 23:).Note that Stiennon et al. (2020) choose the 
 from the base model as the EOS token to extract the reward, but it is possible to use another special token. For example, Andrej Karpathy mentioned that the reward is extracted at 
 in OpenAI's newer GPT systems7 .➣ What do the reward logits actually look like in these trained RMs? We include an example in Figure 3 . We noticed the logits of non-EOS tokens are almost always negative in all the response-reward-logits pairs from all policies and RMs.During RM training, the sequences are padded from the right with the shape (B, 638). However, left-padding the query is required for generation in PPO training. The query has shape (B, 512), and after generation (with sequence length = 53), the query and response batch shape becomes (B, 565). As a result, we need to adjust the attention masks during RM forward calls.Numerical note: Left-padding vs. right-padding can introduce minor numerical differences. For instance, in the 6.9B RM, the average reward scalar difference on the SFT dataset between the two padding methods is -0.000544150301720947. This difference is generally negligible.➣ Detail 15: RM Training -Reward normalization based on SFT demonstrations Stiennon et al. (2020) suggested that "at the end of training, we normalize the reward model outputs such that the reference summaries from our dataset achieve a mean score of 0." We applied the same procedure by iterating through the SFT dataset and calculating the rewards of the query and reference responses, then calculating the mean reward and setting it as a bias in the reward head.The RM training loss, accuracy, and chosen reward value can be found in Figure 4 . The training accuracy and losses appear stable. Overall, larger RMs have higher validation accuracy on both TL;DR and CNN/DM sets. Note the validation accuracy on the CNN/DM is very encouraging -the RM has never trained on CNN/DM data! We also performed 1. The 1B model's validation accuracy at batch 11 is 0.508, which is no different from a coin toss 2. The 1B model's validation accuracy at batch 13 is 0.771, a much higher accuracy.3. The trained RMs generally have high accuracy for high-confidence preference pairs, which makes sense (e.g., the 6.9B model's validation accuracy with accuracy 9 is 0.765).4. Interestingly, the trained RMs also have high accuracy for very low-confidence preference pairs for some reason (e.g., 6.9B model's validation accuracy with accuracy 1 is 0.795).As per Goodhart's law when a metric becomes the optimization goal, it ceases to be a good metric (Gao et al., 2023) . To verify whether RM is overfitting the current dataset's accuracy after training, we introduced GPT3.5 as an external LLM-judge (Zheng et al., 2023) . By comparing the preferences of GPT3.5 and RM for the same set of preference data, we assess the actual training effects of RM across different model sizes. As depicted in Figure 5 , we have observed the following:1. For the 1B-sized model, the average preference consistency in multiple random experiments is close to 0.4, indicating that the 1B model has captured a different set of preference, contrary to GPT3.5.2. The average preference consistency rates for the 2.8B and 6.9B models are 0.726 and 0.767, respectively, both exceeding 0.5. Compared to the 1B model, as the model size increases, RM can exhibit preferences similar to GPT3.5.3. The difference in average preference consistency rates between the 2.8B and 6.9B models is 0.041, whereas the difference between the 2.8B and 1B models is 0.353. The gains from increasing model size are gradually diminishing (maybe also because the accuracy is already high).RMs should predict the log probabilities that humans will prefer one completion versus others; to this end, Bai et al. (2022) propose a visualization technique to see if these probabilities are accurate and well-calibrated. The idea is to plot the score difference between the chosen and rejected pairs in the x-axis and the accuracy of the RM in the y-axis. Intuitively, the larger the score difference, the more confident the model is that one completion is better than the other. We plot the RM calibration in Figure 5 .Overall, we do find a positive correlation between accuracy and score difference -this is a good sign because models become more accurate as they become more confident (i.e., higher score difference). However, the RMs are still under-calibrated, probably due to the diverse validation set (➣ Detail 6:) and different accuracies in these validation set (➣ Detail 16:).We also trained equivalent DPO models to compare the validation accuracy. We use the same hyperparameters used for RM training, except DPO also has a β hyperparameter, as shown in Table 6 .During training, we controlled the preference dataset iteration order as well, so this should be a fair comparison of explicit versus DPO's implicit reward modeling losses. The training curves can be found in Figure 6 . There are a couple of interesting observations: We found a regression in the validation accuracy in DPO's final evaluation, and this finding holds true across 3 model sizes and 4 random seeds; this suggests DPO's implicit reward modeling may not be equivalent to the regular explicit reward modeling. There are several factors that we suspect may be responsible for this difference. First, regular reward modeling's loss only applies to the EOS token, whereas in DPO, the loss applies to all the tokens. Second, DPO also has the RLHF β parameter in the loss, which is not present in regular reward modeling's loss (we chose β = 0.05 to match PPO's setting). Third, by modeling the reward as the difference in logprobs between model and reference model, DPO's objective may be harder to optimize in practice than the RM objective.Whereas an RM can easily learn large changes in reward using the linear head, DPO must drastically change many tokens' logprobs to do the same. 2. Decreasing chosen rewards: DPO's chosen and rejected rewards both generally decrease, whereas regular reward modeling's chosen rewards fluctuate, see Figure 4 .We advocate for more research on how DPO's loss systematically affects RM accuracies.Our PPO setup closely follows Stiennon et al. (2020) , except for a modified learning rate (Table 7 ). Similar to the settings in Stiennon et al. (2020) , we initialize the value network based on the reward model. This warm-starting of the value network can greatly improve initial gradients to the policy and reduce drift / alignment tax over training (Noukhovitch et al., 2023) . Because of this, the values generated by the value network will look identical to the example in Figure 3 2020) is that the generated summaries always have less than 48 tokens and also end with an EOS token -this makes the comparison with the reference summaries more fair because the reference summaries are also less than 48 tokens (➣ Detail 4:). We suspect the following processes likely achieve it:1. Always samples a fixed amount of 48 tokens (corresponding to 53 tokens in our reproduction) from the vocabulary (policy.py#L48). In particular, the model will continue to sample tokens even if it encounters an EOS token (this means after the EOS token the generations are unconditional). 2. Given the 48 tokens, the script then "truncates" at the EOS token, filling the tokens after the EOS token as padding tokens (sample.py#L146, tasks.py#L57-L62). 3. This "truncated" response is then passed to the reward model to get a score; if the response does not contain any EOS token, we suspect Stiennon et al. ( 2020) replaced the score with -1, similar to the procedure described by Ziegler et al. ( 2019); Huang et al. (2024) .The EOS trick serves a couple of purposes for RL:1. Defined reward scores: It guarantees that the PPO model receives a defined reward score. This is important because the RM only backpropagates loss on the EOS token during training. Without an EOS token, the completion's reward is undefined. The EOS trick assigns a constant -1 reward in these cases. 
: 
 RLHF scaling by model size SFT PPO OpenAI 1.3b PPO checkpoint (Stiennon et al., 2020) reference summaryFigure 1 : The win rate of our models' summaries over the human-generated reference summaries on the validation split of the TL;DR dataset, judged by GPT 3.5. Our SFT / RM / PPO models were trained with four random seeds across the 1B, 2.8B, and 6.9B Pythia (Biderman et al., 2023) model sizes using the same 3e-6 learning rate. 
 
 Appendix:  


Decision: The paper outlines a re-implementation of the RLHF pipeline from Stiennon et al. 2020. The main contributions of the paper are (1) reproducing the RLHF pipeline and the scaling behaviors in Stiennon et al 2020, (2) a number of implementation details and insights (e.g. benefits of dropping dropout) that are needed for a stable PPO implementation.

Overall, the reviewers agree that this paper contains useful insights for researchers, especially since PPO training is known to be unstable. 

Cons: 
It is unclear what advantages this provides over existing frameworks like TRL. The paper is also missing a through comparison with the design choices between these frameworks.

Reviews: review: paper_summary: paper_summary: The paper proposes a re-implementation of the RLHF algorithm with PPO. The paper provides insights into the key implementation details for the reproduction of OpenAI's seminal TL;DR summarization work. Each key implementation insight has been backed by an ablation study into its effect on the final performance. This thorough examination enhances our understanding of the algorithm's intricacies and will serve as a valuable resource for the research community.
strength_weakness: strength_weakness: I don't have many concerns with the paper as-is. Here are a few questions about the experiments to the authors.

a) (Detail 2) How do the experiment observations change with different restrictions to the length of the query tokens (say 256 v/s 512 v/s 768)? 

b) (Detail 4) How do the differences in SFT and Preference datasets affect the final performance of the trained model? Is there a control study, where the authors restrict the length of the examples in  both the datasets to check their impact on the final performance?

c) (Detail 21) What's the effect of multiple epoch training on the same SFT prompts? Do the authors observe differences in the quality of the different SFT prompts?

d) "Reward whitening makes the model generate shorter outputs." Is this helpful only for summarization tasks?
questions: questions: Please check my questions above.
paper_summary: paper_summary: The paper does a comprehensive reproducibility experiment on the RLHF results from a past OpenAI paper. The authors experiment with different details of training that are often overlooked or not discussed in detail in most papers.
strength_weakness: strength_weakness: I am not sure how much it might align with the "research novelty" aspect of the conference. I leave that judgement to the editors. 

I would like to point out though that the authors mentioned that many of the practices are not standard in the community. In that note, it would be good to contrast the design choices of the authors against the out-of-the-box choices presented in many open source libraries like deepspeed-chat, trl, etc. Since the main objective of this paper is to make the engineering practices more reproducible and "open", it is important to contrast the presented choices against other available alternatives.
paper_summary: paper_summary: This paper contributes a detailed reproduction of OpenAI’s RLHF work in the TL;DR summarization. The nice thing about the work is that many technical details (datasets with tokenizer, truncation, padding, EOS tokenizing, disabling dropout, the way they extract reward from the EOS token, etc) are transparent. This should make the paper a good reference for future effort to replicate RLHF work in the dataset.

The work also demonstrated the scaling behavior of PPO across different model sizes, this is unsurprising though.
strength_weakness: strength_weakness: * The novelty of the work is limited. I consider it as a "white" paper that to unblock everyone who tries to replicate "RLHF" work, which is to me a significant contribution. But other than that I don't see other novelty of the work.
paper_summary: paper_summary: This paper summarizes the reproduction details of the OpenAI work: RLHF for summarization. The authors study many different items that are specific and important for reproduction. The authors have found several key points that affect the performance.
strength_weakness: strength_weakness: 1. Though the comprehensive study appears, it is somehow hard to show the connections between these different details, this potentially could be crucial. 
2. The application summarization task is specific, however, it would be better to make some details that could be transferable. Or it should be study some of these items.
3. The base model GPT-2 is somehow old now, at least LLaMA would be a better replacement.
paper_summary: paper_summary: The paper discusses more than 20 implementation details of RLHF with PPO using the tl;dr summarization task. The authors achieve great performance on the summarization task, and the authors reproduced the scaling behavior by OpenAI by experimenting with models of different sizes. So, the implementation details must be meaningful.
strength_weakness: strength_weakness: (Detail 19) On DPO vs. PPO reward modeling ability: it’s unclear to me whether the comparison is fair. 
For DPO reward model, do you mean using the probabilities as an implicit reward model? 
In that case, would it be more fair if you not only include your regular reward model validation accuracies, but also PPO-trained LM’s reward modeling accuracy (in other words, similar to DPO, treat the PPO-trained model as an implicit reward model, and compute the validation accuracies)? 

Imo the advantage implementation is quite important but there seems to be minimal discussion. 

There have been quite a few recent “implementation details of PPO” online. I thought it would be best if the authors highlight the least known and the most important details, and also point out which detail likely doesn’t matter much. 

I’m not entirely sure if the example in Table 1 is appropriate so I leave it to other reviewers / the meta-reviewer. I know we are all adults but I also don’t want to run into this genre unprompted, especially in an academic paper (e.g., losing my virginity, leaving me with a boner).

--

Updated score after rebuttal
questions: questions: Detail 23 seems important but it’d be great if the authors can elaborate what the values are. Are values for each batch? The past k steps?

Comments: title: Rebuttal Reminder
Rebuttal Reminder
[Area Chair Comment] Author response
Thanks for the rebuttal
comment: I thank the reviewers for their detailed response to my questions. Looking over the reviews of my fellow reviewers, I see no major concern, other than whether the paper is novel enough. I strongly believe that we need such papers to dive deeper into the design choices of existing algorithms for better understanding of these models. As such, I keep my score as-is but am in favor of acceptance into the conference proceedings.
Given that the discussion period ends tomorrow, let us know if there are any concerns you feel we haven't addressed or clarifications to improve our score. Thank you!
Given that the discussion period ends tomorrow, let us know if there are any concerns you feel we haven't addressed or clarifications to improve our score. Thank you!
Thank you for the reply. In the meantime I'm reading through other reviews and I'll update my review accordingly.
Dear reviewers, 

Please take a look at the author's rebuttals and the other reviews for this paper! If the rebuttals addressed your concerns, please let the authors know about this and update your review. If not, please continue to engage with the authors and the other reviewers in the discussion forum.

Please feel free to comment on reviews (or their author responses) other than your own! 

Thanks!
Thanks the authors for the rebuttal. I look forward to your next versions of these details. It's okay for the model side. 
Therefore, I will raise the score.