The N+ Implementation Details of RLHF with PPO: A Case Study on TL;DR Summarization
Authors: Shengyi Huang
Michael Noukhovitch
Arian Hosseini
Kashif Rasul
Weixun Wang
Lewis Tunstall
Venue: COLM (2024)
Keywords: Reinforcement Learning from Human Feedback, RLHF
TL;DR: Enumerated 20+ implementation details of RLHF and reproduced RLHF scaling behaviors in prior closed-source work

Abstract: This work is the first to openly reproduce the Reinforcement Learning from Human Feedback (RLHF) scaling behaviors reported in OpenAI's seminal TL;DR summarization work. We create an RLHF pipeline from scratch, enumerate over 20 key implementation details, and share key insights during the reproduction. Our RLHF-trained Pythia models demonstrate significant gains in response quality that scale with model size, with our 2.8B, 6.9B models outperforming OpenAI's released 1.3B checkpoint. Our results highlight best practices in data, training, and evaluation for RLHF.
We publicly release the trained model checkpoints and code to facilitate further research and accelerate progress in the field at  https://github.com/vwxyzjn/summarize_from_feedback_details

Published as a conference paper at COLM 2024
The N+ Implementation Details of RLHF with PPO: A Case Study on TL;DR Summarization
Shengyi Huang
Michael Noukhovitch
Arian Hosseini
Kashif Rasul
Weixun Wang
Lewis Tunstall
Hugging Face
Mila, Université de Montréal Fuxi Al Lab, NetEase
costa. costa. www.aaaannnuuoloo.coom com
Abstract
This work is the first to openly reproduce the Reinforcement Learning from Human Feedback (RLHF) scaling behaviors reported in OpenAl's seminal TL;DR summarization work (Stiennon et al., 2020). We create an RLHF pipeline from scratch, enumerate over 20 key implementation details, and share key insights during the reproduction. Our RLHF-trained Pythia models demonstrate significant gains response quality that scale with model size, with our 2.8B, 6.9B models outperforming OpenAl's released 1.3B checkpoint. Our results highlight best practices in data, training, and evaluation for RLHE. We publicly release the trained model checkpoints and code to facilitate further research and accelerate progress in the field at https: / /gi thub. cow/wwxyzjn/ http/////////mmr/raaa//e summarize ffrof________ feoomeeeebaccc...oee reedback_eetails..
RLHF scaling by model size
Win rate against reference summaries
0.7
(according 9pt-3 apt-3.5-turbo-012-
0.6
0.4
0.3
PPO
OpenAl 1.3b PPO checkpoint (Stiennon et al., 2020)
reference summary
10th
Model size
Figure 1: The win rate our models' summaries over the human-generated reference summaries the validation split of the TL,DR dataset, judged by GPT 3.5. Our SFT / RM / PPO models were trained with four random seeds across the 1B, 2.8B, and 6.9B Pythia (Biderman et al., 2023) model sizes using the same 3e-6 learning rate.

---

Published as a conference paper at COLM 2024
1 Introduction
There has been tremendous development in pre-trained large language models (LLMs) over the years (Radford et al., 2018; 2019; Brown et al., 2020; Rae et al., 2021). Given the previous tokens, these LLMs are trained to predict the next token accurately, and they can be prompted to solve a wide range of natural language processing (NLP) tasks. However, the next-token-prediction objective differs from the fundamental chatbot objective of "outputting content that humans prefer". " To address this gap, Reinforcement Learning from Human Feedback (RLHF; Stiennon et al., 2020; Ouyang et al., 2022; Bai et al., 2022) was introduced pipeline to 1) supervised finetune (SFT) on the domain 2) collect pair-wise human preferences, and train a reward model (RM) to model these preferences then 3) use Reinforcement Learning (RL) to optimize a model against the RM to output content that humans prefer.
It has proven challenging to reproduce OpenAl's RLHF pipeline (Ouyang et al., 2022; Ope- nAI et al., 2024) in the open-source community for several reasons: 1) RL and RLHF have many subtle implementation details that can significantly impact training stability (En- gstrom et al., 2020; Huang et al., 2022; 2024), 2) the models are challenging to evaluate for the instruction following tasks (e.g., evaluating the quality of 800 lines of generated code snippet for coding task), 3) they take long time to train and iterate.
This work addresses the aforementioned challenges by taking a step back and reproducing OpenAl's earlier but seminal RLHF work in TL;DR summarization (Stiennon et al., 2020). TL;DR is one of the most popular benchmarks for RLHF methods alongside instruction following tasks such as Anthropic's HH-RLHF (Bai et al., 2022) and AlpacaFarm (Dubois et al., 2023). But summarization tasks are much easier to evaluate than general instruction following tasks because summaries are typically short and bad summaries usually have bad accuracy, coverage, or make-up facts. The reduced context and generation length also mean more efficient training, allowing us to iterate quickly and polish working RLHF pipeline. Specifically, our contributions are as follows:
We reproduced the RLHF scaling behaviors in Stiennon et al. (2020). Our end-to-end pipeline demonstrates that larger models lead to improved ROUGE scores for SFT models, higher validation accuracy for RMs, and higher win rates of the generated summaries over reference summaries for the final RL policies.
We release robust, highly reproducible RLHF pipeline To simplify the setup and improve reproducibility, we use the same learning rate for SFT, RM, and RL training, in contrast to the original setup which ran hyperparameler sweeps separately each one. To ensure resea rchers can reliably reproduce our work, we ran our model training for four random seeds, including failure cases for analysis.
We enumerate over 20 relevant implementation details and offer detailed insights. This paper delves into the details of the TL;DR datasets, including their specifications, tok- enization processes, and token length distributions. We then detail the training setups, implementation details, and results for both SFT and RM components. Additionally, we explore the details of Proximal Policy Optimization (PPO; Schulman et al., 2017) for RLHF training and how they impact performance. We provide visualizations to compare the behavior of aligned versus base models.
Our work is fully open source and transparent. We make our complete source code and release
available at https: thub. wtwp.h....wwwy.....cccccc wttp://wswwuwrrarrr///a model checkpoints and training metrics in Appendix H.
Preliminaries
RLHF trains reward model from human preferences and then performs RL training against the reward model (Christiano et al., 2017) for tasks where is difficult to design a reward function. At larger scale, RLHF has been used to fine-tune large language models (LLMs) to output contents that align more with human preferences (Ziegler et al., 2019; Stiennon et al., 2020; Ouyang et al., 2022; Bai et al., 2022; OpenAT et al., 2024; Gemini Team et al., 2023).
2

---

Published as a conference paper at COLM 2024
Table 1: Query pre-processing example. The left example has 512, which is greater than the max query token length of 512, so the pre-processing truncates the last paragraph of the post. Colors show how the contents are tokenized.
Before: 514 tokens
After: 445 tokens
SUBREDDIT: I rreatiooships
SUBREDDIT: r //relationships
TITLE: Me [19 FJ should be trying to help my brother 16 MJ with his life?
TITLE: Me [19 FJ should be trying to help my
brother 16 MJ with his life?
POST: This is my first Reddit post and I'm not sure if I'm doing it right
POST: This is my first Reddit post and I'm not sure if I'm doing it right
I've tried cutting back his computer usage to
I've tried cutting back his computer usage to three hours on weekdays and five hours on weekends but he gets through it. I've tried countless things. TL;DR:
three hours on weekdays and five hours on weekends but he gets through it. I've tried countless things.
Reddit, should I just leave him be and worry about myself? love him to death and miss talking to him like we used It's been like this for awhile. Maybe it's puberty, don't know. I'm at a loss. Someone tell me what to do
TL;DR:
RLHF typically has three steps: 1) train an SFT policy, 2) Collect preference pairs and train an RM, and 3) Train an RL policy with PPO to optimize scores from the RM.
RL-free approaches have been proposed to optimize the RLHF objective (Rafailov et al., 2023; Azar et al., 2023; Hong et al., 2024). RLHF with PPO can quite computationally expensive because the training program typically needs to load 3-4 models into the GPU memory and policy training autoregressively generates online data. To alleviate these problems, researchers have proposed RL-free approaches such Direct preference optimization (DPO), which implicitly optimizes RLHF by optimizing reward modeling. We describe their specifics and losses in more detail in Appendix A.
TL;DR Summarization is the task of summarizing Reddit posts (Völske et al., 2017). Sti- ennon et al. (2020) showed that optimizing with human preference data can produce summaries that are preferred to those which optimized traditional NLP metrics i.e. ROUGE (Lin & Och, 2004) as well baseline human-written summaries. The only previous open- source codebase TLRX by Phung et al. (2023) focused mostly on creating an example of an RLHF pipeline and did not reproduce the specific data pipeline, model scores, or scaling behaviour.
Dataset Details
We start with a solid understanding of the dataset, the tokenization process, and the token lengths. This section provides an in-depth analysis and visualization of the TL,DR datasets from Stiennon et al. (2020), which includes SFT dataset and preference dataset We include more details in Appendix B.
Detail 1: Dataset Specification. Completions in the preference dataset come from variety of models and ratings include confidence values that may be taken into account.
Inttps://huggingfacee http:aaacc.cccaaaaacaeeeee http:p/////iiiiiiecccc///aa////p http:///p///sumrrree...orrrreeae 2nttps://huggingfacc co/ https///......aaaaa/////////aaaa htt.rarrrrieerrffeeeeeeeaaaaa

---

Published as a conference paper at COLM 2024
Table 2:
The number of unique pairs of policies compared differs in each preference dataset split. In particular, notice the validation set contains highly diverse pairs (see Appendix L for details on the exact policy comparisons and their counts).
Split name train
The number of unique pairs of policies compared
47
validation
241
validation calddatio__enddm cnndm
7
> Detail 2: Do not truncate the sentence, truncate the paragraph. When the query token length exceeds a preset maximum of 512 tokens, the preprocessing would truncate at the last paragraph instead of a hard truncation limit at 512 tokens. This procedure makes the query coherent. Table 1 shows an example.
> Detail 3: Format completions with leading space, an EOS token at the end, and pad with a special padding token [PAD] instead of just EOS. Note the prompt ends with TL;DR: which does not include a trailing space, so we need to prepend a leading space
in the completion when preparing the query and response (e.g., long relationship; fell in love with another person; admi tted it; would like it to di disapper,, though it doesn't t.< endoffext |> [PPAD][PAD][PAD]...
is also important to use special padding token [PAD] distinct from the EOS token, otherwise EOS can be masked as a padding token and the model won't learn to end summaries with EOS tokens.
Detail 4: SFT and preference datasets have different tokenization length. Interest- ingly, the summary lengths in the preference dataset are not controlled to be the same. We show visualizations of the tokenization length in Figures 9 and 10 at Appendix In particular, The chosen/ rejected response token length in the preference dataset can be as long as 169, significantly exceeding the 53 tokens found in the SFT dataset. Also the median chosen response token length is 32, which is slightly longer than that of the rejected response token of 30.
Detail 5: Pre-tokenize the dataset: right pad the concatenation of queries and responses; left pad the queries. To pre-tokenize the dataset for training, we right pad the concatenation of queries and responses for SFT and RM training; we also left pad the
queries for generations during PPO training.
Detail 6: The validation split of the preference dataset has lot of OOD data. As illustrated in Table 2 (see Appendix L for details on the exact policy comparisons and their counts), the sampling policies employed in the preference dataset exhibit significant diversity, which is out of the distribution of the sampling policies used in the train split.
4
General Details
We note a few general details used across all training, with more in Appendix C.
Detail 7: Disable dropout to ensure PPO's ratio calculation still works. We disable dropout during all training, similar to the settings in Ziegler et al. (2019); Huang et al. (2024). This is important for PPO, because when dropout is activated, the log probabilities of tokens are not deterministic. This makes calculating the KL penalty unreliable and means the logprob ratios used in PPO are not equal to during the first epoch, which leads to optimization issues. For consistency, we disable dropout for SFT and RM training, > Detail Setup- Tech stack, We used the transformers (Wolf al., 2020) library's im- plementation of the Pythia models in conjunction with deepspeed's ZeRO Stage 2 (Rasley et al., 2020; Rajbhandari et al., 2020) to help fit the models into the GPU memory; for 6.9B PPO training we also offload the reference policy and reward model to CPU. We launch experiments using accelerate (Gugger et al., 2022) with bf mixed-precision training and track them with Weights and Biases (Biewald, 2020). We use 8xH100 machines and always upload the trained models to Hugging Face's model hub3
com
4

---

Published as a conference paper at COLM 2024
sft
ROUGE scores scaling
2.8
2.8B
model
6.9B
0.260
ROUGE score
Loss
2.6
0.255
2.4
0.250
2.2
0K
2.5K
5K
7.5K
10K
12.5K
15K
10a
10th
Steps
Model size
Figure 2: (Left) Train loss across model scales for one epoch of the SFT dataset (116k steps) (Right) Scaling behaviors of the ROUGE score SFT validation set
PPO Response Token Ids
2325
PPO Response (Decoded)
Be
em
with
GF
for
5
cave
mi
Mone
?
<lendottudt
Reward
-4.87
-5.85 -7.04 -6.57
-4.63
-3.93
Figure 3: A 1B PPO model's response and its corresponding reward logits from 1B RM. Here, we use Pythia's tokenizer, so denotes the EOS token and 0. 65 is the extracted EOS reward. Notice how the logits of non-EOS tokens are almost always negative we see this behavior in all the response-rewane response-rewardlloiis pairs from all policies and RMs.
5 Supervised Finetuning Details
Overall, SFT is fairly straightforward. Our setup closely follows Stiennon et al. (2020), except for a modified learning rate (> Detail 9:); we show all hyperparameters in Table in Appendix D). We finetune base models on the SFT dataset using standard next-token prediction loss.
As shown in Figure 2, unsurprisingly, larger models have smaller next-token-preddee losses. After finishing the training, we generate summaries from our models and evaluate the ROUGE scores against the reference summaries in the SFT validation set and find favorable scaling behavior, similar to Figure 14 (a) in Stiennon et al. (2020).
6 Reward Modeling Details
Reward modeling, despite its relatively simplicity, has few important notes to its setup. See Appendix E for more details.
Detail 10: Setups. We follow Stiennon et al. (2020)'s original setting to train the RM, except that we used different learning rate (see hyperparameters in Table 4 in Appendix E); note the linear head to output reward scalars is initialized with weights
according to (0,1 1/ (dmooeel + 1)).
Detail 12: Extract reward from the EOS token When obtaining the scalar reward, the RM does forward pass on the sequence and extracts the reward only on the EOS token.
This implemented by finding the first index of the padding token and then minus 1. the padding token does not exist, the extracted reward will then be logits corresponding to the last token of the sequence - that token is not the EOS token, its reward won't be used for PPO training,
explained later in PPO's EOS trick at > Detail 23:)
Detail 13: Most values in the reward logits are non-valid and negative; only the reward logit at the EOS token are valid. What do the reward logits actually look like in

---

Published as a conference paper at COLM 2024
reward
reward
0.9
0.8
Training Accuracy
0.7
0.6
0.6
RM
0.5
0.5
0.4
6.98
0.4
1B
0.3
0K
20K
40K
60K
80K
OK
20K
40K
60K
80K
reward
RM scaling
TL:DR
Rejected Rewards
5.0
Validation accuracy
0.70
CNN/DM Set
2.5
0.68
0.66
-2.5
0.64
-5.0
0.62
0K
20K
40K
60K
BOK
10a
1020
Steps
Model size
Figure 4: (Top Left) The RM training accuracy across the model sizes for one epoch of the train split of the preference dataset (92,858 episodes) (Top Right) RM loss (Bottom Left) Chosen reward mean values (Bottom Right) Scaling behavior of reward modeling on the TL;DR set and CNN/DN DM validation sets. TL;DR validation accuracy is lower because the validation set has out-of-distributio (OOD) data, explained in Detail 6:
these trained RMs? We include example in Figure and notice the logits of non-EOS tokens are almost always negative for all RMs on completions for all policies.
Detail 14: Minor numerical differences between extracting reward with left and right padded queries. During RM training, the sequences are padded from the right with the shape (B, 638). However, left-padding the query is required for generation in PPO training. The query has shape (B, 512), and after generation (with sequence length 53), the query and response batch shape becomes (B, 565). As result, need to adjust the attention masks during RM forward calls. Left-padding vs. right- padding can introduce minor numerical differences. For instance, in the 6.9B RM, the average reward scalar difference on the SFT dataset between the two padding methods is -0. 00.000451550 50301 720947. This difference generally negligible.
Detail 15: Reward normalization based on SFT dataset. Stiennon et al. (2020) sug- gested that "at the end of training, we normalize the reward model outputs such that the reference summaries from our dataset achieve mean score of 0." We applied the same procedure by iterating through the SFT dataset and calculating the rewards of the query and reference responses, then calculating the mean reward and setting it as a bias in the reward head.
We show the results of our training (RM training loss, accuracy, and chosen reward value) in Figure 4. The training accuracy and losses appear stable. Overall, larger RMs have higher validation accuracy the CNN/DM is very encouraging the RM has never trained on CNN/DA data so this demonstrates good transfer. We notice several additional details in the RM results:
both TL,DR and CNN/DM Note the validation accuracy
Detail 16: Different batches / confidences have different accuracies. We calculated the aggregated mean and standard deviation of validation accuracy for each batch, split,

---

Published as a conference paper at COLM 2024
Reward Model scaling
Calibration with TLDR Preferences Data
0.8
1.0
0.7
0.9
0.6
Accuracy
consistency
0.5
0.8
sapaaue dopp
-2.8
0.4
0.7
0.3
0.6
0.2
Our PPO Reproduction
perfect calibration
0.5
10
10
Reward Model size
Score difference
Figure 5: (Left) RM agreement rate with GPT3.5 across different model sizes. (Right) RM calibration the black line the perfect calibration 1 where A is the the score difference (Equation 2) (Bai et al., 2022).
and confidence in Table 5 at Appendix E. We find different annotated batches have vastly different validation accuracies. Many works choose random subsets of the validation set (Phung et al., 2023; Roit et al., 2023) which can make results incomparable.
Detail 17: Preference consistency with GPT3.5. To verify whether the RM is overfitting use GPT3.5
the preference dataset,
external LLM-judge (Zheng et al., 2023). We compare GPT3.5 and different RM sizes on the model-generan
vs human baseline data. As depicted in Figure agreement increases with model size and though it diminishes
around 6.9B, ~ 80% agreement suggests our RM does capture general human preference.
> Detail 18: RM calibration. We follow Bai et al. (2022) visualize the calibration of the RM on the preference dataset in Figure 5. Overall, we find a positive correlation between accuracy and score difference models become more accurate as they become more confident (i.e., higher score difference). However, the RMs are still under-calibrated probably due to the diverse validation set (> Detail 6:) and different accuracies across these validation sets(> Detail 16:).
> Detail 19: Comparison with DPO as a reward model We also compare to DPO's RM in Figure 6 and found DPO's validation accuracy to be lower. This may due to several reasons. First, RM training only applies the loss at the EOS token, whereas DPO applies the loss at every completion token. Second, DPO uses f parameter that controls the KL of the language model and therefore also the DPO RM, whereas a regular RM does not. Finally, DPO's objective might be harder to optimize the model needs to make the logprobs sufficiently different from the base model to change the reward, whereas RM can learn a linear head that can much easier faster change the value of the reward.
7 PPO Details
PPO is well known to be an effective algorithm but can require variety of specific tricks (Huang et al., 2022). We delve into the main RLHF-specific tricks and explain other, subtle notes in Appendix F.
Detail 20: Setups. We closely follow Stiennon et al. (2020) in our PPO setup, with our modified learning rate, and detail all hyperparameters in Table in Appendix F.
> Detail 21: Train with the SFT dataset and shuffle between epochs. Stiennon et al. (2020) trains the PPO models for 1M episodes generating from prompts in the SFT dataset. But the train split of the SFT dataset is only of size 116,722, so once we'v ve'v gone through the dataset, we shuffle it and again sample batches without replacement.
> Detail 22: Initialize value model from reward model; trained value model looks like a pek-token RM. Following Stiennon et al. (2020), we find it is important to initialize the value network using the reward model. This warm-starting of the value network can greatly improve initial gradients to the policy and reduce drift / alignment tax over training (Noukhovitch et al., 2023). In actor-critic RL training, the value function aims
7

---

Published as a conference paper at COLM 2024
dpo
dpo
0.7
Training Accuracy
0.8
0.6
DPO Loss
0.7
0.5
0.6
6.98
1B
0.5
0K
20K
40K
60K
80K
0K
20K
40K
60K
80K
dpo
RM scaling
Reward Modeling
Chosen Rewards
Validation accuracy
0.68
DPO Reward Modeling
0.66
-2
0.64
0.62
0K
20K
40K
60K
80K
10a
10t0
Steps
Model size
Figure (Top Left) DPO RM train accuracy (Top Right) DPO train loss (Bottom Left) DPO chosen reward mean values (Bottom Right) RM and DPO's s RM validation set accuracy TL,DR. We find DPO's accuracy to be lower across scales (> Detail 19:).
to predict the end-of-episode return at each timestep / token, effectively acting as per-token RM. See Figure 11 in Appendix F for the rewards and values of completion. Detail 23:"EOS trick" ensures valid rewards from the RM During training, PPO typically samples fixed number tokens but what the completion does not end with an EOS token? The logits of non-EOS tokens are almost always negative and invalid (> Detail 13:). If a completion does not end with EOS, the EOŠ trick sets the reward constant -1. This ensures the reward is valid and encourages the models to output shorter responses that end EOS tokens.
> Detail 24:Reward whitening is optional Huang et al. (2024) note that Ziegler et al. (2019) implement a whiten function that normalizes reward values by subtracting the mean followed by dividing by standard deviation. Optionally, whiten the mean of the whitened values with Shift_mean=True. In each minibatch, PPO could whiten the reward whi ten (rewwaasds, Figure/ 7, we find reward whitening 1) makes the win rate against reference summaries bit lower and 2) makes the completion token length bit shorter. When controlling for summary length, we find it more challenging to compare the results. We therefore consider reward whitening be optional it can shorten completions but might not change summary quality.
shift back
(ett_mma==alleee without shifting the mean. In
Detail 25: Advantage whitening helpful Similar practices identified Engstrom et al. (2020); Andrychowicz et al. (2021); Huang et al. (2022), PPO in Stiennon et al. (2020) whitens the advantages with the shifted mean whitenn(dvantages))
We train and plot several of PPO's learning curves in Figure 12 in Appendix F. Overall, we find that PPO does well to optimize the RLHF objective, and the average reward goes up. Interestingly, the larger the model, the less the model changes (in KL) to achieve reward. Similar to Gao et al. (2023), we also see that larger RMs tend to better estimate the true reward.
8

---

Published as a conference paper at COLM 2024
RLHF scaling by model size
RLHF scaling by token length
0.8
rate againd referenca summarres
0.7
refinntteeeeeene inccardnn to gat- 3pt-3.5-tti 3.S-tutt-00155)
Win rate aqganntt reference summmetic
0.7
0.6
16,PO whitening
(according
0.4
0.4
2.80/PPO w/ reward whitening
6.9tyPPO w/ reword whitening
whitening
16PPPO
PPO
0.3
7.80/PPO
OpenAl chockpoln
6.9p/PPO
OpenAl
chockpoint
reference summary
0.2
(Stiennon
10a
10:4
32
34
36
42
44
Model size
Token length
Figure 7: (Left) RLHF preference scaling behavior across different model sizes with and without reward whitening Detail 24: (Right) Plots the same data with the x-axis being the average summary token length.
RLHF scaling by token length
Model quality for different summary lengths
1b/SFT
PPO
Bb.5FT
rate sgainst reference summares 3-t-utbb---o0i1
6. 9b/5FF
Winrate against referunce seferncesummmrtin zummertes sicrarding to 3Ft gpt-3.-tuubee-
SFT
summary
2 Bb/PPO
0.8
0.6
OpenAi 1. 3b PPO checkpoint (Stiennon et el. 2020)
0.5
0.6
0.4
0.4
0.2
30
32
34
36
0.5
1.0
longth
log(summary length reference length)
Figure 8: (Left) Win rate against reference summaries, plot by average summary token length (Right) Win rate of 6.9B SFT and PPO models for different summary lengths (one random seed; see Appendix for other seeds and model sizes).
We also evaluate the final model checkpoint on the prompts from the validation set of the SFT dataset using GPT 3.5 as judge againng the reference, human-written summaries in Figure (the GPT prompt can be found in Appendix I). We also observed good scaling behaviour for RLHF in terms of win rate. In particular, G.5 prefers our best 6.9B model to human sum- maries nearly 80% of the time. To account for correlation between summary length and pref- erence, we plot the win rate against the log(summary length reference summary length) at Figure S. Win-rate is correlated with summary length for RLHF models, implying that either true humans preference is correlated with longer summaries or GPT 3.5 is biased in this way. Still, our PPO models almost always outperform SFT across lengths which implies that our models not simply learning to generate longer summaries but actually generating higher quality summaries.
Conclusion
This work presents the first high-fidelity reproduction OpenAl's RLHF work TL;DR summarization (Stiennon et al., 2020). We have demonstrated the powerful scaling behavior

---

Published as a conference paper at COLM 2024
of PPO across different Pythia model sizes and shown how RLHF can lead to better summa- rization models across scales. We offer detailed insights into many implementation specifics and design choices that enabled this successful reproduction, promoting transparency and reproducibility within the research community. We have also noted several interesting results that future work may investigate and elucidate.
Many of our implementation details are not currently standard practices within the RLHF community. We hope this work leads collection of best practices in the open-source RLHF community for data, training, and evaluation.
Author Contributions
Shengyi Huang led the overall project.
Michael Noukhovitch helped discuss and verify early design choices/results, led the analysis of RM calibration plots, and edited the paper.
Arian Hosseini led the analysis of the length-controlled summary comparisons (e.g, Figure 8), improved visualization in > Detail 22; and edited the paper.
Kashif Rasul crafted the visualization in Table 8 and edited the paper.
Weixun Wang plotted the GPT3.5 agreement rate in Figure 5 (left) and Table 5 and edited the paper.
Lewis Tunstall advised the project.
Acknowledgments
The experiments presented in this paper were conducted using NVIDIA H100 nodes in the Hugging Face's compute cluster.
References
Marcin Andrychowicz, Anton Raichuk, Piotr Starcczy,, Manu Orsini, Sertan Girgin, Raphaël Marinier, Leonard Hussenot, Matthieu Geist, Olivier Pietquin, Marcin Michalski, Sylvain Gelly, and Olivier Bachem. What matters for on-policy deep actor-critic methods? large-scale study. In International Conference on Learning Representations, 2021. URL https://openreview www..nn//oorumiiddmlliisssiiiii
Mohammad Gheshlaghi Azar, Mark Rowland, Bilal Piot, Daniel Guo, Daniele Calandriello, Michal Valko, and Rémi Munos. A general theoretical paradigm to understand learning from human preferences. arXiv preprint arXiv:2310. 12036, 2023.
Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training helpful and harmless assistant with reinforcement learning from human feedback. ar preprint dxXiv:2044.08862, 2022.
Stella Biderman, USVSN Sai Prashanth, Lintang Sutawika, Hailey Schoelkopf, Quentin Anthony, Shivanshu Purohit, and Edward Raff. Emergent and predictable memorization in large language models. 2023.
Lukas Biewald. Experiment tracking with weights and biases, 2020. URL https://www. wandb. cwm.c Software available from wandb.com.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. Advances in neural information processing systems, 30, 2017.
10

---

Published as a conference paper at COLM 2024
Yann Dubois, Xuechen Li, Rohan Taori, Tianyi Zhang, Ishaan Gulrajani, Jimmy Ba, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. AlpacaFarm: A Simulation Frame- work for Methods that Learn from Human Feedback. In Advances in Neural Information Processing Systems, 2023.
Logan Engstrom, Andrew Ilyas, Shibani Santurkar, Dimitris Tsipras, Firdaus Janoos, Larry Rudolph, and Aleksander Madry. Implementation matters in deep rl: case study ppo and trpo. In International Conference on Learning Representations, 2020. URL https://openreview. http:////eeeereeeeeeeeee/eeoottr
Leo Gao, John Schulman, and Jacob Hilton. Hcaling laws for reward model overoptimization. In International Conference on Machine Learning, pp. 10835-10866. PMLR, 2023.
Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, lean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al. Gemini: family of highly capable multimodal models. ar arv preprint arXi::2312.11800 2023.
Sylvain Gugger, Lysandre Debut, Thomas Wolf, Philipp Schmid, Zachary Mueller, Sourab Mangrulkar, Marc Sun, and Benjamin Bossan. Accelerate: Training and inference scale made simple, efficient and adaptable. https://gi http://ggttth....cc httpss//gtthub..cco/uugggne acceaccceeeeeee 2022.
Jiwoo Hong, Noah Lee, and James Thorne Reference-free monolithic preference optimiza- tion with odds ratio. Xiv preprint ar (xXi:22433.06611, 2024.
Shengyi Huang, Rousslan Fernand Julien Dossa, Antonin Raffin, Anssi Kanervisto, and Weixun Wang. The 37 implementation details of proximal policy optimiza- tion. In ICLR Blog Track, 2022. URL https://iclr-blog-ttrre github. 1o/2022/03/ 25/ppo- iemeoo-impleeeeee ewweeationnnnieeeeeee inmeenttiioneeeaiss// httpr/aaii////////////o/t/////tt ppppp ppo- impleenentaion-detalss...
Shengyi Costa Huang, Tianlin Liu, and Leandro von Werra. The n implementation details of rlhf with ppo. In ICLR Blogposts 2024, 2024. URL https::/
Hamish Ivison, Yizhong Wang, Valentina Pyatkin, Nathan Lambert, Matthew Peters, Pradeep Dasigi, Joel Jang, David Wadden, Noah A Smith, Iz Beltagy, et al. Camels in changing climate: Enhancing Im adaptation with tulu 2. arxiv preprint arXi0:2311.10702, 2023.
Albert Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, al. Mixtral of experts. arXiv preprint ar arXiv:2401.0400
2024.
Bill Yuchen Lin, Abhilasha Ravichander, Ximing Lu, Nouha Dziri, Melanie Sclar, Khy- athi Chandu, Chandra Bhagavatula, and Yejin Choi. The unlocking spell on base llms: Rethinking alignmeet via in-context learning. In International Conference on Learning Representations, 2024. URL httpps///rre arggabbs/2312.115553
Chin-Yew Lin and Franz Josef Och. Automatic Evaluation of Machine Translation Quality Using Longest Common Subsequence and Skip-Bigram Statistics. In Proceedings of the 42nd Annual Meeting of the Association for Computational Linguistics (ACL-04), Barcelona, Spain, 2004.
Michael Noukhovitch, Samuel Lavoie, Florian Strub, and Aaron Courville. Language Model Alignment with Elastic Reset. In Advances in Neural Information Processing Systems, 2023.
11

---

Published as a conference paper at COLM 2024
OpenAI, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, llge Akkaya, Floren- cia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, Red Avila, Igor Babuschkin, Suchir Balaji, Valerie Balcom, Paul Baltescu, Haiming Bao, Mohammad Bavarian, Jeff Belgum, Irwan Bello, Jake Berdine, Gabriel Bernadett-Shaping, Christopher Berner, Lenny Bogdonoff, Oleg Boiko, Madelaine Boyd, Anna-Luisa Brak- man, Greg Brockman, Tim Brooks, Miles Brundage, Kevin Button, Trevor Cai, Rosie Campbell, Andrew Cann, Brittany Carey, Chelsea Carlson, Rory Carmichael, Brooke Chan, Che Chang, Fotis Chantzis, Derek Chen, Sully Chen, Ruby Chen, Jason Chen, Mark Chen, Ben Chess, Chester Cho, Casey Chu, Hyung Won Chung, Dave Cummings, Jeremiah Currier, Yunxing Dai, Cory Decareaux, Thomas Degry, Noah Deutsch, Damien Deville, Arka Dhar, David Dohan, Steve Dowling, Sheila Dunning, Adrien Ecoffet, Atty Eleti, Tyna Eloundou, David Farhi, Liam Fedus, Niko Felix, Simón Posada Fishman, Juston Forte, Isabella Fulford, Leo Gao, Elie Georges, Christian Gibson, Vik Goel, Tarun Gogineni, Gabriel Goh, Rapha Gontijo-Lopes, Jonathan Gordon, Morgan Grafstein, Scott Gray, Ryan Greene, Joshua Gross, Shixiang Shane Gu, Yufei Guo, Chris Hallacy, Jesse Han, Jeff Harris, Yuchen He, Mike Heaton, Johannes Heidecke, Chris Hesse, Alan Hickey, Wade Hickey, Peter Hoeschele, Brandon Houghton, Kenny Hsu, Shengli Hu, Xin Hu, Joost Huizinga, Shantanu Jain, Shawn Jain, Joanne Jang, Angela Jiang, Roger Jiang, Haozhun Jin, Denny Jin, Shino Jomoto, Billie Jonn, Heewoo Jun, Tomer Kaftan, Lukasz Kaiser, Ali Kamali, Ingmar Kanitscheider, Nitish Shirish Keskar, Tabarak Khan, Logan Kilpatrick, Jong Wook Kim, Christina Kim, Yongjik Kim, Jan Hendrik Kirchner, Jamie Kiros, Matt Knight, Daniel Kokotajlo, Lukasz Kondraciuk, Andrew Kondrich, Aris Konstantinidis, Kyle Kosic, Gretchen Krueger, Vishal Kuo, Michael Lampe, Ikai Lan, Teddy Lee, Jan Leike, Jade Leung, Daniel Levy, Chak Ming Li, Rachel Lim, Molly Lin, Stephanie Lin, Mateusz Litwin, Theresa Lopez, Ryan Lowe, Patricia Lue, Anna Makanju, Kim Malfacini, Sam Manning, Todor Markov, Yaniv Markovski, Bianca Martin, Katie Mayer, Andrew Mayne, Bob McGrew, Scott Mayer McKinney, Christine McLeavey, Paul McMillan, Jake McNeil, David Medina, Aalok Mehta, Jacob Menick, Luke Metz, Andrey Mishchenko, Pamela Mishkin, Vinnie Monaco, Evan Morikawa, Daniel Mossing, Tong Mu, Mira Murati, Oleg Murk, David Mély, Ashvin Nair, Reiichiro Nakano, Rajeev Nayak, Arvind Neelakantan, Richard Ngo, Hyeonwoo Noh, Long Ouyang, Cullen O'Keefe, Jakub Pachocki, Alex Paino, Joe Palermo, Ashley Pantuliano, Giambattista Parascandolo, Joel Parish, Emy Parparita, Alex Passos, Mikhail Pavlov, Andrew Peng, Adam Perelman, Filipe de Avila Belbute Peres, Michael Petrov, Henrique Ponde de Oliveira Pinto, Michael, Pokorny, Michelle Pokrass, Vitchyr H. Pong, Tolly Powell, Alethea Power, Boris Power, Elizabeth Proehl, Raul Puri, Alec Radford, Jack Rae, Aditya Ramesh, Cameron Raymond, Fran- cis Real, Kendra Rimbach, Carl Ross, Bob Rotsted, Henri Roussez, Nick Ryder, Mario Saltarelli, Ted Sanders, Shibani Santurkar, Girish Sastry, Heather Schmidt, David Schnurr, John Schulman, Daniel Selsam, Kyla Sheppard, Toki Sherbakov, Jessica Shieh, Sarah Shoker, Pranav Shyam, Szymon Sidor, Eric Sigler, Maddie Simens, Jordan Sitkin, Katarina Slama, Ian Sohl, Benjamin Sokolowsky, Yang Song, Natalie Staudacher, Felipe Petroski Such, Natalie Summers, llya Sutskever, Jie Tang, Nikolas Tezak, Madeleine B. Thomp- son, Phil Tillet, Amin Tootoonchian, Elizabeth Tseng, Preston Tuggle, Nick Turley, Jerry Tworek, Juan Felipe Cerón Uribe, Andrea Vallone, Arun Vijayvergiya, Chelsea Voss, Carroll Wainwright, Justin Jay Wang, Alvin Wang, Ben Wang, Jonathan Ward, Jason Wei, CJ Weinmann, Akila Welihinda, Peter Welinder, Jiayi Weng, Lilian Weng, Matt Wiethotf, Dave Willner, Clemens Winter, Samuel Wolrich, Hannah Wong, Lauren Workman, Sher- win Wu, Jeff Wu, Michael Wu, Kai Xiao, Tao Xu, Sarah Yoo, Kevin Yu, Qiming Yuan, Wojciech Zaremba, Rowan Zellers, Chong Zhang, Marvin Zhang, Shengjia Zhao, Tianhao Zheng, Juntang Zhuang, William Zhuk, and Barret Zoph. Gpt-4 technical report, 2024.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730-27742 27744, 2022.
Duy V. Phung, Ayush Thakur, Louis Castricato, Jonathan Tow, and Alex Havrilla. Implementing RLHF: Learning to Summarize with trlX, March 2023. URL https: //wandb. Implementing- RLHF-Learning-to- twwtttorrize-eeweeeeeee wi- Vm11122oZMZw0ODM
_________LA_____ /reports/
12

---

Published as a conference paper at COLM 2024
Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al. Improving language understanding by generative pre-training. 2018.
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and llya Sutskever. Language models are unsupervised multitask learners. 2019.
Jack W Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susannah Young, et al. Scaling language models: Methods, analysis & insights from training gopher. arviv preprint arXio:2112.11446, 2021.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly reward model. In Thirty--enenth Conference on Neural Information Processing Systems, 2023. URL
18290.
Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. Zero: Memory optimizations toward training trillion parameter models. In SC20: International Conference for High Performance Computing, Networking, Storage and Analysis, pp. 1-16. IEEE, 2020.
Jeff Rasley, Samyam Rajbhandari, Olatunji Ruwase, and Yuxiong He. Deepspeed: System optimizations enable training deep learning models with over 100 billion parameters. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pp. 3505-3506, 2020.
Paul Roit, Johan Ferret, Lior Shani, Roee Aharoni, Geoffrey Cideron, Robert Dadashi, Matthieu Geist, Sertan Girgin, Léonard Hussenot, Orgad Keller, Nikola Momchev, Sabela Ramos, Piotr Stanczyk, Nino Vieillard, Olivier Bachem, Gal Elidan, Avinatan Hassidim, Olivier Pietquin, and Idan Szpektor. Factually Consistent Summarization via Reinforce- ment Learning with Textual Entailment Feedback. In ACL, 2023.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal Policy Optimization Algorithms, 2017.
Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. Learning summarize with human feedback. Advances in Neural Information Processing Systems, 33:3008-3021, 2020.
Lewis Tunstall, Edward Beeching, Nathan Lambert, Nazneen Rajani, Kashif Rasul, Younes Belkada, Shengyi Huang, Leandro von Werra, Clémentine Fourrier, Nathan Habib, Nathan Sarrazin, Omar Sanseviero, Alexander M. Rush, and Thomas Wolf. Zephyr: Direct distillation of Im alignment, 2023.
Leandro von Werra, Younes Belkada, Lewis Tunstall, Edward Beeching, Tristan Thrush, Nathan Lambert, and Shengyi Huang. Trl: Transformer reinforcement learning. https: 2020.
//github. http:/uu...oooogggggga////////te
Michael Völske, Martin Potthast, Shahbaz Syed, and Benno Stein. TL;DR: Mining Reddit to Learn Automatic Summarization. In Proceedings of the Workshop on New Frontiers in Summarization, 2017.
Peiyi Wang, Lei Li, Liang Chen, Dawei Zhu, Binghuai Lin, Yunbo Cao, Qi Liu, Tianyu Liu, and Zhifang Sui. Large Language Models are not Fair Evaluators, May 2023.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Perric Cistac, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. Transformers: State-of-the-Ae Art Natural Language Processing. pp. 38 38-45. Association for Computational Linguistics, October 2020. URL https:/ //ww:///wwwwww// www.b....bb.............. org/ anthology//2220 emnlp-demos. 6.
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging as-jaus-- with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 36, 2023.
13

---

Published as a conference paper at COLM 2024
Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human prefer- ences. Xiv preprint ar Xi0:1999., 2019.
14

---

Published as a conference paper at COLM 2024
A RLHF procedure
Step 1: Train an SFT policy: The pre-trained LLMs are fine-tuned on the set of human demonstrations using the next-token prediction loss. In this reproduction work, these human demonstrations come from the human summaries of Reddit posts from filtered TL;DR dataset (Stiennon et al., 2020). In later work, the human demonstrations could come from paid contracted labelers (Ouyang et al., 2022) on a larger variety of tasks.
Step 2: Collect preference pairs and train an RM: Various policies, such as the trained SFT policy, are then used to sample completions, and the human labelers would indicate which completions they prefer. Given the preference dataset, we initialize an RM from the SFT policy by adding randomly initialized linear head that outputs a scalar score. The RM is trained to predict the log probability that completion would be preferred by the labelers. Specifically, the RM loss is
(R(TØ) -IE (1...,)) )
(rg (4((((())
(1)
(xx.o..)) [log(1 (((()))))))))))))))))))))))))))) (xx)))))))
(2)
where (xx = Ite is the sigmoid function, DrREF the human preference dataset, x the prompt to the model (in this case, the Reddit post), Ye the chosen/ ptrfere/peferree completion by labeler, y, the rejected completion by the labeler, & are the parameters of the RM
Step 3: Train RL policy against the RM: Initializing from the SFT policy, the RL policy then samples completions given prompts and has the RM produce score based on these completions. The reward of the RL policy then includes this score and KL penalty to ensure the RL policy does not deviate too much from the SFT policy. Specifically, the reward of the RL problem is
R(x,y) (Tx( BDLL [y xx)
7l SFT (y x)]]
(3)
where B is a parameter controlling the strength of the KL penalty, 0 the parameters of RL pol- icy Ty. Then, PPO is used to maximize the RLHF objective max (y- R(x where DSFT is the prompts in the SFT dataset.
RL-free approaches: The RLHF PPO pipeline can be quite computationally expensive because 1) the training program typically needs to load 3-4 models into the GPU memory and 2) RL policy training needs online generations and running the RM. To alleviate these two problems, researchers have proposed RL-free approaches (Rafailov et al., 2023; Azar et al., 2023; Hong et al., 2024). One of the most widely-used RL-free approaches is Direct preference optimization (DPO), which has the following loss:
(PPO(TT)) = (00t))))B(xxxeeeaea (x..c.y.)--PREEF logo (Blog
Top TV |x)
B
To x) 7TT (y:)
(4)
(ye x)
Note that DPO implicitly does the reward modeling: we can extract the reward score using the following formula:
Ty x) 7F x)
r(x,y Bog
DPO is a more accessible alignment technique that has been implemented in popular RLHF libraries such as TRL von Werra et al. (2020). DPO has also been used to align larger models effectively (e.g., Zephyr 7B (Tunstall et al., 2023), Tulu 70B (Ivison et al., 2023), and Mixtral 8x7B (Jiang et al., 2024)).
B Dataset Details - Appendix
Detail 1: Dataset - Sperrttaatiiiii
The SFT dataset fairly intuitive - it contains the subreddit, title, post, and reference summary columns. On the other hand, the preference dataset is lot more nuanced.
15

---

Published as a conference paper at COLM 2024
The train split of the preference dataset contains the subreddit, title, and post columns; it also contains two sampled summaries, their sampling policies, an internal batch number, the belonging split, which summary the human rater prefers, and optionally, a note or confidence level.
The validation split of the preference dataset contains the same information above, and definitely includes a confidence level. Furthermore, the validation split contains small batches of data for CNN/DN news articles.
> Detail 2: Dataset Do not truncate the sentence, truncate the paragraph
The next step is to tokenize the query. The query token goes through the following two transfornations httpls/experimen___eeeepe pytll
tasks. tasks.pyti py#LL9--116)
1. Format the query input string using the following template.
SUBREDDIT: r/(subreddit) I (title)\n\nPOST: [post) AntTTL;RR
2. Clever truncation to ensure the query token length is not greater than 512.
The formatted query is tokenized using the tokenizer. If the query token length is not greater than 512, it is padded from the left with either padding tokens or repeated white spaces.
If the query token length exceeds 512, the pre-processigg process will attempt to remove the last paragraph. Specifically, it finds the last index of An in the post and removes the content after. Table shows an example. This is much more sophisticated form of truncation compared to hard truncation on maximum token length.
3. No trailing space after "TL;DR:" to make sure there is no weird generation issues due to the nature of tokenization.
Detail 3: Dataset Prepend a leading space to completion; append an EOS token to the completions; use special padding token [PAD]; do not use EOS token synonymously [PAD]
When tokenizing the concatenation of queries and responses for the SFT and preference dataset, we always do the following:
1. Prepends a leading space to the completion, so there is always a space between TL;DR: and the completion such as below.
2. Append an EOS </endof |> token the completion.
3. When needed to pad the sequence to maximum length, we always use special padding token [PAD].
For example, we would add the EOS token and [PAD] token to the reference summary as follows:
long relationship; fell in love with another person; admitted it; would like it to disappear, though it doesn't t. (PAA))
We do not recommend using the common practice which uses the EOS token synonymously with the [PAD] token (e.g., tokenizer. wwwpnnieer.adddttteeeeeeee pad_token_it
tokenizer cos_token_idj This is because the BOS token would then be masked out as a padding token during SFT training, and the model would not learn to end summary a trained model would often continue to sample summary texts without stopping. This could exacerbate existing issues with RLHF models generating longer outputs (Stiennon et al., 2020; Dubois et al., 2023). With clear EOS token and padding token distinction, our final trained endpoint always learns to end summaries with the EOS token, shown in Figure 8.
16

---

Published as a conference paper at COLM 2024
While Stiennon et al. (2020) choose 
 the EOS token, it may be possible to another token like <lim_end]>> instead as the EOS token. We suspect the key practice is to end the completion with some special token, so the model can learn when to stop.
Detail 4: Dataset SFT and preference datasets have different tokenization length
The SFT dataset had already been filtered such that all the reference summary lengths were controlled they have maximum of 48 tokens using the GPT2 tokenizer. In our case, we used Pythia's tokenizer (Biderman et al., 2023), with which the reference summaries have a maximum of 53 tokens. However, an interesting fact that the summary lengths in the preference dataset are not controlled to be the same. Figures 9 and 10 show the length distribution. Several observations:
1. The chosen /rejece response token length in the preference dataset can be as long as 169, significantly exceeding the 53 tokens found in the SFT dataset.
2. The median chosen response token length is 32, which is slightly longer than that of the rejected response token of 30.
Detail
Dataset Pre-tokenize the dataset: right pad the concatenation of queries and responses; left pad the queries
To pre-tokenize the dataset for training, we right pad the concatenation of queries and responses and left pad the queries, as shown below.
1. SFT dataset for SFT training: we concatenate the query and the reference response together and pad from the right, so during training each sampled batch will have the shape (B, 562).
2. Preference dataset for RM training: we concatenate the query-chosen and query- rejected responses together and pad from the right, so during the RM training, each sampled batch will have the shape (B, 638).
3. Preference dataset for RM evaluation: During RM evaluation, the sampled batch in the TL;DR splits will have shape (B, 638). Note that in the preference dataset, there is also split that measures the RM's generalization ability to the CNN / DM dataset, and this split has much longer token length; in particular, sampled batch from this dataset will have shape (B, 2021).
4. SFT dataset for PPO training: we pad the query from the left to make genera- tions compatible with transformer' (since decoder models require left padding for generations), so each sampled batch will have shape (B, 512).
> Detail 6: Dataset The validation split of the preference dataset has lot of OOD data.
As illustrated in Table (see Appendix for details on the exact policy comparisons and their counts), the sampling policies employed in the preference dataset exhibit significant diversity, which is out of the distribution of the sampling policies used in the train split. As result, the validation set serves great measure of the generalization ability of the (RM).
C General Details Appendix
Detail 7: Model Disable dropout ensure PPO's ratio calculation still works
We disable the dropout layers during training, similar to the settings in Ziegler et al. (2019); Huang et al. (2024). This is important for PPO training, especially because with dropout activated, the log probabilities of tokens will not be reproducible, making calculating the KL penalty unreliable while also causing the ratios of the PPO to be not is during the first
/https: http:///ttttt.ttooooo////////aaa httpa/////aappaaapapoo/e//e/e//e 0/chatml..nnmm md
17

---

Published as a conference paper at COLM 2024
Fleutherkkkyyytii------eeup Tokenizen: Preference dataset token length distribution
train split
train split
train split
chesen token length
rejected taken length
query chasen token length max token length-638 mean token 380.6 token length=386.00
max token length=134 mean token length=33.92
max token length-169 mean taken lengthh31.36
median token length=32.00
median token length=30.00
2000
1500
4000
4000
1000
2000
2000
500
50
100
125
150
600
train split
train split
validation split
query + rejected token length
query token length
chosen token length
max token length=627
max token length=512
max token longth-129 129
mean token length=378.00 04 median taken longth-384.00
m ean token length=346.60 6.
mean token length 40 median token length-35.00
median taken length-352.00
2000
1500
1500
00.00
1000
1000
500
2000
200
600
100
200
300
400
25
100
125
validation split rejected taken length max token length=1311 mean taken length-33.60
validation split query + chesen taken length max token length=6225 m can token fingth381.22 median token lengthh383.00
validation spiit query rejected taken length max taken length=
mean taken length-379.4 379.43 median token length=381.00
median token length=33.00
8000
2000
1500
6000
4000
1600
500
25
75
100
125
200
400
6:00
200
400
600
validation split
validation cnndm split chosen token length max length-155
validation split rejected length max taken longth-1555 mean token length=39
query token length
longth=512
mean token length=345.88
mean token anngth444.50
median token lengthh348800
median token lengthh43.00
median token length=39.00
1590
150
1000
100
590
50
50
1:00
200
300
400
500
50
100
150
50
1:00
150
validation cnndm split query chosen max token length=2021 mean token length-877.22
validation cnndm split query rejected length max token length=2021 mean length-872.51
validation, condm split query length length=1910
length
moan length-833.74 medial an taken length-737.00
median longth-786.00
median taken length-776.00
80
80
60
40
40
20
20
500
1000
1500
2000
500
1000
1500
2000
500
1000
1500
Figure 9: The token length visualization of the preference dataset.
epoch, causing PPO optimization problems. For consistency, we also disable dropout for SFT and RM training.
Detail Setup Tech stack
We used the transformers (Wolf et al., 2020) library's implementation of the Pythia models in conjunction with deepspeed's ZeRO Stage (Rasley et al., 2020; Rajbhandari et al., 2020) to help fit the models into the GPU memory; for 6.9B PPO training we also offload the reference policy and reward model to CPU. We launch experiments using accelerate (Gugger et al.,
18

---

Published as a conference paper at COLM 2024
Bleutherrijyythie-----eeeppp Tokenizer: SFT detaset token length distribution
train split
train split
validation split
reference res ponse taken length max Token length=
reference response token length
reference response token length max taken length- mean token length-33.90 median token length=33.00
max token length-561 m ean token length-379.69 median token length=381.00
mean token 1on7th-33.77
median token length=33.00
2000
5000
2.00
2500
20
30
50
100
200
300
400
500
20
30
50
validation split
test split
best split
query reference response token length length=
reference response taken length langth=S1. taken length-33.82 token length
reference response token lengt token longth-
length-380.08
taken 380.t
median enngth=322.00
Seigth=381.00
100
400
50
50
100
200
300
400
500
30 40 50
100
200
300
400
500
Figure 10: The token length visualization of the preference dataset.
2022) with 6f16 mixed-precisse training and track them with Weights and Biases (Biewald, 2020). We use 8xH100 machines and always upload the trained models to Hugging Face's model hub".
D SFT Details Appendix
Detail 9: SFT Training - Setups
Our SFT setup closely follows Stiennon et al. (2020), except for modified learning rate (Table 3).
Table 3: SFT hyperparameters
Hyperparameter
Default Value
Number of Train Epochs Optimizer
1 (or 116,722 episodes) AdamW (e = le 5, 1r = 3e 6)
Scheduler
Cosine
Batch Size
128
E
Reward Model Details Appendix
Detail 10: RM Training Setups
We follow Stiennon et al. (2020Y's ss original setting to train the RM, except that we used different learning rate (Table 4).
Detail 11: RM Training Reward head initialization
We follow Stiennon et al. (2020)'s s original setting to initialize the RM from the trained SFT model and create linear heard to output reward scalar with weights initialized according to (0, 1/ V (dmodel 1))
model. p 10
hnttps://huggingfaac /ot.////ggig.......coooeeeeeee co/models
Note Stiennon et al. (2020) have minor typo of saying the initialization was according to (O 1/ (dmodel + 1)), but the reference code clearly indicates otherwise.
19

---

Published as a conference paper at COLM 2024
Table Reward modeling hyperparameters
Hyperparameter
Default Value
Number of Train Epochs
(or 92,858 episodes) AdamW (c = le 5, 1r 3e 6)
Optimizer
Scheduler
Cosine 64
Batch Size
Detail 12: RM Training Extract reward from the EOS token
When obtaining the scalar reward, the RM does a forward pass on the sequence and extracts the reward only on the EOS token. (reward_ horrdddooel..___ py) This is implemented by finding the first index of the padding token and then minus 1. If the padding token does not exist, the extracted reward will then be logits corresponding to the last token of the sequence if that token is not the EOS token, its reward won't be used for PPO training, explained later in PPO's EOS trick > Detail 23:).
Note that Stiennon et al. (2020) choose the < </endof t from the base model the EOS token to extract the reward, but it is possible to use another special token. For example, Andrej Karpathy mentioned that the reward is extracted at <| reward] in OpenAl's newer GPT systems?.
Detail 13: RM Training - Most values the reward logits are non-valid and negative; only the reward logit at the EOS token are valid
What do the reward logits actually look like in these trained RMs? We include an example in Figure 3. We noticed the logits of non-EOS tokens are almost always negative in all the response-reward-loghh pairs from all policies and RMs.
Detail 14: RM Training Minor numerical differences between extracting reward with left and right padded queries
During RM training, the sequences are padded from the right with the shape (B, 638). However, left-padding the query is required for generation in PPO training. The query has shape (B, 512), and after generation (with sequence length 53), the query and response batch shape becomes (B, 565). As a result, we need to adjust the attention masks during RM forward calls.
Numerical note: Ledt-padding vs. right-padding can introduce minor numerical differences. For instance, in the 6.9B RM, the average reward scalar difference the SFT dataset between the two padding methods -0. 000.0004453001720000000
This difference is generally negligible
Detail 15: RM Training Reward normalization based on SFT demonstrations
Stiennon et al. (2020) suggested that "at the end of training, we normalize the reward model outputs such that the reference summaries from our dataset achieve a mean score of 0." We applied the same procedure by iterating through the SFT dataset and calculating the rewards of the query and reference responses, then calculating the mean reward and setting it as a bias in the reward head.
E.1 RM training results
The RM training loss, accuracy, and chosen reward value can be found in Figure 4. The training accuracy and losses appear stable. Overall, larger RMs have higher validation accuracy on both TLDDR and CNN/DM DM sets. Note the validation accuracy on the CNN/DM is very encouraging - the RM has never trained on CNN/DM data! We also performed
7https://youti httppps//ooouuub//auuaa///WWWA// de/bbzqumaY4L2A2t=990
20

---

Published as a conference paper at COLM 2024
Table 5: The mean and standard deviation of various metrics of the reward models across four random seeds. The table shows the metric names across different batches, confidences, and splits. There limited documentation from Stiennon (2020) about these batches and splits, but nevertheless interesting to see this table.
Metric Names
1B
2.8B
6.9B
Max
8.273 0.993
5.961 +: 2.45
11.75 2.203
Mean
2.114 = 0.939
0.925 = 2.386
4.783 = 1.545
Reward
Min
-5.461 1.754
-5.039 2.547
-3.016 1.421
Std
1.657 0.086
1.361 0.206
1.912 # 0.078
Overall Accuracy
0.628 0.002
0.669 0.003
0.689 0.004
0.661 = 0.016
0.682 = 0.024
0.709 = 0.009
0.694 0.023
0.718 0.011
0.732 = 0.014
8
0.398 = 0.014
0.63 = 0.008
0.636 = 0.009
0.378 = 0.005
0.687 0.017
0.091
10
0.626 = 0.007
0.655 = 0.015
0.007
0.508 +: 0.01
0.603 0.004
0.653 a 0.021
12
0.686 + 0.007
0.697 0.009
0.704 # 0.007
Batch Number
0.771 = 0.016
0.708 = 0.013
0.745 0.008
14
0.577 = 0.031
0.588 = 0.01
0.634 = 0.011
0.628 = 0.021
0.699
0.671
16
0.707 = 0.017
0.737 0.002
0.761 = 0.006
0.752 = 0.014
0.757
0.003
0.734 0.018
18
0.733 0.015
0.741 0.025
0.771 + 0.011
Validation Accuracy
19
0.636 + 0.02
0.688 = 0.012
0.714 # 0.01
20
0.671 0.005
0.705 0.008
0.711 = 0.007
22
0.587 = 0.006
0.632 = 0.009
0.651 = 0.005
0.693 0.012
0.758 0.005
0.795 0.004
0.669 = 0.011
0.706 = 0.012
0.718 0 0.007
0.635 0.005
0.656 0.011
0.674 = 0.003
4
0.58 0.005
0.562 0.006
0.589 0.009
Confidence
0.563 + 0.006
0.574 + 0.012
0.581 + 0.009
7
0.568 0.006
0.635 = 0.007
0.655 = 0.008
0.609 = 0.011
0.691 = 0.008
0.704 0.007
0.694 0.007
0.744 0.005
0.765
0.009
0.639 = 0.003
0.667 # 0.007
0.69 0.007
Split Valid
0.621 0.003
0.669
0.003
0.688
0.002
Overall Accuracy
0.627 + 0.013
0.665 + 0.01
0.686 + 0.003
0.679 + 0.06
0.714 = 0.027
0.723 + 0.009
Batch
Cnndmü
0.772 0.009
0.677 = 0.017
0.714=0.033
Cnndm2
0.564 = 0.012
0.646 + 0.013
0.666 = 0.005
0.589 * 0.094
0.804 * 0.043
0.815 * 0.022
0.641 0.139
0.661 0.107
0.732
0.036
Cnndm Accuracy
0.5 0.037
0.771 1771000000 0,023
0.736 = 0.014
0.597 0.053
0.028
0.615
Confidence
4.61100055
0.587 = 0.031
1.002
0.743 = 0.095
0.646 0.036
0.741 0.032
8
0.594 = 0.092
0.632 = 0.056
0.662 = 0.056
0.65 0.094
0.777 + 0.054
0.812 + 0.061
Split Valid
2
0.627 + 0.013
0.665 + 0.01
0.686 0.003
a comprehensive evaluation of the trained RM on the validation set and calculated the aggregated mean and standard deviation for each batch, split, and confidence in Table 5.
Detail 16: RM Training Different batches confidences have different accuracies
21

---

Published as a conference paper at COLM 2024
As shown in Table 5, different annotated batches could have different validation accuracies. Several observations:
1. The 1B model's validation accuracy at batch 11 is 0.508, which is no different from coin toss
2. The 1B model's validation accuracy at batch 13 is 0.771, a much higher accuracy.
3. The trained RMs generally have high accuracy for high-confidence preference pairs, which makes sense (e. the 6.9B model's validation accuracy with accuracy is 0.765).
4. Interestingly, the trained RMs also have high accuracy for very low-confidence pref- erence pairs for some reason (e.g., 6.9B model's validation accuracy with accuracy 1 is 0.795).
> Detail 17: RM Training- Preference consistency rate with GPT3.5
As per Goodhart's law when metric becomes the optimization goal, it ceases to be good metric (Gao et al., 2023). To verify whether RM is overfitting the current dataset's accuracy after training, we introduced GPT3.5 as an external LLM-judge (Zheng et al., 2023). By comparing the preferences of GPT3.5 and RM for the same set of preference data, we assess the actual training effects of RM across different model sizes. As depicted in Figure 5, we have observed the following:
1. For the 1B-sized model, the average preference consistency in multiple random experiments is close to 0.4, indicating that the 1B model has captured a different set of preference, contrary GPT3.5.
2. The average preference consistency rates for the 2.8B and 6.9B models are 0.726 and 0.767, respectively, both exceeding 0.5. Compared to the 1B model, the model size increases, RM can exhibit preferences similar to GPT3.5.
3. The difference in average preference consistency rates between the 2.8B and 6.9B models is 0.041, whereas the difference between the 2.8B and 1B models 0.353. The gains from increasing model size are gradually diminishing (maybe also because the accuracy is already high).
> Detail 18: RM Training RM calibration
RMs should predict the log probabilities that humans will prefer one completion versus others; to this end, Bai et al. (2022) propose visualization technique to see if these probabil- ities are accurate and well-calibrated. The idea is to plot the score difference between the chosen and rejected pairs in the x-axis and the accuracy of the RM in the y-axis. Intuitively, the larger the score difference, the more confident the model that one completion better than the other. We plot the RM calibration in Figure 5.
Overall, we do find a positive correlation between accuracy and score difference - this is good sign because models become higher score difference). However, the RMs are still under-calibrated, probably due to the diverse validation set (> Detail 6:) and different accuracies in these validation set (> Detail 16:).
accurate they become more confident (i.e.,
Detail 19: RM Training Comparison with DPO's implicit reward modeling
We also trained equivalent DPO models to compare the validation accuracy. We use the same hyperparameters used for RM training, except DPO also has ß hyperparameter, shown in Table 6.
During training, controlled the preference dataset iteration order well, this should be a fair comparison of explicit versus DPO's implicit reward modeling losses. The training curves can be found in Figure 6. There are couple of interesting observations:
22

---

Published as a conference paper at COLM 2024
Table 6: DPO hyperparameters
Hyperparameter
Default Value
Number Train Epochs
(or 92,858 episodes)
Optimizer
AdamW (e = le - 5, 1r 3e 6)
Scheduler
Cosine
Batch Size
64
ß (KL Penalty Coefficient for RLHF)
0.05
1. Validation accuracy regression in DPO: We found regression in the validation accuracy in DPO's final evaluation, and this finding holds true across model sizes and 4 random seeds; this suggests DPO's implicit reward modeling may not be equivalent to the regular explicit reward modeling. There are several factors that we suspect may be responsible for this difference. First, regular reward modeling's loss only applies to the EOS token, whereas in DPO, the loss applies to all the tokens. Second, DPO also has the RLHF & parameter in the loss, which is not present in regular reward modeling's 's loss (we chose & 0.05 to match PPO's s setting). Third, by modeling the reward as the difference logprobs between model and reference model, DPO's s objective may be harder to optimize in practice than the RM objective. Whereas an RM can easily learn large changes in reward using the linear head, DPO must drastically change many tokens' logprobs to do the same.
2. Decreasing chosen rewards: DPO's chosen and rejected rewards both generally decrease, whereas regular reward modeling's 's chosen rewards fluctuate, see Figure
We advocate for more research on how DPO's loss systematically affects RM accuracies.
F PPO Details Appendix
Detail 20: PPO Training Setups
Our PPO setup closely follows Stiennon et al. (2020), except for modified learning rate (Table 7).
Table PPO hyperparameters.
Hyperparameter
Default Value
Episodes
1,000,000 (or ~8.56 epochs)
Optimizer
AdamW le 5, 1r 6)
Scheduler
Linear
Batch Size
512
β (KL Penalty Coefficient for RLHF)
0.05
Y (Discount Factor)
1.0
À (for GAE)
0.95
Nmb Number of Mini-batches
K (Number of PPO Update Iteration Per Epoch)
4
8 (PPO's Policy Clipping Coefficient)
0.2
? (Value Clipping Coefficient)
0.2
C (Value Function Coefficient)
0.1
Value Function Loss Clipping
True 0.7
Sampling Temperature
Detail 21: PPO Training Re-use the SFT dataset and shuffle when reaches the end
Stiennon et al. (2020) trains the PPO models for 1M episodes, but the train split of the SFT dataset is only of size 116,722, educated guess is that the SFT dataset is re-used
23

---

Published as a conference paper at COLM 2024
repeatedly during PPO training. Specifically, we should shuffle the SFT dataset and sample from without replacement; once the dataset is depleted, we should reshuffle again and sample without replacement; we continue this process unfil we reach 1M episodes. pootasets/__iit____. pY#L27-L33)
Detail 22: PPO Training Value model initializes from the reward model; trained value model looks like per-token RM.
Similar to the settings in Stiennon et al. (2020), we initialize the value network based on the reward model. This warm-starting of the value network can greatly improve initial gradients to the policy and reduce drift alignment tax over training (Noukhovitch et al., 2023). Because of this, the values generated by the value network will look identical to the example in Figure 3 (> Detail 13:), where the values of most tokens are negative numbers except for the BOS token.
However, in RL training, the value function would aim to predict the end-of-episode return at each timestep token, effectively acting a per-token RM. In Figure 11, we show the rewards and values of a completion, where the 4. 5000 is the score from the RM corresponding the EOS token. The other values the rewards are per-token KL penalty. See https: /w/ndw/and/////////////// ai/ //ss/oostaaaaaaaa huang/ wummarize/uuss//ffeeee
logs for the full log.
2.96
2.95
3.07
4.46
Reward
-0.00
-0.02
0.00
4.50
Token
x,
X
EOS
Figure 11: Reward and values of completion. The score from the reward model at the EOS token 4. 50 while the rest of reward numbers are per-token KL penalty scores.
Detail 23: PPO Training "EOS trick" to ensure scores from the RM is valid
One interesting phenomenon we observed with the original checkpoint of Stiennon et al. (2020) is that the generated summaries always have less than 48 tokens and also end with an EOS token - this makes the comparison with the reference summaries more fair because the reference summaries are also less than 48 tokens (> Detail 4:). We suspect the following processes likely achieve it:
1. Always samples a fixed amount of 48 tokens (corresponding to 53 tokens in our reproduction) from the vocabulary (policy, 4y8)L48). In particular, the model will continue to sample tokens even it encounters an EOS token (this means after the EOS token the generations are unconditional).
2. Given the 48 tokens, the script then "truncates" at the EOS token, filling the tokens after the EOS token as padding tokens (sample. wyw.lyyh11111 tasks. py#L57-L62).
3. This "truncated" response is then passed to the reward model to get score; if the response does not contain any EOS token, we suspect Stiennon et al. (2020) replaced the score with -1, similar to the procedure described by Ziegler et al. (2019); Huang et al. (2024).
The EOS trick serves couple of purposes for RL:
1. Defined reward scores: It guarantees that the PPO model receives defined reward score. This is important because the RM only backpropagates loss on the EOS token during training. Without an EOS token, the completion ' reward is undefined. The EOS trick assigns a constant -1 reward these cases.
2. Constraining completion length: The trick encourages the model to generate concise completions - longer completions that lack an EOS token are penalized with a -1 reward.
24

---

Published as a conference paper at COLM 2024
ppo
2.5
80
2.0
RLHF Reward
1.5
60
1.0
from
40
0.5
2.8B
0.0
6,98
20
-0.5
1B
0
OK
200K
400K
600K
800K
1000K
0K
200K
400K
600K
800K
1000K
0.010
RM Score
4
Approx KL
0.008
0.006
2
0.004
0.002
0
OK
200K
400K
600K
BOOK
1000K
0K
200K
400K
600K
800K
1000K
Episode
Figure 12: Top left shows PPO's RLHF's reward Rxx,y) (Equation 3). The top right figure shows the mean of the sum of per-token KL divergence between the RL and SFT policies. The bottom left shows the scores obtained from the reward model.
Essentially, the EOS trick helps ensure the completion ends with EOS token, rewards are well-defined.
> Detail 24: PPO Training (Optional) Reward whitening
As indicated in Huang et al. (2024), Ziegler et (2019) implement a whiten function that looks like below, basically normalizing the values by subtracting its mean fol- lowed by dividing by its standard deviation. Optionally, whiten can shift back the mean of the whitened values with shift_mean-True, In each minibatch, PPO could whiten the reward whiten(rewards, shift_mean=False) without shifting the mean (Im hww.aanwweeeeerreee////aappppipp
25

---

Published as a conference paper at COLM 2024
def whitennvalues, shi shift_mean-True)::
mean, var torch. mean (values), torch. var unbiased-False) whitened = (values mean) * torch. rsqrt(var + 1e-8) if not shift_mean: whitened += mean
return whitened
Detail 25: PPO Training Advantage whitening
Similar to practices identified in Engstrom et al. (2020); Andrychowicz et al. (2021); Huang et al. (2022), PPO whitens the advantages whi (hiteen(dvantagge) with the shifted mean (1m_human_pprett hww.aanorrefrrrrnnestttaaooooooo pyHLL33))
E1 PPO training results
We include several PPO's learning curves in Figure 12. We also evaluate the final model checkpoint on the validation set of the SFT dataset using GPT as a judge against the reference summaries in Figure 1 (the GPT prompt an ablation study in which we used reward whitening (> Detail 24:), and the results are in Figure 7. Finally, to help understand the correlation between summary length and win rate, we plot the win rate against the Figure
be found in Appendix I). We also conducted
length/ rreeerence summary length) at
Several observations:
1. RLHF objective goes up. Our PPO implementation at least optimizes the RLHF objective, increasing the score total.
2. Good scaling behaviors. The preference rate of the PPO models scales nicely with the model checkpoint sizes. In particular, GPT prefers our best 6.9B model nearly 80% of the time.
3. Overooptimiztionn in 1B models. For 1B models, the KL divergence seems high (around 50 and 85 for two runs). From an optimization point of view, there is nothing wrong with them because these two runs got higher RLHF Reward (xx,y) (Equation 3), but GPT then judges these two checkpoints to have poor human preference: less than 20% of time GPT prefers them over reference summaries)
Upon inspection of these overoptimized samples, we find the PPO policy would concatenate the strings like Myfiettrieneeeettyyccceeeeeeeeee htwp://www...oo.oaooooooooooooee opphhonsnndddeeeewuuuubrrraaappp
(see https: //wandw ai/ costa-huang/tldr. sumrrrssumrrrrre www.susssuaaaaaaaunnnaaaaaa httpriie//////nnnneee aq as an example).
4. Reward whitening makes the model generate shorter outputs. We conducted an ablation study with and without reward whitening in Figure Our experiments show that reward whitening makes the model's completions get lower preference rate, and the completions are shorter than those without reward whitening. How- ever, when inspecting the length-controlled comparisons in Figure 8 (right), the models perform similarly with or without reward whitening in different summary lengths.
5. PPO models significantly outperform SFT when controlling for length. As shown in Figure (left), while PPO gets higher win rate than SFT, the models' responses are generally longer compared to SFT responses, so the summary length is a con- founding factor. To address this issue, we control for ratio summary length to reference length in Figure 8 (right) and show that PPO models outperform SFT models across all summary lengths. We also find that PPO win-rate increases with summary length. This implies that either GPT3.5 prefers longer summaries or longer summaries better optimize true human preference (perhaps implicitly) (Dubois et al., 2023).
26

---

Published as a conference paper at COLM 2024
Table 8: Sample query, responses from the 1B SFT, PPO, and DPO models; scores are from 6.9B model. We mark the response tokens ranked top by the pre-trained model blue, meaning unshifted tokens, tokens ranked within the top yellow, meaning marginal to- kens, and tokens ranked beyond the top 3 red, meaning shifted tokens (Lin et al., 2024). Essentially, red and yellow tokens highlight what the SFT, PPO, and DPO models would do differently compared to the pre-trained model. We released the source code to load the model and generate this visualization https: / thub..u....... com/ wwwzjjn/summriizza py.
Type
Content
Score (RM)
Query
SUBREDDIT: r/ AskRedddi
N/A
TITLE: How do you get someone out of your head?
POST: Hi, I'm 22, and I have been with my girl-
friend for years now. We recently moved together. We've always loved each other intensely.
Problem, I recently started to have feelings for other person friend). This person has had boyfriend now years, and has absolutely no ideas. Those feelings were so strong, it was hard to hide them. After 2 months of me being distant and really sad, my girlfriend forced me to say what bothering me. I'm not good liar, and now she knows.
We decided to give us a week alone, went to my parents.
Now, I'm completely lost. keep thinking about this person, and hate that. would like for those feelings to go away, to leave me alone. But can't. What do I do? It's been 3 months now, and I'm just desperate.
TL;DR:
SFT Model Response
haaee feelings for friend, and I'm sure how to get them out of my head,< endoftext
-3.4151
PPO Model Response
Been with GF for 5 years, recently started to have feelings for another person. love her deeply how- ever the feelings are driving me crazy, What do do? Completely lost< endoftext
2.8743
DPO Model Response
recently started to have feelings for another 1.354 person, My girlfriend forced me to say what
bothering me. She now knows. want those feel- ings to go away, but can't, What do do?< endof-
text
Reference response
long relationship; fell in love with an other person; admitted it; would like it to disappear, though it doesn't.< t endoftext!
-1.6587
Base Model Response
How do you get someone out of your head?
-6.7223
A:
think you're in situation where you need to get out your head. You're 're not in a relationship, and you're re not in a relationship with someone who is good fit for you. You're re in relationship with someone who not good fit for you. You're re in
E2 Visualizing the aligned models vs pre-trained models
Lin et al. (2024) proposed an interesting visualization regarding how aligned models would behave differently from pre-trained models. The idea is to sample response from the
27

---

Published as a conference paper at COLM 2024
aligned LLM and check if the pre-trained LLM would greedy sample the same tokens; if so, then color the text blue (unshifted tokens); if the token is within the top probability, color the text yellow; else color the text red (shifted tokens). In simpler terms, the red tokens correspond to what aligned models do differently. We include such visualization of 1B models in Table 8. There are more visualizations of models in the Appendix K. Several observations:
1. Pre-trained model would continue sampling. As result, the generated summary would go significantly beyond the typical lengths of the reference summary or SFT PPO DPO summary.
2. Most tokens are unshifted tokens. Similar the findings in Lin al. (2024), we find most tokens to be unshifted tokens this means arguably that the summarization ability mostly comes from the pre-trained model.
3. Fine-tuned models mostly change behaviors at the beginning and the end. The SFT / PPO / DPO models always alter the initial output and end the summary with an EOS token.
G List of model checkpoints and tracked logs
The list of model checkpoints and tracked logs can be found at Table
H List of model checkpoints and tracked logs
The list of model checkpoints and tracked logs be found at Table 9.
I GPT as judge prompt
We modify the GPT as judge prompt from Rafailov et al. (2023).
Which of the following summaries does better job of summarizing the most importtan points in the given forum post, without including unimportant or \ irrelevant details? Judge based on accuracy, coverage, and coherence.
Post: <post>
Summary A:
<Summary A>
Summary B:
<Summary B>
FIRST provide one-sentence comparison of the two summaries, explaining which you prefer and why. SECOND, new line, state only "A" or "B" to indicate your \ choice. Your response should use the format: Comparison: <one-sentence comparison and explanation> Preferred: <"A" or "B">
Following Wang et al. (2023); Zheng et al. (2023) we randomize the order of the summaries to remove positional bias in GPT-3.5 Turbo.
J Model win rate versus summary lengths
Figure 13 show more plots like Figure 8 (right).
28

---

Published as a conference paper at COLM 2024
Table 9: List of Hugging Face model checkpoints and tracked Weights and Biases logs.
Model Checkpoint
Tracked Wandb Logs
Base Model
Type
Seed
TleutherAA/ppthhaallbdddddp
ppo
44413
Link
Link
55513
Link
Link
66613
Link
Link
77713
Link
Link
reward
44413
Link
Link
55513
Link
Link
66613
Link
Link
77713
Link
Link
sft
44413
Link
Link
55513
Link
Link
66613
Link
Link
77713
Link
Link
EllutherAA/yyyhiss2SBddduup
ppo
44413
Link
Link
55513
Link
Link
66613
Link
Link
77713
Link
Link
reward
44413
Link
Link
55513
Link
Link
66613
Link
Link
77713
Link
Link
sft
44413
Link
Link
55513
Link
Link
66613
Link
Link
77713
Link
Link
EllubherA//pyrhiaa..bddddd
ppo
44413
Link
Link
55513
Link
Link
66613
Link
Link
77713
Link
Link
reward
44413
Link
Link
55513
Link
Link
66613
Link
Link
77713
Link
Link
sft
44413
Link
Link
55513
Link
Link
66613
Link
Link
77713
Link
Link
29

---

Published as a conference paper at COLM 2024
BleutherAAljythie b-deduped seed 44413
Eleuther/Wjpyhhie 2.b-ri--eeeee seed 44413
BeetherAAlgythhiee 9b-deduped seed 44413
Winzate apaine reterance summaress
racolare surt 2uubb-0023
0.8
0.8
0.4
0.2
9.5
1.0
seed 55513
seed 55513
seed 55513
Whrraf aganett miterrce Lammanee
Decoesiig Qu1 (11---155
0.2
0.8-
0.4
0.6
0.4
-1.5
1.0
1.0
1.0
1.0
Whiate sosindt reference summertt
racendin to
0.4
0.2
0.2
-1.0
-0.5
0.0
0.5
1.0
-1.0
-0.5
0.0
0.5
1.0
-1.0
-0.5
0.0
0.5
1.0
77713
seed 77713
seed 77713
1.0
1.0
Wheaate agatns refeennee simmeres
(acorddng 914-35 StufE0-0022 01251
PPO wi whithn rewards
0.9
SFT
0.8
ruference summary
0.7
0.4
0.4-
0.2
0.0
0.5
-1.0 log/summet ary length inferencce lengih)
0.5
log(summerry length relerence length)
logisumeaary length Inferrecce lergth!
Figure 13: The figure shows the win rate for different summary lengths of the SFT and PPO models across random seeds and model sizes. Note that some of the PPO seeds corresponds to overoptimized policies.
K Visualization on aligned models vs base models
This section generates more samples for different-sized models like Table 8.
1. 1B model responses in Table 10, Table 11, and Table 12
2. 2,8B model responses in Table 13, Table 14, Table 11, and Table 12
3. 6.9B model responses in Table 17, Table 18, Table 19, and Table 20
30

---

Published as a conference paper at COLM 2024
Table 10: Sample query, responses from the 1B SFT, PPO, and DPO models; scores are from a 6.9B model. See Table 8 and Section 8 for semantics on colors.
Type
Content
Score (RM)
Query
SUBREDDIT: pettyreeeegegee
N/A
TITLE: So, my mom woke me up with a loud TV.
POST: She was in her living room, watching TV. This was about 8:30 in the morning, and she was exercising. She turned the TV up extra loud to hear it over her excercycle, and woke me up. went in there asking for her to turn it down. She said she didn't have to; explained that always used headphones so she didn't 't have deal with my noise and that she should give me little more respect, given that paid rent at the time. She disagreed. I went back to my room, rather pissed off at the lack of equality. I had no lock my door; but had dresser right next to it, so pulled one of the drawers out enough so that it caused the door to not be openable. Then, turned my speakers up really loud and blasted Gangnam Style on repeat, with the bass cranked up as high as it could go.
If you hate Gangnam Style for being overplayed, you will see why chose that particular song. personally don't mind it. But here's the thing about my bass; it vibrates the walls, making one hell of a lot of noise. Needless to say, my mom was not pleased and shut off the internet. But it was oh so worth it.
TL;DR:
SFT Model Response
Mom woke me up with loud TV. turned down, and she got pissed. turned it up and she got pissed < endoffext
-3.9943
PPO Model Response
While watching TV my mom woke me up with loud racket that vibrated the walls making one hell a racket, resulting her shutting off internet. Very satisfying aftermath< ent endoftext
1.3313
DPO Model Response
an over-played Gangnam Style song on the morning TV wake-up call, and shut it off. It was worth it endooftex
My mom was annoyed
-0.6789
Reference response
Mom had the TV on loud and woke me up, didn't care that I'd respected audio levels in the house, countered with playing Gangnam Style on repeat with the bass thumping through the walls.< endoftext!>
1.3359
Base Model Response
was pissed off at my mom for not giving me more respect, and was pissed off at her for shutting off the internet.
-7.4169
I'm not sure if I'm the only who has this prob- lem, but I've been having problem with my com- puter for a while now. I've been using it for work, and I've been using it for school. I've been using
L Details the comparison pairs in the preference dataset
The comparison pairs and their counts can be found in Table 21, Table 22, Table 23, Table 24, Table 25, Table 26, Table 27.
31

---

Published as a conference paper at COLM 2024
Table 11: Sample query, responses from the 1B SFT, PPO, and DPO models; scores from a 6.9B model. See Table 8 and Section 8 for semantics on colors.
Type
Content
Score (RM)
Query
SUBREDDIT:
r/ rellattiosship
N/A A
TITLE: My girlfriend (20f) of two years cheated on me (20m) by kissing two guys at a Halloween party. POSI: Lately her and have been having few problems, and these problems have been brought up before a few times. One problem being that don't show enough affection. don't tell her she's pretty very often don't 't compliment her much. feel terrible about it, but this time was really trying to change her.
For Halloween she went to visit her step brother at a college and I got drunk with my friends and watched movies. Last night (11/1) huge fight about me not changing and how our relationship won't work out and basically broke up over the phone. So in an effort to try and fix it drove to her house. She told me how at the parties she went that two guys kissed her. The first she pushed away, but the second one asked her if she kissed him back and she said yes and that she did it because it made her feel wanted, which guess I haven't been making her feel that way lately. We cried, we talked about everything, we had great sex, and I stayed over her house just to sleep with her and then snuck out in the morning so her parents wouldn't ' know.
got in
We both obviously want to work things out but aren't sure if we should, love this girl, but the more think about it, all I can think about is her cheating on me, and more importantly, liking it. It makes me sick to my stomach. Should even try to fix it or would be better off cutting all ties. TL;DR:
SFT Model Response
Girlfriend cheated on me by kissing two guys at a Halloween party. don't know if should try to fix it or cut all ties.c < endoftext
1.6686
PPO Model Response
My girlfriend of two years cheated me by kiss- ing two guys at Halloween party. love her but the more think about the more feel sick about the situattion Should try to fix it cut all ties?< lenoottett
3.7295
DPO Model Response
(20m) by kiss- ing at Halloween party and we cried and talked
My girlfriend (20h) cheated
2.7493
about it Should try to fix it or should just cut
ties?< lnnooftext
Reference response
My girlfriend kissed two guys at a Halloween party and liked We love each other but don't know what to do about it.< endoftext |>
0.1145
Base Model Response
I'm not sure if should try to fix it or cut ties. I'm not sure if should try to fix it or cut ties. cut ties.
-5.7284
I'm not sure if should try to fix I'm not sure if should try fix cut ties. I'm not sure if should try to fix or
32

---

Published as a conference paper at COLM 2024
Table 12: Sample query, responses from the 1B SFT, PPO, and DPO models; scores are from 6.9B model. See Table 8 and Section 8 for semantics on colors.
Type
Content
Score (RM)
Query
SUBREDDIT: r/ running
N/A
TITLE: One year post-pregnancy, aches and pains in hips and knees, can't even comfortably do day C25k. Help?
POST: I'll try to keep this short!
*BBacgrround*
again/off again (very ca- sual!) jogger, typically doing 3 5 k My knees have always been finicky, and went to physio who thought had "runner's 's knee" Pre-pregnancy, my "runner's s knee" would flare up when got to the 8 10 distance range, even if I had decent base (doing C210k type program) "Current Problem*
I've always been
run had my baby year ago, so all in all haven't run for about 1.5 years. I'm quite slim and have been doing acrobics-style classes for the past year, I'm not totally out of shape. Body weight exercises, aerobics, bikes and ellipticals are all fine. However, when run even the tinyiest bit, or even go on long walk hike, my pelvis gets very tight, and my knees start hurting very quickly, al- ready am doing general squats/ squats/lunges, stretching type things.
and
I'm starting to feel like running just isn't 't for me anymore. Which is a bummer, because really joy running!
Has anyone had something similar? Can anyone recommend some stretches or exercises that might help? Should see Dr? Or should just see physio? Not quite sure how proceed. Thanks!
TL;DR:
SFT Model Response
have very tight pelviš and knees, and can't run anymore. I'm starting feel like I'm not running anymore. Should see Dr? Or should just see < endoftext 1>
0.2434
PPO Model Response
One year post-pregnancy, my knees hurt when run, can't even comfortably do day of C25k. Has anyone had something similar and what ex- ercises stretches might help (preferably with Dr)?< endoftext
4.8225
DPO Model Response
had my pregnancy year ago and haven't it run for about 1,5 years, but when run my pelvis gets sore and fight, and my knees start hurting very quickly Should T see Dr, or should just see physio? Not quite sure how to proceed, < endoftext
3.1864
Reference response
think pregnancy messed with my body, now can't even even the smallest amount without pain in my pelvis and knees. I'm fairly certain the problem isn't just that I'm completely out of shape.< endoftextl>>>
2.4334
Base Model Response
have very tight pelvis, and my knees are very sore. I'm not sure if it's the pregnancy or the run- ning, but I'm not sure can run anymore.
-8.9523
have had the same problem. I have been running for about year now and have had a lot of pain in my knees. hhve'Been doing

---

Published as a conference paper at COLM 2024
Table 13: Sample query, responses from the 2.8B SFT, PPO, and DPO models; scores are from 6.9B model. See Table 8 and Section 8 for semantics colors.
Type
Content
Score (RM)
Query
SUBREDDIT: r/ AskReddii
N/A
TITLE: How do you get someone out of your head?
POST: Hi, I'm 22, and I have been with my girl- friend for years now. We recently moved together. We've always loved each other intensely.
Problem, I recently started to have feelings for an other person friend). This person has had boyfriend for now years, and has absolutely ideas. Those feelings were so strong, it was hard to hide them. After 2 months of me being distant and really sad, my girlfriend forced me to say what bothering me. I'm not good liar, and now she knows.
We decided to give us a week alone, went to my parents.
Now, I'm completely lost. keep on thinking about this person, and I hate that. I would like for those feelings to go away, to leave me alone. But can't. What do I do? It's been months now, and I'm just desperate.
TL;DR:
SFT Model Response
I'm love with friend, and can get her out of my head. want to get her out of my head, but can't.+ < endoftext
-3.6299
PPO Model Response
Been with my gf for 5 years, recently developed feelings for another person, Can't stop thinking about him, and hate it. Has anyone been simi- lar situation and how did you get past it? endof- text
3.0524
DPO Model Response
Been with gf for 5 years, moved together, started to have strong feelings for someone else. Made clear that needed to explain why was being distant. Now I'm 3 months in and can't go back to normal How do get the strong feeling go away?
-2.9696
[LDRR: Been with gf for 5 years, moved together,
started to have strong feelings for someone else.
Made it clear to me that needed to explain why
Reference response
long relationship: fell in love with an other person; -1.6587 admitted it; would like it to disappear, though it doesn't.< endoftext >
Base Model Response
How do you get someone out of your head?
-6.611
I'm going to assume that you'te 'e not in a relation- ship with this person, and that you're 're not in re- lationship with your girlfriend. If you're rou're in lationship with your girlfriend, then you'le 'e in relationship with her. you're not in relationship with her, then you're not in a relationship with her. If you'te re in a relationship with your girlfriend, then you're re in
34

---

Published as a conference paper at COLM 2024
Table 14: Sample query, responses from the 2.8B SFT, PPO, and DPO models; scores from a 6.9B model. See Table 8 and Section for semantics on colors.
Type
Content
Score (RM)
Query
SUBREDDIT: peetyreevnngge
N/A
TITLE: So, my mom woke me up with a loud TV.
POSI: She was in her living room, watching TV. This was at about 8:30 in the morning, and she exercising. She turned the TV up extra loud to hear it over her excercycle, and woke me up. went in there asking for her to turn it down. She said she didn't 't have to; I explained that always she didn't have to deal with my noise and that she should give me little more respect, given that paid rent the time.
used headphones
She disagreed. I went back to my room, rather pissed off at the lack of equality. had no lock my door; but had dresser right next to it, pulled one of the drawers out enough so that it caused the door to not be openable. Then, turned my speakers up really loud and blasted Gangnam Style repeat, with the bass cranked up as high could go.
If you hate Gangnam Style for being overplayed, you will see why chose that particular song. personally don't mind it. But here's ' the thing about my bass; it vibrates the walls, making one hell of lot of noise. Needless to say, my mom was not pleased and shut off the internet. But it was oh so worth it. TL:DR:
SFT Model Response
My mom woke up with loud TV, and turned it up really loud. She shut off the internet endof- text
-0.3209
PPO Model Response
My mom woke me up with a loud TV, turned my speakers up really loud and blasted Gangnam Style. She shut off the internet afterwards, < endofteet >
2.4708
DPO Model Response
woke me up with the TV turned up loud, so blasted Gangnam Style on her bass-vibrating walls, to show her who's boss endoftext
My sanu--ooluee-wking--uppe
1.8347
Reference response
Mom had the TV on loud and woke me up, didn't care that I'd respected audio levels in the house, so countered with playing Gangnam Style repeat with the bass thumping through the walls.< endoftext >
1.3359
Base Model Response
woke up my mom with loud TVV. She was exer- cising. was pissed. TITLE: I'm not a racist, but I'm not racist. POST: I'm not a racist. I'm not racist. I'm not racist. I'm not racist. I'm not racist. I'm not racist. I'm not a racist. I'm not racist. I'm not racist. I'm not a racist.
-5.5731
35

---

Published as a conference paper at COLM 2024
Table 15: Sample query, responses from the 2.8B SFT, PPO, and DPO models; scores are from a 6.9B model. See Table 8 and Section 8 for semantics on colors.
Type
Content
Score (RM)
Query
SUBREDDIT: r/reaationship
N/A
IIILE: My girlfriend (20f) of two years cheated on me (20m) by kissing two guys at a Halloween party. POSI: Lately her and have been having few problems, and these problems have been brought up before a few times. One problem being that don't show enough affection. don't tell her she's pretty very often or don't compliment her much. feel terrible about it, but this time was really trying to change her.
For Halloween she went to visit her step brother at a college and I got drunk with my friends and watched movies. Last night (11/1) huge fight about relationship won't work out and basically broke up over the phone. So in an effort to try and fix it drove to her house. She told me how at the parties she went that two guys kissed her. The first she pushed away, but the second one asked her if she kissed him back and she said yes and that she did it because it made her feel wanted, which guess I haven't been making her feel that way lately. We cried, had great sex, and I stayed over sleep with her and then snuck out in the morning so her parents wouldn't 't know.
got in not changing and how our
talked about everything, we
house just to
We both obviously want to work things out but aren't It sure if we should. love this girl, but the more think about it, all can think about is her cheating on me, and more importantly, liking it. It makes me sick to my stomach. Should even try to fix it would be better off cutting all ties. TL;DR:
SFT Model Response
Girlfriend cheated on me by kissing two guys at a Halloween party. want fix but don't know
0.3877
if I should, endoftext
PPO Model Response
My girlfriend of two years cheated me by kiss- ing two guys at Halloween party. love her but the thought her cheating makes me sick to my stomach. Should try to fix or would be better off cutting all ties? < endoftext
4.1882
DPO Model Response
My girlfriend cheated on me by kissing two guys at party while she was visiting family, She told me ib during our fight and cried. She says Says she did it because she felt wanted by the guys, but think it goes deeper than that. hate that she did it, and hate myself for not showing her enough affection.
3.3642
Should try fix things cut all ties.<l fies.<Iendof-
text
Reference response
My girlfriend kissed two guys Halloween party and liked We love each other but don't know what to do about it.< endoftext
0.1145
Base Model Response
I'm love with a girl, but I'm not sure if should try to fix things or just cut all ties.
-5.5782
TL;DR: I'm in love with girl, but I'm not sure if just cut all ties.
should try to fix things
TL;DR: I'm in love with girl, but I'm not sure
should try to fix things or just cut all ties.
TLDRR: I'm in 1ove36

---

Published as a conference paper at COLM 2024
Table 16: Sample query, responses from the 2.8B SFT, PPO, and DPO models; scores are from a 6.9B model. See Table 8 and Section 8 for semantics on colors.
Type
Content
Score (RM)
Query
SUBREDDIT: r/ running
N/A
TITLE: One year post-pregn,ncy, aches and pains in hips and knees, can't even comfortably do day C25k. Help?
POST: I'll try to keep this short!
*BBackrround*
again/ again (very ca- sual!) jogger, typically doing 3 5 k My knees have always been finicky, and went to physio who thought had "runner's 's knee" Pre-pregnancy, my "runner's s knee" would flare up when got to the 8 10 distance range, even if I had decent base (doing C210k type program) "Current Problem*
I've always been
run had my baby year ago, so all in all haven't run for about 1.5 years. I'm quite slim and have been doing acrobics-style classes for the past year, I'm not totally out of shape. Body weight exercises, aerobics, bikes and ellipticals are all fine. However, when run even the tinyiest bit, or even go on long walk hike, my pelvis gets very tight, and my knees start hurting very quickly, al- ready am doing general squats/ squats/lunges/ stretching type things.
and
I'm starting to feel like running just isn't ' for me anymore. Which a bummer, because really joy running!
Has anyone had something similar? Can anyone recommend some stretches or exercises that might help? Should see Dr? Or should just see physio? Not quite sure how proceed. Thanks!
TL;DR:
SFT Model Response
I'm runner, but I'm not sure I'm ever going to able to run again. Any advice?< endoffext
-1.774
PPO Model Response
One year post-poegnanee pregnancy, aches and pains in hips and knees, can't even comfortably do day C25k. Has anyone else gone through something simi- la??
4.5098
What exercises exercises/ /stretches might help?< endof- text
DPO Model Response
One year post-pregnancy, my pelvis and knees hurt when run, and I'm not exactly out of shape Has anyone had something similar, and rec- ommend stretches or exercises? Should see Iendoffett
5,5663
Reference response
think pregnancy messed with my body, now can't even run even the smallest amount without pain in my pelvis and knees. I'm fairly certain the problem isn't just that I'm completely out of shape.< endoftext
2.4334
Base Model Response
I'm runner, but I'm not runner anymore I'm not out of shape, but I'm not in great shape either* I'm not sure what to do about my aches and pains A:
-7.651
I'm not doctor, but have few suggestions. would start with general stretching routine. would start with few minutes of static stretching, then move on to some dynamic stretching. I would
37

---

Published as a conference paper at COLM 2024
Table 17: Sample query, responses from the 6.9B SFT, PPO, and DPO models; scores are from a 6.9B model. See Table 8 and Section for semantics on colors.
Type
Content
Score (RM)
Query
SUBREDDIT: r/ AskRRdid
N/A
TITLE: How do you get someone out your head?
POST: Hi, I'm 22, and I have been with my girl-
friend for 5 years now. We recently moved together. We've always loved each other intensely.
Problem, recently started have feelings for other person (a friend). This person has had a boyfriend for now years, and has absolutely no ideas. Those feelings were so strong, it was hard to hide them. After months of me being distant and really sad, my girlfriend forced me to say what was bothering me. I'm not good liar, and now she knows.
We decided to give us week alone, I went to my parents.
Now, completely lost. keep on thinking about this person, and hate that. would like for those feelings to go away, to leave me alone. But can't. What do I do? It's been months now, and I'm just desperate.
TL;DR:
SFT Model Response
have feelings for an other person, and don't
-3,2271
know how get rid of them.< endoftext
PPO Model Response
Been with my girlfriend for years, recently devel- oped feelings for another person. would like for those feelings to go away, but they won't I'm lost and don't know what to do< endoftext >
3.0259
DPO Model Response
In loving relationship with my girlfriend for years. Started having feelings for another person. My girlfriend found out, and talked. Now I'm lost,
2.5583
can't get those other feelings to leave me
alone lenoffeext
>
Reference response
long relationship; fell in love with an other person; admitted it; would like it to disappear, though it doesn't.<l heittteeennde endoftext!>
-1.6587
Base Model Response
I'm 22, and have been with my girlfriend for 5 years now. We recently moved together. We've always loved each other intensely. I recently started to have feelings for an other person (a friend). This person has had boyfriend for now 3 years, and has absolutely strong, was hard to hide them. After months of me being distant and really sad, my girlfriend forced me to say what was bothering me. I'm not
-4.9909
ideas. Those feelings were
38

---

Published as a conference paper at COLM 2024
Table 18: Sample query, responses from the 6.9B SFT, PPO, and DPO models; scores are from a 6.9B model. See Table 8 and Section 8 for semantics on colors.
lype
Content
Score (RM)
Query
SUBREDDIT: r/ pettrreeeeeggee
N/A A
TITLE: So, my mom woke me up with a loud TV. POST: She was in her living room, watching TV. This was about 8:30 in the morning, and she was exercising. She turned the TV up extra loud to hear it over her excercycle, and woke me up. went in there asking for her to turn it down. She said she didn't have to; explained that always used headphones so she didn't 't have deal with my noise and that she should give me little more respect, given that paid rent at the time.
She disagreed. I went back to my room, rather pissed off the lack of equality. I had no lock my door; but had dresser right next to it, so pulled one of the drawers out enough so that it caused the door to not be openable. Then, turned my speakers up really loud and blasted Gangnam Style on repeat, with the bass cranked up as high as it could go.
If you hate Gangnam Style for being overplayed, you will see why chose that particular song. personally don't mind it. But here's the thing about my bass; it vibrates the walls, making one hell of a lot of noise. Needless to say, my mom was not pleased and shut off the internet. But it was oh so worth it.
TL:DR:
SFT Model Response
My mom woke me up with loud TV, blasted Gangnam Style repeat, with the bass cranked
0.9671
up as high as it could go,< endoftext >
PPO Model Response
My mom woke me up with loud TV. blasted Gangnam Style loudly through my speakers, mak- ing lot of noise and annoying her, Got some sweet revenge< lendoftext
3,9558
DPO Model Response
My mom woke me up with loud TV, because she couldn't t hear her excercycle while exercising, She was annoyed with me for having head when paid rent. blasted Gangnam Style loud enough to vibrate the walls. She turned off the internet,< lend- 1
1.3729
Reference response
Mom had the TV on loud and woke me up, didn't care that I'd respected audio levels in the house, countered with playing Gangnam Style on repeat with the bass thumping through the walls.< endoftext
1.3359
Base Model Response
My mom woke me up with a loud TV. SUBREDDIT: r/ pettyrevengee IITLE: I'm a little bit of dick.
-8.1818
POST: I'm little bit of dick.
I'm a little bit of a dick.
I'm a little bit of a dick.
I'm a little bit of dick.
I'm little bit of dick.
I'm
39

---

Published as a conference paper at COLM 2024
Table 19: Sample query, responses from the 6.9B SFT, PPO, and DPO models; scores are from a 6.9B model. See Table 8 and Section 8 for semantics on colors.
Type
Content
Score (RM)
Query
SUBREDDIT: r/ rrelatiiosshis
N/A
TITLE: My girlfriend (20f) of two years cheated on me (20m) by kissing two guys at Halloween party. POST: Lately her and have been having few problems, and these problems have been brought up before a few times. One problem being that don't show enough affection. I don't tell her she's pretty very often don't compliment her much. feel terrible about it, but this time was really trying to change for her.
For Halloween she went to visit her step brother at a college and I got drunk with my friends and watched movies. Last night (11/1) huge fight about me not changing and how our relationship won't work out and basically broke over the phone. So in effort to try and fix it drove to her house. She told me how at the parties she went that two guys kissed her. The first one she pushed away, but the second one asked her if she kissed him back and she said yes and that she did it because it made her feel wanted, which guess I haven't been making her feel that way lately. We cried, we talked about everything, we had great sex, and I stayed over at her house just to sleep with her and then snuck out in the morning her parents wouldn't know.
got in
We both obviously want to work things out but aren't 't sure if we should. love this girl, but the more think about it, all I can think about is her cheating me, and more importantly, liking it. It makes me sick to my stomach. Should even try to fix it or would be better off cutting all ties. TL;DR:
SFT Model Response
Girlfriend cheated on me by kissing two guys at
1.3901
Halloween party. Should try or cut
?ies?< lndoffexx
PPO Model Response
My girlfriend of two years cheated on me by kiss- ing two guys at Halloween party. love her but am conflicted about whether or not to try and fix things Should I cut ties or try salvage what we have left< endoftext
5.2579
DPO Model Response
My girlfriend cheated on me by kissing two guys af Halloween party friendly make out session. She claims she only kissed to feel wanted, but I can't shake the feeling that she cheated and want to cut all lnnoffeen
2.0515
Reference response
My girlfriend kissed two guys Halloween party and liked We love each other but don't know what to do about it.< endoftext 1>
0.1145
Base Model Response
My girlfriend cheated on me with two guys at a Halloween party.
-5.0535
SUB: r/relationships
TITLE: My girlfriend (20f) of two years cheated on me (20m) by kissing two guys at Halloween party.
POST: Lately her and have been having few problems, and these problems have been brought up before few times. One problem being that don't 't' show enough affection. don't tell her she's 40
pretty

---

Published as a conference paper at COLM 2024
Table 20: Sample query, responses from the 6.9B SFT, PPO, and DPO models; scores are from a 6.9B model. See Table 8 and Section 8 for semantics on colors.
Type
Content
Score (RM)
Query
SUBREDDIT: r/ running
N/A
TITLE: One year post-pregn,ncy, aches and pains in hips and knees, can't even comfortably do day C25k. Help?
POST: I'll try to keep this short!
*BBackrround*
again/off again (very ca- sual!) jogger, typically doing 3 5 k My knees have always been finicky, and went to physio who thought had "runner's 's knee" Pre-pregnancy, my "runner's s knee" would flare up when got to the 8 10 distance range, even if I had decent base (doing C210k type program) "Current Problem*
I've always been
run had my baby year ago, so all in all haven't run for about 1.5 years. I'm quite slim and have been doing acrobics-style classes for the past year, I'm not totally out of shape. Body weight exercises, aerobics, bikes and ellipticals are all fine. However, when run even the tinyiest bit, or even go on long walk hike, my pelvis gets very tight, and my knees start hurting very quickly, al- ready am doing general squats/ squats/lunges, stretching type things.
and
I'm starting to feel like running just isn't 't for me anymore. Which is a bummer, because really joy running!
Has anyone had something similar? Can anyone recommend some stretches or exercises that might help? Should see Dr? Or should just see physio? Not quite sure how proceed. Thanks!
TL;DR:
SFT Model Response
L5 years post-pregnancy, aches and pains in hips knees, can't even comfortably do day C25k. Help?< endoffeet
1.1442
>
PPO Model Response
One year post- pregnancy, aches and pains hips and knees when try to run even the smallest bit, and m not sure if running is for me anymore, Has anyone something similar? Any advice would appreciated< endoftext
5.5148
DPO Model Response
One year post-pregnancy, aches and pains hips and knees when run, can't even comfortably do day 1 of C25k, Can anyone recommend some stretches or exercises that might help? Should see Dr?< endoftextt
4.8368
Reference response
think pregnancy messed with my body, now can't even run even the smallest amount without pain in my pelvis and knees. I'm fairly certain the problem isn't just that I'm completely out of shape.< endoftextl
2.4334
Base Model Response
I'm runner, but my knees hurt when I run. Any suggestions?
-6.8827
I'm à runner, but my knees hurt when I run. Any suggestions?
I'm runner, but my knees hurt when I run. Any suggestions?
I'm a runner, but nA knees hurt when I run. Any suggestions?
I'm a runner, but my knees hurt when I run. Any suggestions?

---

Published as a conference paper at COLM 2024
count
policies
ref, sup2 sup2, sup2 ref, supl
18065
16273
7947
supl, supl sup4_ppo_mt _l10, sup4_ppo_rmi kl10
7653
6206
1op4_pp__sst rm3 m_20, sup4_ppo_rrm___ee kl20
6098
sup4_6b t0.7, $t77,uupp_6b_t.. ref, sup3_6b 6b
5614
1788
ref, sup2_bot ____boo_l rml sup2 bo8 mm1, sup2, sup3_6b 6b sup2, sup2 bo8_rm) ref, sup4_t0.7
1786
sups 6b
1751
1748
1738
1667
sup4_t0.7, sup4,tupp_t0) t0.7
1330
ref, sup4_pppp _pp__pppmm_m rm3 ref, sup3 bo8 rm2
1028
958
ref, swwpsspo__sseeeeeepppppppppppppt ppp_rml rml
955
_oo_bo8mmmm 30_ sup4_bot 2a___bo___3,,,____________33____ sup4_ppo_mm ref, sup4 bo128 rm3
sup3_ppo_rm
927
775
669
sup4 bo128 rm3, sup4_uu25666 ref, sup3 Ba_3__o63_______ rm2 ref, sup3 sup3 bo63_rm2, sup3_ sup3_ppo_rmi
649
480
476
470
sup3, sup3_ bes_boo_mmmm rm2 sup3 1yp3_bo63_rm22 sup3_bo8_rm2 sup3, sup3 _pp__pp_rr__ rml sup4 ppo rm3, sup4_tupp t0.7
467
464
451
441
ret, sup4 bo8 rm3
406
sup4_ppo 2pp___3p_,,______________ sup4_ppo_rm3 ref, sup4_bo256 bo256_rmi
384
340
sup4_bo128 rm3, sup4_bo122
322
sup4_ _064__33,,, supt_ppo_rm3
255
ref, sup4 ppo rm3 k110
253
sup4 6b t0.7, sup4_bo128 rm3_6b, ref, ______o122'' ref, suu4__po__mpaa
1ipo_rm3__rm kl20
249
sup4_bo256_rm33
246
246
12p0_rm33kk2) k120 sup4_ 6b _0__6.66_t1..___________ sup4_ppo_rm3
245
k110 kl10
220
sup4_bo512. 1222_m33,00000
218
ref, sup4 6b t0.7
124
ref, sup4 bo256 rm3
121
sup4_bo128_ 1__ ref, sup4 6b t0.7, sup4 Baop4_bo112_mmp rm3
rm3_6bbbm_
116
70
60
sup4 6b t0.7, stpp_uuppbbbb bo64 rm3 ref, sup4_bo512 bo512 rm3
53
Table 21: The unique comparison pairs and their counts in the train split of the preference dataset.
42

---

Published as a conference paper at COLM 2024
count
policies
ref, sup4_t0.7 t0.7
3252
sup4_1007,sup4_10.7
2927
1opp__m3___0 6b 6b_kk15,
2669
sup4_ppe ppl_lpppppp___p__ppppppppppppp moo_rm333__mmm kl20, sup4 ppo rm3 ppo_rm3 ki20
2340
sup4_ ppo rm3 kl10, sup4 ppp_mpppmm rm3 kl10 sup4_6b_tu (_o__t,,,,aaaee..ee sup4_6b_t0.7 pretrain 1_b__tbffe t.5, ref ref, sup446bbttt0 ref, sup4_6b
2070
1828
1682
1628
1167
ref, wtw.sut__b_ppo_rrt____
1154
ref, wttsssp4p_oommta mm4
1141
sup4 12b_t0.,, sup4
ppo_mpppp1___11 rm4 t.7
1097
ref, Bttps__ppoorrr_____1 k19
1084
ref, sup4_12b ref, title
1026
970
sup4_3b
950
sup4
934
pretrain xl t.7, ref
854
pretrain_ 12b_ 12,_ sup4 t0.7
847
pretrain_12b 1.5, Baop4__po_rm____ pretrain_x prrrtinnx__t,,,.om Detrrinn____77supp4__000
818
817
pretrain Settrain__bbtttt t.5, pretrain xl t.7
816
ref, sup4__po_rmmm _pt__pp_______ t.7
814
ref, ref, sup4_ppo__mm
kl6
812
ref, hupp4_ppormm' kl69
810
ref, sup4_ppo__mme rm3 kl22
810
pretrain_12b_t5, sup4_66 1144__bbt0..7
799
sup4_6b_tt0 po04_6b_t0..,,50 gup4_6o__t..,ssup_pppe Swp____p___or__ttt
796
ref, ref, suf_sup4pppe pplplppppp________pp rm3 kl2
794
pretrain 6b, ref pretrain_12b, ref
786
770
ref, sup4_ppoo_rm__kle
740
ref, _up4_66_ppo_rm3 66__b_l111
726
ref, sup4_ 1eb_t0.bb77 sup4_122 1eb__ttbb,,,___..
719
_e_______pp_r_____ kl15 ref, sup4_3b 11144_3b_ttt_ t0.7 3b_
715
709
pretrain Saeaiinnn,,rrr ref
707
sup4_6b 1___6b__pp_ @b_ppo_rm4 him__6b_k114, wwwpu_u__ppp_eeeeeee 6bt__pb__________1__________1 _61_________6444 kl14
670
sup4_6b_pp 1_6___________________ 66__6__k kl15,
sup__66_ppo_rm3___m kl15
669
sup4_3b_t0) stp,ttup_tt.77
661
pretrain_XL xl, ref ref, sup3_6b ref, sup2 ref, sup2 bo8 rm1
567
545
543
542
sup2_bo8_rms _t._boo__mmm____m sup3 6b
535
ref, sup4_ppo_rms 1ac00_r333___0 kl10
527
ref, sup4 xl bo64 rouge
525
sup2, sup2 bo8 rml
523
sup2, sup3_66 6b ref, sup4_bo644_ms
517
507
Table 22: The unique comparison pairs and their counts in the validation split of the prefer- ence dataset. (Part 1)
43

---

Published as a conference paper at COLM 2024
count
policies
ref, sup4_ xl b___bo512____1 2012__44_66 _6
497
sup4_xl x__boll1l8 rosgerooueeeee_
495
sup4_xl _____ololl2__mmmma
483
ref, sup4_ bo512___m_
481
ref, sup4 xl bo64_mmn rm4
467
ref, haop__bo22__nge
464
sup4_xl x1
bol28 _14__xlbb1228
rm4, sup4_xl xl bo64 rm4
453
pretrain citi__bbbt.,.ee ref
443
ref, 1o0-4_6b_ppo___ rm__mm_660 _1144
kl14
442
1l14_6bbkk1,,, sup4_6b_t0.7
440
ref, sup4 xl bolbbo5122111 rm4
439
pretrain 6b t.7, Br__s_u_____PP_P
_b___o___p__________pp_pp__p_p__ 6b kl14
439
ref, sup4_xl 1i__x__bo512_roua
438
pretrain_6b_t-7 6t7_66__t,,00 sup4_6b_10.
436
ref, sup4_xl_bo64__mgs
436
sup4 _lboo022__rme sup4 bo_12 rm
432
sup4 xl ___o128 rm4 _b,, sup4 xl 2o64_oo4_00000000_ 6b
427
sup4 bo128 rm3, sup4 bo64 rm3
417
sup4 xl bo1024 rouge, sup4_xx bup522l_bb5111111_11111____ (o512_rough sup4_ hop4__o512_aig rm3, sup4_x[bbo024 rm3 hop4_ppo_rm's 10__rm33k6,______ sup4_ppo_rm3 kl9
407
403
387
sup4 rm3 k169, sup4_ ppo rm3 k19
380
spp4_ppo rm3 kl2, sup4 s rm3 kl9 ref, sup4_ppo rm3
351
345
sup4_ fo4__po__mpe rm3 kl22, sup4_ppo _pp__ppppp_mm r0___m3_________ k19
325
sup4_ppo rm3 k19, sup4_tó.7 t0.7
322
ref, _aup3_pp__rmp rml
315
sup4_ppo_rmt 11t0_rm3____ rm3_k1260, ref, sup3 bo8 rm2
307
306
sup3 bo8 rm2, Soup3_pp__mms rml
302
sup4_ bo64 rm3, sup4_ppo_
_0c__ppop________________ rm3 kl10
302
sup4_6b_tt. sup4_ppo_rmm3_10
274
ref, sup4_xl Xi__xxlolllreee rouge
btpb__xx_oll2_ruuee
262
sup4 bo8 103_ sup4_ppo_rms
261
sup4_ppo_mme 111pp__m3_____________p_ 316, Sup____p____rm__k00
260
ref, sup4 bo8 rm3 ref, sup4 6b p.95 ref, 1up44xl_bollin _____oo0224_mm__66666
248
248
247
sup4_ 6b, sup4_6b_10) _6._4__0_.07
245
sup4 xl bo64 rouge, sup4 xl bo64 bb6444oooueeee rouge
244
sup4_6b_p 6b 6b_5.95, sup4_bbup_bbbtt t0.7
244
sup4 6b, sup4 6b95.95
244
sup4 bo128 rm3, sup4 bo256 rm3
su24_bo256
244
sup4 xl bo512 rm4 6b, sup4 xl bo512 rm4 6b
242
sup4_ppo_rmn haa-po__m33kkp_____ mm3_k122, sup4_ppo_rm3_kl69
242
ref, sup4_ppo_rr3 mm2_kl20 kl20
238
sup4 20,, sptpsppp__ppp__pppppppppppppop ppo__mpmmm rm3 kl20
234
sup4_ rm3 kl2, sup4_ ppo rm3 kl22
234
_epp__xl_bol22 ref, sup4_xl bo128 rm4 sup4_ppo_rms rmp k12, sup4_t0.7
231
228
Table 23: The unique comparison pairs and their counts in the validation split of the prefer- ence dataset. (Part 2)
44

---

Published as a conference paper at COLM 2024
count
policies
sup4 sup4_ppo_mtt -po__mpppp rm3 kl22, sup4_pp app_rm33_s
228
supt_ppo_rmy 100_rm33____m kl2, supt_ppo_rms rm3_k169
226
sup4_xl 11p4__x_bb64_____1
sup4_xl_boba xi_bo64_rm_
224
sup4 bo512 rm3, sup4_ Sapp_ppo_aatt ppo_rm331 kl20
224
ret, sup4 xl bo1024 rouge
219
sup4_ppo_mey iipo__mm_11260,, sup4_ppo_rm3_ ref, sup4_ xl bo128 rm4_6b ref, sup4 xl bb10244_mmp rm4
219
218
218
sup4_xl_bos cl_bo64___ae rm4_6b. sup4_xl
217
htcp-ppoo_m3_22221111 sup4_t0.7
217
sup4 xl bo512 rm4, sup4 xl bo512 rm4 sup4 Ppo___p rm3 k122, sup4_pppp _Bpe__pP__________ rm3
217
upo_rm3311 kl260
216
_po_rm3 rm3 k12, sup4_ppo _pr__m33 kl6 sup4_ppo
215
rm3, 1up4_bo64_rme rm3 ref, sup4 xl bo102440mmp sup4 bl5122 rouge, sup4 xl bo512_rouge sup4 bo512 rm3, sup4_bo512 2_m33333_____m33
212
212
204
203
sup4 _po_mppppmeeeppppepppppppppppppp rm3 kl260, sup4_ppo. rm3 kl69 sup4 ppp__m3 k12, sup4_ppo_rm3 kl6, sup4_t0.7 sup4_ppo_mm kl69, sup4_t0)
196
kl260
186
184
183
sup4 Bit__po_mmae rm3 k1260, sup4 t0.7 sup3
179
158
ref, sup3_bo63 b__3bo63__mmm
157
sup4 6b ppo_ _po_rm4 6b, sup4 _wwp__pp__pppaapepp_ep_ppp ppo_rm4 rm4
156
sup3, sup3 bo8 rm2
155
sup4_ 1114_ppo__o_e ppp_pppop3 pro__m3,,am sup4_t0.7
152
sup3, sup3 pop3_ppo_rml sup3 bo63 rm2, sup3 _pp
151
ppo mll
151
sup3 bo63 rm2, sup3 bo8 rm2
148
ref, sup4_66 _.3 t.3 ref, sup4_6b_tt. sup4_ppo_ rm4, title ref, sup4_6b
146
144
143
141
sup4_ 6b _.3, sup4_6b, 6b t1 sup4 12b, sup4 ppo pmp sup4 6b t.5, sup4 6b
140
rm4
137
137
sup4_ 6b 1.3, sup4_6b 6b t. ref, sup4_uupeo 6b ________p__p_____ _pp__p_mmmmm @po_rm4_6b_Lt sup4, sup4 pplppppppmmmppppppppppppppp rm4 pm
136
135
134
ref, _po ippo_mm4660 6b sup4 6b ppo rm4 6b t.5, www______ppooppaee_e_eep_ppppppe rm4 6b t.7
134
_b__po_rm4__ sup4 3b, sup4 _ppe_ppp____ rm4
132
_pp_rm4
132
sup4 6b _b_ppo______ _po_rm4 6b, sup4_mpe 6b Bll__po__mpgg rm4 6b t.7
132
sup4 sup4_ppo_rm3 sup4_ppo_rm3 sup4_ 12b, sup4_6b sup4 6b palpppo_rr ref, sup4 bo256 rm3 sup4 _up4_12b, title sup4_6b, sup4_6b 1e14__6b_pp________ ppopppppop__ _pp__ppo____6bb0_____11111
131
131
_po_mmppp_-__ppppppppppppppp rm4 6b, sup4 6b ppo_pppottteeepppppppppppppppppp 6b_t.5
127
127
127
127
Table 24: The unique comparison pairs and their counts in the validation split of the prefer- ence dataset. (Part 3)
45

---

Published as a conference paper at COLM 2024
count
policies
ref, sup4_ 2_4___bb_____ _p_p. ref, sup4_6b__ppppppe _p.9 ref, sup4_upppo _p__p. ref, sup4 xl bo8 rm4
124
124
124
124
sup4 bo128 rm3, sup4 bo128 rm3
119
sup4_xl_bos Barl__bo___rr_ bo16_rm4, sup4_ xl ___08__m44 sup4, sup4_ 12b
118
116
pretrain_123 sup4 1up4_6b_____ apo_rm4_66
114
sup4_ 6b 6b_pp7,,000 sup4_6b_ppp sup4, title
114
113
sup4_3i sup4 6b Sappppp___pee
pptppppppppppm 6b
113
ref, sup4 xl bo8 rouge sup4 ___p6__pp___ _b_p.8,,_____...p sup4_666 _b____ sup4 6b _1_p_6_ppp_______ cb_p.7,sup44 s.p4_66b46b.2. 6b 9.9 sup4 bo8 rm3, supp_ rm3 sup4 6b, title sup4 6b, sup4 3b, title
113
113
112
111
107
106
106
sup4_tt t0.7, sup4_xx4x xl rouge sup4_t0.77 10.7, sup4_xl_baa _1__o2_rm3 sup4_xl_boll cool__rouge, sup4, sup4_3b 3b
106
104
103
102
sup4 12b, sup4442ppppppp_epp _b______ppp_________ 110_-po__mm____000 6b
100
pretrain 12b, sup4_ppo_mm
98
pretrain_30 sup4
97
sup4 6b ppo hal_ppo__mm____eeeee1eee title sup4_12b, sup4_3b
_b__________________________
97
95
sup$_10..,spp4____bo sup4_xl_bo2_rm4
sup4 xl bol6 rm4 6b, sup4 xl bo8 rm4 6b
94
pretrain 6b, sup4_ppp_rmma rm4
tretrain_12b, sup4_12b
93
ref, sup4_xlboo8_rmn pretrain Deetrain_bb,sup__12 pretrain 6b, title pretrain_3b, pretrain_ 6b
92
91
90
90
pretrain Leciiin__bbaaaaa sup4_3b 3b
90
pretrain 6b, sup4_u446 6b
88
sup4_ppo_rm's kl10, sup4_ppo_rm3
86
3_10,, sup4_ppo_rrm3_1
sup4_t0.7, t0.7, sup4 ro0__m___6666
84
sup4, sup4_6b 6b sup4_30 3b, sup4_6b
84
pretrain 12b, sup4 6b
82
pretrain 6b, sup4 6b ppo rm4 6b
81
sup4, sup4_6b_ppppppeee 6bp______poo_____hh pretrain_3h 3b, sup4_ppo_rm4 pretrain_31 3b, sup4 3b
81
79
79
sup4_ppo ppo pppppppptttppp_ppppppppppppppp rooormm3.mmmmmm kl10, sup4_upppppp _pp rm3 kl22
77
pretrain_30 sup4_6b_ppppppp_pp_ppeppppppp 6b scb_ppo_rm__0_11 rmb_6b
75
pretrain_6b sup4_3b 3b
74
Table 25: The unique comparison pairs and their counts in the validation split of the prefer- ence dataset. (Part
46

---

Published as a conference paper at COLM 2024
count
policies
sup4_6b _____b___tt_ sup4 sup_bbo44 bo64 rm3
74
pretrain_6b, sup4
72
pretrain_xl, sup4 6b plo__po_rmm__
68
sup4_pp__mtt m1o_rm3___ f0m__k10,,00 sup4_t0.7
pretrain_3b, suppii___,ssup_12222 sup4_12b
67
sup4_ppo_mme rm3 __10,,10,,011111111111111 sup4_ patpppppp___1__1_111111 rm3 k1260
65
pretrain xl, sup4 12b
64
pretrain_12b, sup4
63
sup4_bo128 10___m33_bb________
sup4_bo256__ga rm3 6b
63
ref, huup4_bo12__me @ho128_rm3_6b
62
ref, sup4_supp__c 1144____bol__mm_
pretrain_3k title
60
sup4_6b 6b s0..,,sup__boo1221111..
bo512_
60
pretrain __,,,,,,________, sup4_3b
60
sup4_ppo_rm3_kkn supp pretrain_xl, sup4
58
sup4_xl bo8 bo8 rm4, sup4 xl bo8 rm4
58
pretrain xl, sup4_ pmp rm4
56
ref, sup4 x1_ bo2 rm3
ref, sup4 xl sl_boo6__mee rm3 rm3
56
sup4_ bo8 1008_m3,________01 sup4_bo8_rmi bo3_mm3 pretrain_xi, title ref, sup4_x__bo2_ropp
56
56
55
ref, sup4 xl bol6 rouge
bol6rrouge
55
_e__rouge,,,,,_____1 sup4 xl rouge, sup4 xl bo8 rouge
55
pretrain 12b, pretrain
54
pretrain_12b pretrain_3a _bb
51
ref, suup__l_bbo2_ueee ____bo2_mm rm4 pretrain_125 title
48
47
pretrain_33 _b, pretrain ref, sup4 xl bo2 rm4 6b
47
47
pretrain (ittrain__lb,,, pretrain 6b
47
ref, sup4_xl xl _d_bo16_rm4-_
47
sup4_xl_bos 1op4_xx_boo_rm__00 sub4,suppxpbpp Bac__oo8_________ pretrain_3b, _b
46
45
pretrain_xl,supt sup4_6b
45
pretrain_66 6b, pretrain_xl
44
bo128 sup4_bo128 rm3_mmm66 6b, Bop4__bo128_mm_
33
bo256 ref, sup4_bo256 11166_m33__mm_mm 6b
human_editor, sup4_6b_t0.7 human_editor, ref
Table 26: The unique comparison pairs and their counts in the validation split of the prefer- ence dataset. (Part 5)
47

---

Published as a conference paper at COLM 2024
count
policies
supenndm3 6b t.3, supcnndm3 6b t.3
1410
pretrain sactiin____tttt _t___,bbbtutt_tt.. (ttp__pp_______________ee_ rm4 6b kl14
148
ref, sup4_66 6b t0.7
148
ref, sup4_66 11i__po_rm___0000 kl14
148
pretrain_6b_tt2 ref
146
kl14, sup4_6b_t0.7
144
pretrain 6b t.7, sup4_uup 6b t0.7
140
Table 27: The unique comparison pairs and their counts in the validation littaaatiinnnnnmmaee split of the preference dataset. (Part 5)
48


Reviews:
[{"review_id": "sBZ7KyPqDN", "reviewer": "Reviewer_BkcZ", "summary": "The paper proposes a re-implementation of the RLHF algorithm with PPO. The paper provides insights into the key implementation details for the reproduction of OpenAI's seminal TL;DR summarization work. Each key implementation insight has been backed by an ablation study into its effect on the final performance. This thorough examination enhances our understanding of the algorithm's intricacies and will serve as a valuable resource for the research community.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are glad that the reviewers found our paper contributing to advancing the understanding and application of RLHF. We hope our careful study is helpful for future practitioners when building out their RLHF pipeline.\n\n> a) (Detail 2) How do the experiment observations change with different restrictions to the length of the query tokens (say 256 v/s 512 v/s 768)?\n\nThat is a great question! However, it might be difficult to evaluate because the reference summary has less than 53 tokens using the Pythia tokenizer, so comparing with 256 / 512/ 768 tokens might not be as meaningful.\n\nThat said, length is definitely a confounding factor, so it motivated us to do length-controlled comparisons in Figures 12 and 13.\n\n> How do the differences in SFT and Preference datasets affect the final performance of the trained model? Is there a control study, where the authors restrict the length of the examples in both datasets to check their impact on the final performance?\n\nA reasonable comparison might be to filter out the portion of the preference dataset that has more than 53 tokens. This might be an interesting experiment for the camera-ready version if our computes allow.\n\n> What's the effect of multiple epoch training on the same SFT prompts? Do the authors observe differences in the quality of the different SFT prompts?\n\nGreat question. Multiple epochs on the SFT dataset might cause overfitting (e.g., https://www.fast.ai/posts/2023-09-04-learning-jumps/) In this work, we have used a single epoch SFT training to follow the setting in Stiennon et al., (2020) closely (e.g., “For supervised baselines, … We use a batch size of 128, and run for a single epoch.”). We find preliminary results at 1.3B scale show overfitting at more than one epoch. Nevertheless, performing multiple epochs training would be a good experiment to be included in the camera-ready version\n\n> d) \"Reward whitening makes the model generate shorter outputs.\" Is this helpful only for summarization tasks?\n\nHuman preference datasets generally have longer answers as preferred causing length bias, so reward whitening may be a possible tool that can remedy this in regular chat tasks as well.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper's strength lies in its careful study of the different implementation details in RLHF.  The paper demonstrates the scalability of RLHF with various model sizes and validates the effectiveness of the RLHF approach for the pythia models at different scales.\n\nFurthermore, the authors have made the complete source code, model checkpoints, and training metrics publicly available. I believe, future works can build on these models for a deeper dive into the model's behavior with RLHF.\n\nFurthermore, the paper conducts consistency and calibration studies on the trained reward models. The authors showcase the consistency of these models with human preferences, as judged by external metrics like GPT-3.5, and also report comparisons of their trained reward model to dpo's reward model. They point to the specific differences between the two algorithms.\n\nThese strengths highlight the paper's contributions to advancing the understanding and application of RLHF, particularly in the context of summarization tasks.", "reasons_to_reject": "I don't have many concerns with the paper as-is. Here are a few questions about the experiments to the authors.\n\na) (Detail 2) How do the experiment observations change with different restrictions to the length of the query tokens (say 256 v/s 512 v/s 768)? \n\nb) (Detail 4) How do the differences in SFT and Preference datasets affect the final performance of the trained model? Is there a control study, where the authors restrict the length of the examples in  both the datasets to check their impact on the final performance?\n\nc) (Detail 21) What's the effect of multiple epoch training on the same SFT prompts? Do the authors observe differences in the quality of the different SFT prompts?\n\nd) \"Reward whitening makes the model generate shorter outputs.\" Is this helpful only for summarization tasks?", "questions_to_authors": "Please check my questions above.", "extra_scores": {}}, {"review_id": "xggOkZPRWJ", "reviewer": "Reviewer_EUPs", "summary": "The paper does a comprehensive reproducibility experiment on the RLHF results from a past OpenAI paper. The authors experiment with different details of training that are often overlooked or not discussed in detail in most papers.", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are glad the reviewer found our work to be a great guide to all practitioners in the community. While we agree our work does not propose a novel method, we believe there is novelty in our investigation of details and the strength of our open and reproducible baselines. Our work helps the community build upon a better foundation (and create better science).\n\nFor example, our PPO models are well-calibrated and have a very high GPT4 win rate. The DPO paper (https://arxiv.org/pdf/2305.18290) reported its PPO model to have 57% max win rate against reference summaries (judged by GPT4). Whereas our same-sized model has a ~80% win rate. Our paper can become a strong baseline for future works and reviews to refer to. Below, we take the example completions at Table 5 from the DPO paper and compare them to ours :\n\n**DPO TL;DR:** About to propose to my girlfriend of almost 2 years, but I am extremely\nanxious and can’t sleep. How can I relax?\n\n**PPO TL;DR:** I am about to ask my girlfriend of almost 2 years to\nmarry me, and I can’t calm myself down. Any tips on how to relax? I would like to be able to sleep tonight (and the night before)..............................................................................................................................\n\n**Our PPO TL;DR:** I am about to propose to my girlfriend of almost 2 years in a couple days and I am freaking out. I can't seem to relax leading up to it. Any suggestions on how to calm myself down and relax?\n\nAs demonstrated, the PPO baseline used in the DPO paper seems broken (with excessive periods), whereas our completion is coherent and does nothing like that.\n\nComparing deepspeed-chat and trl: This is a good point. We noticed that TRL’s PPO 1) does not use reward normalization, 2) does not use a separate value network, 3) does not use the EOS trick, 4) reported issues with negative KL divergence (https://github.com/huggingface/trl/issues/355). We noticed deepspeed-chat does not use reward normalization. \n\nOverall, direct comparisons are difficult to make because the full-stack setups are very different. It may be better to compare the final model checkpoints, but deepspeed-chat / trl did not release checkpoints for TL;DR summarization (the exact challenge this work aims to address).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "I think this paper is a great read and a guide to all practitioners in the community working in the intersection of LLM and RLHF. It would", "reasons_to_reject": "I am not sure how much it might align with the \"research novelty\" aspect of the conference. I leave that judgement to the editors. \n\nI would like to point out though that the authors mentioned that many of the practices are not standard in the community. In that note, it would be good to contrast the design choices of the authors against the out-of-the-box choices presented in many open source libraries like deepspeed-chat, trl, etc. Since the main objective of this paper is to make the engineering practices more reproducible and \"open\", it is important to contrast the presented choices against other available alternatives.", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "YyQfc9RRvj", "reviewer": "Reviewer_5BXq", "summary": "This paper contributes a detailed reproduction of OpenAI’s RLHF work in the TL;DR summarization. The nice thing about the work is that many technical details (datasets with tokenizer, truncation, padding, EOS tokenizing, disabling dropout, the way they extract reward from the EOS token, etc) are transparent. This should make the paper a good reference for future effort to replicate RLHF work in the dataset.\n\nThe work also demonstrated the scaling behavior of PPO across different model sizes, this is unsurprising though.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are glad the reviewer found our work to be a good reference and appreciates how significant our contribution is. We agree our work does not propose a novel method, but we do provide novel, open results and novel findings about the most common benchmark and baseline in the field. \n\nMany of our details are not commonplace in standard RLHF libraries which highlights their novelty and necessity to researchers e.g. TRL’s PPO 1) does not use reward normalization, 2) does not use a separate value network, 3) does not use the EOS trick, 4) reported issues with negative KL divergence (https://github.com/huggingface/trl/issues/355). \n\nThe strength of our open and reproducible baselines is also a novelty. For example, the DPO paper (https://arxiv.org/pdf/2305.18290) reported its PPO model to have 57% max win rate against reference summaries (judged by GPT4). Whereas our same-sized model has a ~80% win rate (in a separate GPT4 eval; just one random seed). By releasing code and baselines, we hope to ground the field in reproducible and significant results. \n\nFinally, we believe there is some novelty in the correctness of our preprocessing and eval guidelines that enables the validity of the benchmark. For example, we are the first to openly replicate the scaling behaviors on summarization and we hope there is novelty for the community to now use this as a benchmark for scaling properties of RLHF algorithms.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* This paper will be a good reference for future work on replicating RLHF work in the TL;DR summarization.", "reasons_to_reject": "* The novelty of the work is limited. I consider it as a \"white\" paper that to unblock everyone who tries to replicate \"RLHF\" work, which is to me a significant contribution. But other than that I don't see other novelty of the work.", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "HipEt8ITdN", "reviewer": "Reviewer_UfCo", "summary": "This paper summarizes the reproduction details of the OpenAI work: RLHF for summarization. The authors study many different items that are specific and important for reproduction. The authors have found several key points that affect the performance.", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are glad the reviewer found our work to be a good manuscript to reproduce RLHF summarization. \n> Though the comprehensive study appears, it is somehow hard to show the connections between these different details, this potentially could be crucial.\n\nWe address this by releasing the source code and have tried to make it as minimal as possible, reducing the mental burden of understanding how these details connect with each other.\n\n> The application summarization task is specific, however, it would be better to make some details that could be transferable. Or it should be study some of these items.\n\nThis is a great point. The main objective of this work is to dive really deep into TL;DR summarization to make sure the RLHF pipeline works (and scales with model sizes). Given this knowledge, our next steps include adding other benchmark tasks such as anthropic’s HH.\n\n> The base model GPT-2 is somehow old now, at least LLaMA would be a better replacement.\n\nWe actually used pythia models instead of gpt-2 models. Furthermore, we choose pythia models for several reasons:\n\n* **pythia models compare more fairly with base models used in Stiennon et al., (2020)**. In particular, Pythia models were trained with ~207B tokens, and Stiennon et al., (2020) suggested that they trained the base models for “200-300 billion tokens”. In comparison, even LLaMA 1 models are trained with 1000 to 1400 billion tokens (https://arxiv.org/pdf/2302.13971). \n* **LLaMA models don’t have 1B models**. LLaMA 1 models only have 8B to 60B models, which makes it difficult to conduct the RLHF scaling experiments shown in this paper.\n* **Pythia models are truly open**. Unlike many open weights models like LLaMA, Pythia models release the training code, source code, and are fully transparent.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. This is a good manuscript for others to reproduce the work of RLHF for summarization.\n2. The tricks are detailed in lots of ways and aspects, which are important for reproduction.\n3. The experience could be possible to transfer to other applications.", "reasons_to_reject": "1. Though the comprehensive study appears, it is somehow hard to show the connections between these different details, this potentially could be crucial. \n2. The application summarization task is specific, however, it would be better to make some details that could be transferable. Or it should be study some of these items.\n3. The base model GPT-2 is somehow old now, at least LLaMA would be a better replacement.", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Yq7WxvOo8W", "reviewer": "Reviewer_toWU", "summary": "The paper discusses more than 20 implementation details of RLHF with PPO using the tl;dr summarization task. The authors achieve great performance on the summarization task, and the authors reproduced the scaling behavior by OpenAI by experimenting with models of different sizes. So, the implementation details must be meaningful.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their in-depth review and are glad they find our paper to be an important contribution to the community and furthering open science. We agree that these details are important and deserve to be discussed. \n\n**On DPO vs PPO RM**\n\nThough the title of the DPO paper is “Your Language Model is Secretly a Reward Model”, only by training with DPO’s objective is this true. In particular, DPO has a formula that allows the trained LLM to be used as a reward model in conjunction with the reference LM), as defined in page 5 of [https://arxiv.org/pdf/2305.18290](https://arxiv.org/pdf/2305.18290):  \n\n$$\\hat{r}_\\theta(x, y) = \\beta \\log\\frac{\\pi_\\theta(y|x)}{\\pi_{\\text{ref}}(y|x)}$$\n\nBut this isn’t the case for PPO. PPO’s policy LM is trained exclusively to maximize RM and so its logprobs will not correspond to an implicit RM since it isn’t part of the objective function. Since both methods use an RM (DPO’s implicit RM, and PPO’s separate, explicit RM) we compare those. \n\n**Novelty of our details**\n\nThe most similar work to ours is Delve into PPO by Zheng et al, 2023. Notably, our work distinguishes itself by building a scaling behavior – we showed our RLHF pipeline scales with model sizes and works across random seeds. Our investigation is also more extensive, and our findings differ (we find reward whitening to be optional and advantage whitening to be beneficial, whereas Zheng et al find the opposite). Though we find the EOS trick to be the most underreported detail, our work proposes a collection of different details to form an overall “correct” setup for the popular benchmark summarization task, which we hope will be adopted by the community. In contrast, Zheng et al focus on the best combination of PPO details for a single model on HH-RLHF.\n\n**Table 1**\n\nThis is a good point. We have already changed it to another, more appropriate sample. \n\n```\nSUBREDDIT: r/relationships\nTITLE: Me [19 F] should I be trying to help my\nbrother[16 M] with his life?\nPOST: This is my first Reddit post and I’m not\nsure if I’m doing it right\n```\n\n\n\n**Detail 23 Values**\n\nAre you referring to “values'' in Detail 13 (not 23)? We propose to treat the reward function as a per-token value function and get its outputs at non-eos tokens. We find that, in general, the reward model does not learn to be a per-token RM and gives nonsensical outputs at non-eos tokens.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "In general I believe this discussion is a great contribution to the community, as top-performing models’ RLHF procedures are sometimes hidden and not published. \n\nThe authors also promise to open-source the code. The appendix is helpful (with implementation details and detailed results and prompts). \n\nSome of the details are great: for example, “including the leading space in completion” (detail 3) is a common bug in implementation and people rarely openly discuss it, so I really appreciate the authors for writing it out. \n\nAnother example is detail 7. It is also great because people don’t usually discuss it often. \n\nDetail 23 (eos trick; if there’s no eos in the trajectory then assign it a negative reward) is also something I used before, but I didn’t hear others discussing it. So it’s great to see some confirmation that others are using it too and it works.", "reasons_to_reject": "(Detail 19) On DPO vs. PPO reward modeling ability: it’s unclear to me whether the comparison is fair. \nFor DPO reward model, do you mean using the probabilities as an implicit reward model? \nIn that case, would it be more fair if you not only include your regular reward model validation accuracies, but also PPO-trained LM’s reward modeling accuracy (in other words, similar to DPO, treat the PPO-trained model as an implicit reward model, and compute the validation accuracies)? \n\nImo the advantage implementation is quite important but there seems to be minimal discussion. \n\nThere have been quite a few recent “implementation details of PPO” online. I thought it would be best if the authors highlight the least known and the most important details, and also point out which detail likely doesn’t matter much. \n\nI’m not entirely sure if the example in Table 1 is appropriate so I leave it to other reviewers / the meta-reviewer. I know we are all adults but I also don’t want to run into this genre unprompted, especially in an academic paper (e.g., losing my virginity, leaving me with a boner).\n\n--\n\nUpdated score after rebuttal", "questions_to_authors": "Detail 23 seems important but it’d be great if the authors can elaborate what the values are. Are values for each batch? The past k steps?", "extra_scores": {"ethics_concerns_details": "Mildly sexual content in Table 1 but fixable"}}]

Author rebuttal:

Meta-review: The paper outlines a re-implementation of the RLHF pipeline from Stiennon et al. 2020. The main contributions of the paper are (1) reproducing the RLHF pipeline and the scaling behaviors in Stiennon et al 2020, (2) a number of implementation details and insights (e.g. benefits of dropping dropout) that are needed for a stable PPO implementation.

Overall, the reviewers agree that this paper contains useful insights for researchers, especially since PPO training is known to be unstable. 

Cons: 
It is unclear what advantages this provides over existing frameworks like TRL. The paper is also missing a through comparison with the design choices between these frameworks.
Decision: Accept