Understanding the Effects of RLHF on LLM Generalisation and Diversity
Authors: Robert Kirk
Ishita Mediratta
Christoforos Nalmpantis
Jelena Luketina
Eric Hambro
Edward Grefenstette
Roberta Raileanu
Venue: ICLR 2024 poster (2024)
Keywords: reinforcement learning
large language models
rlhf
ood generalisation
diversity
Primary area: generative models
TL;DR: We analyse the effects of RLHF fine-tuning on LLMs in terms of OOD generalisation and output diversity, finding that RLHF makes better-generalising but less diverse models

Abstract: Large language models (LLMs) fine-tuned with reinforcement learning from human feedback (RLHF) have been used in some of the most widely deployed AI models to date, such as OpenAI's ChatGPT or Anthropic's Claude. While there has been significant work developing these methods, our understanding of the benefits and downsides of each stage in RLHF is still limited. To fill this gap, we present an extensive analysis of how each stage of the process (i.e. supervised fine-tuning (SFT), reward modelling, and RLHF) affects two key properties: out-of-distribution (OOD) generalisation and output diversity. OOD generalisation is crucial given the wide range of real-world scenarios in which these models are being used, while output diversity refers to the model's ability to generate varied outputs and is important for a variety of use cases. We perform our analysis across two base models on both summarisation and instruction following tasks, the latter being highly relevant for current LLM use cases. We find that RLHF generalises better than SFT to new inputs, particularly as the distribution shift between train and test becomes larger. However, RLHF significantly reduces output diversity compared to SFT across a variety of measures, implying a tradeoff in current LLM fine-tuning methods between generalisation and diversity. Our results provide guidance on which fine-tuning method should be used depending on the application, and show that more research is needed to improve the tradeoff between generalisation and diversity.

Published as a conference paper at ICLR 2024
UNDERSTANDING THE EFFECTS OF RLHF ON LLM GENERALISATION AND DIVERSITY
Robert Kirk Ishita Mediratta 8 Christoforos Nalmpantis 9 Jelena Luketina"
Eric Hambro 8 Edwward Grefenstette " Roberta Raileanu 3
University College London, Meta, University of Oxford
ABSTRACT
Large language models (LLMs) fine-tuned with reinforcement learning from human feedback (RLHF) have been used in some of the most widely deployed Al models date, such as OpenAl's ChatGPT or Anthropic's 's Claude. While there has been significant work developing these methods, our understanding of the benefits and downsides of each stage in RLHF still limited. To fill this gap, we present an extensive analysis of how each stage of the process (i.e. supervised fine-tuning (SFT), reward modelling, and RLHF) affects two key properties: duitt--disttibution (OOD) generalisation and output diversity. OOD generalisation is crucial given the wide range of real-world scenarios in which these models are being used, while output diversity refers to the model's ability to generate varied outputs and is important for a variety of use cases. We perform our analysis across two base models on both summarisation and instruction following tasks, the latter being highly relevant for current I.I.M use cases. We find that RI.FF generalises better than SFT to new inputs, particularly as the distribution shift between train and test becomes larger. However, RLHF significantly reduces output diversity compared to SFT across a variety of measures, implying tradeoff in current LLM fine-tuning methods between generalisation and diversity. Our results provide guidance on which tin-ttnnig method should be used depending on the application, and show that more research is needed to improve the tradeoff between generalisation and diversity.
INTRODUCTION
Large language models (LLMs) have become a standard approach to solving natural language processing (NLP) tasks. As these models become more capable, the tasks we want them to solve become more complex, which makes it more difficult to provide training data and to evaluate performance. For such tasks it may be casier and faster for humans to evaluate or rank model outputs than provide full demonstrations. Thus, there has been much recent work on using human preferences dominant approach being reinforcement learning from
in this form fine-tune LLMs, with human feedback (Christiano et al. 2017 Ziegler et al. 2020) RLHF). This approach has been used to produce some of the most impressive Al systems that exist today (Glaese et al.) 2022 OpenAl 2022 2023 Anthropic 2023).
The standard RLHF fine-tuning pipeline generally consists of three stages: supervised fine-tuning (SFT), where the pretrained model is fine-tuned with the language modelling loss on demonstrations of the desired behaviour; reward modelling (RM), where the pretrained model is fine-tuned predict human preferences between pairs of outputs for a given input; and reinforcement learning (RL), where the reward model is used fine-tune the model produced by the SFT stage using an on-policy RL algorithm like PPO (Schulman et al. 2017). While this pipeline has been used to seemingly great success, there is little understanding about how each component contributes to the behaviour of the final model. Two important areas where the effects of each component of the pipeline have been underexplored are the ouitt--diitributtin (OOD) generalisation and output diversity.
WWrk partly done during internship at Meta, Correspondence to robert ki rk. 200uc1 ack.uk For details of author contrihutions, see Author Contributions

---

Published as a conference paper at ICLR 2024
OOD generalisation is important for the widespread adoption of these models, since it is necessary to ensure that LLMs are performant and reliable in a wide variety of situations that go beyond the distribution of the training data. While anecdotally models seem to perform well across a wide range of settings, it is unknown which stage of the pipeline is responsible for this strong generalisation, and even whether the observed generality is due to the training or fine-tuning methods used, the large model size, or purely from the very large and diverse distribution of data.
Further, these models are used in creative open-ended domains such as story generation (Castricato et al. 2022), scientific research (Boiko el al.) 2023), or other tasks where a diverse output distribution
required, such as red teaming (Perez et al. 2022) J. In these situations training models that produce
diverse (but still high-quality) outputs is of crucial importance. There has been some speculation as to the possible effects of different steps of the RLHF pipeline on diversity (janus] 2022), ), and some work has shown a decrease in diversity from RLHF (Khalifa et al., 2021; Perez et al. (022). However, no rigorous analysis has been done of the effects of different parts the RLHF pipeline on output diversity across different tasks. In addition, in contrast with our paper, prior work has been limited to evaluating diversity using simple token-level metrics such as BLEU (Papineni et al. 2002) and on use cases which are not common in practice.
In this work. we evaluate each stuge of the RLHF pipeline (i.e. supervised fine-tuning reward modeling, and reinforcement learning) as well as best-of-N (BoN) sampling in terms of their effeets on in-distribution (ID) performance, OOD performance, and output diversity. We disentangle the effects of the data set and the training method on generalisation by using OOD test datasets that induce realistic distribution shifts between training and testing, and evaluate generalisation using these test sets each stage of the pipeline (Section 5.1 While diversity is a difficult concept to operationalise, we take a pragmatic approach and measure a range of diversity metrics at each step of the pipeline, covering syntactic, semantic, and logical diversity (Section[5.2) diversity of outputs sampled for single input and for range of inputs. Evaluating BoN as well as SFT and RLHF enables us to uncover whether differences between RL.FF and SFT are due the use of reward model or the type of optimisation applied.
We evaluate both
In summary, we find:
RLHF improves in-distribution performance (as expected from previous work) and also OOD performance in comparison to SFT.
However, RLHF substantially decreases the diversity of outputs sampled for a given input compared to SFT.
Even when sampling outputs for different inputs, RLHF produces less diverse text on some metrics, implying that such models tend to produce more similar text regardless of the input
These findings reveal an inherent tension between generalisation and diversity when applying current fine-tuning techniques. This underscores the necessity for novel methods that improve both these attributes without sacrificing one for the other, and for research to understand whether this tension is fundamental to fine-tuning or a deficit of current techniques. We open source our code to enable re- producible research
2 BACKGROUND AND RELATED WORK
Fine-tuning Large Language Models. The current common practice in NLP is to fine-tune large pre-trained language models (LLM) for downstream tasks. The standard approach for fine-tuning is supervised fine-tuning (SFT), which trains the model on demonstrations of solving the task using supervised learning. When it is easier to evaluate or rank model outputs than it is to gather demonstrations that accurately perform the desired task, an alternative method called reinforcement learning from human feedback (Christiano et al. 2017 Ziegler et al.) 2020 RLHF) can be used. Most previous work on RLHF uses on-policy RL algorithms such as Proximal Policy Optimization (PPO) (Schulman et al., 2017) or Advantage Actor Critic (A2C) (Mnih et al. 2016). but offline RL methods have also been proposed (Snell et al., 2022). Once RM has been trained, it also be used to do Best-of-N (BoN) sampling (also called rejection sampling) of the model outputs. (Casper et al.
2023) survey existing problems with RLHF method for fine-tuning LLMss, and work further
2

---

Published as a conference paper at ICLR 2024
investigates some of these problems (specifically policy generalisation and output diversity or "mode collapse").
Dubois et al. (2023 AlpacaFarm) introduces framework for developing methods for learning from human feedback in an instruction following setting, and in this framework demonstrate that approaches that learn from human feedback (including RLHF) generally perform better than SFT on a specific evaluation set they introduce. We use the AlpacaFarm models and evaluation dataset in our experiments on instruction following. In that work they do not evaluate out-of-distribution generalisation or output diversity, and only present results on single evaluation dataset, while we address all of these issues, while also performing the analysis in an additional task (summarisation).
We discuss other recent approaches for tne-tuning LLMs in Appendix C but note that while these works sometimes show improvements, they are not used by most large-scale systems being deployed currently, and hence we focus our analysis on the more popular and widely used RLHF pipeline, as that is where understanding will be most relevant and useful.
Generalisation and Diversity in NLP. Almost all prior work using RLHF has evaluated models on the same distribution of inputs used for fine-tuning (Bai et al. 2022 Glaese et al. 2022 Ouyang et al. 2022 Stiennon et al. 2022), meaning that the generalisation properties of such methods isn't I understood. One notable exception is [Stieennon et al. (2022), who perform some experiments evaluating their models trained the TL;DR dataset (Völske et al. (017)) (reddit post summarisation) on the CNN/Daily Mail dataset (Nallapati et al.) 2016) (news article summarisation). However, they didn't t investigate how different parts of the pipelime affected generalisation, and the investigation was less rigorous and involved than ours is. (Hupkes et al. 2023 provide a comprehensive survey of generalisation in the wider NLP literature.
Several works (Kaliffa et al. 2021 Perez et al.
2022) have shown in specific settings that RLHF
fine-tuning produces models with less output diversity, as measured by self-BI.EU (Zhu et al.) 2018). Our work extends these works by making diversity evaluation a primary focus and using diversity metries beyond self-BLEU which have been externlly validated (Tevet & Berant 202I) and measure diversity in range of different ways.
We discuss more related work, including details on LLMs, SFT and RLHF, in Appendix
MODEL TRAINING
Here briefly describe the details of how the models we evaluate are trained. For more detailed description of model training see Appendix B.
Pretrained Models. We use the I.LaMa pretrained 7 billion parameter model (Touvron et al. 2023a). This is standard decoder-only transformer-basee causal language model trained on a large corpus of web text. This size of model has been shown to be effective in the tasks we investigate (Stiennon ct al. 2022 Dubois et al. 2023).
In Appendix we perform experiments with OPT (Zhang et al. 2022) models, using five model sizes. These models have worse performance in general, but the experiments allow us to see trends across several model scales.
We train Reward Models(RMs) following (Stiennon et al. 2022). We initialise the RM as the base model, and we add a scalar head before the unembedding layer. The model is then fine-tuned on inputs and pairs of outputs with one output labelled as preferred, and trained to output scalar for each input-output pair representing which output is preferred.
Supervised Fine- Tine-Tuning (SFT) models are trained on reference input-output pairs using the cross- entropy loss on the output conditioned on the input.
Reinforcement Learning from Human Feedback (RLHF) models are trained using PPO (Schulman) et al., 2017) as our base RL algorithm, following previous work (Stiennon et al. 2022). We initialise the model with the corresponding SFT model, use the language modelling head as the policy output, and learn a value function using a linear layer (an identical architecture to the RM), with the policy and value function sharing the same model backbone. We optimise the policy to maximise the RM described above, and use a KL divergence term as an auxiliary reward to ensure that the language model stays close the SFT model, in previous work (Jaques et al. 2017 Stiennon al. 2022
3

---

Published as a conference paper at ICLR 2024
Ziegler et al. 2020). The final reward for the policy is
(I) where RM denotes the reward model trained as described above; ONL. Prs and Osrr are the parameters of the policy, RM and SFT model respectively; æ, y are the input and output; and fK is hyperparameter that controls the weight of the KL penalty. We use 3KL 0.05 throughout this work, following (Stiennon et al. 2022) and our own early experiments that found this choice struck a good balance between model performance and overoptimisation (Gao et al. 2022).
R(z.y) y) = RM (z, y)
Best-of-N. The reward model can also be used to filter samples from another model; this is called Best-of-N (BoN) sampling and has been used in multiple previous works achieve good performance (Menick et al. 2022; Nakano et al. 2022)). N summaries are sampled from the SFT model, and then the RM is used to select the best one. We sample with temperature 0.7 and use N 16, as that is what is used in previous works (Rafailov et al. (023)) and strikes good balance between improved performance and computational cost. Evaluating BoN performance gives use a way to evaluate whether the differences between RLHF and SFT models are due to the use of a RM, or the type of optimisation applied. However, due to the increased inference time compute cost, this method is generally not used in practice, focus our analysis on RLHF and SFT policies.
DATASETS AND TASKS
We investigate the effects of RLHF in two tasks: text summarisation and instruction following.
Summarisation. We follow Stiennon et (2022) : Ziegler et al. 2020) in evaluating LLM fine- tuning algorithms on summarisation task. Models are trained to produce summaries of Reddit posts, given the post as input. We use the same dataset as (Stiennon et al. (2022). which is a filtered version of the TL:DR dataset (Völske et al.) (017). consisting of approximately 120,000 Reddit posts with accompanying summaries. See Stiennon et al. (2022) for full details on the filtering procedure and the analysis of the resulting data distribution.
We use the preference data gathered by Stiennon et al. 2022) for reward model training. This consists of approximately 64,000 summary comparisons. These data consist of inputs (reddit posts) along with pairs of outputs (summaries), with one summary labelled as preferred. The preferences are from human annotators contracted by /Stiennon et al. (2022) to choose summaries according list of criteria designed to select high quality summaries.
Instruction Following (Chung al. 2022 Dubois al. 2023;
Wang al. 2022 is of the main use cases for the I.L. fine-tuning techniques we investigate, so we also vaae models trained in this setting.
We use the SFT, RLHF, and RM models released by Dubois et al. 2023 AlpacaFarm). These models were trained in very similar way how train our summarisation models, and all based on the LLaMa 7B base model. Models take text instruction as input and are trained to output preferred answers those instructions. The outputs used for the SFT model are from the cext-davin nci 003 model from the OpenAl API, and the human preferences used to train the reward model are gathered by the authors of AlpacaFarm based on outputs the SFT model. For precise details on how these models were trained, refer to Dubois al. (2023).
5
MODEL EVALUATION
We now describe how we evaluate both out-of-distribution generalisation and output diversity for the different fine-tuning techniques.
5.1 GENERALISATION EVALUATION
To evaluate the performance of our trained models, use GPT-4 (OpenAl 2023) through the OpenAl API as a simulated human annotator. While we ultimately care about human preferences, these expensive, difficult to gather, and often noisy. Previous work has shown that GPT-4 accurately
https://platform. openai
4

---

Published as a conference paper at ICLR 2024
simulates human preferences in both summarisation (Rafailov al. 2023) and instruction following (Dubois et al. 2023: Zheng et al. (023), so we use it as the main performance metric in both tasks.
To evaluate model's performance
given dataset of inputs and reference outputs with GPT-4, we prompt GPT-4 with the input, the reference output, and the model output, and prompt it to decide which output is better. We use variant of the prompts from (Rafailov al., 2023) for summarisation, and alpaca_eve _vvllll (Li et al. 2023) with the standard prompts for instruction fo following. See Appendix [ for the precise prompts and other details. This gives us a percentage win rate of the model being evaluated versus the human-annotate reference oulput, which we refer to as preference vs reference (PvR). In Appendix [D.] we validate that this evaluation is good proxy for human preferences. We also perform head-to wead-to-heede to-head comparisons between two policies. by prompting GPT-4 in the same way to decide which of two model model outputs is better.
To evaluate oittr--distribttion (OOD) generalisation, we specify an in-distribution (ID) test set and one or more out-of- of-dissribuuton (OOD) test sets for each task, which have inputs drawn from a different distribution. In each of these sets we have evaluation inputs and corresponding reference outputs (produced either by humans in summarisation or text text-davines davincii -00- in instruction-folloinng
For summarisation, the ID test set is the original TL;DR test set from (Stiennon et al. 2022), and the 0OD test set is the CNN/DailyMail test set, a dataset of news articles and corresponding summaries Nallapati al. 2016). This tests the ability of the model to have learnt the more general skill of summarisation and to apply it in a very different domain.
For instruction following, the ID test set is new test set generated in the same way the training set was for AlpacaFarm, using the AlpacaFarm variant of Self-Instruct (Dubois et al., [2023] Wang et al. 2023). Regenerating the test set ensures that it not seen during training or model selection for AlpacaFarm models. For the first OOD test sets, we use the AlpacaEval evaluation test sel proposed in the original paper. This is set of inputs taken from variety of open-source instruction following and dialogue training and evaluation datasets (Bai et al. [022] Ganguli et al. 2022 Gudibande 2023
Köpf et al. 2023] Wang et al., 2023 Zheng et al. 2023), curated by Dubois et al. (2023). For an additional OOD test set, we generate set of Sequential Instructions using an adjusted Self- Sesf-Instruct protocol. These instructions contain multiple steps in a single input, often building on each other, and require the model to ensure that they complete all the steps to produce satisfactory outcome. For more details on this dataset see Appendix H
We also report the generalisation gap (the difference between in-distribution and -if-riiiiitiittionn performance) which provides a measure of generalisation specifically. Lower generalisation gaps imply the model generalises better, model performance does not drop as much when the model is evaluated out of distribution. For the head-to-head comparisons between models, we look at the change in head-to head winrate to give different evaluation of generalisation; the model whose winrate increases generalises better.
5.2 DIVERSITY EVALUATION
To evaluate the output diversity of trained policies,
several diversity measures which are well- supported by prior work. namely distinct N-grams (Li et al. 2016). Sentence-BERT embedding cosine similarity (Reimers & Gurevych| 2019) and NLI diversity (Stasaski & Hearst 2022). All of these metrics have been shown to align well with human diversity evaluations and with underlying diversity generators by Tevet & Berant (202T)
Each diversity metric D takes set of model outputs O, and produces a scalar score representing how diverse the set is. Distinct N-grams counts the number of distinct N-grams (averaging over = .s) in the set of outputs, and following (L.iu et al., 2022) we use the irrectation-djjutee distinct N-grams (EAD) formula to remove the bias towards shorter outputs. The Sentence-BERT metric embeds each element of the output set using a sentence transformer, and then measures the average cosine similarity between the embeddings. The metric is then 1 minus the average similarity.
The NLI diversity metric measures the number of entailments and contradictions between pairs of elements in the output set using a natural language inference (NLI) model and rates an output set as more diverse if it produces more contradictions and fewer entailments. We pass sentences sampled from elements of the output set (rather than complete elements) to the NLI model so that the inputs closer to the model's training distribution.
5

---

Published as a conference paper at ICLR 2024
ID (TL;DR)
OOD (CNN/DailyMaiil)
Generalisation Gap
GPT-4- 1---- R
60
50
60
25
40
40
20
30
15
20
20
10
0
0
BoN
RLHF
SFT
BoN
RLHF
SFT
BoN
RLHF
SFT
Figure 1: Summarisation Gemeralisation Results. GPT-4-PvR for SFT, BoN and RL policies, based on LLaMa 7B, trained on the summarisation task. In-distribution is performance on TL;DR, and out-of-diistribution is on CNN/DailyMail, and generalisation gap is IDD OOD performance.
For each policy produce of outputs from the model for each subset of N inputs from the test set, sampling with temperature 1:
for 1... N O {ıj ا -...
In this work we use K 16, N = 500. For diversity metric D we then evaluate the per-input diversity which is defined the average diversity of the output sets over inputs (i.e. the diversity of π (ylz:)) as well as across-input diversity defined as the diversity of outputs across inputs (i.e. the diversity of (t))::
N (अ) == N Σ D(OO.)
(rosslaputDiiverstty(())) := D (UOC
Peresiut )
Here Oi [1] is the first element of the set O For conciseness, we refer to aepectation-adjustt distinct N- N-grams, sentence- BERT average cosine similarity and NLI diversity as EAD, Sent BERT and NLI respectively. We can view them measuring syntactic, semantic and logical diversity, and hence using all of them ensures that we are evaluating diversity in a wide range ways.
EXPERIMENTAL RESULTS
In this section we present the results of our experiments on generalisation and output diversity. In general we find that RLHF performs better than SFT in an absolute sense both ID and OOD, but generalisation gap and head-to-head metrics tell more nuanced story. However, RL.FF does reduce output diversity substantially in the per-input setting, and still reduces it to lesser extent in the cross-input setting. These conclusions are supported by similar results for OPT models across a range of model scales in the summarisation task, presented in Appendix
6.1 GENERALISATION
Summarisation. Fig. shows the GPT4-PvR for SFT, Bo16 and RLHF policies trained on the summarisation task, showing the ID (TL:DR) and OOD (CNN/DailyMaill performance and the generalisation gap; Bo16 outperforms RLHF which outperforms SFT, both ID and OOD. While Bol6 outperforming RLHF somewhat surprising. there are examples of this in the literature (Nakano) et al., 2022). and Bol6 has a substantially higher inference practical applications. The performance of all policies drops OOD, which unsurprising given the difficulty of the shift (reddit post summarisation to news article summarisation). The generalisation gap is fairly similar between methods, implying that none of these methods has particular advantage with respect to generalisation specifically in this setting.
cost, making RLHF better for
These results also show that the generalisation gap for SFT and Bol6 policies are the same (and they are still the same for different choice of temperature for Bol6 see Fig. 8). This can be explained by the fact that the reward model generalises near near-perfects serfecerfyttly to the CNNDM preference dataset from (Stiennon et al. 2022). ID and OOD accuracy is 75.8 and 71.6 respectively, and considering that the maximum inter annotatoo agreement the CNNDM preference dataset is lower than TL;DR (and hence the maximum accuracy attainable is lower), it is plausible that the RM suffering any real

---

Published as a conference paper at ICLR 2024
drop in OOD performance in this case. This implies that all of the drop in performance for Bol6 is driven by the drop in performance of the SFT model, as if both SFT and RM performed worse OOD would expect those drops in performance to compound. Overall, this shows that if you expect your reward model to generalise well then BoN is good choice of policy, although is limited by the generalisation abilities of the underlying model heing sampled from (the SFT model in this case), and is more expensive at inference time than RLHF and SFT.
In Appendix ! we present results for a range of models based on OPT (Zhang al. 2022). trained in a similar way to the LLaMa models but on split versions of the TL:DR dataset, and evaluated with a LLaMa proxy reward model. These results show similar trends: BoN outperforms RLHF which in turn outperforms SFT at the largest model size, and the ordering holds OOD for 3 different splits of the training dataset. This shows our results are robust across different evaluation metrics and base models.
AlpacaFarm Self-Instruct (ID)
Alpaca Eval (OOD) 1
Sequential Instructions (OOD)
50
50
50
30
30
0
0
0
BoN
RLHF
SFT
BoN
RLHF
SFT
BoN
RLHF
SFT
Generalisation Gap
Generalisation Gap
30
30
&. 20
20
10
10
5
BcN
RLHF
SFT
BcN
RLHF
SFT
Figure 2: Instruction Following Generalisation Results. GPT-4 PyR for SFT, BoN and RL policies, based on I.LaMa 7B, trained on the AlpacaFarm Self-Instruct instruction following task. ID is AlpacaFarm Self- Instruct, OOD is the AlpacaEval and Sequential Instructions datasets respectively, and generalisation gap ID OOD performance.
shows the results of BoN, SFT and RLHF models trained in Al- pacaFarm Self-Instruct (Dubois et al., 2023) evaluated ID) and OOD on AlpacaFval and Sequential Instructions. Similar to summarisation, we see that RLHF and Bo16 both outperform SFT, but here RLHF outperforms Bo16 across all datasets, in contrast to the summarisation task. As the focus of this paper is mostly comparing RLHF and SFT, we did not investigate this result further, but there could be many possible reasons for the change in ordering between RLHF and Bol6, as the two tasks are very different and the model training procedures are not identical.
Instruction Following. Fig.
We that AlpacaEval (the easier OOD generalisation task), models all generalise equally well, but on the harder Sequential Instructions OOD task, RLHF generalises much better. This suggests that RLHF may generalise better relative SFT for larger distribution shifts, which potentially explains why models fine-tuned with RLHFF have been observed to be much better in practice when interacting with users (Touvron al.) 2023b Ouyang et al., [2022] inter alia); when users interact with these models the distribution shift is quite pronounced and hence many inputs are more OOD, and this where RLHF model performance continues be high.
While GPT-4 PvR a useful metric, it does not show difference in generalisation (as measured by generalisation gap) between SFT, RLHF and Bol6 models on the easier AlpacaEval dataset. This could be due to these models having similar generalisation properties, or a deficiency of the metric. To investigate this further, we use can look at the GPT-4 Head-to- -o--Hooddd winrate of SFT vs Bo16 and vs RLHF, which is shown in Fig. 3 These results show that both RLHF and Bol6 winrates improves SFT by approx-
Winrate SFT RUHF
Wirrate of SFT BoN
Serumar 200 Siaws
U8in mes
Vioee COD
None COD
Figure 3: GPT4 Head-to-Head Winrate SFT vs RLHF and Bo16 in AlpacaFarm Self-Instruct, AlpacaF. val and Sequential Instructions datasets.

---

Published as a conference paper at ICLR 2024
EAD
Sent BERT T
NLI
1.0
0.6
0.925
0.5
0.8
0.4
0.900
0.6
0.3
0.875
0.2
0.4
0.1
0.850
0.2
0.0
RLHF SFT
RLHF
SFT
RLHF
SFT
Figure 4: Per-input diversity metrics for RLHF and SFT models. For these scores the outputs used to calculate the diversity are sample of outputs from the model for single input. These per-input scores are then averaged, in Eq. (2). Error hars are standard deviation of the per-input diversity score across different inputs. Note that some plots have broken y-axis for better visualisation.
imately 3.5% from ID to AlpacaEval OOD. This implies that RLHF and Bo16 generalise better than SFT even in this case, emphasising the need for a range metrics when evaluating model generalisation.
6.2 DIVERSITY
For the diversity evaluations, we focus on the summarisation task specifically, as it has the most compelling results. We ran some initial experiments evaluating diversity for the instruction-folloviin models, but we did not see any meaningful differences. We hypothesise this due to the diversity metrics use being designed for settings where the model output is relatively short (e.g. single sentence), whereas in the instruction-folloiiin setting outputs are generally much longer. Further- more, RLHF models tend to produce longer outputs than other models, which confound the evaluation of output diversity, since most metrics are not invariant to output length.
Fig. shows the per-input diversity scores for RLHF and SFT models in the summarisation task. We see that across the first two metrics, RLHF has much lower output diversity than SFT. Fig. 5 shows the across-input diversity scores in the same setting. Here has slightly higher diversity as before, the difference is much smaller than in the per-input case. The drop in across-input diversity cannot be explained purely by the use of the reward model, as BoN has similar or higher across-input diversity that SFT for the first two metrics. Both these trends are the same for OPT across model sizes (see Appendix f.4]or full results) We find that NLI does not show meaningful difference between models in either per-input or across input, showing that in a logical sense all models are similarly diverse.
that while SFT generally
The difference in across-input diversity between RLHF and SFT, while small, can also be taken as evidence of the phenomena of "mode collapse" hypothesised to occur under RL.FF fine-tuning (aanu]] 2022). The hypothesised effect is that even for different inputs, RLHF models can be biased towards outputting text of a specific style or "mode", meaning that even changing the inputs to a model is not sufficient to generate truly diverse outputs. We believe that this is the first rigorous empirical demonstration of scross-input mode collapse emerging from RLHF training specifically.
For most metrics and models, the across- input diversity scores higher than the per-input diversity scores, which is expected given the across-input diversity distribution is much broader. However, for EAD (which measures diversity the n-gram level), SFT has similar levels of per-input and across-input diversity. This is likely due to SFT effectively approaching the maximum EAD diversity even the per-input case, so that the across-input diversity cannot be much higher.
6.3 THE IMPACT OF THE KL PENALTY
We have shown that while RLHF improves performance and OOD in an absolute sense, this comes at the cost of substantial drops in output diversity relative to SFT. Motivated by the fact that the KL penalty coefficient (see Eq. (I) encourages the RLHF policy to stay closer to the SFT policy, investigate whether adjusting this coefficient trades off between generalisation and diversity. The results show that this does not work increasing the KL penalty coefficient leads to a drop in performance expected, but also drop in per-input diversity, rather than gain (see Appendix
8

---

Published as a conference paper at ICLR 2024
EAD T
Sent BERT
NLI 1
0,900
0.800
0.92
0.875
0.775
0.90
0.850
0.750
0.88
0.825
0.725
0.86
0.800
0.700 x
BoN
RLHF
SFT
BoN
RLHF
SFT
BoN
RLHF
SFT
Figure 5: Across-input diversity metrics for RLHF, BoN and SFT models. For these scores the outputs used calculate the diversity are set of single outputs from range of inputs, in Eq. (3). Note that all plots have broken y-axis for better visualisation; the differences between SFT and RLHI much smaller in this than the per-input diversity metrics Fig. Error bars (where present) are standard deviation of the across-input scores over different samples from the set of outputs for each input.
for details). This emphasises that more research needed to investigate whether more sophisticated methods can improve the trade-off between generalisation and diversity.
7 DISCUSSION AND CONCLUSION
Summary of Contributions. In this work, we analyse three methods for fine-tuning LLMs (RLHF, SFT, and BoN) across two problem settings (summarisation and instruction following) in terms of OOD generalisation and output diversity. We demonstrate an inherent tradeoff between generalisation performance and output diversity when choosing between these methods: RLHF produces more performant models both in-distribution and -iitt---ittrittiionn but at the cost of lower output diversity, both per-input and across-input. It is unclear whether this tradeoff is fundamental one in fine-tuning LLMs with RLHF or just demonstrates deficiency in current methods. We suspect the answer will be a combination of both explanations: There will be pareto-frontier of output diversity vs generalisation performance on which tradeoff's have to be made, but current methods do not yet seem be that frontier. Future work could investigate producing methods that are closer to this frontier, either through increasing the performance of SFT or increasing the output diversity of RLHF.
When looking at generalisation metrics that control for ID performance, results are mixed. RLHF generalises better for the most difficult distribution shift in the instruction following setting, but in less difficult shifts RI.HF generalises similarly or slightly worse than SFT (as measured by generalisation gap and head-to-head performance drop). While RLHF still performs best OOD in absolute terms, these results demonstrate the need for the multifaceted evaluation we perform this paper as opposed to focusing on single metric of performance.
Implications for Practical Applications. Our results have implications for which fine-tuning method should be used in different situations. The OOD performance of RLHF the most difficult instruction following task is evidence for the utility of RLHF when large distribution shifts are likely occur, such when training models to be used chat bots by users (OpenAI 2022| Anthropic 2023). However, in use cases where the model needs to generate a wide variety of outputs, such as story generation (Castricato et 2022), red-teaming (Perez et (022), and when doing rejection sampling (Cobbe et al. 2021), supervised fine-tuning may be desirable. In cases where you can expect the reward model generalise very well (for example, it is likely easier to spot whether text toxic or not than to never generate toxic text), then BoN may produce better generalisation results, although its performance will always be limited by the generalisation of the underlying model being sampled from, and its inference time cost is much greater than that of SFT or RLHF models.
Future Research Directions. This work also suggests areas for further research. Future work should investigate why RLHF reduces the output diversity so much, and whether this diversity can be recovered without the loss of performance. Inspiration could be taken from the deep reinforcement learning literature, where several works specifically inject diversity into the RL optimisation process to increase the policy's 's diversity Eysenbach et al. (2019): Haarnoja (2018): : Osa et al. (002);; Kumar ett al. (2020). Also, while there some hypotheses about why RLHF generalises better than SFT (Goldberg] 2023), it is important to experimentally validate these in order to build our understanding of how these methods work and when they should be used.

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGEMENTS
We would like to thank (in alphabetical order) Akbir Khan, Amy Zhang. Dieuwke Hupkes, Ethan Perez, Jacob Hilton, Kyle McDonnell, Laura Ruis, Louis Castricato, Patrick Lewis, Sehastian Riedel, Stephen Roller, Susan Zhang, Tim Rocktäschel and Verna Dankers for discussions and feedback on ideas related to this project. Robert Kirk is supported by the UCL CDT in Foundational AI.
AUTHOR CONTRIBUTIONS
Robert Kirk lead the project, set the direction, designed, programmed and ran the majority of the experiments, and wrote much of the paper. Ishita Mediratta assisted with programming and running the RLHF training experiments. Christoforos Nalmpantís programmed parts of the GPT-4 evaluation code and initial RLHF training code. Jelena Luketina assisted in the AlpacaFarm evaluations. Eric Hambro programmed earlier versions of the RLHF training code. Edward Grefenstette advised on project direction and paper writing. Roberta Raileanu advised project direction, experiment design, programming and paper writing. All authors participated in discussions over experiment design and paper editing.
REFERENCES
Marcin Andrychowiez, Dwight Crow, Alex Ray, Jonas Schneider, Rachel Fong, Peter Welinder, Bob McGrew, Josh Tobin, Pieter Abbeel, and Wojciech Zaremba. Hindsight experience replay. In Isabelle Guyon, Ulrike Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pp. 5048-5058, 2017. URL UR//ttpss.///rrceediinng. neurips. cC/
Anthropic. Introducing Claude, 2023.
URL
index
introducing-claate
Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, Nicholas Joseph, Saurav Kadavath, Jackson Kernion, Tom Conerly, Sheer El-Showk, Nelson Elhage, 7ac Hatfield-Dedds, Danny Hernandez, Tristan Hume, Scott Johnston, Shauna Kravec, Liane Lovitt, Neel Nanda, Catherine Olsson, Dario Amodei, Tom Brown, Jack Clark, Sam McCandlish, Chris Olah, Ben Mann, and Jared Kaplan. Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback. 05866 [cs], 2022.
/204 05862
Daniil A. Boiko, Robert MacKnight, and Gabe Gomes Emergent autonomoou scientific research capabilities of large language models, 2023.
05332
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agar- wal, Ariel Herbert-Voss Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu. Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCan- dlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In Hugo Larochelle, Mare' Aurelio Ranzato, Raia Hadsell, Maria-Flome and Hsuan- Tien Lin (eds.), Advances in Neural Information Processing Systems 33: Annual Conference Neural Information Processing Systems 2020, NewrIPS 2020, December 6-12, 2020, virtual, 2020. URL httpt::/t///ttttttt http:ttt////oocceeddigge//// newrips pttp://ppppep//////aaaaeeee html
Balcan,
Stephen Casper, Xander Davies, Claudia Shi, Thomas Krendl Gilbert, Jérémy Scheurer, Javier Rando, Rachel Freedman, Tomasz Korbak, David Lindner, Pedro Freire, Tony Wang, Samuel Marks, Charbel-Raphaël Segerie, Micah Carroll, Andi Peng, Phillip Christoffersen, Mehul Damani, Stewart Slocum, Usman Anwar, Anand Siththaranjan. Max Nadcau, Eric J. Michaud, Jacob Pfau, Dmitrii Krasheninnikov, Xin Chen, Lauro Langosco, Peter Hase, Erdem Biyık, Anca Dragan,
10

---

Published as a conference paper at ICLR 2024
David Krueger, Dorsa Sadigh, and Dylan Hadfield-Menell. Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback, 2023. URL http:://rxii:
erq/abs/ 2307 15217
Louis Castricato, Alexander Havrilla, Shahbuland Matiana, Michael Pieler, Anbang Ye, lan Yang, Spencer Frazier, and Mark Riedl. Robust Preference Learning for Storytelling via Contrastive Reinforcement Learning, 2022. URL www.wwwwwwwwwaaaa.aaaa.op/aaappa
07792
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsyyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vinodkumar Prabhakaran, Emily Reif, Nan Du, Ben Hutchinson, Reiner Pope, James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Lev- skaya, Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski, Xavier Garcia, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito, David Luan, Hyeontaek Lim, Barret Zoph, Alexander Spiridonov, Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omernick, Andrew M. Dai, Thanumalayan Sankaranarayana Pillai, Marie Pellat, Aitor Lewkowycz, Erica Moreira, Rewon Child, Oleksandr Polozov, Katherine Lee, Zongwei Zhou, Xuezhi Wang, Brennan Saeta, Mark Diaz, Orhan Firat, Michele Catasta, Jason Wei, Kathy Heiier-Hlllstem, Douglas Eck, Jeff Dean, Slav Petrov, and Noah Fiedel. PaLM: Scaling Language Modeling with Pathways, 2022. URL
org/ abs/ 2204 002311
Paul F. Christiano, Jan Leike, Tom B. Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 30: Annual Conference on Neu- ral Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pp. 4309. 2017. URL https:: neuruig.g..eeussssoe nwwrips wwweeurisscccppppeeeeeeeee wwtpppppee//////aaaaaaaaaeeeeeea
html
Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, Albert Webson, Shixiang Shane Gu, Zhuyun Dai, Mirac Suzgun, Xinyun Chen, Aakanksha Chowdhery, Sharan Narang, Gaurav Mishra, Adams Yu, Vincent Zhao, Yanping Huang, Andrew Dai, Hongkun Yu. Slav Petrov, Ed H. Chi, Jeff Dean, Jacob Devlin, Adam Roberts, Denny Zhou, Quoc V. Le, and Jason Wei. Scaling
Certuuttoo-FFinttnee
Language Models, 2022. URL http:// http//aaxvvv... www.iiiw.............o..ooo..ooe abs / 221022211... 11416
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training Verifiers to Solve Math Word Problems, 2021. URL http::pppp///....o////to
abs 2oss/2100.1113
Yann Dubois, Xuechen Li, Rohan Taori, Tianyi Zhang, Ishaan Guirajani, Jimmy Ba, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedback, 2023. URL StLtbtt::// /awpvva.ivvvvvv org/ /200//335200 14387
Benjamin Eysenbach, Abhishek Gupta, Julian Ibarz, and Sergey Levine
Diversity is all you need: Learning skills without reward function. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019. URL 1RgEm
lhttps://peezzet
net
Deep Ganguli, Liane Lovitt, Jackson Kernion, Amanda Askell, Yuntao Bai, Saurav Kadavath, Ben Mann, Ethan Perez, Nicholas Schiefer, Kamal Ndousse, Andy Jones, Sam Bowman, Anna Chen, Tom Conerly, Nova DasSarma, Dawn Drain, Nelson Elhage, Sheer El-Showk, Stanislav Fort, Zac Hatfield-Dodds, Tom Henighan, Danny Hernandez, Tristan Hume, Josh Jacobson, Scott Johnston, Shauna Kravec, Catherine Olsson, Sam Ringer, Eli Tran-Johnan
Dario Amodei, Tom Brown, Nicholas Joseph, Sam McCandlish, Chris Olah, Jared Kaplan, and Jack Clark. Red Teaming Language Models Reduce Harms: Methods, Scaling Behaviors, and Lessons Learned, 2022. URL httpthttpa/aiiiie o abs/ /tass/2200 220.
Leo Gao. John Schulman, and Jacob Hilton. Scaling Laws for Reward Model Overoptimization, 2022. URL http::ttt:rrrt wwwg/raaaaa / 1076660
11

---

Published as a conference paper at ICLR 2024
Amelia Glaese, Nat McAleese, Maja Trębacz, John Aslanides, Vlad Firoiu, Timo Ewalds, Maribeth Rauh, Laura Weidinger, Martin Chadwick, Phoebe Thacker, Lucy Campbell-Gillnghamm. Jonathan Uesato, Po-Sen Huang, Ramona Comanescu, Fan Yang, Abigail See, Sumanth Dathathri, Rory Greig, Charlie Chen, Doug Fritz, Jaume Sanchez Elias, Richard Green, Soña Mokrá, Nicholas Fernando, Boxi Wu, Rachel Foley, Susannah Young, lason Gabriel, William Isaac, John Mellor, Demis Hassabis, Koray Kavukcuoglu, Lisa Anne Hendricks, and Geoffrey Irving. Improving alignment of dialogue agents via targeted human judgements, 2022. URL attp://arx: org/ hte/abs//a 2209. 14375
Yoav Goldberg. Reinforcement Learning for Language Models, 2023. URL https: github.
Arav Gudibande. Koala Evaluation Set, 2023.
URL.
srrav--uudiaandeekcaaa--ttttttt
Tuomas Haarnoja. Acquiring Diverse Robot Skills via Maximum Entropy Deep Reinforcement Learning. PhD thesis, UC Berkeley, 2018.
et
25g6573w
Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas. Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hennigan, Eric Noland, Katie Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osindero, Karen Simonyan, Erich Elsen, Jack W. Rae, Oriol Vinyals, and Laurent Sifre. Training Camputee-ptimall Large Language Models, 2022. URL ptttt:://axxv..o/g//aa.
2203. 15556
Dieuwke Hupkes, Mario Giulianelli, Verna Dankers, Mikel Artetxe, Yanai Elazar, Tiago Pimentel, Christos Christodoulopoulls, Karim Lasri, Naomi Saphra, Arabella Sinclair, Dennis Ulmer, Florian Schottmann, Khuyagbaatar Batsuren, Kaiser Sun, Koustuv Sinha, Leila Khalatbari, Maria Ryskina, Rita Frieske, Ryan Cotterell, and Zhijing Jin. State-of-the-art generalisation research in NLP: A taxonomy and review, 2023. Utthttp::/// http://aa......oo///////10/////0 03050
janus. Mysteries of mode collapse, 2022. URL httpt::///// V5swA22MiF550aa333///stttttt
-oe--oee---o----------oo collapse
Natasha Jaques, Shixiang Gu, Dzmitry Bahdanau, José Miguel Hernández-LLoato Richard E. Turner, and Douglas Eck. Sequence tutor: Conservative fine-tuning of sequence generation models with kl-control. In Doina Precup and Yee Whye Teh (eds.), Proceedings of the 34th International Conference on Machine Learning, ICML 20/7, Sydney, NSW, Australia, 6-11 August 2017, volume 70 of Proceedings of Machine Learning Research, pp. .hmml
PMLR, 2017.
URLhttpp//prcceeaingg www.ang..mmmrrr...............mm mlr. wttppppessssssppppppppe htt.://cc.............ee////aaaa
Muhammad Khalifa, Hady Elsahar, and Mare Dymetman. A distributional approach to controlled text generation. In 9th International Conference Learning Representattions, ICLR 2021, Virtull Event, Austria, May 3-7, 2021. OpenReview.net, 2021. URL/attps://ppenew/// net/
forummid=jikkw 5 9AbL
Andreas Köpf, Yannic Kilcher, Dimitri von Rütte, Sotiris Anagnostidis, Zhi-Rui Tam, Keith Stevens, Abdullah Barhoum, Nguyen Minh Duc, Oliver Stanley, Richard Nagyfi, Shahul ES, Sameer Suri, David Glushkov, Arnav Dantuluri, Andrew Maguire, Christoph Schuhmann, Huu Nguyen, and Alexander Mattick. OpenAssistant Conversations - Democratizing Large Language Model Alignment, 2023. htLLtt/://ee http::///xxxxv.xg/aaa..
org. abs///b/////// 2304 07327
Saurabh Kumar, Aviral Kumar, Sergey Levine, and Chelsea Finn. One solution is not all you need: Few-shot extrapolation via structured maxent RL. In Hugo Larochelle, Marc' Aurelio Ranzato, Raia Hadsell,
Balcan, and Hsuan- Tien Lin (eds.), Advances in Neural Information Pro- cessing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6- 12, 2020, virtual, 2020. URR. igg new.ieerrip. iesurips.. cc/ paper/2020/hash/
Abst .tml
Jiwei Li, Michel Galley, Chris Brockett, Jianfeng Gao, and Bill Dolan. A objective function for neural conversation models. In Proceedings of the 2016 Conference of the
12

---

Published as a conference paper at ICLR 2024
North American Chapter of the Association for Computational Linguistics: Human Language Technologies. pp. 110-119. San Diego, Califormia, 2016. Association for Computational Linguistics. doi 1/N16-1016 URUAttpp::/aalattoooooo www.//oloo..///666/ 1014
Xuechen Li, Tianyi Zhang, Yann Dubois, Rohan Taori, Ishaan Gulrajani, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. AlpacaEval: An automatic evaluator of instruction-following models. GitHub, 2023. URLhttps://gitaun com www.//otttttttttoooooooooo http//alpaaaaeeeee
Hao Liu, Carmelo Sferrazza, and Pieter Abbeel. Chain of Hindsight Aligns Language Models with Feedback, 2023a. URL httt:://aaxiii httpt::://xxivooo//aaa/.o wwt.oogg//aooo..oooooo 2302 02676
Siyang Liu, Sahand Sabour, Yinhe Zheng, Pei Ke, Xiaoyan Zhu, and Minlie Huang. Rethinking and refining the distinct metric. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Pupers). pp. 762-770, Dublin, Ireland, 2022.
Association for Computational Linguistics. 1oi.10118633/11/222aclllll
URL https
-cwp::/.....//2222..lla--------a short.86
Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment, 2023b. URL httpp://axxiv.org
abs/2300 16634
Steven Loria TextBlob: Simplified Text Processing TextBlob 0.16.0 documentation, 2013. URL
https://textbloa http:eeettt..b...etteteeeeeeeeee weadthedocs..s.oooo 10/ www.i////////eee dev/
Rui Mao, Guanyi Chen, Xulang Zhang, Frank Guerin, and Erik Cambria. GPTEval: A Survey on Assessments of ChatGPT and GPT-4, 2023. URLhttpp. http:::/aaxxv....gp. abs /aabb//2330 2200 12488
Jacob Menick, Maja Trebacz, Vladimir Mikulik, John Aslanides, Francis Song, Martin Chadwick, Mia Glaese, Susannah Young, Lucy Camphell-Gillingee Geoffrey Irving, and Nat McAleese. Teaching language models to support answers with verified quotes, 2022. URL http::/ttaaxtt
org/ abs/ bzg//bs//ee 2203.22332 11147
Volodymyr Mnih, Adrià Puigdomènech Badia, Mehdi Mirza, Alex Graves, Timothy P. Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforcement learning. In Maria-Florina Balcan and Kilian Q. Weinberger (eds.), Proceedings of the 33nd International Conference on Machine Learning, ICML 2016, New York City, NY. USA, June 19-24, 2016, volume 48 of JMLR Workshop and Conference Proceedings, pp. 1928-1937. JMLR.org. 2016. URL URLpttp:: http:/::///ccceeddeeee mlr http://ns.m.....reee/eeee www.ppesss/vvv.. mn haa 1alha1.hhhe htmml
Retichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christopher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, Xu Jiang, Karl Cobbe, Tyna Eloundou, Gretchen Krueger, Kevin Button, Matthew Knight, Benjamin Chess, and John Schulman. WebGPT: Browser-assisted
with human feedhack, 2022. URL /tttp://arxivt
abs/2112. 09332]
Ramesh Nallapati.
Bowen Zhou, Cicero dos Santos, Çağlar Gulçehre, and Bing Xiang. Abstractive to-suqeence RNNs and beyond. In Proceedings of the 20th SIGNLL Conference on Computational Natural Language Learning. pp. 280-290, Berlin, Germany, 2016. Association for Computational Linguistics.
text summarization using
URL https:
//aclannhoolooy..or///666--1
OpenAl. Introducing ChatGPT, 2022. URL Bttpps:/ppehaaa.cco/loo///aaaaaa
OpenAl. GPT-4 Technical Report, 2023. URL httpttpp///e211..r//////33333113
Takayuki Osa, Voot Tangkaratt, and Masashi Sugiyama. Discovering Diverse Solutions in Deep Reinforcement Learning by Maximizing State-Action Mutual Information, 2022. URL
http://arxiv..rr /abrr//bbs///
abs / 2103. 07084
Long Ouyang. Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback, 2022. URL http::/arxii or. wttpr/wwwwwrg//b////////////// 02155
13

---

Published as a conference paper at ICLR 2024
Kishore Papineni, Salim Roukos, Todd Ward. and Wei-Jing Zhu. Bleu: method for automatic evaluation of machine translation. In Proceedings of the 40th Anmual Meeting the the Association for Computational Linguistics, pp. 311-318 Philadelphia. Pennsylvania, USA. 2002. Association for Computational Linguistics. 1o11100.3113//1073333110003
URL htt.h:ttps://ett antho 1 logy
orq/PO2-1040)
Baolin Peng, Chunyuan Li, Pengcheng He, Michel Galley, and Jianfeng Gao. Instruction Tuning with GPT-4, 2023. halltttp//aaxa wrg/ 2304. 03277
Ethan Perez, Saffron Huang, Francis Song. Trevor Cai, Roman Ring, John Aslanides, Amelia Glaese, Nat McAleese, and Geoffrey Irving. Red teaming language models with language models. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp. 1419-3448, Abu Dhabi, United Arab Emirates, 2022. Association for Computational Linguistics. URL https:
/tr//acaannntooo http:///aaaaaanooy..rrrrooo 2022. emnlp mainnmainn.... 225
Jack W. Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susannah Young, Eliza Rutherford, Tom Hennigan, Jacob Menick, Albin Cassirer, Richard Powell, George van den Driessche, Lisa Anne Hendricks, Maribeth Rauh. Po-Sen Huang. Amelia Glaese, Johannes Welbl, Sumanth Dathathri, Saffron Huang. Jonathan Uesato, John Mellor, Irina Higgins, Antonia Creswell, Nat McAleese, Amy Wu, Erich Elsen, Siddhant Jayakumar, Elena Buchatskaya, David Budden, Esme Sutherland, Karen Simonyan, Michela Paganini, Laurent Sifre, Lena Martens, Xiang Lorraine Li, Adhiguna Kuncoro, Aida Nematzadeh, Elena Gribovskaya, Domenic Donato, Angeliki Lazaridou, Arthur Mensch, Jean-Baptiste Lespiau, Maria Tsimpoukelli, Nikolai Grigorev, Doug Fritz, Thibault Sottiaux, Mantas Pajarskas, Toby Pohlen, Zhitao Gong, Daniel Toyama, Cyprien de Masson d' dutuume, Yujia Li, Tayfun Terzi, Vladimir Mikulik, Igor Babuschkin, Aidan Clark, Diego de Las Casas, Aurelia Guy, Chris Jones, James Bradbury, Matthew Johnson, Blake Hechtman, Laura Weidinger, Iason Gabriel, William Isaac. Fd Lockhart, Simon Osindero, Laura Rimell, Chris Dyer, Oriol Vinyals, Kareem Ayoub, Jeff Stanway, Lorrayne Bennett, Demis Hassabis, Koray Kavukcuoglu, and Geoffrey Irving. Scaling Language Models: Methods, Analysis & Insights from Training Gopher, 2022. StLhttt::/ http:://axxiv.ooo awwiv. org/ abs/
Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning, and Chelsea Finn. Direct Preference Optimization: Your Language Model is Secretly Reward Model, 2023.
URL http:://axii..or/ abs/ 18290
Rajkumar Ramamurthy, Prithviraj Ammanabrolu, Kianté Brantley, Jack Hessel, Rafet Sifa, Christian Bauckhage, Hannaneh Hajishirzi, and Yejin Choi. Is Reinforcement Learning (Not) for Natural Language Processing?: Benchmarks, Baselines, and Building Blocks for Natural Language Policy Optimization, 2022. http: org/ www.///s....... 01241
Nils Reimers and Iryna Gurevych. Sentence-BERT: Sentence embeddings using Siamese BERT- networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP- IJCNLP), pp. 18110..166//11/DD/
Hong Kong. China, 2019. Association for Computational Linguistics. URL http:p:://oclaatttoooooooo//////
1410
Jérémy Scheurer, Jon Ander Campos, Tomasz Korbak, Jun Shern Chan, Angelica Chen, Kyunghyun Cho, and Ethan Perez. Training Language Models with Language Feedback at Scale, 2023. URL
http:://axix> wwwprriwwv.vvvvoooeeeee orooorr///aa.. a3taaab///3331.0
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal Policy Optimization Algorithms. arXiv:1707.06347 [cs], 2017. URL htttp://arxiv.orgg/ abs/
Charlie Snell, Ilya Kostrikov, Yi Su, Mengjiao Yang, and Sergey Levine. Offline RI. for Natural Language Generation with Implicit Language Q Learning, 2022. URL. htt.:ttpp//arxiii.rrr
abs/ / 020/2226..111 11871 11871
Katherine Stasaski and Marti Hearst. Semantic diversity in dialogue with natural language inference. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 85-98, Seattle, United States,
14

---

Published as a conference paper at ICLR 2024
2022. Association for Computational Linguisties. doi:10.1655/v112222aaccma-- URL https:///acLanthoooyy..coo
Nisan Stiennon, Long Ouyang. Jeff Wu, Daniel M. Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford,
Dario Amodei, and Paul Christiano. Learning to summarize from human feedback, 2022. URL
http:://rxiv/
wttp:/v...gggbb/////o// http:://arriv.orrg/aa//222
.01325
Guy Tevet and Jonathan Berant. Evaluating the evaluation of diversity in natural language gener- ation. In Proceedings of the 16th Conference of the European Chapter of the Association for Computationnl Linguistics: Main Volume, pp. 326-346, Online, 2021. Association for Compu- tational Linguistics. doi 100018666/1120021.ccllmmama
URL https: httppss
2xg/202L.eacl-mainn
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. LLaMA: Open and Efficient Foundation Language Models, 2023a. URL http: http://axxiv..ooo org/ 220. 13971
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaci, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cris- tian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open Foundation and Fine-Tuned Chat Models, 2023b. URL http: /ttp///axvvvvv//a org abs/ / 2307 09288
Michael Völske, Martin Potthast, Shahbaz Syed, and Benno Stein. TL:DR: Mining Reddit to learn automatic summarization. Proceedings of the Workshop on New Frontiers in Summa- rization, pp. 59-63, Copenhagen, Denmark, 2017. Association for Computational Linguistics. doi
htlltttps:// //aci
Yizhong Wang, Swaroop Mishra, Pegah Alipoormolabashi, Yeganeh Kordi, Amirreza Mirzaei, Atharva Naik, Arjun Ashok, Arut Selvan Dhanasckaran, Anjana Arunkumar, David Stap. Eshaan Pathak, Giannis Karamanolakis, Haizhi Lai, Ishan Purohit, Ishani Mondal, Jacob Anderson, Kirby Kuznia, Krima Doshi, Kuntal Kumar Pal, Maitreya Patel, Mehrad Moradshahi, Mihir Parmar, Mirali Purohit, Neeraj Varshney, Phani Rohitha Kaza, Pulkit Verma, Ravsehaj Singh Puri, Rushang Karia, Savan Doshi, Shailaja Keyur Sampat, Siddhartha Mishra, Sujan Reddy A, Sumanta Patro, Tanay Dixit, and Xudong Shen. Super-NaturalInstrucce
Generalization via declarative instructions on 1600+ NLP tasks. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing. pp. 5085-5109, Abu Dhabi, United Arab Emirates, 2022. Association for Computational Linguistics. URL https: wtattss://aclaahhh...t http//aaaannhhccccooooooo org/ 2022. emn --maminnnnnnimmm
340
Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith, Daniel Khashabi, and Hannaneh Hajishirzi. Self-Instruct: Aligning Language Models with Self-Generated Instructions, 2023. BRLLhtt::/aaxiii http///xxxiv..r//bbs././//
2212.1111 10560
Sean Welleck, Ilia Kulikov, Stephen Roller, Emily Dinan, Kyunghyun Cho, and Jason Weston. Neural text generation with unlikelihood training. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa. Ethiopia, April 26-30, 2020. OpenReview.net, 2020. URL
net htt.://ewwe....eeeoooooooooeoeee
Tian Xu, Ziniu Li, Yang Yu, and Zhi-Quan Luo. On Generalization of Adversarial Imitation Learning and Beyond, 2022. htt.:/tt///aarv...o//////////1/0
15

---

Published as a conference paper at ICLR 2024
Zheng Yuan, Hongyi Yuan, Chuanqi Tan, Wei Wang, Songfang Huang, and Fei Huang. RRHF: Rank Responses to Align Language Models with Human Feedback without tears, 2023. URL
http://arxiv..o org/
hwtg/bb//////a////////////////e/ abs 2304 05302
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, Todor Mihaylov, Myle Out, Sam Shleifer, Kurt Shuster, Daniel Simig, Punit Singh Koura, Anjali Sridhar, Tianlu Wang. and Luke Zettlemoyer. OPT: Open Pre-trained Transformer Language Models, 2022. URL http: http:://axxive/ arxiv.org/
abs/2205.01063
Tianjun Zhang. Fangchen Liu, Justin Wong, Pieter Abbeel, and Joseph E. Gonzalez. The Wisdom of Hindsight Makes Language Models Better Instruction Followers, 2023. hRtt::/://aax
org/aba/2300 05206
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li. Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and lon Stoica. Judging LLM-as-a-juuee judge with MT-Bench and Chatbot Arena, 2023. URL http:
org/aba 2306 05685
Yaoming Zhu, Sidi Lu, Lei Zheng, Jiaxian Guo, Weinan Zhang, Jun Wang. and Yong Yu. Texygen: A benchmarking platform for text generation models. In Kevyn Collins-Thompson Qiaozhu Mei, Brian D. Davison, Yiqun Liu, and Emine Yilmaz (eds.), The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval, SIGIR 2018, Ann Arbor, MI, USA, July 08-12, 2018, pp. 1097-1100. ACM, 2018. doi 10000.144//209978322000 321 C080
URL
pktpss//doot .wr./...rrg/oo........ /tl.rgg10..0 10.....11352999990
Daniel M. Ziegler, Nisan Stiennon, Jeffrey Wu, Tom Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-Tuning Language Models from Human Preferences, 2020.
URL http: org/ abs 08593
16

---

Published as a conference paper at ICLR 2024
A LIMITATIONS AND FUTURE WORK
Here we discuss some potential limitations of our work and possible future directions for research pointed to by our results. While our work shows the effects of RLHF on generalisation and output diversity empirically, we do not provide theoretical explanation for these results. Furthermore, while we demonstrate results on multiple base models and tasks, more combinations base models and tasks could be experimented on, well as other methods. Future work could investigate whether these effects are more general and why they arise.
Our work also only investigates SFT, RLHF and BoN as methods for fine-tuning language models with human preferences, but there are many other methods as described in Appendix Understanding the effects of these methods generalisation and output diversity would be beneficial, especially some of those methods are able to provide the generalisation benefits of RLHF without harming output diversity to the same extent.
Finally, we only evaluate our models on automatic metrics and do not perform any human evaluation (although validate that our metrics align well with human preferences in Appendix D.Te While automatic metrics are useful for comparing models, they are not a perfect proxy for human judgement. Future work could investigate the effects of RLHF on human judgement of model outputs.
B BROADER IMPACT
Large Language Models are increasingly used in production systems, and so is important to understand the effects of different fine-tuning methods on the properties of the resulting models. Our work shows that RLHF produces better- generaliena models than SFT, but those models are less diverse. This could be beneficial for some use cases, but harmful for others. For example, if a model is being used to generate text for a chathot, it is important that the model is able to generalise to new inputs, but also that it is able produce diverse outputs. On the other hand, if the model is being used to generate text for a summarisation system, it is important that the model is able to generalise new inputs, but less important that it is able produce diverse outputs.
C RELATED WORK
More RLHF and SFT details Often, SFT and RI.HF are combined by performing SFT followed by
RLHF (Glaese et al., 2022 Menick et al. 2022. Nakano et al. 2022 Ouyang al. 2022). We call the process of doing SFT followed by RL. "The RLHF Pipeline", " as it s the standard approach used in the literature and in deployed products (that use RLHF) (OpenAl 2022 Anthropic 2023). Other work has directly used RLHF top of prompt-distilled language model Bai et al. 2022). Prompt distillation gathers demonstrations from a prompted version of base model, and then performs SFT on the base model with these outputs, effectively fine-tuning the model to behave as if it was always prompted.
Ramamurthy al. (002)) introduced an adaptation of PPO specifically for RLHF called Natural Language Policy Optimisation (NLPO), which calculates an action mask with top-p sampling and applies this the language model, resulting in slightly improved performance on range of tasks when using automated reward functions (not trained with human preferences for the task). |Ramamurthy al. 2022) demonstrate that RL generally outperforms SFT, but that their combination performs the best. However, their work only investigates relatively small models (220 million parameters), does not evaluate OOD performance or diversity, and does not use reward functions trained with human feedback, instead using mostly hard-coded reward functions from the literature. While hard-coded reward functions can sometimes useful, RLHF is most widely applied in settings where we do not have a hard-coded reward function, and hence need to learn one from human data.
Large Language Models (LLMs) have recently been shown to solve a wide variety of language- based tasks, often without additional gradient-based training. Examples of such models include GPT-3 (Brown et al. 2020), Gopher (Rae et al. 2022). Chinchilla (Hoffmann et al. 2022). OZhngg et al. (202)) PaLM (Chowdhery et al. 2022), Claude (Anthropic (202) and LLaMad Touvron et al. 2023a). These models are trained on large amounts of text data, with a simple language modelling objective, and can often be prompted perform tasks zero-shot few-shot, without additional
17

---

Published as a conference paper at ICLR 2024
2020). Such tasks include translation, question-answe -nswwring. and other standard NLP tasks, as well as newer tasks such as using LLMs to simulate human annotators (Dubois et al. content generators for improving other models (Peng
training (Brown et al.
2023
Mao et al. 2023 Liu et al. 2023b)
et al.
2023
Wang et al. 2023).
Other methods for fine-tuning LLMs Recently, multiple approaches for fine-tuning LLMs have been proposed: Chain of Hindsight (Liu et al. 2023a) fine-tunes models using SFT on sequences of increasingly better outputs for a given input: CoOp CARP (Castricato al. 2022) uses dataset of story-critique pairs combined with contrastive learning and Sacudo-labelling to learn a preference model that is then used in the RLHF pipeline: RRHF Yuan et al. 2023), uses RM rank multiple outputs from the model, and then optimises the model with weighted SFT, with a negative weight (similar to unlikelihood training (Welleck et al.) 2020)) on lower-ranked samples; HIR (Zhang et al.) 2023) relabels outputs using goal-conditioned reward function or feedback function and then trains goal-conditioned policy on these outputs (similar to (Andrychowicz et al. 2017); ): and ILF (Scheurer et al. 2023), which uses natural language human feedback to prompt the model to produce better outputs than its original inputs, and then optimises the model with SFT on this dataset of improved outputs. While these works sometimes show improvements, they not used by most large-scate scale systems being deployed currently, and hence we focus our analysis on the more popular and widely used RLHF pipeline, that is where understanding will be most relevant and useful.
Possible Explanations for Results. et al. (2022) present results for adversarial imitation learning (AIL) as compared behavioural cloning (BC) in more classical RL setting, showing that often AIL methods can generalise better than BC because they optimise the policy on out-of-distribee states. Mapped to the LLM fine-tuning regime, AIL somewhat analogous to RLHF, and BC is identical to SFT, so this result may somewhat explain why RLHF generalises better than SFT.
Goldberg 2023)
hypothesises that RLHF may generalise better than SFT because RLHF does not force models to produce outputs that are not implied their internal world model (to the extent that exists), whereas SFT trains models produce outputs even if the model "believes" those outputs to be false.
D GPT-4 EVALUATION DETAILS
For GPT-4 evaluations for both summarisation and instruction following, we use the AlpacaEval (Li al. 2023) software package to query GPT-4. For the instruction-followinn prompts, use the standard annotator configuration recommended for that dataset, alpaca_ seraa__ev____g__eeeeeaeae _gppt. 4. For summarisation, we use the same configuration, but change the prompts to utilise variant of those provided in (Rafailov et al., 2023). For the exact prompts see Figs. and
D.1 VALIDATING GPT-4 EVALUATION
Summarisation. We validate the use of our GPT-4 evaluator for summarisation in two ways. First, we use the evaluator label the preference validation datasets for both TL:DR and MNNi released by Stiennon et al. (2022) and measure their accuracy. On TL;DR our evaluators gets 71.7% accuracy and on Mali it gets 65.5% accuracy. Comparing this the inter-annotator agreement reported by Stiemnon et al. (022) or 70% for TL.DR and 66% for CNN/DailyMMii demonstrates that our annotator has strong agreement with the human raters that generated the preference data we use for training our reward models.
For the second validation of GPT-4 as an evaluator, we measure the agreement between human labellers and GPT-4 on a subset of 25 inputs for every test set we use, comparing both SFT and RLHF model outputs with the reference output. This results in a total of 100 datapoints, each labelled by two human labellers giving 200 total annotations. This tests whether GPT-4 is in agreement with human preferences on the models we evaluate this work. Table | shows the preference rating for GPT-4 and human labellers for each dataset and model, and the agreement between labellers and GPT-4, and we see that both at an aggregate level and at a per-example level our GPT-4 evaluator has good agreement with expert labellers.
18

---

Published as a conference paper at ICLR 2024
[lim_start >systen
You are a helpful assistant, that ranks models by the quality of their answers.
<|im_end]>>
<lim_start|/Psst
Which of the following summaries does better job of summarizing the most important points in the given article, without including unimportant or irrelevant details? good summary is both precise and concise. Post: (instruction) "**
Summary A:
"model": "model 1", "summary": "." (output_1)"'
Summary B:
"model": "model_2" "summary"; u." (output_2)""
Now please rank the models by the quality of their summaries, so that the model with rank 1 has the best summary. Then return list the model. names and ranks, 1.e,, produce the following output:
(' m'moell: <model-name>, 'rank': <model-rank>>, ('model'; <model -nam>,, rrank"; <model-rankks)
Your response must be valid Python dictionary and should contain nothing else because will directly execute it in Python. Please provide the ranking that the majority of humans would give. <li_endd>>
Figure GPT-4 evaluation prompt the CNN DailyMail dataset.
Table I: GPT-4 agreement with human raters at an aggregate and individual level, across both summarisation datascts and RLHF and SFT model types. We see that both aggregate level and per-example level GPT-4 evaluator has good agreement with expert labellers.
Dataset
TL:DR RLHF
CNNDM RLHF
Model Type
SFT
SFT
GPT-4 winrate
0.40
0.40
0.24
0.08
Human winrate
0.48
0.40
0.30
0.30
H-GPT-4 agreement
69%
72%
74%
72%
19

---

Published as a conference paper at ICLR 2024
<lim starttrtra (syssen
You are a helpful assistant, that ranks models by the quality of their answers.

>
<lim_start/>>uer
Which of the following summaries does better job of summarizing the important peints in the given ferum post, without including unimportant or irrelevant details? A good sunmary is both precise and concise. Post: ""* (instruction) un* Summary A:
Mmodel": :: "madel_1" "summary"; """ [output_1)""
Summary B:
model": "model_2", "summary"; """ """(outuut_])" ***
Now please rank the models by the quality of their summaries, so that the model with rank 1 has the best summary. Then return list of the model. names and ranks, i.e., produce the following output:
(' moael'; <model-name>, 'rank': <model-ranke), ('model'; <model -name>, 'rank'; <model-ranks)
]
Your response must be valid Python dictionary should contain nothing else because we will directly execute in Python. Please provide the ranking that the majority of humans would give. ili__ddd>>
Figure 7: GPT-4 evaluation prompt for the TL:DR dataset.
20

---

Published as a conference paper at ICLR 2024
Instruction Following. For instruction following, we use the evaluator released in (Li et al. 2023). which has been rigorously show to agree well with human labellers both at a per-example and aggregate level. Hence, do not validate this ourselves.
E MODEL TRAINING DETAILS
E. REWARD MODEL TRAINING.
To train the Reward Model (RM), we again follow (Stiennon al. 2022) We initialise the RM as the base model, and we add scalar head before the unembedding layer. The model is then fine-tuned on inputs and pairs outputs with one output labelled as preferred. The RM takes the full input and output, and outputs scalar value. This gives us scalar value for each output in the pair, and these values are treated as logits in a softmax to predict the correct preference label. We train the R.M with cross- enos-py loss. Formulating the RM in this way means it can (in theory) learn to predict any transitive ranking over possible outputs, while still maintaining a type signature suitable for use as a reward function in an RL context (producing scalar value given single input-output pair which acts as proxy for the quality of the output given the input, as evaluated by human annotators).
E.2 POLICY TRAINING
We treat the autoregressive language model reinforcement learning policy, where the action space is the set of possible tokens, the state space is the the outputted action the input. The episode terminates when the maximum number of tokens is generated, or the model outputs an end-of-sequence token.
current input, and the transition function adds
Supervised Fine- Vine-Tuning. In this interpretation of the LLM policy, Supervised Fine-Tunine (SFT) can be seen as behavioural cloning, form of imitation learning in RL, where the behaviour being cloned is that of the human who produced the original output. The policy is trained with the cross-entropy loss on batches of inputs concatenated with outputs, with the loss calculated only on the output tokens.
Reinforcement Learning from Human Feedback. Again following previous work we use PPO (Schulman et al., 2017) as our base RL algorithm. We initialise the model with the corresponding SFT model. We treat the language modelling head as the policy output, and learn separate value function which takes the final hidden state of the language model and outputs a scalar using a MLP layer (an identical architecture to the reward model). We a shared backbone for the policy and value function, with only the two heads being different. We train with the reward function described in Appendix [ and stays close to the SFT model, as in prior work (Jaques et al. 2017 al. 2020). The final reward for the policy is
KL divergence term an auxiliary reward to ensure the language model
Stiennon et al. 2022
Ziegler
R(x, y) = R.Ma (x, v) 3KL (TeRL (wl=)]] (y(x))
where RM denotes the reward model trained as described in Appendix E. PRL, ORM and F the parameters of the policy, reward model and SFT model respectively: x, y are the input and output; and 3xL is a hyperparameter that controls the weight of the KL penalty. We use 3KL. = 0.05 throughout this work, following (Stiennon al. 2022) and our own early experiments that found this choice struck a good balance between model performance and overoptimisation (Gao et al.) (022).
Best-of-N. Instead of using the RM to train a policy via PPO, the reward model can be used to filter samples from another model; this is called Best-of-N (BoN) sampling, and has been used in multiple previous works to achieve good performance (Menick et al., 2002) Nakano al. 2022). N summaries sampled from the pretrained model, and then the RM is used select the best one. This method removes the need to perform RL policy training, but makes inference time more computationally expensive, as require N generations from the policy and N passes through the reward model to produce a single output. In this work we do BoN sampling on top of the SFT model, and mostly use N 16, that is what is used in previous works and strikes good balance between improved performance and computational cost.
We summarise the differences in training from (Steennon et al. 2022 in Appendix
21

---

Published as a conference paper at ICLR 2024
Table 2: Hyperparameters for SFT model training. These are fixed across all dataset splits and model sizes and types for summarisation.
Hyperparameter
Value
batch size
128
epochs
1
adam betal
0.9
adam beta2
0.999
adam epsilon
1e-8
frozen layers 80%
Table 3: Hyperparameters for RM training. These are fixed across all dataset splits and model sizes and types for summarisation.
Hyperparameter
Value
batch size
64
epochs
adam betal
0.9
adam beta2
0.999
adam epsilon
1e-8
frozen layers
80%
E.3 MODEL SELECTION
For the results reported in the paper, we sweep over 3-5 learning rates for each model type, and select the best model the validation using appropriate metric (accuracy for RMs, loss for SFT, reward for RL) see Appendix E..4 For both the in-distribution and out-of-distribution results, we always use validation set drawn from the same distribution training. This more closely matches real-world setting where we would not have access to OOD data to do model selection, and would
have draw model selection data from the same distribution used in training.
E.4 HYPERPARAMETERS
Summarisation. For each model type (SFT, RM, RLHF) we do a sweep over learning rate, choosing ranges of values informed by choices in previous work (Stiennon et al.) 2022) and early experimenta- tion. The results in the paper are the best model with the learning rate chosen an in-distribution validation set using loss, accuracy and reward respectively for SFT, RM and RLHF training. The learning rates for SFT are 3e-4, 1e-4, 3e-5, with 3e-5 selected; for RMs are 3e-4, le-4, 3e-5, le-5, 3e-6, with 3e-5 selected; for RLHF are: 1.5e-6, 3e-6, 6e-6, 1.5e-5, 3e-5, with 1.5e-5 selected.
We list the other hyperparameters (which are unchanged between all runs) for SFT, RM and RLHF training Table 2 Table and Table respectively. We chose these following prior work (Stiennon)
al. 2022)
Instruction Following. For the instruction following results, we use the models released by (Dubois al. 2023). and so the hyperparameters can be found in that work.
F DIFFERENCES FROM STIENNON ET AL. (2022)
As we mostly follow (Stiennon et al.) 2022) in the training of our summarisation models, we here describe the main differences between our work and theirs in terms of training. For RLHF, we train single model with policy and value head, rather than separate policy and value functions. This is much more computationally efficient, and follows other recent work that still achieves impressive results (Glaese al. 2022). This means that randomly initialise the value function head, rather than intialising the value function from the reward model as done in (Stiennon et al.) (022).. We pretrained
LLaMMa (Touvron et al. 2023a) (and OPT (Zhang al. 2022 in Appendix
22

---

Published as a conference paper at ICLR 2024
Table 4: Hyperparameters for RLHF model training. These fixed across all dataset splits and model sizes types for summarisation. One PPO step consists generating batch size samples, and then performing ppo epochs of optimization on them, split into ppo minibatch size minibatches.
Hyperparameter
Value
batch size ppo epochs
256
4
ppo steps ppo minibatch size
750
256
KL penalty coefficient
0.05
normalise advantages
True
adam betal adam beta2
0.9
0.999
adam epsilon
1e-8
frozen layers
80%
(TL;DR)
OOD (CNN/DaillMaiil)
Generalisation Gap
GPT-4-PVR
Gap in PvR
20
40
40
20
20
10
0
Bo16 (t=0.7)
Bo16 (t=1.0) 0)
RLHF
SFT
Bo16 (t=0.7)
Bo16 (t=1.0) 0)
RLHF
SFT
Bo16 (t=0.7)
Bo16 (t=1.0)
RLHF
SFT
Figure 8: GPT-4 API evaluation win rate reference summaries for SFT, Bol6 with two different temperatures, models, trained on the summarisation task. In-distribution is performance on TL:DR, and out-of- distribution is on CNN/DailyMaii, Effectively version of Fig. with addition Bol6 results with a worse- performing temperature
models, while they use unrelcased models which were trained in similar way to GPT-3 Brown al. 2020).
We freeze the first 80% of the layers and the embedding and unembedding layers, as more recent work (Glaese et al. 2022, Menick et 2022) has shown that good results can still be achieved, and training is much more computationally efficient. In T4 we show the drop in performance for smaller OPT models between freezing 80% of the layers and not freezing. There a drop of performance, but it is not catastrophic (equivalent to about drop in model size among the three model sizes used), which justifics the tradeoff training models with partially frozen weights.
G BEST OF N TEMPERATURE EXPERIMENT
In the main paper report results for BoN using temperature 0.7. Here we show results for BoN with temperature 1, and show that temperature 0.7 performs better, hence our choice of it as the hyperparamter we use. Fig. shows the results of BoN with two temperatures, as well RLHF and SFT for comparison.
H SEQUENTIAL INSTRUCTIONS DATASET DETAILS
For the sequential instructions dataset, we build on the AlpacaFarm variant of the Self-Instruct protocol (Dubois et al. 2023) Wang al.) 2023), but adjust the seed instructions and prompt gather more sequential instructions. Fig. (Jshows the prompt used to generate these instructions, and Table 3 shows examples of generated from the dataset.
23

---

Published as a conference paper at ICLR 2024
You are asked to come up with set of 20 diverse task instructions. These task instructios will be giver to GPT model and we will evaluate the GPT model for completing the instructions.
Here are the requirements:
1. Try not to repeat the verbs for each instruction to maximize diversity.
2. The language used for the instruction also should be diverse. For example, you should combine questions with imperative instrucitons.
3, The type of instructions should be diverse. The 1ist should include diverse types of tasks like open-ended generation, classification, editing, etc.
4. GPT language model should be able to complete the instruction. For example, do not ask the assistant create any visual audio output. For another example, do not ask the assistant to wake you up at 5pm or set a reminder because it cannet perform any action.
5. The instructions should be in English.
compositional instruction containing multiple steps, where each step is related to the previous steps. Either an imporative sentence or a question is permitted,
6. The instructions should be sequential
7. Try not to repoat the verbs used for each part of the instruction across instructions to maximize diversity.
8. The output should be apprepriate response the instruction and the input. Make sure the output is less than 100 words.
List of 20 tasks:
Figure 9: The prompt for text-davinci-003 produce instructions for the sequential instructions dataset using the Self-Instruct protocol (Wang al. 2023
Table 5: Example inputs from the sequential instructions dataset
Generate list of items that may be found in a first aid kit, along with description on why each item important.
Sort a list of emotions (sadness, joy, anger, fear, disgust) into two categories, and explain why each emotion fits into the categories created.
Explain the concept of Renormalization Group in simple words, describe three uses for Renormaliza- tion Group in theoretical physics, and discuss its relationship with scaling laws. Summarize the history of the Cold War, explain the outcome of the war. and discuss its significance the world today.
24

---

Published as a conference paper at ICLR 2024
TL;DR (ID)
CNN/DailyMail (OOD)
Generalisation Gap
30
Win rate
40
40
20
20
20
Gap
10
0
0
0.01
0.05
0.5
1.0
0.01
0.05
0.5
1.0
0.01
0.05
0.5
1.0
Figure 10: GPT-4 API evaluation win rate reference (text-davinci-O0 003) outputs RLHF models, based on LLaMa 7B, trained on the summarisation task, sweeping over KL penalty coefficient, In-distribution is perfor- mance on TI;DR, out-of-distribution is on CNN/DailyMail, and generalisation gap is II) - OOD performance.
EAD
Sent BERT
NLI
0,950
0,925
0,900
0.875
0.850
0.0
0.0
0.01
0.05
1.0
0.01
0.05
0.5
1.0
0.01
0.05
0.5
1.0
KL Penalty
KL Penalty
KL Penalty
Figure 11: Per-input diversity metries for RLHF summarisation models with different KL penalty coefficients. For these scores the outputs used calculate the diversity sample of outputs from the model for single input. These per-input scores are then averaged, as Eq. (2) Error bars are standard deviation per-input diversity score across different inputs. Note that some plots have broken y-axis for better visualisation.
I SUMMARISATION KL SWEEP RESULTS
Here we present results for generalisation and diversity for LLLMaa models trained with RLHF on the summarisation task, sweeping over the KI. penalty coefficient. This hyperparameter determines the weight of the KL penalty in the RLHF reward (the 9к in Eq. (T). It could be the case that this KL penalty can control the tradeoff between diversity and generalisation, and so we try multiple different values of the penalty and include the results here.
found that higher KL penalties actually resulted in less output diversity (Figs. and 12 and also generally worse performance (Fig. [Op, showing that the choice of KL penalty did not seem to provide a way to tradeoff between diversity and generalisation.
EAD
Sent BERT T
NLI
0.86
0.800
0.92
0.775
0.90
0.84
0.750
0.88
0.725
0.82
0.700
0.86
0.84
0.01
0.05
0.5
1.0
0.01
0.05
0.5
1.0
0.01
0.05
0.5
1.0
Penalty
KL Penaily
KL Penalty
Figure 12: Across-input diversity metrics for RLHF, Bol6 and SFT policies. For these scores the outputs used to calculate the diversity set of single outputs from range of inputs, in Eq. (3). Note that all plots have broken y a-axis for better visualisation.
25

---

Published as a conference paper at ICLR 2024
Table 6: Size of the train, validation, test and OOD test datasets for each split for the SFT and RLHF models.
length
sentiment
relationships
Train
58770
58361
63324
Validation
3234
3223
3462
Test
3303
3276
3539
OOD Test
3250
3277
3014
J SUMMARISATION EXPERIMENTS WITH OPT
In addition to the experiments in the main paper, we did several experiments using OPT (Zhang et al. 2022) in the summarisation task with a different choice of ID and OOD test sets.
J.1 DATASET SPLITTING
We create split versions of these datasets along several factors of variation in their inputs: length, sentiment, and subreddit. For each of these factors of variation, we create a train/test split where the train and test inputs drawn from different parts of the distribution for that factor. For the length split, the training set consists of data where the post is less than 245 words (the median number of words in the SFT training distribution)) for the sentiment split, we classifier (Loria 2013), and the training set consists of summaries with sentiment less than the median sentiment in the dataset; for the subreddit split, the training set consists of summaries from a specific subreddit, r/relationships. In all cases the test set is the complement of the training set in the full dataset, meaning that the trained models will be evaluated on imputs from different distribution than the one seen during training.
off-the-shet sentiment
In all cases, we apply the same splitting procedure to both the preference data and the input/output pairs, to ensure that the training and test sets are consistent across the different methods. Each of these splits was chosen to produce a roughly 50-50 split between the training and testing distributions. In the case of length and sentiment this is exact, and in the case of subreddit the r/relationships subreddit contains approximately 60% the data.
While we do not expect these splits to capture the full range of distribution shifts models may experience when deployed, using a range of splits will give us a more robust measure of how well the policies trained with different methods generalise under distribution shifts.
The dataset we use from (Stiennon et al. 2022 (filtered from (Nallapati et al., 2016 comes with train, validation and test splits, which have size 16722,, 6553 and 6447 respectively, and for the RM dataset they have size 92858, 33083, 50718 respectively. The SFT and RLHF splits the same apart from the RLHF dataset does not require the summaries (outputs), just the posts (inputs). To create the dataset splits used for the OOD generalisation experiments in Section 6. we split each of the train, validation and test sets into an in-distribution (ID) and
throughout work. For the SFT dataset these splits
(OOD) train, validation and test set. We then train on the ID train set, do model selection using the ID validation set, and evaluate on the ID and OOD test sets to measure the in-distribution and cattof-distrrbution performance.
For the sentiment split, we first measure the sentiment of each post using an off-the-shelf sentiment classifier (Loria 2013). For a given subset of the dataset (i.e. train, validation, test), the ID version of that subset the set of all inputs with posts whose sentiment is lower than the median sentiment, and the OOD version is the complement of that set. For the length split, we take the same approach using the length (in words) of the post, and the ID version the subset is the set with posts of length less than the median length. For the relationships split, take the ID version of the subset to be all posts in the r/relationships subreddit, and the OOD version to be the complement.
We apply this same splitting procedure to both the SFT and RM training datasets. Table and Table 7 show the size of the training, validation, testing and OOD testing sets for the RLHF and SFT. and RM training, respectively. For the results in this work randomly sample from the test and OOD test sets for those metrics, and we randomly sample from the validation set (which is in in-distribution) for calculating metrics used for model selection.
26

---

Published as a conference paper at ICLR 2024
Table 7: Size of the train, validation, test and OOD test datasets for each split for the reward models.
length
sentiment
relationships
Train
45395
46411
52346
Validation
16513
16529
17687
Test
25539
25353
27492
OOD Test
25180
25366
23227
SFT Dataset Post Length Distribution
RM Dataset Post Length Distribution
3000
Length Split
2500
Length Split
ID
ID
2500
OOD
OOD
2000
2000
Count
Count
1500
1500
1000
1000
500
500
0
0
100
200
300
400
100
200
400
length
length
Figure The distribution post lengths across SFT and RM datasets. ID in-distribation version of the dataset, and OOD is the out-of-distribution version.
To understand the distribution shifts these splits entail, show density plots for post length and sentiment across the full SFT and RM dataset in Fig. I3d Fig. [4| and show the number of posts in each subreddit in Fig. 13
J.2 HYPERPARAMETERS
We use the same hyperparameters in LLLMMa training (see Appendix E.4 but sweep over learning rates for each model size. We detail the learning rates swept for each model size and the chosen learning rate, for SFT, RM and RLHF training in Table M Table g and Table I0 respectively. In general for SFT and RLHF matching prior work (Stiennon et al. 2022). For the RLHF training with the largest two model sizes, due to the large amount of compute required to run multiple training runs, for several model size and dataset shift combinations we chose a single learning rate based on what we thought would give the best results the time started training. For the combinations where did vary the learning rate we did not see much variation in performance on the metrics we measured, so we do not expect these choices affect the results.
did not much variance with seeds, but did in RM training,
Table 8: Learning rates for different model sizes and dataset splits for SFT models. Underlined learning rate is the chosen one. ke means 10-
Dataset
125m
350m
1.3b
2.7h
6.7h
relationships
1e" 3e 1.15e
lc 3e 1.B6-
le 3. 1
1.66
le-1 10.3-- ..5e -3
le 1e
3. 1.5e
length
1e 1. 1.5e
le 1e_ 1.5e
le 1.3 1.5e
1e 3e
1.5e
1e 4.3e 4. - 1.56
sentiment
1e 1-3 ....e
le 3e 1.5e
le A. 1.5e
le 1e- -"
11.5e ""..5e
le Se -_1.e
27

---

Published as a conference paper at ICLR 2024
SFT Dataset Post Sentiment Distribution
RM Dataset Post Sentiment Distribution
Sentiment Split
Sentiment Split
2500
ID
ID
OOD
2000
OOD
2000
1500
Count
1500
Count
1000
500
500
0
0
-0.4
-0.2
0.0
0.2
0.4
-044
-0.2
0.0
0.2
0.4
sentiment
sentiment
The distribution of post sentiment the full SFT and RM datasets. IDD is the in-distribution version of dataset, OOD the out-of-distribution version.
Figure 14:
SFT Dataset Subreddic Distribution
Dataset Subreddit Distribution
Subreddit Spit
Subreddi Splt
50000
000
00D
50000
40000
40000
Dount
Gount
30000
30000
20000
20000
10000
affmor
mooitii frave
pets
boggre ning
callege
GotModhsstd
books
resstioni pa
Ackitudii
relatiorhil advice dating sdvio
personall lins
Advice
argeladviioe
affincchsst
salf
jobs
ning
Dogtra
surning
needudiic e
cats
GetModiv
colleg hais
books
Coolding
Subreddit
Subreddit
Figure number of posts in each subreddit across full SFT and RM datasets, ID is in-distribution version of the dataset, and OOD is the out-of-distributte version.
Table 9: Learning rates for different model sizes and dataset splits for reward models. Underlined learning rate is the chosen one. be means i 10
Daraset
125m
350m
6.7b
relationships
5e 1.3e Se -use
5e 54 1.5e
Se 0 1.5e 5.5e
5e 9 1.5e 0.5e
5e 5 1.5e 0.5e
length
Se 1.5e -9.5e
5e
Se-11.50 1,5e
Semb
Ge 1.5e 5e
Se 1.5e 5.5 5e-t
5e 3.55e 5.
sentiment
Se 1.5e 50,56
Se 1.5e
Se 1.5e 5,5m
Se 11.5e
Se 9.15e 0,5e
Table Learning rates for different model sizes and dataset splits for RLHF models, Underlined learning rate the chosen one. ke means & x 10"".
Datasct
125m
350m
1.3b
2.7b
6,7b
4 3. 1e-6
le 1.-4.3 5 5
16-5.ge
9.1e
5
1e
length
Se "le
le 4. 5
3.
0
relationships
Se "e
lc 4.3 "ee"
Se
5c
Se
1.7.
Se
Be
",1.7e
28

---

Published as a conference paper at ICLR 2024
Distribution reward
Out of Distribution reward
Generalisation gap for reward
0.08
0.5
0.5
0.06
SFT
RLHF
0.0
0.0
0.04
BoN
-0.5
0.02
125m
350m
1.3b 2.7b
6.7b
125m 350m
1.3b 2.7b
6.7b
125m
350m
1.3b 2.7b
6.7b
Model size
Model size
Model size
Figure 16: Proxy RM Score for SFT, BoN and RL models, averaged over dataset splits. for both in-distribution and out-o--distriuution performance, and the generalisation gap. Arrows indicate whether higher or lower scores are better.
In Distribution reward
Out of Distribution reward f
Generalisation gap for reward 4
0.4
0.0
0.0
length
relationships
sentiment
-0.5
-.5
Average
125m
1.3b Model size
6.70
125m
350m
1.36 2.76
125m
350m
1.3b
2.76
6.7b
Model size
Model size
Figure 17: Proxy RM Score for SFT models for each dataset split, both in-distribution and out-of distriiiitiittionn performance, and the generalisation gap. Arrows 1, indicate whether higher or lower scores are hetter.
J.3 GENERALISATION EVALUATION
For evaluation these experiments, we train an RM as described in Appendix E..1 dataset of summaries and preferences without splitting and the 6.7 billion parameter OPT model. We then use this proxy RM to evaluate the performance SFT, BoN and RLHF models. As this reward model is trained with a different random seed and different data distribution, it serves held-out automated evaluation of models, and a good proxy for human preferences.
uuing the full
We first discuss the results from the OPT version of the generalisation experiments described in Section 5.I Fig. [6] shows the proxy RM score for SFT, BoN and RLHF models, averaged over dataset sizes. The important result here is that BoN generalises better than RLHF, which generalises better than SFT.
We that middling model sizes, RLHF outperforms BoN, but BoN scales better than RLHF, eventually outperforming it for models with more than 2.7b parameters. SFT comes out worst in this comparison, both scaling worse than BoN and the same RLHF, and having worse absolute performance and generalisation. We see that BoN sampling does not see diminishing returns as model size increases, implying will continue be a useful yet simple technique. This experiment highlights the importance of training a reward model on human feedback and using it to select the best outputs at test time, potentially after fine-tuning the model.
Supervised Fine- Tine-Tuning. Fig. 17 shows the proxy RM score for the SFT models. Performance increases as model size increases, and in general performance drops OOD, which is unsurprising. There is a slight downward trend in the average generalisation gap across dataset splits model size increases, implying that larger models fine-tuned with SFT generalise better (in terms of automated metrics). The relationships and sentiment splits produce negligible generalisation gap for the proxy RM score while the length split is more difficult.
Best-of-N. Fig. 18 shows the proxy reward score for the BoN models. Here we almost linear increase in performance as the model size increases. Given that the RM scores for smaller models were below chance, the fact the performance requires explanation. We hypothesise that the smooth increase here comes from two factors: the improvement in the SFT model being sampled from, and the improvement from the
smooth
for smaller models BoN results improve
29

---

Published as a conference paper at ICLR 2024
In Distribution reward
Out of Distribution reward
Generalisation gap for reward
1.0
1.0
0.5
0.5
length
relationships
0.0
0.0
sentiment
Average
0.5
0.5
125m
125m
350m
2.7b
125m
350m
Model size
Madel size
Model size
Figure 18: Proxy RM Score for BoN models for each dataset split, both in-distribution and out-of-distriuutio performance, and the generalisation gap. Arrows 1, indicate whether higher or lower scores are better.
Distribution reward improvement 1
Out Distribution reward improvement
Generalisation gap for reward Improvement
0.75
0.75
0.1
0.50
0.50
length
relatiorships
0.0
sentiment Average
0.00
0.00
-0.1
0.25
0.25
125m
1.3b
350m
125m
350m
6.7b
Model size
Model size
Model size
Figure 19: Proxy RM Score improvement using BoN on top of SFT models for each dataset split, both in- distribution and out-of-distribution performance, and the generalisation gap. Arrows f, indicate whether higher lower scores better. This plot is highlights the improvement from Fig. Fig.
RM. At smaller model sizes increasing the number of parameters leads to improvements SFT performance but not RM performance, while at larger model sizes increasing the number of parameters leads to improvements both SFT and RM performance, but both to a lesser extent.
Fig. 19 shows the improvement of BoN over SFT. We can see that BoN only starts improving SFT as model size passes 350 million parameters, and the improvement grows model size grows. This implies that
increase model size BoN is likely to become to SFT. Further, we note that BoN uniformly improves the generalisation across model sizes (as shown by the black dashed line), implying that scaling models, even using SFT+BoN, will still result in non-zero generalisation gap.
performant choice compared
Reinforcement Learning from Human Feedback. Fig, 20 shows the proxy RM score for the RLHF models. Again, we see that increasing model size improves performance. Here we see a clearer trend of increasing model sizes reducing generalisation gap this implies that RL models larger, they are likely to generalise better, Given the difference between this trend and the generalisation gap trend for SFT models Fig. 17 this partially justifies why RLHF is used in fine-tuning LLMs at a very large scale (Glaese et al. 2022 OpenAl 2023 Ouyang el al. 2022): RLHF produces better-gent seater-gnerralsing models at larger model sizes than SFT.
make
J.4 DIVERSITY EVALUATION
Table 11 shows the per-input diversity scores (Eq. (2) for both RLHF and SFT models. The RLHF models have much lower diversity than SFT models according to all three metries. While RLHF leads to better-gemerallsingg policies, those policies generate much less diverse outputs. We also see that diversity does not seem to change much with model size, apart from a slight downward trend for diversity in SFT models model size increases.
shows the across-inpus input diversity scores (Eq. Here the corpus of text over which the diversity is measured is single input sampled from the model for range of outputs. Even if a model produces less diverse outputs for a single input, could still produce different inputs for different outputs. This is the case for the EAD score, which is proxy for diverse vocabulary and syntax, as
Table 12
30

---

Published as a conference paper at ICLR 2024
In Distribution reward 1
Out Distribution reward
Generalisation gap reward
0.2
length
relationships
-1
-1
0.0
sentiment
Average
-2
-2
350m
350m
Model size
Madel size
Model size
Figure 20: Proxy RM Score for RL models for each dataset split, both in distribution and out-of of-distributton performance, and the generalisation gap. Arrows 1, indicate whether higher or lower scores are better.
Table 11: Per-input diversity scores for both RLHF and SFT models averaged over dataset splits. For these
scores the outputs used to calculate the diversity are sample of outputs from the model for single input. These per-input scores are then averaged, as Eq. (2). Bolded results are better scores for each model size.
Model Size
125m
350m
1.3b
2.7b
6.7b
Model Type
RLHF
SFT
RLHF
SFT
RLHF
SFT
RLHF
SFT
RLHF
SFT
EAD
0.15
0.81
0.15
0.8
0.13
0.81
0.16
0.8
0.07
0.79
Sent BERT
0.15
0.5
0.12
0.46
0.15
0.48
0.13
0.45
0.06
0.45
NLI
-1.08
0.26
-0.96
0.2
-1.0
0.12
-1.07
0.09
-1.54
0.06
Average
-0.26
0.52
-0.23
0.49
-0.24
0.47
-0.26
0.45
-0.47
0.44
Table Across-input diversity scores for both RLHF and SFT models averaged over dataset splits. For these scores the outputs used to calculate the diversity are set of single outputs from a range of inputs, as in Eq. 3 Bolded results are better scores for each model size.
Model Size
125m
350m
1.3b
2.7b
6.7b
Model Type
RLHF
SFT
BoN
RLHF
SFT
BoN
RLHF
SFT
BoN
RLHF
SFT
BoN
RLHF
SFT
BoN
EAD
0.69
0.86
0.85
0.87
0.87
0.86
0.86
0.87
0.86
0.86
0.87
0.86
0.87
0.87
0.86
Sent BERT
0.63
0.73
0.72
0.71
0.73
0.71
0.68
0.73
0.71
0.7
0.74
0.71
0.73
0.73
0.71
NLI
0.05
0.35
0.32
0.2
0.38
0.3
0.25
0.36
0.27
0.3
0.28
0.27
0.32
0.32
0.3
Average
0.46
0.64
0.63
0.59
0.66
0.62
0.6
0.65
0.61
0.62
0.63
0.62
0.64
0.64
0.62
Table 13: Across-input diversity scores for BoN models averaged over dataset splits. For these scores the outputs used calculate the diversity of single outputs from range of inputs, Eq. (3). This is equivalent Table the main paper for BoN policies. See response reviewer hdr6 for more details.
Model Size
125m
350m
1.3b
2.7b
6.7b
EAD
0.85
0.86
0.86
0.86
0.86
Sent BERT
0.72
0.71
0.71
0.71
0.71
NLI
0.32
0.3
0.27
0.27
0.3
Average
0.63
0.62
0.61
0.62
0.62
31

---

Published as a conference paper at ICLR 2024
Table 14: SFT Model ROUGEI and reward model accuracy for the 3 smallest OPT model sizes, comparing freezing 80% of layers vs no layer freezing. Freezing generally results less performance, as expected.
Model Size
SFT Model Rougel
RM Accuracy
80% Frozen
0% Frozen
80% Frozen
0% Frozen
125m
0.217
0.221
0.482
0.496
350m
0.2241
0.2233
0.498
0.508
1.3b
0.221
0.2347
0.538
0.559
In Distribution Reward
Out Of Distribution Reward
Generalisation Gap
0.10
0.0
0.0
0.06
Dest of
-0.5
-0.5
0.04
32
0.00
1.5
125m
350m
1.3b
2.76
6.7b
125m
350m
1.36
2.7b
6.7b
125m
350m
1.30
2.7b
6.7b
Model
Model
Model
Figure 21: Proxy RM score for BoN sampling with varying N. All metrics are averaged over the 3 dataset splits.
for models with more than 125 million parameters both SFT and RLHF produce policies with very similar scores.
However, for the Sent BERT score, which is proxy for diverse content and semantics, RLHF models consistently less diverse than SFT models. This implies that RLHF produces models that have tendency to generate outputs about certain topics or content regardless of the input. For the NLI score, which is proxy for logical diversity, we that RLHF model size increases the score increases, eventually reaching the diversity of SFT models. Low NLI score for smaller models implies they have tendency to make logically consistent claims their outputs on top of producing outputs about certain topics of content.
J.5 MODEL FREEZING EXPERIMENTS
We perform a small experiment to evaluate the effects of freezing the first 80% of layers during fine-tuning. The results are shown in Table and show that while performance drops for the models evaluated in the experiment, the drop is not catastrophic, justifying the use of model freezing.
J.5.1 BON PERFORMANCE FOR DIFFERENT N
In the previous OPT results we use N 64 samples in best of N sampling. Here we show proxy RM scores for N 2, 8, 16, 32, show how choices of N trade off against performance. Fig. 21 shows proxy RM score in-distribution, cot-of-distribution and the generalisation gap for these different choices of N. We see that increasing N does lead to improved performance. At lower model sizes leads to a larger generalisation gap, but this does not hold for larger model sizes.
K RLHF AND RM TRAINING CURVES
Here we present training curves for PPO and reward model training, for the summarisation task, for the models used in the main paper. In Fig. 22 we show the reward model validation accuracy throughout training. This is 1 epoch of training. In Fig. 23 we show the KL divergence and reward model score throughout training. PPO training has converged by approximately 250 PPO training steps, and terminated training early to save compute.
32

---

Published as a conference paper at ICLR 2024
Reward Model Validation Accuracy
Reward Model Validation Accuracy during Training
0.75
0.70
0.65
0.60
0.55
0.50
200
400
600
800
1000
1200
1400
Training steps
Figure 22: Reward model validation accuracy during training for the summarisation task.
KL Divergence of PPO Run during Training
Mean Reward of PPO Run during Training
1.0
15
KL Divergence
Mean Reward
0.5
10
0.0
5
-0.5
-1.0
200
200
600
Training steps
Training steps
Figure 23: PPO KL divergence and reward model score curves for the summarisation task.
33

---

Published as a conference paper at ICLR 2024
X
30
20
20
GPT.4 PVT.4P4
0.4
0.5
Per-input EAD
0.6
0.7
0.8
Per-input Sent BERT
0.3
0.4
0.899
0.900
Per-input
0.901
0.902
BoN
SFT
X
x
X ID (TLDB)
OOD (CNN/DailyMai)
40
20
0.85
0.86
0.67
0.75
0.76
0.77
0.896
0.898
Across-Input Sent BERT
Across-input NLI
Figure 24: We plot the diversity vs gpt4 win rate trade-off for the summarisation task, across in-distribution and out-of-distrihution winrates and per- and across-input diversity metrics.
L GENERALISATION VS DIVERSITY TRADE-OFF PLOTS
Fig. 24 shows the tradeoff between diversity and win rate the summarisation task, across the three policy types investigate. This reinforces the inherent tradeoff between generalisation and diversity present in existing language model fine-tuning techniques.
34


Reviews:
[{"review_id": "ixVwVTMf6f", "reviewer": "Reviewer_De3y", "summary": "This paper empirically investigates the difference in generalization and generation diversity for LLMs trained with supervised learning and reinforcement learning for text summarization and instruction following. Moreover, they evaluate best of N (BoN), a very strong method for text summarization, as an additional method to test generalization of language models. They ultimately find evidence for RLHF improving generalization over supervised learning but at the cost of generation diversity.", "questions": "Please refer the questions posed in the weaknesses section.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- Their thorough investigation of RLHF vs SFT generation quality is very valuable. This work helps improve our understanding of why RLHF policies have empirically seemed to perform well in practice with users where more OOD data is likely encountered. \n- The paper is very clearly presented and investigates two popular settings for RLHF finetuning.", "weaknesses": "Minor Comments:\nFor summarization, it appears that pretrained models already perform very well for CNN daily mail. Would the same diversity, generalization, performance relationships be seen when evaluating OOD performance on a different summarization dataset where Llama2 7B does not perform as well? Or perhaps more simply, when trained on CNN as in-distribution, how is OOD performance to the harder TL;DR task?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "bejX7iBCG0", "reviewer": "Reviewer_6G1j", "summary": "This paper aims to study the effects of RLHF for fine-tuning LLMs, focusing on out-of-distribution generalization and output diversity metrics. Through empirical experiments, this paper finds that RLHF can outperform SFT in terms of out-of-distribution generalization but at the cost of a decrease in output diversity. Such observations may help in better applying RLHF or SFT in specific applications.", "questions": "The major concerns stem from the fact that the empirical evaluation heavily relies on the training quality of each algorithm, and the reviewer is uncertain about whether RLHF is trained to a high standard.\n\n**Question 1**: Do empirical conclusions heavily depend on the training status of the reward model and PPO? The reviewer observed that this paper freezes some layers of LLMs when using RLHF, which may limit the representation capacity. Thus, the reviewer questions whether the training of RLHF is of good quality. Could this paper provide the evaluation accuracy of the reward model and training curves of PPO?\n\n**Question 2**: Why not use entropy as a metric of diversity (although existing evaluation methods are acceptable)?\n\n**Discussion**: This paper mentions that \"Future work should investigate why RLHF reduces the output diversity so much,\" and the reviewer would like to point out some observations: the optimal policy by RL algorithms is deterministic (i.e., less diversity), if there is no tie in the reward value, there is no KL penalty, and the optimization is done perfectly. When there is a KL penalty, a recent paper shows that this corresponds to \"soft Q-learning\" [2]. In that case, the reward model is optimized perfectly. Although the algorithm in [2] is not applicable to the true RLHF setting where we only have out-of-distribution prompts and no preference labels, the viewpoint in [2] is insightful for in-distribution training.\n\n[2] Rafailov, Rafael, et al. \"Direct preference optimization: Your language model is secretly a reward model.\" *arXiv preprint arXiv:2305.18290* (2023).", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "- This paper conducted extensive experiments to elucidate why RLHF behaves differently from SFT. The experimental setup is sound, and the empirical results may inspire future progress in this direction.\n\n- The paper is well-writen and easy to follow in general.", "weaknesses": "- Missing Related Work\n\nIn fact, there is a theoretical comparison of RLHF and SFT-style methods in the framework of imitation learning [1]. Indeed, LLMs are imitating human speech. In that framework, RLHF corresponds to adversarial imitation learning (AIL) methods, and SFT corresponds to behavioral cloning (BC). To the best knowledge of the reviewer, that theoretical study reveals that AIL (RLHF) methods can have better out-of-distribution generalization performance than BC (SFT) because AIL methods optimize their policy on out-of-distribution states (prompts) and rigorously prove this phenomenon. I believe this related work is insightful for studying the advantages of RLHF over SFT, and this paper should be mentioned in the related work.\n\n[1] Xu, Tian, et al. \"On the generalization of adversarial imitation learning and beyond.\" *arXiv preprint arXiv:2106.10424* (2021).\n\n\n- Typos and Writing Suggestions\n\n1. There are two minus symbols in Equation (1).\n2. It seems unusual to draw a conclusion in Section 6.3 while presenting empirical evidence in Appendix I.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "BnlOb7FVbX", "reviewer": "Reviewer_GaEc", "summary": "This paper studies the effects of RLHF on generalization and diversity. Specifically, the authors look at the three stages in RLHF: supervised fine-tuning, reward modeling, and reinforcement learning. They conduct experiments that show that RLHF generalizes better than SFT to new inputs, but reduces output diversity.", "questions": "The paper makes some unspecified claim that would need justification or further explanation. For example, on page 2, summary of contributions, the third bullet point: \"...implying that such models tend to produce text of a specific style regardless of the input\". How does one arrive at the \"style\" conclusion?\n\nWhy is there no error bars in Figure 5? Could you plot error bars over different choices of outputs from among the set of outputs to the same input?\n\nAre the OOD datasets considered OOD for fine-tuning, or both fine-tuning and pretraining? The CNN/DailyMail dataset is most probably included in the pretraining dataset of Llama-7B.", "limitations": "", "rating": 3, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 1, "strengths": "The writing of the paper is clear and easy to follow. The paper studies three different aspects of performance, including in-distribution generalization, out-of-distribution generalization, and diversity. As far as I know, this covers a more comprehensive study on RLHF -ine-tuned model behavior than most observational studies in the literature.", "weaknesses": "This paper does not offer any new insight or novel methods compared to existing work in the literature, and no new methods have been proposed. First of all, the generalization capabilities offered by RLHF has been widely observed in state-of-the-art models, with clear comparisons and case studies of output from pretrained, instruction fine-tuned, and RLHF fine-tuned models (see, for example, the Llama-2 paper). The mode collapse phenomenon from RLHF has also been widely observed and measured. Maybe the only novelty this paper offers is evaluation on an array of sentence-level diversity metrics. Furthermore, the claims made in the paper are not very well-justified by experiment results, and some experiment details are lacking. Only two sets of experiments, namely summarization and instruction following, are conducted on one single model (Llama-7B), yet the paper makes a general claim about the effects of RLHF. More experiments on different models at potentially different scales could be helpful, but still, the contribution seems to be incremental. \n\nMy main concern is the contribution. Some additional questions are listed below for clarification, but unless the authors could justify their contribution through substantial experiments (on different models at different scales) and more in-depth analysis, I still lean towards rejection.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)