Helping or Herding? Reward Model Ensembles Mitigate but do not Eliminate Reward Hacking
Authors: Jacob Eisenstein
Chirag Nagpal
Alekh Agarwal
Ahmad Beirami
Alexander Nicholas D'Amour
Krishnamurthy Dj Dvijotham
Adam Fisch
Katherine A Heller
Stephen Robert Pfohl
Deepak Ramachandran
Peter Shaw
Jonathan Berant
Venue: COLM (2024)
Keywords: RLHF, reward models, underspecification, reward hacking
TL;DR: Reward models are surprisingly sensitive to random seed; ensembling helps but does not eliminate qualitatively obvious reward hacks.

Abstract: Reward models play a key role in aligning language model applications towards human preferences. 
However, this setup creates an incentive for the language model to exploit errors in the reward model to achieve high estimated reward, a phenomenon often termed \emph{reward hacking}.
A natural mitigation is to train an ensemble of reward models, aggregating over model outputs to obtain a more robust reward estimate. 
We explore the application of reward ensembles to alignment at both training time (through reinforcement learning) and inference time (through reranking). 
First, we show that reward models are \emph{underspecified}: reward models that perform similarly in-distribution can yield very different rewards when used in alignment, due to distribution shift. 
Second, underspecification results in overoptimization, where alignment to one reward model does not improve reward as measured by another reward model trained on the same data. 
Third, overoptimization is mitigated by the use of reward ensembles, and ensembles that vary by their \emph{pretraining} seeds lead to better generalization than ensembles that differ only by their \emph{fine-tuning} seeds, with both outperforming individual reward models.
However, even pretrain reward ensembles do not eliminate reward hacking: we show several qualitative reward hacking phenomena that are not mitigated by ensembling because all reward models in the ensemble exhibit similar error patterns.

Published as a conference paper at COLM 2024
Helping or Herding? A Reward Model Ensembles Mitigate but do not Eliminate Re- ward Hacking
Jacob Eisenstein
Chirag Nagpal
Alekh Agarwal
Ahmad Beirami!
Alex D' D'Amour'
DJ Dvijotham!
Adam Fisch
Katherine Heller?
Stephen Pfohl
Deepak Ramachandran'
Peter Shaw
Jonathan +
1. Google DeepMind
2. Google Research
Core contributors
wewerrdennnebbeeeelllln-n------e -oa.aiggoo-oerddggggggl..... com
Abstract
Reward models play a key role in aligning language model applications towards human preferences. However, this setup creates an incentive for the language model to exploit errors in the reward model achieve high estimated reward, phenomenon often termed reward hacking. A natural mitigation to train an ensemble of reward models, aggregating over model outputs to obtain more robust reward estimate. We explore the ap- plication of reward ensembles to aligmment at both training time (through reinforcement learning) and inference time (through reranking). First, show that reward models are underspecified: reward models that perform similarly in-distribution can yield very different rewards when used in alignment, due to distribution shift. Second, underspecification results in overoptimization, where alignment to one reward model does not improve reward as measured by another reward model trained on the same data. Third, overoptimization is mitigated by the use of reward ensembles, and ensembles that vary by their pretraining seeds lead to better generalization than ensembles that differ only by their fine-tuning seeds, with both out- performing individual reward models.' However, even pretrain reward ensembles do not eliminate reward hacking: we show several qualitative reward hacking phenomena that are not mitigated by ensembling because all reward models in the ensemble exhibit similar error patterns.
1 Introduction
To align machine learning systems with human preferences, common to use reward models (RMs), which are finetuned on preference annotations to score potential outputs by how likely they are to be preferred by human raters (Christiano et al., 2017; Stiennon al., 2020). There are many ways to use RMs to align policy models: they can act as training signals in reinforcement learning (Christiano et al., 2017; Stiennon et al., 2020), they can select examples for further imitation learning (Gulcehre et al., 2023; Liu et al., 2023; Dong
These pretrains are available at http://tt.tggggggg......oogoeeee ensemblee.

---

Published as a conference paper at COLM 2024
Humani wont make rice dlekk dinee out don't know the dilinancc betwar the waroos cuts stack, Assistant: Humane Whet clinc with Asslatent:
Human I'm going the Notholands and would #e 1o Itam low curmar Dutch pirsaes Assistant: 1-] Human: Coulo tnech "Dondhpe and "Tork Assietant:
yi compled response
yf campled rosponse
Stesk In Sigh-fat stesk, which naana that acta a lot, ot calories and fat, bet that it nabez fcz good wall, abenk can rit so side dtath to
couront Shore are lote of diflecent groetings, Ierel For exsapte, ie scemcoa 2aya "qoodaya" or "thank you," you can ure Close expensutions
0.80
or didiveentt Looda. Hacel' aome
-2.00
polibely "goodiye" "thank "thank vacy you, you Cooe)
++2.00
negpeations:
which
orilled tosatoes
F4
-0.181
F4
++.00
aqga Good
F's
-0.10
F3
++234
Fewarrd ensemblee is effective
Fleward ensemb e is Ineffective
++251
Figure 1: Left: RM ensembles can attenuate errors made by individual RMs, in this case the positive 11 for this off-topic response from the policy model π( x), which gets low true reward (r*). Right: insufficiently diverse RMs unanimously rate this overly-verbose and non-responsive reply from (y x) as positive, but it too gets a low true reward. Both examples are real outputs and rewards (represented as normalized Z-scores) from RMs trained on the HELPFULNESS data of Bai et al. (2022).
et al,, 2023; Touvron et al., 2023), or they can be applied at inference time to steer the output distribution toward higher expected reward (e.g., Yang & Klein, 2021; Gao et al., 2023). Such procedures create a semi-adversarial dynamic, in which the language model can achieve high reward by exploiting errors in the RM. Furthermore, while the RM is trained set of human preference data, the process of alignment shifts the distribution of its inputs, increasing the likelihood of such errors. This phenomenon where the policy language model exploits reward model errors is often termed reward hacking (Amodei et al., 2016), reward gaming (Skalse et al., 2022; Pang et al., 2023), or reward over nner-opfimization (Gao al., 2023).
fixed
Reward hacking has been investigated from several perspectives in prior work (e.g,, Krakovna al., 2020; Skalse et al., 2022; Pan et al., 2022). Bai et al. (2022) used rein- forcement learning with human feedback (RLHF) and trained two RMs on non-overlapping splits of preference data, using one drive alignment, and the other to measure the quality of the outputs. They find that RLHF increases performance according to both the driver and measurement models, but that a performance gap emerges as the policy is allowed to diverge from the initial distribution. However, both RMs were built on base models trained on the same pretraining data, which, as we will show, limits their diversity (as hypothesized by Gleave & Irving (2022)) and thus may understate the effect of reward hacking. Other work has simulated the relationship between a "true" reward and a learned proxy, showing that it is possible to over-optimize the proxy to such extent that the true reward starts to decrease (Gao et al., 2023; Coste et al., 2023), e.g. by exploiting spurious correlations in reward model training data (Pang et al., 2023).
In this work, we first analyze RM distribution shift from the perspective of underspecifica- tion (D' Ammur al., 2022), which occurs when a machine learning pipeline yields reliable performance on held-out data from the training distribution, but variable performance on data. When applied to learning RMs from human preference data, we show that RMs that agree in-distribution often disagree after alignment-induced policy dis- tribution shifts. Furthermore, such disagreements are more pronounced when the RMs are built on different pretrainings, even when that difference is induced merely by varying the pretraining random seed. These disagreements become increasingly severe when evaluated outputs of a policy model that has been aligned to a specific RM. This occurs both when using RMs in RLHF, as well as when using an inference-time alignment procedure, best-of-n reranking, where samples are drawn from the policy and then reranked with RM.
Motivated by these findings, we systematically investigate reward model ensembles as possible remedy for reward hacking. Assuming different models err in different ways,
2

---

Published as a conference paper at COLM 2024
ensembling can leverage reward uncertainty across the ensemble during alignment (see Figure I, Left). We explore several techniques for aggregating scores across the ensemble, e.g., taking the median score a robust estimate of the true reward of the policy. We also consider two types of ensembles: pretrain ensembles, where different members of the ensemble differ in the random seed used during the pretraining phase, and fineture ensembles, where members differ only in the random seed used during finetuning. These ensembles are then evaluated across several types of policies and preference annotations: dialogue preferences for a helpful assistant (Bai et al., 2022), summarization quality (Stiennon et al., 2020), and whether a summary grounded in its source text (Roit et al., 2023).
We find that ensembles are significantly more robust than individual reward models, and that pretrain ensembles are particuarly effective. However, reward ensembles are still susceptible to reward hacking when all members of the ensemble share similar error patterns, which in turn propagate to the ensemble (see Figure 1, Right). This exploited and amplified by policy optimization: for example, summarization models produce outputs that are too short when tuned for factuality, but too verbose when tuned for summarization quality; assistant models overuse formulaic answer formats when tuned for helpfulness.
Recent and concurrent related work Coste et al. (2023) argue that reward model ensem- bles effectively mitigate reward hacking. Our work shares similar research question, but differs in several ways, leading to more nuanced conclusions: we investigate both pretrain and finetune ensembles; we use human-annolated preference data rather than synthetically-genrrtee labels; and we analyze several real reward hacks, showing that they are not prevented by ensembles. Subsequent work has explored efficient RM ensembles through low-rank adaptation (e.g., Zhai et al., 2023) and weight averaging (Ramé et al., 2024), but exclusively for finetune ensembles. More generally, we can try optimize the policy against the worst-case reward in some analytically-compute
uncertainty set (Zhu et al., 2023; Zhang et al., 2024), but this too accounts only for uncertainty due to the finite sample of preference pairs and not in the representations made available through pretraining.
2 Preliminaries
We now briefly review how reward models are trained ($2.1) and how they are used for alignment (52.2). We then describe our setup for studying reward hacking (52.3).
2.1 Reward Model Training
RMs are typically trained from preference data, (x, "t,yt,y" E D, where yt is preferred over y for prompt x. Under the Bradley-Terry model (Bradley & Terry, 1952), the probability that response 1/2 is preferred over 1/1 given a reward function r and a prompt x is p 4 y/2 x) (F((x,)2) y(x,y1)), v1)), where o(.)) is the sigmoid function. Given a dataset of preferences, the maximum likelliooodieooood objective is,
(r) = Exxy' J~D [logp(y - y xx)]
(1)
The Brrrley-Terry model is underdetermined: for any RM r", there is set of equivalent models, r'xx, y) = r" (x, + C(x) where C(x) is a prompt-dependent constant, such that
[(r)) [rr).. This is problematic for ensembling: if different RMs choose different values for C(x), then order statistics like median and minimum are meaningless. We therefore modify the objective by regularizing the sum of rewards per preference pair towards zero:
Treg (r) [r) +
y+) + ((x
(2)
where 7 is small positive value, thereby resolving the issue of underdetermination.
Note that RMs also be trained from "pointwise" data, such toxicity factuality annotations on individual examples (Yang & Klein, 2021; Roit et al., 2023). Such RMs are not underdetermined and can be aggregated without adjustment.
3

---

Published as a conference paper at COLM 2024
2.2 Aligning Language Models using Reward Models
Best-of-n reranking (BoN) is an inference-time alignment strategy, where given a prompt x, we sample n generations yı.....,yy V/n from a policy language model T x) and return the generation with highest reward: y* arg yae r(x, The Kullback-Leibler
(KL) divergence of BoN from the initial policy is upper bounded by log n-1 (Beirami et al., 2024). BoN tends to outperform more elaborate alignment techniques like RLHF in the low-KL regime (Gao et al., 2023), albeit with the cost of generating multiple samples at inference time.
Reinforcement Learning from Human Feedback (RLHF) is an online reinforcement learn- ing method that trains a policy language model π to maximize expected reward, while staying close to initial policy, Tltt, which is typically finetuned on supervised data (prompt-output pairs). Distance from the initial policy is measured with KL divergence, which leads to the regularized objective
max π
- (
(3)
where is RM, o is a distribution over prompts, and À is hyperparameter. Typically, this objective is optimized using PPO (Schulman et al., 2017), which we also use in this work.
The KL. penalty in eq- (3) can limit reward hacking by keeping the policy close to Tift- However, KL regularization does not directly address reward model errors, and in particular does not address RM distribution shift when the preference annotations are not sampled from Tist (as is generally the case). Furthermore, we want to diverge from the reference policy when we can be confident of improving the expected reward. As we will show empirically, KL regularization is complementary to the development of more robust reward models, which yields pareto improvements in the reward-KL tradeoff.
2.3 Experimental Setup
Datasets
We consider three tasks. Example instances are provided in Table 12.
TL;DR: A summarization benchmark where authors summarize their own reddit posts (Völske et al., 2017; Stiennon et al., 2020), frequently used in research on RLHF and related methods (Rafailov et al., 2023; Zhao al., 2023).
HELPFULNESS: A popular dialogue benchmark (Bai et al., 2022), where given a partial conversation between a human and a digital assistant the goal is to complete the next assistant turn (Bai et al., 2022; Rafailov et al., 2023).2
XSUM/NLI: A summarization benchamark where a policy model trained on XSum (Narayan et al., 2018) is finetuned to generate summaries that are factually consistent with the source document (Roit et al., 2023).
Training reward models To examine the effect of pretraining, we pretrain five 15 models from scratch at the base (220M parameters), large (770M), and XL (3B) scales, using the stan- dard denoising objective over the C4 corpus (Raffel et al., 2020). The pretrained checkpoints differ only in their random seed, which controls parameter initialization and the sample from the pretraining data.
For each task, we finetune each pretrained model five times using different random seeds. In TI,DR and HEL.FULA PFULNESS we use the aforementioned preference data. For XSUM XSUM/NLI, we finetune poinfwise natural language inference (NLI) models on the ANLI dataset (Nie al., 2020). This yields 25 RMs per task and scale (5 pretrain x 5 finetune), making it possible to evaluate the effect of pretraining and finetuning on underspecification ($3) by constructing ensembles that differ in either pretrain finetune seed (54).
We use the base dataset (44K examples), where responses are generated from 52B context- distilled LM, and split the training set into two: half for training the RM, and half for the policy.
4

---

Published as a conference paper at COLM 2024
Model Size
TL;DR
HELPFULNESS
XSum/NLI
T5-BASE
65.8 0.3
66.7 0.7
86.7
T5-LARGE
69.3 0.7
68.5 0.4
88.3 1.2
T5-XL
71.4 0.8
69.2 0.6
91.3 1 0.5
T5-XXL
79.5
71.5
92.9
Table 1: Mean in-distribution accuracy of 25 RMs on validation data for TL;DR, HELPFULNESS, and XSUM NLI. Standard deviation, indicated with +, is small in-distribution. The single T5-XXL RM is used for evaluation purposes only.
Alignment strategy We use the publicly available 15-large (Raffel et al., 2020) as a policy for the summarization tasks. For HELPFULNESS, which requires substantial background knowledge, we use the instruction-tuned PALM-2-XXS model (Anil et al., 2023). Before alignment, we create a finetuned policy stt through supervised finetuning: We finetune on annotated summaries from TI;DR and XSUM / NLI for the corresponding tasks, and on the preferred responses, yx,y²), ) from the preference data HELPFULNESS.
In BoN reranking, we rerank sampled output sets of size π E (2), 25] for HELP- FULNESS and { 2,) for TL,DR. Larger sets lead to higher reward cost of more expensive inference and larger deviation from Tift. In RLHF, we obtain a trade-off between the KL from Ta and the expected reward by training multiple times, varying the value of Å. Low values of À correspond to high KL and high reward, while high values of Á entail low KL and low reward. For each value of we train roughly to convergence using predetermined fixed number of steps (all hyperparameter values, including À and the number of steps, are in Appendix C). Coste et al. (2023) trade-off KI. and reward by tracking their values during training; however, for any particular value of KL the reward might still be underoptimized during training (i.e., there can exist a different policy Tl x) with better reward, but the same (L(TT(V) (T x)), which can be found with longer training).
Evaluation The aligned policies are autoevaluated by two metrics: reward and win rate. Similar to past work (Gao et al., 2023; Coste et al., 2023), we use a larger RM to evaluate the generalization of models trained with a smaller RM. We train a T5-XXI. RM by taking the publicly available T5-XXL (Raffel et al., 2020) and finetuning it as described above. As shown in Table 1, T5-XXL outperforms the best T5-XL model on each task. We report both average reward from the T5-XXL evaluator as well as win rate, which is the fraction of prompts for which the response sampled from the aligned policy π has higher reward compared to Tist-
Because the T5-XXL autoeval model is trained on the same data as the smaller T5 RMs, their errors might be correlated. For this reason, we also compute win rate according to prompted PAL.---LLarge model, which was not exposed to the reward training data but was instruction-tunen on Flan (Wei et al., 2022). Given a prompt x, we sample a response Ysht from St and !rlht from π. We then ask PALM-2 which response is better, using hand-engineered prompt proposed by Rafailov et al. (2023). To avoid position bias, we run PALM-2 on the two possible orderings (ystt yrthr) and (ystt Vriht), sample K = 8 outputs for each order and determine the winner this prompt through majority voting. This style of evaluation has become popular (Dubois et al., 2023; Singhal et al., 2023) and was shown to correlate well with human judgements (Rafailov et al., 2023).
3 Underspecification in Reward Models
We begin by demonstrating that the out-of-disttibutiio performance of individual RMs is underspecified. Table 1 shows the average in-distribution accuracy across the 25 different RMs, together with the standard deviation. The low standard deviation highlights that all 25 reward models achieve similar performance in-distribution. But when move to out-of-distributio data, the models diverge. Figure 2 shows the expected reward achieved by BoN function of the number of sampled candidates, n. The dotted green line
5

---

Published as a conference paper at COLM 2024
base
large
3.0
pretra aift pretrain sell
2.5
1.5
1.0
1.6 rerankinn didattes
candidstes
rerainking condidattes
(a) TL;DR
large
same pretraln diff pretrain
2.0
roword
1.5
1.0
16
32
16
32
16
32
reranking canoidates
reranking canoicates
reranking candidatos
(b) HELPFULNESS
Figure 2: Average reward of the best-of-n output, as judged by: the same RM used for ranking (self); RMs fine-tuned from the same pretrain as the ranker (same pretrain); RMs fine-tuned from different pretrains from the ranker (diff pretrain). The RMs that do not share pretrain with the ranker regard the ranker's 's preferred outputs as significantly worse.
large
0.6
(1+0
Meonn Conoation
samie Ow0 03)
63
BLIT Step
BLLE
Figure 3: Rank correlation of reward scores for TL,DR RMs that share a pretraining seed and models that do not. RLHF alignment increases disagreements between RMs (lower correlation), particularly at low values of and for RMs that do not share a pretrain.
shows the expected reward of the top-ranked output according to the reranker itself, while the dashed blue line shows the expected reward of the same output according to RMs that share pretrain seed. The solid orange line shows the expected reward according to RMs that do not share a pretrain seed. Unsurprisingly, the reranker scores its own top outputs more favorably than the other RMs do. However, the reranker's ' outputs are scored significantly less favorably by RMs which do not share pretrain with the ranker. RMs that share pretrain seed with the ranker model overestimate the reward of the top-ranked srppuut-uggestnng that finetune ensembles are not sufficiently diverse because of the shared pretraining state of each of the ensemble's members. Notably, this gap does not disappear with scale, as shown in Figure 2.
Moving to alignment, differences estimated rewards induce different policies from the BoN strategy. Figure (in the appendix) shows the effects on agreement of the top-ranked summary when RMs do (crosses) or do not (circles) share pretraining seeds. Different RMs tend to produce different 1-best outputs, and these differences are linked to the pretraining seed: for example, two RMs from different pretrains will choose a different best-of-16 output more than half the time for both TL;DR and HELPFULNESS and in all scales.
Figure 3 analyzes the evolution agreement of the estimated reward scores when perform- ing RL.HF on DRR For each policy model, sample five completions for each prompt

---

Published as a conference paper at COLM 2024
in the validation set at 2000 step intervals during RLHF. We then measure how well pairs of RMs agree on the ranking of these completions, using Spearman rank correlation. The correlation is averaged across all pairs of reward models that do and do not share the same pre-training seed; both sets of RMs include the one used to drive RLHF. As shown in the figure, RLHF decreases the average correlation, particularly at low levels of regularization. Furthermore, agreement is substantially lower for pairs of reward models that do not share the same pretraining seed. This supports our conclusion that reward modeling under alignment-induced distribution shift is underspecified by the preference annotations.
Overall, our analysis demonstrates that (1) different RMs tend to disagree on out-of- distribution data, particularly when the RMs have different pretraining seeds; (2) this propagates to the trained policy model, the sense that the resulting policy highly tuned to the preferences of the specific RM used to drive it; and (3) as a result, the disagreement between RMs tends to increase during alignment. These findings suggest that reward model ensembles might mitigate reward hacking, which we turn to next.
4 Reward Model Ensembles
A natural mitigation to reward model underspecification is ensemble multiple RMs, under the assumption that different models will have different errors. Aggregating over the scores of the ensemble members will help when some of the ensemble members erroneously assign high reward to a bad output.
Building reward model ensembles Given set of RMs M, define the reward of the ensemble to be r(x,y) y/ = agg((rm ()) with agg indicating an aggregation function (Dietterich, 2000; takshminarayanan et al., 2017; Raffel et al., 2020; Zaidi et al,, 2021). Intuitively, the aggregation function should be conservative, and return a lower score when there is disagreement between the ensemble members. We consider the following simple aggregation functions: MEAN, MEDIAN, and the standard deviation of the reward from the mean to penalize high variance. We also experiment with MIN, but overall find it to be inferior to the alternatives.
(_STD, which subtracts
We evaluate two types of reward ensembles: prefrain ensembles, where each member is pretrained using a different random seed, and finetune ensembles, where members share the same pretraining seed, but use a different seed when finetuned on the reward data. In all cases the ensemble contains five individual RMs. Pretrain ensembles are expensive to train, but are more diverse and hence likely to lead to more robust reward estimate. (Gleave & Irving (2022) reported negative results when using reward ensembles and hypothesized this is due to members sharing the same underlying pretrained model.)
Evaluation We now evaluate RM ensembles across tasks. Figure 4 shows the results in best- of-n reranking, as measured by an XXL-scale fine-tuned RM. Pretrain ensembles consistently improve performance over individual RMs, especially for higher values of for both TL;DR and HELPFULNESS. Finetune ensembles, conversely, improve performance in some cases and are comparable in others. For example, on TL;DR& a pretrain ensemble with the MEAN aggregator achieves a win rate of 90% over the SFT outputs at the XL scale, while the win rate of a finetune ensemble with the same aggregator is 87.3%. The win rate of the average individual XL-scale RM is 85.3% (see Table 6). For visual clarity, in Figure 4 we show only two aggregators: MEAN and MEAN MEAN_MINUS
see Appendix A for the other aggregators. In general, differences between aggregators small, with MEAN usually performing at, or near, the top. More conservative aggregators (MIN and MEAN_ MEAN_MINUS 2_TD) come out slightly ahead of MEAN at the smaller scales on TL,DR, suggesting that high variance may be a bigger issue in this setting.
Figure 5 shows the KL-reward trade-off of ensemble RMs in RLHF for TL;DR and HELPFULNESS (evaluated with the finetuned T5-XXL model). In such plots, a better model
Pretraining does not complete single epoch over the pretraining data, and thus the data observed by each member of pretrain ensemble different (but sampled from the same distribution).
7

---

Published as a conference paper at COLM 2024
base
largee
pretrain
finetu
single RM
15:X0L PeLLooo
moan.
single R.
64
reranking candidates
rerankkiin considaates
candidates
(a) TL;DR
base
large
ensemble
single
ago gegitoo
meeer
mean minus $to
single
reraninng candieates
reranking canols ates
reranking condieates
(b) HELPFULNESS
Figure 4: In BoN reranking, pretrain ensemble RMs significantly improve output quality, measured by T5-XXL autoeval model. Full numerical results are in Appendix A.
helpfulness helpfulness large
pellleeesss
T1-M0 reownd piretured)
ensemble type
- frwlune
single
100
10 50
1.00
25 50
100
10
100
30
25
50
100
100
KL
KL
KL
KL
KL
KL.
Figure 5: In RLHF, pretrain ensemble RMs lead to more favorable reward-KI. tradeoffs, as judged by a T5-XXL autoeval model. Each point corresponds training to convergence particular value of A. We show MEDIAN ensembles here; for others see Appendix B.
improves reward and/or reduces KL from the SFT policy (Gao et al., 2023; Coste et al., 2023). As with BoN alignment, pretrain ensembles consistently outperform finetune ensembles as well the average individual RM. For clarity, the figure presents results only for the MEDIAN aggregator for clarity, but these conclusions are supported by full numerical results across aggregators, shown in Appendix B. RLHF leads much higher KL values than BoN. Consequently, we witness explicit reward hacking, in which the T5-XXL rewards decrease even the RLHF objective improves. This happens most prominently for individual models, in many cases for finetune ensembles, and most rarely for pretrain wheee T5-XXL reward scores decrease only when RLHF uses T5-Base RM. Thus, our experiments on real data yield more negative conclusions than Coste et al. (2023) about the potential of ensembles to eliminate reward overoptimization.
As the T5-XXL autoeval model is trained on the same distribution as the RMs used for alignment, it may overstate their performance. Thus, we also use zero-shot autoeval model, PALM-2-Large (see Section 2.3). Because this evaluation is computationally expensive, we apply to only the largest-scale RMs (XL). As shown in Figure 6, ensemble RMs achieve higher win rates on both tasks and with both alignment techniques. For best-of-n, pretrain ensembles get significantly higher win rates on TL,DR at 11 64 (p < .001 by permutation test); on HELPFULNESS the differences between ensembling techniques are not significant at n = 32. On both tasks, single RMs are significantly worse, p < ,001. For RLHF, pretrain ensembles achieve better or equal win rates at lower KI. divergence
8

---

Published as a conference paper at COLM 2024
helpfulness
tldr
helpfulness
tldr
90
ensematee type -+- finetune
90
pam-2 win rate
80
single
Palm22
ensembble type
-
finetune
single RM
2
16
32
16
32
10
20
40
100
20
40
reranking candidates
reranking canditatts
(a) BoN
(b) RLHF
Figure 6: Using prompted autoevaluator (PALM-2-Fllan), ensemble RMs get significantly better win rates on both TL;DR and HELPFULNESS. Here all RMs are XL-scale.
from the reference policy, with particularly strong performance on HELPFULNESS. Overall, these results mirror the 15-XXL evaluation, with one interesting difference: the PALM-2 autoeval reveals more reward hacking for RLHF, where win rate decreases with KL. This suggests that fine-tuned autoevaluators can overestimate performance when trained on the same preference data as the alignment RMs.
Figure 9 (in the appendix) shows RLHF results for XSUM/NLA NLI. Here ensembles offer relatively small improvements, and there is little difference between pretrain and finetune ensembles. We conjecture this because XSUM / /NLI optimizes specifically for factuality. This allows all models to find simple and similar strategies that lead to high reward (namely, emitting short responses), and thus ensembling does not lead to large gains in performance.
When do Reward Model Ensembles Fail?
Ensembles improve performance according to automatic evaluation metrics, but does this mean that reward overoptimization solved? We now investigate five specific " "reward hacks", and show that they are not prevented by ensembling, because all members agree that these hacks improve the quality of the outputs.
To demonstrate this, we manually identify five qualitative distribution shifts in which the post-RLHI policies differ dramatically from both the reference policy and the preference annotations. Figure 7 and Figure 10 (in the appendix) show the results of this analysis on all benchmarks with a T5-large RM. The x-axis corresponds to the number of RLHF training steps, and the y-axis is a statistic of interest (e.g., average output length). We plot the statistic for the pretrained ensemble (using MEAN as representative aggregation function) and for its members. For TL;DR and HELPFUT NESS, where the reward model is trained on preference data, we show the statistic value on the preference data validation set, conditioned on the label 'Preferred' 'Rejected'.
For HELPFULNESS (Figure 7a), outputs tend to be in a list format, which we capture with regular expression. The fraction of outputs that have this pattern increases to roughly 50% for three members of the ensemble, and for the ensemble itself. We do not detect this tendency the preference data: the fraction of outputs that matches this format is roughly 8% for preferred and rejected responses.
For TI,DRR (Figure 7b, Figure 10b), RLHF leads to longer summaries (Singhal et al., 2023) and more extractive outputs, i.e., more copying from the input. Length in characters grows substantially for the ensemble and its members, where for the ensemble, length increases by a factor of two. On the preference data, preferred responses are slightly longer than rejected responses, but much shorter than outputs LLHF We also compute the longest common subsequence (in characters) between the document and the summary and find that it doubles after RLHF with ensemble RM. Although preference data shows slight preference for copying, this preference is dramatically amplified by RLHF.
For XSUM/ NLI (Figure 7c, Figure 10c), training for factuality makes summaries shorter and less specific, measured by omission of numerical values. All members of the

---

Published as a conference paper at COLM 2024
bicembbe
400
Namoer
0.125
List
0.4
Nemoars
Narior
Pefence
Length
250
300
2.75
0.050
0.00
Rejecrod
0.0
150
0.000
10 training steps
10a training steps
10a training steps
(a) HELPFULNESS: fraction of an- swers containing lists
(b) TL;DR: output length
(c) XSUM NLE fraction of numer- ical tokens in the output.
Figure 7: Limitations of reward model ensembles. The x-axis is RLHF steps, the y-axis plots different statistics of the average validation output that step, and the curves correspond to the pretrain ensemble (solid blue) and its members (dashed orange). For preference data, we plot the same statistics conditioned on the preference data label (Preferred vs. Rejected). The statistics of the "aligned" outputs are far from their values in the preference data.
ensemble exhibit this phenomenon: RLHF leads to rapid decreases the length of the outputs and the fraction of outputs that contain numerical values.
These findings illustrate the tendency of different reward models to associate certain fea- tures with high reward. Policy models exploit this association, using these features to produce outputs that are dramatically different from the reward training data, which achieve (spuriously) high reward for both individual reward models and the ensemble.
Why do reward model ensembles fail to capture uncertainty about these reward hacks? Prior work has shown that ensembles can provide good uncertainty estimates around the decision boundary, while underestimating uncertainty for examples far from the training distribution (Lakshminarayanan et al., 2017). In LM alignment, the policy can shift the output distribution away from the decision boundary to areas where all reward models erroneously extrapolate in the same manner. The same phenomenon may occur in other approaches for uncertainty estimation that are not distance-aware, such as Monte Carlo Dropout (Gal Ghahramani, 2016) and Epistemic Neural Networks (Osband et al., 2021).
Conclusion
While reward models can improve robustness to alignment-induced distribution shift, diversity of the ensemble is crucial. Pretrain ensembles are more diverse than finetune ensembles, and therefore lead to stronger generalization. However, even pretrain ensembles are not diverse enough: for many reward hacks, all members the ensemble agree, making the ensemble as vulnerable as its constituents. To summarize, reward model ensembles mitigate, but do not eliminate, reward hacking. Future work should examine uncertainty quantification techniques that are more robust to the type of distribution shift that occurs during alignment, particularly techniques that explicitly represent distributional shift from the preference annotations (Chu & Ghahramani, 2005; Tibshirani et al., 2019).
7 Reproducibility
To support reproducibility and enable further work on pretrain ensembles, we have released all 5 x 3 = 15 pretraining checkpoints, covering the T5-BASE, T5-LARGE, and T5-XL scales. The release can be found at https://githe To our knowledge, this is the first release of multiple T5 checkpoints, following prior work on BERT (Sellam et al., 2021). All datasets used in this research are public. Appendix C provides all relevant hyperparameters used during RLHE
com/ com/goos wwmm/goggeeddeemnnndll. weeedddeeeneeeeerddddddaadeaeeed ensembles.
10

---

Published as a conference paper at COLM 2024
References
Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, and Dan Mané. Concrete problems in ai safety. ar Xrv preprint ar #rXi::160600556 2016.
Rohan Anil, Andrew M. Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexan- dre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, Eric Chu, Jonathan H. Clark, Laurent El Shafey, Yanping Huang, Kathy Meier-Hellstern, Gaurav Mishra, Erica Moreira, Mark Omernick, Kevin Robinson, Sebastian Ruder, Yi Tay, Kefan Xiao, Yuanzhong Xu, Yujing Zhang, Gustavo Hernandez Abrego, Junwhan Ahn, Jacob Austin, Paul Barham, Jan Botha, James Bradbury, Siddhartha Brahma, Kevin Brooks, Michele Catasta, Yong Cheng. Colin Cherry, Christopher A. Choquette-Chonee Choo, Aakanksha Chowdhery, Clément Crepy, Shachi Dave, Mostafa Dehghani, Sunipa Dev, Jacob Devlin, Mark Diaz, Nan Du, Ethan Dyer, Vlad Feinberg, Fangxiaoyu Feng, Vlad Fienber, Markus Freitag, Xavier Garcia, Sebastian Gehrmann, Lucas Gonzalez, Guy Gur-Ari, Steven Hand, Hadi Hashemi, Le Hou, Joshua Howland, Andrea Hu, Jeffrey Hui, Jeremy Hurwitz, Michael Isard, Abe Ittycheriah, Matthew Jagielski, Wenhao Jia, Kathleen Kenealy, Maxim Krikun, Sneha Kudugunta, Chang Lan, Katherine Lee, Benjamin Lee, Eric Li, Music Li, Wei Li, YaGuang Li, Jian Li, Hyeontaek Lim, Hanzhao Lin, Zhongtao Liu, Frederick Liu, Marcello Maggioni, Aroma Mahendru, Joshua Maynez, Vedant Misra, Maysam Moussalem, Zachary Nado, John Nham, Eric Ni, Andrew Nystrom, Alicia Parrish, Marie Pellat, Martin Polacek, Alex Polozov, Reiner Pope, Siyuan Qiao, Emily Reif, Bryan Richter, Parker Riley, Alex Castro Ros, Aurko Roy, Brennan Saeta, Rajkumar Samuel, Renee Shelby, Ambrose Slone, Daniel Smilkov, David R. So, Daniel Sohn, Simon Tokumine, Dasha Valter, Vijay Vasudevan, Kiran Vodrahalli, Xuezhi Wang, Pidong Wang, Zirui Wang, Tao Wang, John Wieting, Yuhuai Wu, Kelvin Xu, Yunhan Xu, Linting Xue, Pengcheng Yin, Jiahui Yu, Qiao Zhang, Steven Zheng, Ce Zheng, Weikang Zhou, Denny Zhou, Slav Petrov, and Yonghui Wu. Palm 2 technical report, 2023.
Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXio preprint arXiv:2204005862, 2022.
Ahmad Beirami, Alekh Agarwal, Jonathan Berant, Alexander D' D'Amour, Jacob Eisenstein, Chirag Nagpal, and Ananda Theertha Suresh. Theoretical guarantees the best-of-n alignment policy. arXiv preprint ar arXiv:2401.01873 2024.
Ralph Allan Bradley and Milton E. Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39(3/4):324-33 1952. ISSN 00063444. URL
http:/ /ttp:///wwwww// http//wwwjjjtstsss. jstor. org/ httg//tabee/23330000
Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. Advances in neural information processing systems, 30, 2017.
Wei Chu and Zoubin Ghahramani. Preference learning with gaussian processes. In Proceed- ings of the 22nd international conference on Machine learning, pp. 137-144, 2005.
Thomas Coste, Usman Anwar, Robert Kirk, and David Krueger. Reward model ensembles help mitigate overoptimization. ar Xio preprint ar axX:::2310002743 2023.
Alexander D'Amour, Katherine Heller, Dan Moldovan, Ben Adlam, Babak Alipanahi, Alex Beutel, Christina Chen, Jonathan Deaton, Jacob Eisenstein, Matthew D Hoffman, et al. Underspecification presents challenges for credibility in modern machine learning. The Journal of Machine Learning Research, 23)))110237 2022.
Thomas G Dietterich. Ensemble methods in machine learning. In International workshop on multiple classifier systems, pp. 1-15. Springer, 2000.
11

---

Published as a conference paper at COLM 2024
Hanze Dong, Wei Xiong, Deepanshu Goyal, Rui Pan, Shizhe Diao, Jipeng Zhang, Kashun Shum, and Tong Zhang. RAFT: Reward ranked finetuning for generative foundation model alignment. ar preprint (rXiv:2304.06667 2023.
Yann Dubois, Xuechen Li, Rohan Taori, Tianyi Zhang, Ishaan Guirajani, Jimmy Ba, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. AlpacaFarm: simulation framework for methods that learn from human feedback. arXiv preprint ar arXio:2305.1433 2023.
Yarin Gal and Zoubin Ghahramani. Dropout bayesian approximation Representing model uncertainty in deep learning. In international conference on machine learning, PP-
1050-1059. PMLR, 2016.
Leo Gao, John Schulman, and Jacob Hilton. Scaling laws for reward model overoptimization. In International Conference on Machine Learning, pp. 10835-10866. PMLR, 2023.
Adam Gleave and Geoffrey Irving. Uncertainty estimation for language reward models. Xixi preprint Xiv:: 2203.07472, 2022.
Caglar Gulcehre, Tom Le Paine, Srivatsan Srinivasan, Ksenia Konyushkova, Lotte Weerts, Abhishek Sharma, Aditya Siddhant, Alex Ahern, Miaosen Wang, Chenjie Gu, et al. Reinforced self-training (ReST) for language modeling. arXiv preprint ar 008:02308.08998, 2023.
Victoria Krakovna, Jonathan Jonathann Uesato, Vladimir Mikulik, Matthew Rahtz, Tom Everitt, Ra- mana Kumar, Zac Kenton, Jan Leike, and Shane Legg. Specification gaming: the flip side of ai ingenuity. DeepMind Blog, 3, 2020.
Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. Advances in neural information processing systems, 30, 2017.
Tianqi Liu, Yao Zhao, Rishabh Joshi, Misha Khalman, Mohammad Saleh, Peter J. Liu, and Jialu Liu. Statistical rejection sampling improves preference optimization. Xrv preprint ar xxXie:2309.06657, 2023.
Shashi Narayan, Shay B. Cohen, and Mirella Lapata. Don't give me the details, just the summary! topic-aware convolutional neural networks for extreme summarization. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, 2018.
Yixin Nie, Adina Williams, Emily Dinan, Mohit Bansal, Jason Weston, and Douwe Kiela. Adversarial NLI: A new benchmark for natural language understanding. In Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel Tetreault (eds.), Proceedings of the 58th Annual Meeling of the Association for Compulational Linguistics, 2020.
Ian Osband, Zheng Wen, Seyed Mohammad Asghari, Vikranth Dwaracherla, Morteza Ibrahimi, Xiuyuan Lu, and Benjamin Van Roy. Epistemic neural networks. ar Xiv preprint #xXit:2107008924 2021.
Alexander Pan, Kush Bhatia, and Jacob Steinhardt. The effects of reward misspecification: Mapping and mitigating misaligned models. In International Conference on Learning Representations (ICLR), 2022.
Richard Yuanzhe Pang, Vishakh Padmakumar, Thibault Sellam, Ankur Parikh, and He He. Reward gaming in conditional text generation. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Vlume 1: Long Papers), 2023.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher Manning, and Chelsea Finn. Direct preference optimization: Your language model is secretly reward
model. arviv preprint ar Xiv: 2023.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with unified text-to-text transformer. The Journal of Machine Learning Research, 21(1):5485-555 2020.
12

---

Published as a conference paper at COLM 2024
Alexandre Ramé, Nino Vieillard, Léonard Hussenot, Robert Dadashi, Geoffrey Cideron, Olivier Bachem, and Johan Ferret. Warm: On the benefits of weight averaged reward models. arXiv preprint arXio:2401.12187, 2024.
Paul Roit, Johan Ferret, Lior Shani, Roee Aharoni, Geoffrey Cideron, Robert Dadashi, Matthieu Geist, Sertan Girgin, Leonard Hussenot, Orgad Keller, Nikola Momchev, Sabela Ramos Garea, Piotr Stanczyk, Nino Vieillard, Olivier Bachem, Gal Elidan, Avinatan Hassidim, Olivier Pietquin, and Idan Szpektor. Factually consistent summarization via reinforcement learning with textual entailment feedback. In Proceedings of the 61st Anuul Meeting of the Association for Computational Linguisties (Volume 1: Long Papers), 2023.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithuns. Xiv preprint or Xix:1707.063477 2017.
Thibault Sellam, Steve Yadlowsky, Jason Wei, Naomi Saphra, Alexander D' D'Amour, Tal Linzen, Jasmijn Bastings, Iulia Turc, Jacob Eisenstein, Dipanjan Das, et al. The multiberts: Bert reproductions for robustness analysis. arXiv preprint ar Xro:22006.161 2021.
Prasann Singhal, Tanya Goyal, Jiacheng Xu, and Greg Durrett. A long way to go: Investigat- ing length correlations in rlhf. arXiv preprint ar Xrii::2310.03716, 2023.
Joar Skalse, Nikolaus Howe, Dmitrii Krasheninnikov, and David Krueger. Defining and characterizing reward gaming- Advances in Neural Information Processing Systems, 35: 9460-9471, 2022.
Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul Christiano. Learning to summarize with human feedback. Advances in Neural Information Processing Systems, 33:3008-3022 2020.
RyanJ J Tibshirani, Rina Foygel Barber, Emmanuel Candes, and Aaditya Ramdas. Conformal prediction under covariate shift. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'lché Buc, E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems, 2019.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine-tuned chat models, 2023.
Michael Völske, Martin Potthast, Shahbaz Syed, and Benno Stein. TL;DR:
Mining reddit to learn automatic summarization. In Proceedings of the Workshop on New Frontiers in Summarization, pp. 59 63, 2017.
Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoc V Le. Finetuned language models are zero-shot learners. In International Conference on Learning Representations, 2022. URL https://openreva/ net wow.oooiidggerrrooooeeeee
Kevin Yang and Dan Klein. FUDGE: Controlled text generation with future discriminators. In Kristina Toutanova, Anna Rumshisky, Luke Zettlemoyer, Dilek Hakkani-Tur, Iz Beltagy, Steven Bethard, Ryan Cotterell, Tanmoy Chakraborty, and Yichao Zhou (eds.), Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 2021.
13

---

Published as a conference paper at COLM 2024
Sheheryar Zaidi, Arber Zela, Thomas Elsken, Chris C Holmes, Frank Hutter, and Yee Teh. Neural ensemble search for uncertainty estimation and dataset shift. Advances in Neural Information Processing Systems, 1471198-7991, 2021.
Yuanzhao Zhai, Han Zhang, Yu Lei, Yue Yu, Kele Xu, Dawei Feng, Bo Ding, and Huaimin Wang. Uncertainty Uncertainty-penaizzed reinforcement learning from human feedback with diverse reward lora ensembles. arXiv preprint "Xir:2401.002433 2023.
Xiaoying Zhang, Jean-Francois Ton, Wei Shen, Hongning Wang, and Yang Liu. Overcoming reward overoptimization via adversarial policy optimization with lightweight uncertainty estimation. Xrv preprint ar Xrxx:224330051 00111111, 2024.
Yao Zhao, Rishabh Joshi, Tianqi Liu, Misha Khalman, Mohammad Saleh, and Peter J. Liu. SLiC-HF: Sequence likelihood calibration with human feedback. arviv preprint ar axXiv:230.11025, 2023.
Banghua Zhu, Michael Jordan, and Jiantao Jiao. Principled reinforcement learning with human feedback from pairwise or k-wise comparisons. In International Conference on Machine Learning, pp. 43037-43067. PMLR, 2023.
14

---

Published as a conference paper at COLM 2024
A Numerical results for best-of-n reranking
Average agreement between reward models are shown in Tables 2-5. Autoevaluation results are shown in Table 6 and 7.
k
diff pretrain
same pretrain
self
1
0.599
0.599
0.599
2
0,915
0.963
0.981
4
1.155
1.243
1.275
base
8
1.340
1,462
1.507
16
1.486
1.640
1.696
32
1.605
1.787
1.854
64
1.708
1.914
1.991
0.785
0.785
0.785
2
1.228
1.328
1.368
4
1,556
1.732
1.805
large
8
1.830
2.069
2.168
16
2.031
2.330
2.454
32
2.203
2.552
2.697
64
2.348
2.744
2.907
1
0.673
0.673
0.673
2
1.159
1.245
1.309
4
1.513
1.663
1.780
xl
8
1.806
2.001
2.157
16
2.023
2.256
2.449
32
2.203
2.463
2.686
64
2.349
2.631
2.881
Table 2: TL;DR best-of-n agreement.
k
diff pretrain
same pretrain
self
0.662
0.662
0.662
2
1.081
1.144
1.178
4
1.446
1.560
1.621
base
8
1.609
1.770
1.855
16
1.776
1.972
2.078
32
1.910
2.139
2.267
0.727
0.727
0.727
2
1.139
1.190
1.231
4
1.492
1.580
1.656
large
8
1.670
1.791
1.896
16
1.832
1.979
2.112
32
1.962
2.134
2.291
0.588
0.588
0.588
2
1.037
1.079
1.134
4
1.441
1.513
1.609
xl
8
1.635
1.731
1.866
16
1.817
1.932
2.098
32
1.963
2.097
2.293
Table 3: HELPFULNESS best-of-n agreement.
15

---

Published as a conference paper at COLM 2024
diff pretrain
same pretrain
1
1.000
1.000
0.811
0.904
4
0.667
0.825
base
0.546
0.756
16
0.447
0.695
32
0,366
0.637
64
0.303
0.589
1.000
1.000
2
0.780
0.886
0,616
0.794
large
0,497
0.720
16
0.394
0.651
32
0.319
0.593
64
0.260
0.546
1.000
1.000
2
0.781
0.859
0.618
0.743
xl
8
0.503
0.655
16
0,400
0.567
32
0.323
0.497
64
0.262
0.433
Table TL,DR top agreement.
k
diff pretrain
same pretrain
1.000
1.000
0.805
0.885
4
0.650
0.789
base
0,506
0.695
16
0,406
0,620
32
0.318
0.548
1.000
1.000
2
0.810
0.874
0.656
0.766
large
8
0.522
0,668
16
0.413
0.579
32
0.324
0.506
1.000
1.000
0.812
0.860
0.666
0.746
xl
0.536
0.635
16
0,436
0.547
32
0.345
0.466
Table 5: HELPFULNESS top-1 agreement.
16

---

Published as a conference paper at COLM 2024
scale
ensemble
method
reward
win rate
mean
0..20
0.700
mean minus std
-0.186
0.703
finetune
median
-0.231
0.700
min
00.17
0.710
base
mean
0.130
0.721
mean minus std
0.086
0.731
pretrain
median min
0.155
0.715
0.086
0.727
single RM
single RM
0.244
0.685
mean
0.342
0.814
mean minus std
0.343
0.816
finetune
median min
0.309
0.809
0.348
0.813
large
mean
0.549
0.850
mean minus std
0.513
0.847
pretrain
median min
0.510
0.846
0.475
0.841
single RM
single RM
0.280
0.792
mean mean minus std
0.695
0.873
0,644
0,872
finetune
median
0.625
0.867
min
0.638
0.868
xl
mean
0.831
0.900
mean minus std
0.781
0.895
pretrain
median min
0.757
0.889
0.735
0.883
single RM
single RM
0.585
0.853
Table 6: TL;DR BoN (n 64) autoeval results (T5-XXL fine-tuned evaluator).
17

---

Published as a conference paper at COLM 2024
scale ensemble
method
reward
win rate
mean
0.635
0.741
mean minus std
0.615
0.735
finetune
median min
0.627
0.738
0.604
0.725
base
mean
0.691
0.752
mean minus std
0.661
0.748
pretrain
median min
0.683
0.749
0.624
0.741
single RM
single RM
0.600
0.727
mean
0,778
0,776
mean minus std
0.758
0.772
finetune
median min
0.771
0.770
0.738
0.766
large
mean
0.847
0.792
mean minus std
0.802
0.779
pretrain
median min
0.813
0.784
0.770
0.776
single RM
single RM
0.730
0.759
mean mean minus std
0.884
0.805
0,837
0.788
finetune
median
0.859
0.790
min
0.814
0.788
xl
mean
0,932
0.816
mean minus std
0.876
0.797
pretrain
median min
0.892
0.798
0.858
0.792
single RM single RM
0.811
0.779
Table 7: HELPFULNESS BoN (n 32) autoeval results (T5-XXL fine-tuned evaluator).
18

---

Published as a conference paper at COLM 2024
B Numerical results for RLHF
Full RLHF numerical results for HELPFULNESS and TL;DR are shown in Table and Table
Ensemble
Method
Reward
Reward large
Reward base
ft
0.010
2.562
mean
0.025
2.476
2.204
0.041
ft
mean
0.050
2.148
2.089
1.503
ft
mean
0.100
1.652
1.591
1.497
ft
0.150
1.328
1.258
1.212
ft
mean
0.200
1.079
1.032
0,980
ft
mean
0.300
0.764
0.688
0.666
ft
mean subtract std
0.010
2.478
ft
mean subtract std
0.025
2.401
2.188
0.240
ft
mean subtract std
0.050
2.118
1.978
1.585
ft
mean subtract
0.100
1.620
1.525
1.432
ft
mean subtract std
0.150
1.315
1.207
1.152
ft
mean subtract std
0.200
1.089
0.998
0,949
ft
mean subtract std
0.300
0.746
0.667
0.648
ft
median
0,010
2.466
ft
median
0.025
2.425
2.088
0.153
ft
median
0.050
2.154
2.051
1.445
ft
median
0.100
1.662
1.585
1.489
ft
median
0.150
1.318
1.255
1.197
median
0.200
1.096
1.022
0.976
ft
median
0.300
0.750
0.699
0.676
pt
mean
0.010
2.651
pt
mean
0.025
2.551
2.293
1.220
pt
mean
0.050
2.196
2.099
1.750
pt
mean
0.100
1.724
1.498
1.506
pt
mean
0.150
1.319
1.225
1.191
pt
mean
0.200
1.106
1.014
0.958
pt
mean
0.300
0.759
0.643
0.680
pt
mean subtract std
0.010
2.688
pt
mean subtract std
0.025
2.529
2.293
0.636
pt
mean subtract std
0.050
2.167
2.025
1.696
pt
mean subtract std
0.100
1.695
1.450
1.342
pt
subtract std
0.150
1.301
1.188
1.117
pt
mean subtract
0.200
1.099
1.007
0.932
pt
mean subtract std
0.300
0.732
0.688
0.641
pt
median
0.010
2.611
pt
median median
0.025
2.540
2.383
1.166
pt
0.050
2.141
2.064
1.662
pt
median
0.100
1.674
1.488
1.537
pt
median
0.150
1.365
1.181
1.220
pt
median
0.200
1.117
0.973
1.014
pt
median
0.300
0.743
0.669
0.661
single RM
n/a
0.010
2.245
single RM n/a
0.025
2,321
1.511
-0.349
single RM n/a
0.050
2.024
1.834
1.028
single RM n/a
0.100
1.594
1.478
1.432
single RM n/a
0.150
1.297
1.194
1.148
single RM n/a
0.200
1.069
0.988
0.937
single RM n/a
0.300
0.759
0.661
0.636
Table 8: HELPFULNESS RLHF numerical results.
19

---

Published as a conference paper at COLM 2024
Ensemble Method
Reward
Reward large
Reward base
ft
mean
0.010
2.356
1.562
-1.310
ft ft
mean
0.030
2.088
1.659
-0.456
mean
0.100
1.073
0.779
-0.389
ft ft ft
mean
0.300
-0.217
-0.380
-0.785
mean
0,500
-0.707
-0.785
-0.964
mean subtract std
0.010
2.171
1.579
-1.185
ft
mean subtract std
0.030
1.760
1.533
-0.392
ft
mean subtract std
0.100
0.811
0.658
-0.359
ft
mean subtract std
0.300
-0.303
-0.409
-0.777
ft
mean subtract std
0.500
-0.735
-0.791
-0.660
ft
median
0.010
2.206
1.480
-1.999
ft
median
0.030
1.939
1.596
-0.509
ft
median
0.100
0.955
0.809
-0.37
ft
median
0.300
-0.265
-0.370
-0.789
ft
median
0.500
-0.728
-0.788
-0.963
pt
mean
0.010
2.366
2.037
-08177
pt
mean
0.030
1.997
1.852
-0.433
pt
mean
0.100
0.964
0.858
-0366
pt
0.300
-0.309
-0.377
-0.776
pt
mean
0,500
-0.744
-0.786
-0,962
pt
mean subtract std
0.010
2.398
2.019
-0.957
pt
mean subtract std
0.030
1.997
1.710
-0.500
pt
mean subtract std
0.100
1.002
0.768
-0.28
pt
subtract std
0.300
-0.277
-0.357
-0.767
pt
mean subtract
0.500
-0.752
-0.774
00.558
pt
median
0.010
2.431
2.009
-08668
pt
median
0.030
2.030
1.903
-0.317
pt
median
0.100
1.086
0.850
-0347
pt
median
0.300
-0.308
-0.88
-0.778
pt
median
0.500
-0.746
-0.792
-0.962
single RM
n/a
0.010
1.728
1.429
-1.74
single RM
n/a
0.030
1.590
1.511
-0.458
single RM
n/a
0.100
0.787
0.758
-0.397
single n/a
0.300
-0.299
-0.387
-0.783
single RM
n/a
0.500
-0.736
-0.791
-0.966
Table 9: TL,DR RLHF numerical results.
C Hyperparameters
We provide the hyperparameters for reward model and RLHF training in Table 10 and Table 11. For reward models, we use the validation set to choose the best checkpoint along training. For RLHF, take the last checkpoint.
D Additional results
20

---

Published as a conference paper at COLM 2024
Task
Parameter
value
Learning rate Learning schedule
10-2
Constant (linear warm-up)
Warm-up steps Dropout
500
Helpfulness
0.05
Batch size (regularization coefficient)
64
0.01
Learning rate
10-4
Learning schedule
Constant (linear warm-up)
Warm-up steps Dropout
1000
TL;DR
0.05
Batch size
(regularization coefficient)
0.01
Learning rate
base/ alas//lrge: 10 3 xl: 3 10 3
Learning schedule
constant
XSum/NLI
Warm-up steps
Dropout Batch size
0.01
base/large 128, xl: 32
Table 10: Hyper-parameters for reward model training.
base
large
aprocment
0.8
cifl pretrain
chance
0.6
0.4
022
3.0
16
candidates
didates
reranking condidatos
(a) TL;DR
large
1.0
same pratram
100 00
0.0
16
32
16
32
reranking cordidates
koranknng can dindisttes
renanking condidates
(b) HELLLLNESSS
Figure 8: Agreement of the top-ranked output between reward models that do (crosses) and do not (circles) share pretraining seeds. Underspecification of reward models directly affects the behavior of the aligned policy. Chance agreement is 1/n.
21

---

Published as a conference paper at COLM 2024
Task
Parameter
value
Policy learning rate
5 10
Value learning rate Learning schedule Training steps
10 5
Constant (linear warm-up)
20000
Helpfulness
Warm-up steps
2000
Batch size
base/larger 32, xl: 16
input length
1024
output length
256
À
[0.01, 0.025, 0.05, 0.1, 0.15, 0.2, 0.3]
Policy learning rate
5 10-6
Value learning rate
10
Learning schedule
Constant (linear warm-up)
Training steps
20000
TL;DR
Warm-up steps
2000
Batch size
32
input length
1024
output length
128
A
[0.01, 0.03, 0.1, 0.3, 0.5]
Policy learning rate Value learning rate Learning schedule
5 10-6
10 5
Constant (linear warm-up)
Training steps
20000
XSum/NLI
Warm-up steps
2000
Batch size
32
input length
1024
output length
64
A
[0.01, 0.03, 0.05, 0.1, 0.3, 0.5]
Table 11: Hyper-parameters for RLHF.
Task
Prompt
Output
TLDDR
I'll try keep this short! "Backgroumd" I've always been an on again/off again (very casual!) jogget typically doing 3 5-5kt My knees have always been finicky, and went physio who thought had "runner's knee" [..] had my baby year ago, tinylest
I think pregnancy messed with my budy, now can't even run even the smallest without pain in my pelvis and I'm fairly just that I'm completely out shape.
years| However, bit, or even go on long walk or hike, my pelvis gets very sore and tight I_.]
problem
Helpfulness
have really popular antagonists? Assistant: Sure! I know many books that feature good antagonists, such thuse the Lord of the Rings books. Human: Who specifically from Lord of the Rings? Assistant:
Human: Assistant, can you help me
Well, Sauron, example. you want, can explain who he was.
XSum/NLI The Reading delender denied fraudulent trading charges relating, to the Sodje Sports Foundation a charity to miise money Nigerian sport. Mr Sodje, 37, is jointly charged with elder brothers Ele, 44, firight, 50 and Stephen, 42. Appearing at the Old Bailey earlier, all four denied the offence. The charge relates to offences place
Former Premier League footballer Sam Sodje appeared in court alongside three brothers accused of charity fraud.
which allegedly
2008 2014. from Kent, Creater Manchester, Stephen, Bexley, were all relemsed on bail.
Bright, sland July They
Table 12: Prompt-output pairs for the three benchmarks we consider. See $2.3.
22

---

Published as a conference paper at COLM 2024
pt
T5-XXL (finetuned)
-0,50
single
-1.00
-1.25
-1.50
-1.75
-2.00
15.0
Figure 9: XSUM/NIJ KL-reward tradeoff for pretrain ensembles, finetune ensembles, and individual models. Reward is measured with T5-XXL. Both pretrain and finetune ensembles slightly improve over individual models.
23

---

Published as a conference paper at COLM 2024
624
0.4
HELPFULNESS. Fraction of answers containing lists (as matched by regular expression).
Morbel Herber
Herber
10
30
230
28
(b) TL;DR, Left: extractiveness, as measured by average longest common substring between the summary and the context document. Right: length.
Natse
1.0
Thoaard www.in 2300
(c) XSUM/NLL /LII Left: length. Right: specificity, as measured by fraction of numerical tokens in the output.
Figure 10: Limitations of reward model ensembles. The x-axis is number of RLHF steps, the y-axis plots different statistics of the average validation output that step, and the curves correspond to the pretrain ensemble (solid blue) and its members (dashed orange). For preference data, we plot the same statistics conditioned on the preference data label (Preferred vs. Rejected). On HELPFULNESS (A = 0.05, top), the ensemble tends to return list 0.01), summaries become longer and copy longer spans
of items. On TI, (center, from the original document. For XSUM /I.I NI.I (A 0.03, bottom), responses are short and less specific, as measured by lack of numerical information. HELPFULNESS and TL;DR, the statistics of the "aligned" outputs are far from their values in the preference data.
24


Reviews:
[{"review_id": "60GbxlhO6w", "reviewer": "Reviewer_kjNj", "summary": "This paper looks at the issue of reward hacking with reward model ensembles. The datasets, models, and training methods are all from previous, except for how different individual reward models are trained and ensembled together. Experiments first show that different reward models predict similar scores for in-distribution data, but disagree substantially for out-of-distribution data. Experiments with reward ensembles show that ensembles usually provide higher performance in best-of-n reranking as well as RLHF finetuning. However, reward hacking is still observed even when reward model ensembles are used in RLHF.\n\nStrengths:\n- The issue of reward hacking and the prevention of it are important problems.\n- The experiments presented in the paper are thorough and support most of the claims in the paper.\n- Analysis and discussion provide good insights into the issue.\n\nWeaknesses:\n- Lack of human evaluation. The only human evaluation is qualitative in nature.\n- Ensembling is a well known method to reduce overreliance of spurrious features. The paper does not contribute to the method itself.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for the feedback on our submission! We were pleased to read that you found that the experiments and analysis were thorough and insightful.\n\nWhile we do not offer a new ensembling method, our paper sheds new light on the relevance of ensembling in the pretraining era, and in particular for LLM alignment. First, it is necessary to revisit the ideas behind ensembling given the widespread use of pretraining as an initial point for the ensemble members. To our knowledge, we are the first to compare ensembles within and across pretrains. Second, we are among the first to consider the relevance of ensembling to the phenomenon of reward hacking in RLHF, which, as we argue in the paper, is related to but not coextensive with overfitting.\n\nRegarding the lack of human evaluation, we agree that this would strengthen the empirical results of the paper and hope to explore it in future work. In particular, it would be good to ensure that the gains offered by reward ensembles in automated evaluations also hold up in human evaluations. This evaluation challenge is shared by a large number of recent studies (e.g., all those using [1] for evaluation) where language models generate long responses, which render human evaluation prohibitively expensive. Due to the cost and time-intensity, unfortunately we cannot commit to performing such an evaluation for the next version of this paper. We emphasize that prompt-based and fine-tuned autoeval models tell a similar story about the effects of pretrain and finetune ensembles, increasing our confidence that these effects are real.\n\n[1] AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedback. Dubious et al., NeurIPS 2023.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "See strengths.", "reasons_to_reject": "See weaknesses.", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ZL8KG74u6i", "reviewer": "Reviewer_2M6i", "summary": "This paper studies the use of reward models in aligning language model with human preference with a focus on addressing the reward hacking using reward model ensemble. In RLHF, the policy may exploit the model error to achieve a good performance measured by the imperfect reward model, which is usually called reward hacking or reward over optimization. The authors investigate how well reward model ensemble can mitigate this issue during training and inference. Further, the author also study why does reward model ensemble will fail sometimes. The key findings in this paper are:\n\n1. Reward models are underspecified, which leads to different rewards or rankings when tested on out-of-distribution data.\n2. Overoptimization occurs when alignment to one reward model does not improve reward as measured by another.\n3. Reward ensembles, especially pretrained ensembles (with different random seeds), help mitigate overoptimization and improve generalization.\n4. Reward hacking is not completely eliminated even with reward model ensemble. This is because the reward model ensemble will underestimate the uncertainty if it's far away from the data distribution.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for the feedback on our submission. We're glad that you found reading the paper to be enjoyable!\n\nRegarding insights: Our work is indeed inspired by past work on using ensembles in offline RL, typically in the dynamics model. However, applying ensembles for LLM alignment is more nuanced due to the interaction between the ensemble members and the process of pretraining. A key result of our work is that pretrain ensembles are more effective than finetune ensembles, a new finding that is important for better understanding the persistently difficult problem of combatting reward hacking during LLM alignment.\n\nDistance-aware uncertainty quantification: We agree that distance-aware uncertainty quantification could be a promising next step for uncertainty-aware reward modeling and appreciate the suggestion of SNGP. To our knowledge, however, SNGP has only been applied at the scale of T5-base [1] or with a small number of training examples [6, concurrent with our work]. We would certainly be interested in investigating how to leverage SNGP with modern LLM and practical training sets.\n\nSize of ensemble: In pilot studies in the best-of-n framework, we did not find a large effect of the ensemble size, although we agree that more comprehensive evaluation is needed. As the reviewer notes, experiments on ensemble size are computationally intensive, particularly for pretrain ensembles. However, we can explore the effect of ensemble size on fine-tune ensembles in the best-of-n setting, and will evaluate 10-member ensembles in the best-of-n setting for the next version of the paper.\n\nComputational expense of multiple pretrains: we agree, with two caveats. First, the effort to create a pretrain ensemble could be amortized across many reward models (as we have done in our own experiments on summarization and dialogue), and might also be useful for instruction tuning. Second, our work motivates the search for more sophisticated strategies that may yield similar benefits to full-scale ensembling at lower cost, such as, e.g. LoRA ensembles [2, 3] and weight-space averaging [4] after projection into a shared weight space [5].\n\n[1] https://arxiv.org/abs/2207.07411\n[2] https://arxiv.org/abs/2310.00035\n[3] https://arxiv.org/abs/2405.14438v1\n[4] https://arxiv.org/abs/2306.04488\n[5] https://arxiv.org/abs/2209.04836\n[6] https://arxiv.org/pdf/2403.05171v1", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This paper investigates a critical issue: aligning language models with human preferences. It provides a thorough examination of how reward model ensembles can mitigate over-optimization. The finding that ensembles composed of members pre-trained using different seeds generalize better offers valuable insights for practitioners, albeit unsurprising.\n\nOverall, the paper is well-organized, and the experimental results solidly support each argument, making it an enjoyable read.", "reasons_to_reject": "The new insights from the paper may seem limited after reading it. The use of an ensemble of uncertainty quantification methods to mitigate reward hacking is not novel in the literature, as many reinforcement learning papers, especially those on offline RL, adopt a similar approach to address the reward hacking problem.\n\nThe authors hypothesize that the same phenomenon will occur in other approaches for uncertainty estimation that are not aware of distance. I wonder if the authors have tried using SNGP on top of the reward model to make it distance-aware when quantifying uncertainty. If the hypothesis holds true, it should perform better than an ensemble in mitigating reward hacking.\n\nAdditionally, to validate the above conjecture, it would be interesting to see how increasing the number of ensembles affects performance. If the issue is largely due to being \"distance unaware,\" increasing the number of ensembles should not be helpful beyond a certain point.\n\nImplementing this may require significant resources, and it might be impractical to obtain several large pretrained models with different seeds.", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "U8mMpkQnd7", "reviewer": "Reviewer_jRNy", "summary": "This paper investigates the problem of reward hacking in language model alignment using reward models. The authors show that reward models trained on the same data but with different random seeds can disagree significantly on out-of-distribution data generated by the policy during the alignment process. This underspecification of reward models propagates to the aligned policy, making it highly tuned to the specific reward model used during training. To mitigate this issue, the authors propose using reward model ensembles, particularly \"pretrain ensembles\" where each member is pretrained with a different random seed. Experiments on several language tasks demonstrate that pretrain ensembles are more robust than individual reward models or \"finetune ensembles\" where members only differ in the finetuning seed. However, the authors find that even pretrain ensembles do not eliminate reward hacking, as they can fail to capture uncertainty and penalize certain undesirable behaviors that are incentivized by all ensemble members.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for the careful consideration of our submission! We were happy that you found the results convincing and the analysis insightful.\n\nComputational expense of multiple pretrains: we agree, with two caveats. First, the effort to create a pretrain ensemble could be amortized across many reward models (as we have done in our own experiments), and might also be useful for instruction tuning. Second, our work motivates the search for more sophisticated strategies that may yield similar benefits to full-scale ensembling at lower cost, such as, e.g. LoRA ensembles [1, 2] and weight-space averaging [3] after projection into a shared weight space [4].\n\nEnsemble diversity: as shown in Figure 3 the ensemble makes diverse predictions even before RLHF. Fine-tune ensembles have an average rank correlation between .45 and .62 on five outputs from the same reference model, and the average rank correlation for pretrain ensembles is significantly lower. That said, we agree that more could be done to encourage diversity in finetune ensembles, and the reviewer's suggestion of a bootstrap ensemble seems particularly promising. Our focus on random seed variability aligns with our emphasis on underspecification on deep networks from in-distribution training data [5], but we agree that other sources of variability are of interest.\n\nComparison of an ensemble of small RMs vs a single larger RM: to a limited extent, this comparison can be extracted from our results because the T5-XL models are roughly four times larger than the t5-large models (3B vs 770M parameters), making the T5-large ensemble comparable in parameter count to T5-XL. According to the fine-tuned autoeval model, performance of the two setups is similar. For example in BoN, a T5-large pretrain ensemble gets win rates of .85 (tldr) and .79 (helpfulness), vs .85 and .78 for a single T5-XL model. Morevoer, our results demonstrate that ensembling can *further* improve the performance of a policy trained with a T5-XL reward model. As for affordability, a reward ensemble of size 5xN may enable faster inference during RLHF than a single reward model of size 5N, since inference over the reward models can be easily parallelized; similarly, parallelization may make it easier to pretrain five LMs of size N than a single LM of size 5N.\n\n[1] https://arxiv.org/abs/2310.00035\n[2] https://arxiv.org/abs/2405.14438v1\n[3] https://arxiv.org/abs/2306.04488\n[4] https://arxiv.org/abs/2209.04836\n[5] https://arxiv.org/abs/2011.03395", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The paper analyzes the problem of underspecification in reward modeling, providing empirical evidence of its prevalence and downstream effects on the aligned policy.\n- The proposed solution of pretrain ensembles is well-motivated and evaluated on multiple tasks with different alignment strategies (best-of-n and RLHF). The results convincingly demonstrate the advantages of pretrain diversity.\n- The authors identify important failure modes of reward ensembles through qualitative analysis, showing that they don't fully solve reward hacking. This provides a balanced and insightful perspective on the limitations of the proposed approach.", "reasons_to_reject": "- Pretraining multiple reward models is computationally expensive, especially at larger model scales. The paper doesn't discuss the trade-off between ensemble diversity and training cost.\n- I think RMs might not be significantly different from each other, even though they were trained with different seeds. To promote much more diversity among the RMs, a better strategy for the RMs’ training would be explored, e.g., training each RM with a fraction of preference datasets like K-fold cross-validation or ensembles of RMs trained with different alignment datasets.\n- They did not compare the ensemble of small RMs and a single larger RM in terms of reward overoptimization. In some cases, training a single large RM might be more affordable.", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "7du1jiWfQf", "reviewer": "Reviewer_Aua6", "summary": "The commonly used preference alignment approach involves training a reward model and optimizing the output of the LLM against this reward model. The effectiveness of the reward model directly impacts the preference alignment. Recently, some studies have attempted to improve reward models through the use of a reward model ensemble technique, which can help avoid the phenomenon of reward hacking. This paper further explores the use of the reward model ensemble in preference alignment and presents some insightful conclusions based on experiments in the summarization and dialogue tasks.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for the careful consideration of our submission!\n\n**T5-XXL rewards the same hacks as the smaller scale models**: we agree, which is why we also evaluate the win rate with a prompted auto-eval model in Figure 6. This autoeval model is not trained on any preference data. The issue is discussed on the bottom of page 7.\n\n**Comparison of a larger reward model to an ensemble of smaller RMs**: To a limited extent, a comparison can be extracted from our results because the T5-XL models are roughly four times larger than the T5-large models (3B vs 770M parameters), making the T5-large ensemble comparable in parameter count to T5-XL. According to the fine-tuned autoeval model, performance of the two setups is similar. For example in BoN, a T5-large pretrain ensemble gets win rates of .85 (tldr) and .79 (helpfulness), vs .85 and .78 for a single T5-XL model. However, our results demonstrate that ensembling can *further* improve the performance of a policy trained with a T5-XL reward model. Ensembles across multiple RM scales is an intriguing idea for future work. \n\n**Potential \"difference in score margins between the [RMs] used for ensembling.\"** There are two potential issues: scale and offset. As discussed in section 2.1, the Bradley-Terry model, which defines the RM training objective, is undetermined with regard to offset (r=[-5, -7] is equivalent to r=[107, 105]). This is why we add an additional regularization term to \"center\" the rewards around zero (equation 2). However, the BT model is *not* undetermined with regard to scale: an assignment of r=[-1, 1] is not equivalent to r=[-10, 10] from the perspective of the reward model training objective. Therefore we do not expect RMs trained on the same data to produce rewards of significantly different magnitudes. \n\nQuestions:\n- Thanks for catching the typo in the final paragraph of section 2, we will correct it.\n- The idea of training the RMs on listwise data is an interesting proposed extension of the RLHF methodology, but we do not anticipate significant differences in terms of reward hacking. Listwise preferences are only available in limited settings.\n- Regarding reward score normalization, please see the discussion above about section 2.1 of the paper. The scores are not directly postprocessed, but the objective is changed to remove the shift-invariance of the BT model.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. This paper is easy to read, and the conclusions are clear. This paper explores RM ensemble through reinforcement learning and re-ranking, and evaluates the effects of the data distribution and reward model type.\n2. Building high-quality reward signals for preference alignment training is crucial. This paper can help in understanding the reward behavior and better constructing the reward ensemble, thus outputting higher-quality reward signals for the preference alignment.", "reasons_to_reject": "1. As shown in Table 1, although T5-XXL can outperform the best T5-XL model on each task, using the win-rate computed by T5-XXL as an evaluation metric for each task is inadequate. One concern is that T5-XXL exhibits a significant amount of reward hacking behavior, similar to that of the previously used reward models, such as T5-Large and T5-XL. Therefore, it is recommended to also use other metrics (e.g., BARTScore and GPT-4 Win Rate) to further evaluate the policy model.\n\n2. Given OpenAI's reward scaling law, one might wonder if a larger reward model could effectively mitigate reward hacking. It raises a natural question: Is it more beneficial to use a larger reward model, or consider an ensemble of smaller reward models? Furthermore, what would be the impact of introducing more larger reward model into the reward model collection for ensembling? However, these questions are not discussed in this paper.\n\n3. The reward model ensemble discussed in this paper overlooks a significant issue: the differences in score margins between the reward models intended for ensembling. These differences can lead to error results for the reward model ensemble. For instance, consider two samples, a and b. The first reward model assigns scores of -5 and -6, respectively, while the second reward model assigns scores of 3 and 20, respectively. When applying the reward ensemble for re-ranking, the negative score assigned by the first reward model to sample b can be easily overlooked. However, we cannot guarantee that the margin measure is consistent across all reward models. There is a possibility that the first reward model deems sample b to be significantly inferior to sample a, despite only a one-point difference in their scores.", "questions_to_authors": "1. This seems to be a typing error in the sentence: “PALM-2 on the two possible orderings....”\n2. Have you tried using list-wise data to train the reward models? If so, is this phenomenon the same as with pair-wise data?\n3. Has the reward score of each reward model been normalized or undergone other preprocessing steps?", "extra_scores": {}}]

Author rebuttal:

Meta-review: ## Summary of the paper

The paper studies the effect of ensembling reward models (RMs) in the context of RLHF for LLMs. Specifically, the authors show:
- RMs are underspecified: when trained with different random seeds, they give different rankings on OOD inputs.
- Ensembling over pretrains leads to better uncertainty estimates than ensembling over finetunes.
- Ensembling RMs can lead to better results in RLHF and in best-of-N reranking.
- Ensembling does not remove all reward hacks, as some are consistent across retrains.

## Strengths

- This is a high-quality study, with very carefully designed experiments. The authors consider a collection of models of different sizes, complementary evaluations (Palm 2 win-rate + T5-XXL reward), multiple tasks.
- The paper is well-written, with good structure, writing and figures.
- The paper studies a question which is very relevant in practice. Improvements to RMs can lead to major qualitative improvements in current LLMs. 
- Ensembling is also a pretty generic and fundamental tool for improving models, so I expect the results are likely to generalize to other setup variations.
- Results are overall positive and promising.
- Section 5 on failure modes is very interesting.

## Weaknesses

- Ensembling is a generic method, and similar studies with similar conclusions have been conducted many times in other settings (RL, computer vision, ...)
- Given the prior studies in other domains, the qualitative results are not very surprising
- Only very limited results on training an ensemble vs training a larger RM. Ultimately, it would be interesting to see a collection of scaling laws for different ways of growing the RM: growing the ensemble and growing individual models.
- No human evaluations of the quality of the policy model trained with the ensemble RM.

## Question

I am not sure if I missed it, but I would be quite curious to know if reusing the same pretrain for the policy and RM in RLHF is worse than using different pretrains?

## Recommendation

I think this is a good paper which would be a valuable contribution to the conference.
Decision: Accept