Helping or Herding? Reward Model Ensembles Mitigate but do not Eliminate Reward Hacking

Authors: jacob  eisenstein
chirag  nagpal
alekh  agarwal
ahmad  beirami
alex  d'amour
d j dvijotham
adam  fisch
katherine  heller
stephen  pfohl
deepak  ramachandran
peter  shaw
jonathan  berant
  google deepmind
  google research
Venue: colmweb_org_COLM_2024_Conference
Published: 2024-03-22 05:59:35

Abstract: Reward models play a key role in aligning language model applications towards human preferences. 
However, this setup creates an incentive for the language model to exploit errors in the reward model to achieve high estimated reward, a phenomenon often termed \emph{reward hacking}.
A natural mitigation is to train an ensemble of reward models, aggregating over model outputs to obtain a more robust reward estimate. 
We explore the application of reward ensembles to alignment at both training time (through reinforcement learning) and inference time (through reranking). 
First, we show that reward models are \emph{underspecified}: reward models that perform similarly in-distribution can yield very different rewards when used in alignment, due to distribution shift. 
Second, underspecification results in overoptimization, where alignment to one reward model does not improve reward as measured by another reward model trained on the same data. 
Third, overoptimization is mitigated by the use of reward ensembles, and ensembles that vary by their \emph{pretraining} seeds lead to better generalization than ensembles that differ only by their \emph{fine-tuning} seeds, with both outperforming individual reward models.
However, even pretrain reward ensembles do not eliminate reward hacking: we show several qualitative reward hacking phenomena that are not mitigated by ensembling because all reward models in the ensemble exhibit similar error patterns.

Main paper: 
 introduction: 
 To align machine learning systems with human preferences, it is common to use reward models (RMs), which are finetuned on preference annotations to score potential outputs by how likely they are to be preferred by human raters (Christiano et al., 2017; Stiennon et al., 2020) . There are many ways to use RMs to align policy models: they can act as training signals in reinforcement learning (Christiano et al., 2017; Stiennon et al., 2020) , they can select examples for further imitation learning (Gulcehre et al., 2023; Liu et al., 2023; Dong Steak is a high-fat steak, which means that it's a lot of calories and fat, but that it makes for a good meal. Steak can sit as a side dish to a lot of different foods. Bai et al. (2022 Bai et al. ( ). et al., 2023;; Touvron et al., 2023) , or they can be applied at inference time to steer the output distribution toward higher expected reward (e.g., Yang & Klein, 2021; Gao et al., 2023) . Such procedures create a semi-adversarial dynamic, in which the language model can achieve high reward by exploiting errors in the RM. Furthermore, while the RM is trained on a fixed set of human preference data, the process of alignment shifts the distribution of its inputs, increasing the likelihood of such errors. This phenomenon where the policy language model exploits reward model errors is often termed reward hacking (Amodei et al., 2016) , reward gaming (Skalse et al., 2022; Pang et al., 2023) , or reward over-optimization (Gao et al., 2023) .Reward hacking has been investigated from several perspectives in prior work (e.g., Krakovna et al., 2020; Skalse et al., 2022; Pan et al., 2022) . Bai et al. (2022) used reinforcement learning with human feedback (RLHF) and trained two RMs on non-overlapping splits of preference data, using one to drive alignment, and the other to measure the quality of the outputs. They find that RLHF increases performance according to both the driver and measurement models, but that a performance gap emerges as the policy is allowed to diverge from the initial distribution. However, both RMs were built on base models trained on the same pretraining data, which, as we will show, limits their diversity (as hypothesized by Gleave & Irving (2022) ) and thus may understate the effect of reward hacking. Other work has simulated the relationship between a "true" reward and a learned proxy, showing that it is possible to over-optimize the proxy to such an extent that the true reward starts to decrease (Gao et al., 2023; Coste et al., 2023) , e.g. by exploiting spurious correlations in reward model training data (Pang et al., 2023) .In this work, we first analyze RM distribution shift from the perspective of underspecification (D'Amour et al., 2022) , which occurs when a machine learning pipeline yields reliable performance on held-out data from the training distribution, but variable performance on out-of-distribution data. When applied to learning RMs from human preference data, we show that RMs that agree in-distribution often disagree after alignment-induced policy distribution shifts. Furthermore, such disagreements are more pronounced when the RMs are built on different pretrainings, even when that difference is induced merely by varying the pretraining random seed. These disagreements become increasingly severe when evaluated on outputs of a policy model that has been aligned to a specific RM. This occurs both when using RMs in RLHF, as well as when using an inference-time alignment procedure, best-of-n reranking, where n samples are drawn from the policy and then reranked with a RM.Motivated by these findings, we systematically investigate reward model ensembles as a possible remedy for reward hacking. Assuming different models err in different ways, ensembling can leverage reward uncertainty across the ensemble during alignment (see Figure 1 , Left). We explore several techniques for aggregating scores across the ensemble, e.g., taking the median score as a robust estimate of the true reward of the policy. We also consider two types of ensembles: pretrain ensembles, where different members of the ensemble differ in the random seed used during the pretraining phase, and finetune ensembles, where members differ only in the random seed used during finetuning. These ensembles are then evaluated across several types of policies and preference annotations: dialogue preferences for a helpful assistant (Bai et al., 2022) , summarization quality (Stiennon et al., 2020) , and whether a summary is grounded in its source text (Roit et al., 2023) .We find that ensembles are significantly more robust than individual reward models, and that pretrain ensembles are particuarly effective. However, reward ensembles are still susceptible to reward hacking when all members of the ensemble share similar error patterns, which in turn propagate to the ensemble (see Figure 1 , Right). This is exploited and amplified by policy optimization: for example, summarization models produce outputs that are too short when tuned for factuality, but too verbose when tuned for summarization quality; assistant models overuse formulaic answer formats when tuned for helpfulness. Coste et al. (2023) argue that reward model ensembles effectively mitigate reward hacking. Our work shares a similar research question, but differs in several ways, leading to more nuanced conclusions: we investigate both pretrain and finetune ensembles; we use human-annotated preference data rather than synthetically-generated labels; and we analyze several real reward hacks, showing that they are not prevented by ensembles. Subsequent work has explored efficient RM ensembles through low-rank adaptation (e.g., Zhai et al., 2023) and weight averaging (Ramé et al., 2024) , but exclusively for finetune ensembles. More generally, we can try to optimize the policy against the worst-case reward in some analytically-computed uncertainty set (Zhu et al., 2023; Zhang et al., 2024) , but this too accounts only for uncertainty due to the finite sample of preference pairs and not in the representations made available through pretraining. 
preliminaries: 
 We now briefly review how reward models are trained ( §2.1) and how they are used for alignment ( §2.2). We then describe our setup for studying reward hacking ( §2.3).RMs are typically trained from preference data, (x, y + , y -) ∈ D, where y + is preferred over y -for prompt x. Under the Bradley-Terry model (Bradley & Terry, 1952) , the probability that response y 2 is preferred over y 1 given a reward function r and a prompt x is p(y 1 ≺ y 2 | x) = σ(r(x, y 2 )r(x, y 1 )), where σ(•) is the sigmoid function. Given a dataset of preferences, the maximum-likelihood objective is,J (r) = E (x,y + ,y -)∼D log p(y -≺ y + | x) .(1)The Bradley-Terry model is underdetermined: for any RM r * , there is a set of equivalent models, r (x, y) = r * (x, y) + C(x) where C(x) is a prompt-dependent constant, such that J (r * ) = J (r ). This is problematic for ensembling: if different RMs choose different values for C(x), then order statistics like median and minimum are meaningless. We therefore modify the objective by regularizing the sum of rewards per preference pair towards zero:EQUATION)where η is a small positive value, thereby resolving the issue of underdetermination.Note that RMs can also be trained from "pointwise" data, such as toxicity or factuality annotations on individual examples (Yang & Klein, 2021; Roit et al., 2023) . Such RMs are not underdetermined and so can be aggregated without adjustment.Best-of-n reranking (BoN) is an inference-time alignment strategy, where given a prompt x, we sample n generations y 1 , . . . , y n from a policy language model π(y | x) and return the generation with highest reward: y * = arg max y k ∈{y 1 ,...,y n } r(x, y k ). The Kullback-Leibler (KL) divergence of BoN from the initial policy is upper bounded by log n -n-1 n (Beirami et al., 2024) . BoN tends to outperform more elaborate alignment techniques like RLHF in the low-KL regime (Gao et al., 2023) , albeit with the cost of generating multiple samples at inference time.Reinforcement Learning from Human Feedback (RLHF) is an online reinforcement learning method that trains a policy language model π to maximize expected reward, while staying close to an initial policy, π sft , which is typically finetuned on supervised data (prompt-output pairs). Distance from the initial policy is measured with KL divergence, which leads to the regularized objectiveEQUATION)where r is an RM, ρ is a distribution over prompts, and λ is a hyperparameter. Typically, this objective is optimized using PPO (Schulman et al., 2017) , which we also use in this work.The KL penalty in eq. ( 3) can limit reward hacking by keeping the policy close to π sft . However, KL regularization does not directly address reward model errors, and in particular does not address RM distribution shift when the preference annotations are not sampled from π sft (as is generally the case). Furthermore, we want to diverge from the reference policy when we can be confident of improving the expected reward. As we will show empirically, KL regularization is complementary to the development of more robust reward models, which yields pareto improvements in the reward-KL tradeoff.Datasets We consider three tasks. Example instances are provided in Table 12 .• TL;DR: A summarization benchmark where authors summarize their own reddit posts (Völske et al., 2017; Stiennon et al., 2020) , frequently used in research on RLHF and related methods (Rafailov et al., 2023; Zhao et al., 2023 ). • HELPFULNESS: A popular dialogue benchmark (Bai et al., 2022) , where given a partial conversation between a human and a digital assistant the goal is to complete the next assistant turn (Bai et al., 2022; Rafailov et al., 2023) .2 • XSUM/NLI: A summarization benchamark where a policy model trained on XSum (Narayan et al., 2018) is finetuned to generate summaries that are factually consistent with the source document (Roit et al., 2023) .Training reward models To examine the effect of pretraining, we pretrain five T5 models from scratch at the base (220M parameters), large (770M), and XL (3B) scales, using the standard denoising objective over the C4 corpus (Raffel et al., 2020) . The pretrained checkpoints differ only in their random seed, which controls parameter initialization and the sample from the pretraining data.For each task, we finetune each pretrained model five times using different random seeds.In TL;DR and HELPFULNESS we use the aforementioned preference data. For XSUM/NLI, we finetune pointwise natural language inference (NLI) models on the ANLI dataset (Nie et al., 2020) . This yields 25 RMs per task and scale (5 pretrain × 5 finetune), making it possible to evaluate the effect of pretraining and finetuning on underspecification ( §3) by constructing ensembles that differ in either pretrain or finetune seed ( §4). We use the publicly available T5-large (Raffel et al., 2020) as a policy for the summarization tasks. For HELPFULNESS, which requires substantial background knowledge, we use the instruction-tuned PALM-2-XXS model (Anil et al., 2023) . Before alignment, we create a finetuned policy π sft through supervised finetuning: We finetune on annotated summaries from TL;DR and XSUM/NLI for the corresponding tasks, and on the preferred responses, (x, y + ), from the preference data in HELPFULNESS.In BoN reranking, we rerank sampled output sets of size n ∈ {2 1 , 2 2 , . . . , 2 5 } for HELP-FULNESS and {2 1 , . . . , 2 6 } for TL;DR. Larger sets lead to higher reward at a cost of more expensive inference and larger deviation from π sft . In RLHF, we obtain a trade-off between the KL from π sft and the expected reward by training multiple times, varying the value of λ. Low values of λ correspond to high KL and high reward, while high values of λ entail low KL and low reward. For each value of λ we train roughly to convergence using a predetermined fixed number of steps (all hyperparameter values, including λ and the number of steps, are in Appendix C). Evaluation The aligned policies are autoevaluated by two metrics: reward and win rate. Similar to past work (Gao et al., 2023; Coste et al., 2023) , we use a larger RM to evaluate the generalization of models trained with a smaller RM. We train a T5-XXL RM by taking the publicly available T5-XXL (Raffel et al., 2020) and finetuning it as described above. As shown in Table 1 , T5-XXL outperforms the best T5-XL model on each task. We report both average reward from the T5-XXL evaluator as well as win rate, which is the fraction of prompts for which the response sampled from the aligned policy π has higher reward compared to π sft .Because the T5-XXL autoeval model is trained on the same data as the smaller T5 RMs, their errors might be correlated. For this reason, we also compute win rate according to a prompted PALM-2-Large model, which was not exposed to the reward training data but was instruction-tuned on Flan (Wei et al., 2022) . Given a prompt x, we sample a response y sft from π sft and y rlhf from π. We then ask PALM-2 which response is better, using a hand-engineered prompt proposed by Rafailov et al. (2023) . To avoid position bias, we run PALM-2 on the two possible orderings (y sft , y rlhf ) and (y sft , y rlhf ), sample K = 8 outputs for each order and determine the winner on this prompt through majority voting. This style of evaluation has become popular (Dubois et al., 2023; Singhal et al., 2023) and was shown to correlate well with human judgements (Rafailov et al., 2023) . 
underspecification in reward models: 
 We begin by demonstrating that the out-of-distribution performance of individual RMs is underspecified. Table 1 shows the average in-distribution accuracy across the 25 different RMs, together with the standard deviation. The low standard deviation highlights that all 25 reward models achieve similar performance in-distribution. But when we move to out-of-distribution data, the models diverge. Figure 2 shows the expected reward achieved by BoN as a function of the number of sampled candidates, n. The dotted green line 2 : Average reward of the best-of-n output, as judged by: the same RM used for ranking (self ); RMs fine-tuned from the same pretrain as the ranker (same pretrain); RMs fine-tuned from different pretrains from the ranker (diff pretrain). The RMs that do not share a pretrain with the ranker regard the ranker's preferred outputs as significantly worse. shows the expected reward of the top-ranked output according to the reranker itself, while the dashed blue line shows the expected reward of the same output according to RMs that share a pretrain seed. The solid orange line shows the expected reward according to RMs that do not share a pretrain seed. Unsurprisingly, the reranker scores its own top outputs more favorably than the other RMs do. However, the reranker's outputs are scored significantly less favorably by RMs which do not share a pretrain with the ranker. RMs that share a pretrain seed with the ranker model overestimate the true reward of the top-ranked output-suggesting that finetune ensembles are not sufficiently diverse because of the shared pretraining state of each of the ensemble's members. Notably, this gap does not disappear with scale, as shown in Figure 2 .Moving to alignment, differences in estimated rewards induce different policies from the BoN strategy. Figure 8 (in the appendix) shows the effects on agreement of the top-ranked summary when RMs do (crosses) or do not (circles) share pretraining seeds. Different RMs tend to produce different 1-best outputs, and these differences are linked to the pretraining seed: for example, two RMs from different pretrains will choose a different best-of-16 output more than half the time for both TL;DR and HELPFULNESS and in all scales.Figure 3 analyzes the evolution of agreement of the estimated reward scores when performing RLHF on TL;DR. For each policy model, we sample five completions for each prompt in the validation set at 2000 step intervals during RLHF. We then measure how well pairs of RMs agree on the ranking of these completions, using Spearman rank correlation. The correlation is averaged across all pairs of reward models that do and do not share the same pre-training seed; both sets of RMs include the one used to drive RLHF. As shown in the figure, RLHF decreases the average correlation, particularly at low levels of regularization. Furthermore, agreement is substantially lower for pairs of reward models that do not share the same pretraining seed. This supports our conclusion that reward modeling under alignment-induced distribution shift is underspecified by the preference annotations.Overall, our analysis demonstrates that (1) different RMs tend to disagree on out-ofdistribution data, particularly when the RMs have different pretraining seeds; (2) this propagates to the trained policy model, in the sense that the resulting policy is highly tuned to the preferences of the specific RM used to drive it; and (3) as a result, the disagreement between RMs tends to increase during alignment. These findings suggest that reward model ensembles might mitigate reward hacking, which we turn to next. 
reward model ensembles: 
 A natural mitigation to reward model underspecification is to ensemble multiple RMs, under the assumption that different models will have different errors. Aggregating over the scores of the ensemble members will help when some of the ensemble members erroneously assign high reward to a bad output.Building reward model ensembles Given a set of RMs M, we define the reward of the ensemble to be r(x, y) = agg({r m (x, y)} m∈M ), with agg indicating an aggregation function (Dietterich, 2000; Lakshminarayanan et al., 2017; Raffel et al., 2020; Zaidi et al., 2021) . Intuitively, the aggregation function should be conservative, and return a lower score when there is disagreement between the ensemble members. We consider the following simple aggregation functions: MEAN, MEDIAN, and MEAN_MINUS_STD, which subtracts the standard deviation of the reward from the mean to penalize high variance. We also experiment with MIN, but overall find it to be inferior to the alternatives.We evaluate two types of reward ensembles: pretrain ensembles, where each member is pretrained using a different random seed,3 and finetune ensembles, where members share the same pretraining seed, but use a different seed when finetuned on the reward data. In all cases the ensemble contains five individual RMs. Pretrain ensembles are expensive to train, but are more diverse and hence likely to lead to a more robust reward estimate. (Gleave & Irving (2022) reported negative results when using reward ensembles and hypothesized this is due to members sharing the same underlying pretrained model.)Evaluation We now evaluate RM ensembles across tasks. Figure 4 shows the results in bestof-n reranking, as measured by an XXL-scale fine-tuned RM. Pretrain ensembles consistently improve performance over individual RMs, especially for higher values of n for both TL;DR and HELPFULNESS. Finetune ensembles, conversely, improve performance in some cases and are comparable in others. For example, on TL;DR a pretrain ensemble with the MEAN aggregator achieves a win rate of 90% over the SFT outputs at the XL scale, while the win rate of a finetune ensemble with the same aggregator is 87.3%. The win rate of the average individual XL-scale RM is 85.3% (see Table 6 ). For visual clarity, in Figure 4 we show only two aggregators: MEAN and MEAN_MINUS_STD; see Appendix A for the other aggregators. In general, differences between aggregators are small, with MEAN usually performing at, or near, the top. More conservative aggregators (MIN and MEAN_MINUS_STD) come out slightly ahead of MEAN at the smaller scales on TL;DR, suggesting that high variance may be a bigger issue in this setting.Figure 5 shows the KL-reward trade-off of ensemble RMs in RLHF for TL;DR and HELPFULNESS (evaluated with the finetuned T5-XXL model). In such plots, a better model improves reward and/or reduces KL from the SFT policy (Gao et al., 2023; Coste et al., 2023) . As with BoN alignment, pretrain ensembles consistently outperform finetune ensembles as well as the average individual RM. For clarity, the figure presents results only for the MEDIAN aggregator for clarity, but these conclusions are supported by full numerical results across aggregators, shown in Appendix B. RLHF leads to much higher KL values than BoN. Consequently, we witness explicit reward hacking, in which the T5-XXL rewards decrease even as the RLHF objective improves. This happens most prominently for individual models, in many cases for finetune ensembles, and most rarely for pretrain ensembles-where T5-XXL reward scores decrease only when RLHF uses a T5-Base RM. Thus, our experiments on real data yield more negative conclusions than Coste et al. (2023) about the potential of ensembles to eliminate reward overoptimization.As the T5-XXL autoeval model is trained on the same distribution as the RMs used for alignment, it may overstate their performance. Thus, we also use a zero-shot autoeval model, PALM-2-Large (see Section 2.3). Because this evaluation is computationally expensive, we apply it to only the largest-scale RMs (XL). As shown in Figure 6 , ensemble RMs achieve higher win rates on both tasks and with both alignment techniques. For best-of-n, pretrain ensembles get significantly higher win rates on TL;DR at n = 64 (p < .001 by a permutation test); on HELPFULNESS the differences between ensembling techniques are not significant at n = 32. On both tasks, single RMs are significantly worse, p < .001. For RLHF, pretrain ensembles achieve better or equal win rates at lower KL divergence from the reference policy, with particularly strong performance on HELPFULNESS. Overall, these results mirror the T5-XXL evaluation, with one interesting difference: the PALM-2 autoeval reveals more reward hacking for RLHF, where win rate decreases with KL. This suggests that fine-tuned autoevaluators can overestimate performance when trained on the same preference data as the alignment RMs.Figure 9 (in the appendix) shows RLHF results for XSUM/NLI. Here ensembles offer relatively small improvements, and there is little difference between pretrain and finetune ensembles. We conjecture this is because XSUM/NLI optimizes specifically for factuality. This allows all models to find simple and similar strategies that lead to high reward (namely, emitting short responses), and thus ensembling does not lead to large gains in performance. 
when do reward model ensembles fail?: 
 Ensembles improve performance according to automatic evaluation metrics, but does this mean that reward overoptimization is solved? We now investigate five specific "reward hacks", and show that they are not prevented by ensembling, because all members agree that these hacks improve the quality of the outputs.To demonstrate this, we manually identify five qualitative distribution shifts in which the post-RLHF policies differ dramatically from both the reference policy and the preference annotations. Figure 7 and Figure 10 (in the appendix) show the results of this analysis on all benchmarks with a T5-large RM. The x-axis corresponds to the number of RLHF training steps, and the y-axis is a statistic of interest (e.g., average output length). We plot the statistic for the pretrained ensemble (using MEAN as a representative aggregation function) and for its members. For TL;DR and HELPFULNESS, where the reward model is trained on preference data, we show the statistic value on the preference data validation set, conditioned on the label 'Preferred' or 'Rejected'.• For HELPFULNESS (Figure 7a ), outputs tend to be in a list format, which we capture with a regular expression. The fraction of outputs that have this pattern increases to roughly 50% for three members of the ensemble, and for the ensemble itself. We do not detect this tendency in the preference data: the fraction of outputs that matches this format is roughly 8% for preferred and rejected responses.• For TL;DR (Figure 7b , Figure 10b ), RLHF leads to longer summaries (Singhal et al., 2023 ) and more extractive outputs, i.e., more copying from the input. Length in characters grows substantially for the ensemble and its members, where for the ensemble, length increases by a factor of two. On the preference data, preferred responses are slightly longer than rejected responses, but much shorter than outputs post-RLHF. We also compute the longest common subsequence (in characters) between the document and the summary and find that it doubles after RLHF with an ensemble RM. Although preference data shows a slight preference for copying, this preference is dramatically amplified by RLHF.• For XSUM/NLI (Figure 7c , Figure 10c ), training for factuality makes summaries shorter and less specific, as measured by omission of numerical values. All members of the Figure 7 : Limitations of reward model ensembles. The x-axis is RLHF steps, the y-axis plots different statistics of the average validation output at that step, and the curves correspond to the pretrain ensemble (solid blue) and its members (dashed orange). For preference data, we plot the same statistics conditioned on the preference data label (Preferred vs. Rejected).The statistics of the "aligned" outputs are far from their values in the preference data.ensemble exhibit this phenomenon: RLHF leads to rapid decreases in the length of the outputs and the fraction of outputs that contain numerical values.These findings illustrate the tendency of different reward models to associate certain features with high reward. Policy models can exploit this association, using these features to produce outputs that are dramatically different from the reward training data, which achieve (spuriously) high reward for both individual reward models and the ensemble.Why do reward model ensembles fail to capture uncertainty about these reward hacks? Prior work has shown that ensembles can provide good uncertainty estimates around the decision boundary, while underestimating uncertainty for examples far from the training distribution (Lakshminarayanan et al., 2017) . In LM alignment, the policy can shift the output distribution away from the decision boundary to areas where all reward models erroneously extrapolate in the same manner. The same phenomenon may occur in other approaches for uncertainty estimation that are not distance-aware, such as Monte Carlo Dropout (Gal & Ghahramani, 2016) and Epistemic Neural Networks (Osband et al., 2021) . 
conclusion: 
 While reward models can improve robustness to alignment-induced distribution shift, diversity of the ensemble is crucial. Pretrain ensembles are more diverse than finetune ensembles, and therefore lead to stronger generalization. However, even pretrain ensembles are not diverse enough: for many reward hacks, all members of the ensemble agree, making the ensemble as vulnerable as its constituents. To summarize, reward model ensembles mitigate, but do not eliminate, reward hacking. Future work should examine uncertainty quantification techniques that are more robust to the type of distribution shift that occurs during alignment, particularly techniques that explicitly represent distributional shift from the preference annotations (Chu & Ghahramani, 2005; Tibshirani et al., 2019) . 
reproducibility: 
 To support reproducibility and enable further work on pretrain ensembles, we have released all 5 × 3 = 15 pretraining checkpoints, covering the T5-BASE, T5-LARGE, and T5-XL scales. The release can be found at https://github.com/google-deepmind/reward-ensembles.To our knowledge, this is the first release of multiple T5 checkpoints, following prior work on BERT (Sellam et al., 2021) . All datasets used in this research are public. Appendix C provides all relevant hyperparameters used during RLHF. 
 
 Appendix: Average agreement between reward models are shown in Tables 2 3 4 5 . Autoevaluation results are shown in We provide the hyperparameters for reward model and RLHF training in Table 10 and Table 11 . For reward models, we use the validation set to choose the best checkpoint along training. For RLHF, we take the last checkpoint. Figure 10 : Limitations of reward model ensembles. The x-axis is number of RLHF steps, the y-axis plots different statistics of the average validation output at that step, and the curves correspond to the pretrain ensemble (solid blue) and its members (dashed orange). For preference data, we plot the same statistics conditioned on the preference data label (Preferred vs. Rejected). On HELPFULNESS (λ = 0.05, top), the ensemble tends to return a list of items. On TL;DR (center, λ = 0.01), summaries become longer and copy longer spans from the original document. For XSUM/NLI (λ = 0.03, bottom), responses are short and less specific, as measured by lack of numerical information. In HELPFULNESS and TL;DR, the statistics of the "aligned" outputs are far from their values in the preference data.These pretrains are available at https://github.com/google-deepmind/reward-ensembles.We use the base dataset (44K examples), where responses are generated from a 52B contextdistilled LM, and split the training set into two: half for training the RM, and half for the policy.Pretraining does not complete a single epoch over the pretraining data, and thus the data observed by each member of a pretrain ensemble is different (but sampled from the same distribution). 


Decision: ## Summary of the paper

The paper studies the effect of ensembling reward models (RMs) in the context of RLHF for LLMs. Specifically, the authors show:
- RMs are underspecified: when trained with different random seeds, they give different rankings on OOD inputs.
- Ensembling over pretrains leads to better uncertainty estimates than ensembling over finetunes.
- Ensembling RMs can lead to better results in RLHF and in best-of-N reranking.
- Ensembling does not remove all reward hacks, as some are consistent across retrains.

## Strengths

- This is a high-quality study, with very carefully designed experiments. The authors consider a collection of models of different sizes, complementary evaluations (Palm 2 win-rate + T5-XXL reward), multiple tasks.
- The paper is well-written, with good structure, writing and figures.
- The paper studies a question which is very relevant in practice. Improvements to RMs can lead to major qualitative improvements in current LLMs. 
- Ensembling is also a pretty generic and fundamental tool for improving models, so I expect the results are likely to generalize to other setup variations.
- Results are overall positive and promising.
- Section 5 on failure modes is very interesting.

## Weaknesses

- Ensembling is a generic method, and similar studies with similar conclusions have been conducted many times in other settings (RL, computer vision, ...)
- Given the prior studies in other domains, the qualitative results are not very surprising
- Only very limited results on training an ensemble vs training a larger RM. Ultimately, it would be interesting to see a collection of scaling laws for different ways of growing the RM: growing the ensemble and growing individual models.
- No human evaluations of the quality of the policy model trained with the ensemble RM.

## Question

I am not sure if I missed it, but I would be quite curious to know if reusing the same pretrain for the policy and RM in RLHF is worse than using different pretrains?

## Recommendation

I think this is a good paper which would be a valuable contribution to the conference.

Reviews: review: paper_summary: paper_summary: This paper looks at the issue of reward hacking with reward model ensembles. The datasets, models, and training methods are all from previous, except for how different individual reward models are trained and ensembled together. Experiments first show that different reward models predict similar scores for in-distribution data, but disagree substantially for out-of-distribution data. Experiments with reward ensembles show that ensembles usually provide higher performance in best-of-n reranking as well as RLHF finetuning. However, reward hacking is still observed even when reward model ensembles are used in RLHF.

Strengths:
- The issue of reward hacking and the prevention of it are important problems.
- The experiments presented in the paper are thorough and support most of the claims in the paper.
- Analysis and discussion provide good insights into the issue.

Weaknesses:
- Lack of human evaluation. The only human evaluation is qualitative in nature.
- Ensembling is a well known method to reduce overreliance of spurrious features. The paper does not contribute to the method itself.
strength_weakness: strength_weakness: See weaknesses.
paper_summary: paper_summary: This paper studies the use of reward models in aligning language model with human preference with a focus on addressing the reward hacking using reward model ensemble. In RLHF, the policy may exploit the model error to achieve a good performance measured by the imperfect reward model, which is usually called reward hacking or reward over optimization. The authors investigate how well reward model ensemble can mitigate this issue during training and inference. Further, the author also study why does reward model ensemble will fail sometimes. The key findings in this paper are:

1. Reward models are underspecified, which leads to different rewards or rankings when tested on out-of-distribution data.
2. Overoptimization occurs when alignment to one reward model does not improve reward as measured by another.
3. Reward ensembles, especially pretrained ensembles (with different random seeds), help mitigate overoptimization and improve generalization.
4. Reward hacking is not completely eliminated even with reward model ensemble. This is because the reward model ensemble will underestimate the uncertainty if it's far away from the data distribution.
strength_weakness: strength_weakness: The new insights from the paper may seem limited after reading it. The use of an ensemble of uncertainty quantification methods to mitigate reward hacking is not novel in the literature, as many reinforcement learning papers, especially those on offline RL, adopt a similar approach to address the reward hacking problem.

The authors hypothesize that the same phenomenon will occur in other approaches for uncertainty estimation that are not aware of distance. I wonder if the authors have tried using SNGP on top of the reward model to make it distance-aware when quantifying uncertainty. If the hypothesis holds true, it should perform better than an ensemble in mitigating reward hacking.

Additionally, to validate the above conjecture, it would be interesting to see how increasing the number of ensembles affects performance. If the issue is largely due to being "distance unaware," increasing the number of ensembles should not be helpful beyond a certain point.

Implementing this may require significant resources, and it might be impractical to obtain several large pretrained models with different seeds.
paper_summary: paper_summary: This paper investigates the problem of reward hacking in language model alignment using reward models. The authors show that reward models trained on the same data but with different random seeds can disagree significantly on out-of-distribution data generated by the policy during the alignment process. This underspecification of reward models propagates to the aligned policy, making it highly tuned to the specific reward model used during training. To mitigate this issue, the authors propose using reward model ensembles, particularly "pretrain ensembles" where each member is pretrained with a different random seed. Experiments on several language tasks demonstrate that pretrain ensembles are more robust than individual reward models or "finetune ensembles" where members only differ in the finetuning seed. However, the authors find that even pretrain ensembles do not eliminate reward hacking, as they can fail to capture uncertainty and penalize certain undesirable behaviors that are incentivized by all ensemble members.
strength_weakness: strength_weakness: - Pretraining multiple reward models is computationally expensive, especially at larger model scales. The paper doesn't discuss the trade-off between ensemble diversity and training cost.
- I think RMs might not be significantly different from each other, even though they were trained with different seeds. To promote much more diversity among the RMs, a better strategy for the RMs’ training would be explored, e.g., training each RM with a fraction of preference datasets like K-fold cross-validation or ensembles of RMs trained with different alignment datasets.
- They did not compare the ensemble of small RMs and a single larger RM in terms of reward overoptimization. In some cases, training a single large RM might be more affordable.
paper_summary: paper_summary: The commonly used preference alignment approach involves training a reward model and optimizing the output of the LLM against this reward model. The effectiveness of the reward model directly impacts the preference alignment. Recently, some studies have attempted to improve reward models through the use of a reward model ensemble technique, which can help avoid the phenomenon of reward hacking. This paper further explores the use of the reward model ensemble in preference alignment and presents some insightful conclusions based on experiments in the summarization and dialogue tasks.
strength_weakness: strength_weakness: 1. As shown in Table 1, although T5-XXL can outperform the best T5-XL model on each task, using the win-rate computed by T5-XXL as an evaluation metric for each task is inadequate. One concern is that T5-XXL exhibits a significant amount of reward hacking behavior, similar to that of the previously used reward models, such as T5-Large and T5-XL. Therefore, it is recommended to also use other metrics (e.g., BARTScore and GPT-4 Win Rate) to further evaluate the policy model.

2. Given OpenAI's reward scaling law, one might wonder if a larger reward model could effectively mitigate reward hacking. It raises a natural question: Is it more beneficial to use a larger reward model, or consider an ensemble of smaller reward models? Furthermore, what would be the impact of introducing more larger reward model into the reward model collection for ensembling? However, these questions are not discussed in this paper.

3. The reward model ensemble discussed in this paper overlooks a significant issue: the differences in score margins between the reward models intended for ensembling. These differences can lead to error results for the reward model ensemble. For instance, consider two samples, a and b. The first reward model assigns scores of -5 and -6, respectively, while the second reward model assigns scores of 3 and 20, respectively. When applying the reward ensemble for re-ranking, the negative score assigned by the first reward model to sample b can be easily overlooked. However, we cannot guarantee that the margin measure is consistent across all reward models. There is a possibility that the first reward model deems sample b to be significantly inferior to sample a, despite only a one-point difference in their scores.
questions: questions: 1. This seems to be a typing error in the sentence: “PALM-2 on the two possible orderings....”
2. Have you tried using list-wise data to train the reward models? If so, is this phenomenon the same as with pair-wise data?
3. Has the reward score of each reward model been normalized or undergone other preprocessing steps?

Comments: comment: Thanks for the authors' response. I don't have additional questions / concerns.
Thank you for the response. I'll keep my positive score.
Thank you for your response. Some of my concerns have been addressed. I'd like to see this submission accepted.