Reward Model Ensembles Help Mitigate Overoptimization
Authors: Thomas Coste
Usman Anwar
Robert Kirk
David Krueger
Venue: ICLR 2024 poster (2024)
Keywords: ensembles
overoptimization
RLHF
reinforcement learning from human feedback
language models
uncertainty weighted optimization
Primary area: reinforcement learning
TL;DR: Using an ensembles of (learned) reward models in RLHF helps mitigate overoptimization.

Abstract: Reinforcement learning from human feedback (RLHF) is a standard approach for fine-tuning large language models to follow instructions. As part of this process, learned reward models are used to approximately model human preferences. However, as imperfect representations of the “true” reward, these learned reward models are susceptible to overoptimization. Gao et al. (2023) studied this phenomenon in a synthetic human feedback setup with a significantly larger “gold” reward model acting as the true reward (instead of humans) and showed that overoptimization remains a persistent problem regardless of the size of the proxy reward model and training data used. Using a similar setup, we conduct a systematic study to evaluate the efficacy of using ensemble-based conservative optimization objectives, specifically worst-case optimization (WCO) and uncertainty-weighted optimization (UWO), for mitigating reward model overoptimization when using two optimization methods: (a) best-of-n sampling (BoN) (b) proximal policy optimization (PPO). We additionally extend the setup of Gao et al. (2023) to include 25% label noise to better mirror real-world conditions. Both with and without label noise we find that conservative optimization practically eliminates overoptimization and improves performance by up to 70% for BoN sampling. For PPO, ensemble-based conservative optimization always reduces overoptimization and outperforms single reward model optimization. Moreover, combining it with a small KL penalty successfully prevents overoptimization at no performance cost. Overall, our results demonstrate that ensemble-based conservative optimization can effectively counter overoptimization.

Published as a conference paper at ICLR 2024
REWARD MODEL OVEROPTIMIZATION
ENSEMBLES
HELP MITIGATE
Thomas Coste', Usman Anwar', Robert Kirk², David Krueger' University of Cambridge, Uniierriity College London {tc628, ua237, dsk30) ( @cam. ac. uk, robert warert.kikk... 20eecl. acc.clllac.... uk
ABSTRACT
Reinforcement learning from human feedback (RLHF) is standard approach for fine-tuning large language models follow instructions. As part of this process, learned reward models are used to approximately model human preferences. How- ever, as imperfect representations of the "true" reward, these learned reward mod- els are susceptible to overoptimization. Gao et al. (2023) studied this phenomenon in a synthetic human feedback setup with a significantly larger "gold" reward model acting the true reward (instead of humans) and showed that overopti- mization remains a persistent problem regardless of the size of the proxy reward model and training data used. Using similar setup, conduct systematic study to evaluate the efficacy of using ensemble-based conservative optimization objectives, specifically worst-case optimization (WCO) and uncertainty-weigheel optimization (UWO), for mitigating reward model overoptimization when using two optimization methods: (a) best-of-n sampling (BoN) (b) proximal policy op- timization (PPO). We additionally extend the setup of Gao et al. (202)) to include 25% label noise to better mirror real-world conditions. Both with and without label noise, we find that conservative optimization practically eliminates overop- timization and improves performance by up to 70% for BoN sampling. For PPO, ensemble-basee conservative optimization always reduces overoptimization and outperforms single reward model optimization. Moreover, combining with a performance cost.
small KL penalty successfully prevents overoptimization Overall, our results demonstrate that ensemble- based conservative optimization can effectively counter overoptimization.
INTRODUCTION
With the advent of large language models, reinforcement learning from human feedback (RLHF) has emerged as a powerful technique to fine-tune and enhance models' behaviors (Ziegler et al.) 2019; Ouyang et al.) 2022 Bai et al. 2022a). However, despite its empirical success, RLHF remains a fickle method suffering from many failure modes (Casper et al. 2023). One such failure mode overoptimization, phenomenon in which policy optimization appears to be making progress according to the learned reward model, but in reality begins to regress with respect to the true reward function (Ziegler al. 2019 Stiennon al. 2020 Gao et al., 2023). While many works on RLHF contain anecdotal evidence of overoptimization (Ziegler et al. 2019 Stiennon et al,, 2020 Dubois
al., 2023), Gao al. (2023) is the only work that studies overoptimization in systematic way. As working directly with human labelers is expensive, Gao et al. (2023) introduce synthetic setup to study overoptimization which a much larger language model is first trained as a "gold" reward model and is then used to generate preference labels for training of proxy reward models.
In this work, we conduct systematic study investigating whether combining ensembles with con- servative optimization can help mitigate overoptimization. Our results indicate that not only does ensemble-ber conservative optimization help mitigate overoptimization,
also results in im- proved performance. Our setup is identical to that of Gao et al. (003)) with modification: the addition of label noise. [Ga et al.] assume that the preference labels used to train the proxy reward model do not contain any noise. However, this does not mirror the real-world RLHF setup, in which agreement rates among human annotators are typically between 60 75% (Ziegler et al.
The code is available http:::/gittubbccmm/tttecc///
1pt ontion

---

Published as a conference paper at ICLR 2024
SFT stage
RM Training Stage
Policy Optimization Stage
nolsy preferences
SFT Model 1 (Policy)
generate
Preference Dataset
initial polti
Saseline
train
train
ensemble of functions
SFT Model (RM)
change
Optimized Policy
&.L.S.
Proxy RMs
Conservative Optimization
Seed
Figure 1: RLHF pipeline used in this work modifications top of the standard RLHF setup used in Gao et al. (2023) are highlighted green.
2019 Stiennon et al. 2020; Dubois et al. 2023). To simulate that disagreement and to better reflect the world RLHF, we extend the setup to include 25% label noise as well. In both the cases of no label noise and 25% label noise, we provide strong evidence that ensemble-based conservative optimization methods are effective in mitigating overoptimization and improving performance.
Scaling laws for reward model overoptimization discovered by Gao al. (2023) indicate that in- creasing the size of the proxy reward model reduces overoptimization as well. However, reward models derived from pretrained language models. Thus, acquiring a larger reward model would require significant pretraining, which is not always feasible and can be very costly (Morgan 2022 Venigalla & Li 2022). However, our approach, using ensembles of reward models, only requires fine-tuning muluple copies of an already pretrained reward model, which is relatively inexpensive. Moreover, our model and data scaling results (Figures and 9 indicate that the gains provided by our method are orthogonal to the gains achieved by increasing the reward model size; thus, the two approaches can combined seamlessly for even better results.
Our main contributions are as follows:
We conduct the first study of using ensembles to counter overoptimization in RLHF-based fine-tuning of language models.
Our results indicate that using ensembles and conservative optimization eliminates overop- timization for BoN and results in up to 70% improvement some cases.
For PPO, ensemble-based conservative optimization typically outperforms single reward model optimization, and when combined with a suitable KI. penalty weight successfully eliminates overoptimization.
We further conduct studies to establish the robustness of the ensemble-based conservative optimization methods to any new hyperparameters it introduces (e.g. size of the ensemble).
2
BACKGROUND
In this section, we briefly review two commonly used policy optimization methods: best-of-n sam- pling (BoN) and proximal policy optimization (PPO), followed by discussion of overoptimization.
2.1 BEST-OF-N SAMPLING (BoN)
Best-of-n (BoN) sampling, also called rejection sampling. is a simple inference-time optimization method (Ouyang al.) 2022 Nakano al.) (021). For given prompt, responses are generated from the policy model, and the answer with the highest proxy reward model score is returned. To evaluate the degree of optimization, the KL distance defined analytically as function of n::
KL = log n
(1)
2

---

Published as a conference paper at ICLR 2024
2.2 PROXIMAL POLICY OPTIMIZATION (PPO)
Proximal Policy Optimization (Schulman et al. 2017) is a policy-gradient-baeed online reinforce- ment learning method that maximizes given reward function by repeatedly performing small in- cremental updates to the policy. PPO is the standard algorithm used in fine-tuning language models based on human feedback @uuyang al. 2022 Bai al. 2022a Stiennon et al. 2020; Zheng al.) 2023). When using PPO to fine-tune a language model, KL
penalty term is added during the reward calculation to regularize the policy by preventing it from deviating far from the initial policy:
PPO 7PPPO(al)) (alg)
R (g. a) R(g. a) Blog
π i-nit (alg)
where x PPO is the policy being optimized and π inir is the initial (pretrained) language model. The degree of optimization is measured in terms of KL distance between the initial policy and the one being optimized, with details for this calculation provided in Appendix
2.3 OVEROPTIMIZATION
In reinforcement learning from human feedback (RLHF) reward model used to approximate human preferences, to remove the need for querying humans for every policy generation. As the learned reward model is only a proxy for the true reward function, optimizing may not always result in an improvement according to true human preferences. In practice, optimizing a (fixed) learned reward model almost always leads to improvement according to this learned reward model, but only improves according to the true reward model (i.e. humans) for some initial period, after which performance often begins to regress. This phenomenon is referred as overoptimization, an example of which is shown in Figure
To study the problem overoptimization, Gao et al. (2023) introduced synthetic setup in which, instead of human annotators, a gold reward model is used to score responses and generate preferences to train proxy reward models. The gold reward model is generally chosen to be much larger than the proxy reward model, to simulate the fact that in a real setup, human preferences are too complex be captured by a neural network with finite capacity. Within this setup, Gao et al. (2023) discover overoptimization be a persistent issue, although they note that larger proxy reward model sizes and greater amounts of training data can help reduce overoptimiza- tion. However, scaling up is not always a feasible so- lution as the proxy reward models are generally derived from extensively pretrained language models.
1.50
Dold Score
1225
0.75
0.50
0.0
025
25 50 KL. Devergence
75 100
125
Figure 2: An example of overoptimiza- tion. The KL divergence is the distance between the initial and current policy, measuring the degree of optimization.
3 METHOD
Standard RLHF often learns a single reward model to estimate the true reward, which is then used to optimize the policy. However, many works in wider machine learning have shown that learning multiple estimators and combining them can help improve robustness (Lakshminarayanan et al.)
2017 Ovadia al. 2019; Yu et al., 2020b). Taking inspiration from this insight, we propose to learn an ensemble of reward models [R,, R& in the reward model training stage. During policy optimization, we combine the reward estimates from different reward models within the ensemble according to the following three methods.
Mean Optimization: Mean optimization (Boyd & Vandenberghe 2004 Wright 2006) simply takes the mean of the outputs of the different ensemble members:
Rg a) Σ R.(g,a)
3
E=1
where q is the prompt given to language model and a is the corresponding response sampled from the language model.
3

---

Published as a conference paper at ICLR 2024
We note that mean optimization is not conservative optimization technique; if at least member of the ensemble overestimates the reward (even while other members accurately estimate it), mean optimization will not prevent the policy from exploiting the faulty reward model.
Worst-case optimization (WCO) (Boyd & Vandenberghe 2004 Wright 2006) creates a conservative estimate by choosing the lowest reward from the ensemble at every step. Choosing the lowest reward helps ensure that as long as at least one ensemble member does not overestimate the reward, policy optimization will not result in overoptimization.
Worst-Case Optimization:
Rwcolg, a) min (g.
major advantage of WCO is that it does not have any hyperparameters that might require tuning. However, it sometimes result in performance penalty due its highly conservative nature.
uncertainty-wcightee optimization (UWO) (Wu et al, 2021b Yu et al. 2020b Brantley et al. 2019), the reward for sample is calculated by combining the average reward across all models in an ensemble with the intra-ensemble variance, weighted by coefficient A. Intuitively, UWO works by penalizing the policy for generating responses for which there is high disagreement among reward models within the ensemble. This helps prevent the exploitation of a single faulty reward model which might be erroneously assigning high rewards to incorrect or low-quality responses. Mathematically, this objective is given by:
Uncertainty- Weighted Optimization:
- [. 2
1 Σ Σ Ri - -
Ruwo a)
(2( -
(5)
meun
variance
where À is hyperparameter which controls the weight of the uncertainty component.
4 EXPERIMENTAI. SETUP
We based our experiments on the setup proposed by Gao et al. (2023), however, we use open source models [Biderman al., [023) Dubois al., 2023), open source datasets (Taori al. (023), and train our proxy reward models under 25% label noise. We also present qualitative reproduction of the results of Gao et al. Appendix F
4.1 DATA
In order to train proxy reward models, we use the Alpaca dataset (Taori et al. 2023 with 52, 000 in- structions covering range of commands and corresponding demonstrations generated by OpenAl's text-davinci-O03. (OpenAl, 2023a). Each entry is composed of an instruction, an optional additional the AlpacaFarm (Dubois et al., (023) variant of the dataset, as it provides splits for use in the different RLHF stages and for validation, well human preference annotations. Further details concerning the splits, prompt format, and examples are given in Appendix D
input, and demonstrator output. More specifically,
4.2 PRETRAINED MODELS
For the policy model and (proxy) reward model, we use pretrained language models provided in the Pythia suite (Biderman et al. (023). The policy model used everywhere this work is the 1.4B Pythia model. For proxy reward models, we remove the unembedding layers from Pythia models of sizes 14M, 70M, and 1.4B and add scalar head to output the reward to get proxy reward models of sizes 7M, 44M, and 1.3B.
Finally, the 7B human preference reward model from AlpacaFarm (Dubois et al. 2023 is used as the gold reward model. It is of very similar size the (closed-source) 6.9B gold reward model used in Gao et al. (2023) and is significantly larger than any of our proxy RMs (with the largest being 1.3B) and such is reasonable choice for gold standard.
4

---

Published as a conference paper at ICLR 2024
4.3 RLHF PIPELINE
Our RLHF pipeline is similar to that of Gao et al. (2023) with several modifications highlighted in Figure
We explain the full setup below for completeness.
Supervised Fine-tuning: As the first step in our RLHF pipeline, both the policy model and the proxy reward model undergo supervised fine-tuning using 10k instruction demonstrations from the "sft" split of the AlpacaFarm dataset (see Section 4. and Appendix D.2 for details). This fine-tunes the models have better instruction-following capabilities.
Preference Generation and Labeling: To generate a preference dataset, the SFT model is prompted using instructions from the AlpacaFarm dataset, for which it produces two responses per instruc- tion. Relevant hyperparameters for sampling these responses are given in Appendix [.] The two responses are then scored with the gold reward model, which assigns a score to each them. Hu- man annotators tend to have high disagreement rates, often around 25% (Stiennon et al. 2020) or more (Ziegler et al. 2019 Dubois et al. 2023). To simulate this, we optionally mislabel 25% of the dataset. This results in two datasets: one with no label noise and one with 25% label noise.
Proxy Reward Model Training: We train our proxy reward model by minimizing cross-entropy loss the preference dataset generated in the previous timestep. Unless mentioned otherwise, we use the complete dataset of 46, 000 prompts for reward model training and train all the reward models for 5 epochs. We give other hyperparameters for reward model training in Appendix ann and example validation loss curves for a set of reward models in Appendix Trained reward models reach between 60-75% validation accuracy.
Ensemble Creation:
To create an ensemble, we train a fixed number of proxy reward models using identical data and hyperparameters but with different random seeds. This results in different random initialization for the scalar reward head added on top of the pretrained language model and a different data shuffling order. We use all the available training data for the training of every reward model as training on lesser data results in higher validation loss and poorer performance (Lakshminarayanan al. 2017). Unless stated otherwise, we train an ensemble consisting of five reward models.
Policy Optimization: Similar to Gao et al. (2023). we use BoN sampling and PPO as optimization algorithms. For BoN, the evaluation cost for greater KL nats increases exponentially, As result, due to constraints available compute, we only evaluate BoN for maximum of Amaz 12, 500 samples which roughly equals 8.4 nats of KL. For PPO, we train for 3000 PPO steps. We give further details on implementation and other hyperparameters in Appendix D.I
5 RESULTS
In this section, we report the results of our experiments. To summarize, our main findings are:
Using an ensemble, with any of the objectives highlighted in Section 3 almost always helps over using a single reward model.
For BoN, we do not observe any overoptimization when using WCO and UWO as optimization methods; however, in the 25% label noise case, mean optimization does overoptimize (Figure 3).
For BoN, all three ensemble-based optimization methods result in up to ~ 30% improvement in final performance over the average performance attained by optimizing individual reward models in the no label noise case and up to ~ 75% improvement in the 25% label noise case (Figure 3)
For PPO, WCO, and UWO do reduce overoptimization (Figure 5) but completely mitigating overoptimization requires combining them with small KL penalty (Figure 4).
For PPO, WCO, and UWO always match or outperform single reward model optimization in terms of final performance for all values of KL penalties (Figure 7.
Our findings are robust to scaling of model size and training dataset size (Figures and 9..
To present our results, we primarily rely on policy optimization curves where we show the gold reward model score on the left y-axis and the proxy reward model score the right y-axis. For
'Generating the n 12, 500 answers for 1000 prompts and then relabeling them with proxy and gold reward model takes approximately 700 1100 GPU hours.
5

---

Published as a conference paper at ICLR 2024
RM Type
2.3
RM Type
3.0
0.8
Single
0.6
Metn
2.0
Mean
T
WCO
I
0.5
WCO
2.0
1.3
Pron Score
Gold Score:
0.4
Prody Soore
1.5
Gold
0.4
1.0
0.3
1.0
0.2
0.2
0.5
0.1
0.5
2
Davergence
Divergence
(a) With no label noise.
(b) With 25% label noise.
Figure 3: BoN results for 44M reward model size. In this and future BoN Figures, KL divergence is defined as per Eq. and measures the degree of optimization.
2.0
0.6
1,75
0.5
1.30
0.4
1.25
Cold Score
0.3
1.0
Prwxy Stare
Gold Score
0.3
StoRe
0.2
RM Type
0.75
Single
Single RM
0.1
Meen
0.1
Moan
WCn
0.25
0.0
UWO
25
50
75
100 125150
25
50
75
100 125150
KL. Divergence
KT. Divergenoe
(a) With no label noise. For all four methods, KL penalty 0.01 used.
(b) With 25% label noise. KL penalty 0.1 for and 0.01 for all other methods is used.
Figure 4: PPO results for 44M reward model size, over three PPO seeds. The KI divergence scale different from BoN due to differences in the algorithm and the KL calculation (Appendix C). For each method, we choose the KL penalty that gives the best final performance.
the x-axis, following the common practice (Bai et al.) 2022a Gao et al. 2023), we use the KL. divergence defined in Eq. orr Appendix C on a square root scale for clarity. For PPO, most runs do not exceed KL divergence of 150, but those that do are truncated to maintain more legible plot. Additionally, runs in Figure 4 the main PPO optimization illustration, are averaged over three PPO seeds, with standard deviation shown. For other PPO optimization figures, due to the low variance observed, single seed is used. However, the results for single RMs are averaged over the five RMs which make up the ensemble for the other methods. When comparing many experiment results at once, we present bar plots showing the final performance on the gold reward model and the variable of interest the x-axis. As an additional evaluation metric for quality, Appendix P. compares the final policy win-rate of different ensemble methods. Finally, we also include some qualitative samples in Appendix F.10
5.1 BEST-OF-N SAMPLING
In Figure 3 we present results for BoN sampling for a 44M size reward model. Across both noiseless and noisy settings, ensembles help improve performance by 30% and 75% respectively and successfully avoid overoptimization, except for mean optimization in the case of noisy labels. For UWO, we show results for 0.5 which we found to be most performant; however, in Section 5.4 we show that many reasonable values of À work well. Additional results for different sizes of reward models are presented in Appendices and
5.2 PROXIMAL POLICY OPTIMIZATION
For PPO observe that WCO and UWO reduce overoptimization and outperform other methods in terms of final performance in the absence of KL penalty, although they do not eliminate overop- timization completely (as shown in Figure 5). However, in Figure show that with small KI.

---

Published as a conference paper at ICLR 2024
2.0
Gold Soore 7
0.3
1.5
Poady Score
Cold Score
0.2
Score
0.2
1.0
1.0
0.1
RM Type
Sing RM
0,5
0.1
RM
Stegle
0,54
0.0
WC0
UWO,
0.0
UWO, A 0.1
0.3
100 125150
125
Divergence
Divergence
(a) With label noise.
(b) With 25% label noise.
Figure 5: PPO results for different optimization objectives without using KL penalty,
0.4
Gold Score
0.3
Cold Score
0.4
0.2
Penulty
0.2
0.65
0.1
0.001
0.001
0.005
0.01 Kl. Ponalty
0.05
0.1
0.0
0.01
75
KT. Divergence
Single RM
Mean
WCD
UWO
Figure 6: Individual reward model opti- mization (i.e. no ensemble) for different values of KL penalty in the 25% label noise case.
Figure 7: Effectiveness of conserva- tive optimization methods across KI. penalty weights in the 25% label noise case.
penalty coefficient of 0.01, WCO and UWO both successfully prevent overoptimization, without any notable reduction performance. On the other hand, when using KL penalty on its own, a 20x larger KL penalty weight of 0.2 is required to eliminate overoptimization which incurs significant performance penalty (Figure 6). In Figure 7 we show that for all penalties, WCO and UWO match outperform the average final performance achieved by optimizing single reward model. More- over, for small KL penalties, they comprehensively outperform single reward model optimization.
5.3 MODEL AND DATA SCALING RESULTS
Prior work (Gao et 2023) has shown that increasing reward model size or training dataset size helps improve performance as measured by the gold reward model. In Figure 8, we vary the size of reward models (keeping the dataset size fixed at 46k samples and policy size fixed 1.4B) and plot the final performance of each method for the fixed training budget. Recall that for BoN, this is n = 12, 500 samples and for PPO, this is 3000 timesteps. Our results show that the improvement due to the use of ensembles is orthogonal to the improvement in performance achieved by scaling the reward model size. particular, with the 1.3B reward model, we highlight that even when the policy and reward model have similar number of parameters, WCO and UWO still provide non- trivial gains over using a single reward model. A more detailed illustration of this example can be found in Appendix F4
Similarly, in Figure at 44M) and plot the final performance for each method for the fixed training budget. The results again indicate that using ensembles provides additional improvement in addition to increasing the size of the training dataset.
9 we vary the size of the training dataset (keeping the reward model size fixed

---

Published as a conference paper at ICLR 2024
0.0
Cold Score
1.0
0.6
Single RM
0.5
Gold
0.4
UNO
WCO
0.0
0.0
44M
(a) BoN.
(b) PPO.
Figure 8: Final gold reward model performance achieved by different objectives when optimizing reward models with varying parameter sizes, but trained with the same dataset containing 25% label noise. The 1.3B PPO models are exceptionally trained for 6000 PPO steps, due to slower policy optimization rate with large reward models (Appendix
0.4
Gold Scoce
Gold Soore
Mean
0.4
0.0
0.2
-0.2
UWO
10k
20k
46k
1.kk
20k
46k
Data Size
Data Size
(a) BoN.
(b) PPO.
Figure 9: Final gold reward model performance achieved by different objectives when optimizing (44M) reward models trained under varying amounts of data. A label noise ratio of 25% was main- tained for all dataset sizes.
5.4 EVALUATING ROBUSTNESS TO HYPERPARAMETER
Using an ensemble-base based approach introduces an additional hyperparameter: the cardinality, or size, of the ensemble. In Figure T1 plot the final performance for BoN and PPO for the three ensemble-based approaches: mean optimization, WCO, and UWO. We note that while there is no- ticeable gap between 3-member and 4-member ensembles, in most cases the performance is highly similar for 4-member and 5-member ensembles, indicating that 4-member or 5-member ensembles are likely work best, and diminishing returns will seen after this point.
For UWO, the value of the uncertainty penalty is another hyperparameter. Our results in Figure 12 indicate that most reasonable values of uncertainty penalty actually work well, indicating that there is potentially no need to tune this hyperparameter.
5.5 EFFECTS OF LABEL NOISE AND UNCERTAINTY PENALTY
In Figure 10 we show intra-ensemble variance for an ensemble of 44M parameter reward models optimized via UWO and mean optimization objectives using PPO. The variance among the ensemble members starts small value in the no label noise case, and increases by relatively small amount during training for UWO. However, the uncertainty increases by almost times for mean optimization during training.
Yoein ezsecbble the Elps, no KL
UnO roneecile Ear flpe.
exacatée Tips
ersccble fips, UNO ensemste dwl 5.23 flps. KLO01
02
For the case of 25% label noise, the variance starts much higher and during the course of training, creases by about 2.5 times for mean optimization. However, the variance only increases by about 20% for UWO (using X 0.1). Further, using KL penalty of 0.01 results in a slight reduction in the variance at the end of training. This hints at the fact that while mean optimization is able exploit any reward model that overestimates the reward, the uncertainty penalty UWO prevents that resulting in better final performance (shown in Figure 7 and reduced overoptimization.
500
1000
1208
2000
2500
2000
Styt
Figure 10: Intra ensemble variance for different optimization objectives under different levels of noise for PPO.
8

---

Published as a conference paper at ICLR 2024
God Boore
0.6
0.4
Mean
WCO
0.2
Cold
0.2
UNO
0.0
0.0
Members
Members
(a) BoN.
(b) PPO.
Figure 11: Impact of the cardinality of the ensemble on the final performance of mean, UWO, and WCO for the 25% label noise case.
0.6
Gold Score
Cold Score
0.4
A
A 0.5
0.2
1=10
Members
Members
(a) BoN
(b) PPO.
Figure 12: Impact of uncertainty penalty weight final performance for different numbers of reward models within the ensemble. We note that there considerable robustness to the value of the uncertainty penalty.
DISCUSSION
We have demonstrated that ensemble-based conservative optimization methods improve perfor- mance and are highly effective in combating the overoptimization problem in RLHE. This work opens up the possibility of several interesting follow-ups. Firstly, future work should consider repli- cating findings in other environments (i.e. other RLHF datasets) and with larger-scale language models. Secondly, our setup is based on offline RLHF, in which human feedback is collected up- front and there are updates made to the reward model throughout the policy optimization process. This contrasts with online RLHF (Bai et al.) [022aa] Christiano et al. 2017), where reward models periodically retrained on freshly collected data from humans. Does ensembling result in similar gains in this setup as well or not?
7 RELATED WORKS
Overoptimization in RLHF: Several works (Ibarz et al. 2018 Ziegler et al. 2019 Stiennon et al,
2020 have provided anecdotal evidence of overoptimization in RLHF, However, the best of our knowledge, Gao et al. (2023) are the only ones who study it systematically within the setup of fine- tuning of L.Mss. Our work utilizes their setup, reproduces their results, and performs a systematic study evaluating the effectiveness of using ensembles for mitigating overoptimization.
Use of Ensembles in (Model-Based) RL: Ensembles are often used in deep learning as a tool to estimate uncertainty (Lakshminarayanan) et al.) 2017; Dietterich] 2000) and can often outperform more complex Bayesian deep learning methods (Ovadia et al. 2019). Learning an ensemble of dynamics models is especially popular in model-based reinforcement learning where often reliable uncertainty estimate over state space is critical for avoiding distribution shift (Depeweg et al.) 2016
Gal et al. 2016 Chua et al.) 2018 Yu et al.) (020a). Disagreement among ensemble members also a popular approach for driving the exploration
learning agents in an environment (Henafi) 2019 Pathak et al. 2019 [Shyam al. 2019 Lee al. 202T). Brantley al. (2019)
ensemble- based approach to estimate the support of expert policies in an imitation learning setup. Within language models setting, Gleave & Irving (2022) explore the use of ensembles for uncertainty- based active learning to improve sample efficiency of reward models. However, to the best of our knowledge, there no prior work that explores the use of ensembles for improving the robustness of RLHF; especially in the setting of language models fine-tuning.

---

Published as a conference paper at ICLR 2024
REFERENCES
Josh Abramson, Arun Ahuja, Federico Carnevale, Petko Georgiev, Alex Goldin, Alden Hung, Jessica Landon, Jirka Lhotka, Timothy Lillicrap, Alistair Muldal, et al. Improving multi- modal interactive agents with reinforcement learning from human feedback. arXiv preprint arXiv:2211.11602 2022. A
Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXhv:2204.0586, 2022a.
2.2
A
Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al. Constitutional ai: Harm- lessness from ai feedback. arXiv preprint arXiv:2212.08073. 2022b. A
Stella Biderman, Hailey Schoelkopf, Quentin Gregory Anthony, Herbie Bradley, Kyle O'Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, al. Pythia: A suite for analyzing large language models across training and scaling. In International Conference on Machine Learning, pp. 2397-2430. PMLR, 2023.
Stephen Boyd and Lieven Vandenberghe. Convex optimization. Cambridge university press, 2004.
Kiante Brantley, Wen Sun, and Mikael Henaff. Disagreement--eggllaiizee imitation learning. In
International Conference Learning Representations, 2019.
Stephen Casper, Xander Davies, Claudia Shi, Thomas Krendl Gilbert, Jérémy Scheurer, Javier Rando, Rachel Freedman, Tomasz Korbak, David Lindner, Pedro Freire, et al. Open problems and fundamental limitations of reinforcement learning from human feedback. arXiv preprint
arXiv:2307.15217. 2023. I
Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. Advances neural information processing sys- tems, 30, 2017. A
Kurtland Chua, Roberto Calandra, Rowan McAllister, and Sergey Levine. Deep reinforcement learn- ing in a handful of trials using probabilistic dynamics models. Advances in neural information processing systems, 31, 2018.
Jack Clark and Dario Amodei. Faulty reward functions in the wild. https://opena: com/ aeeearchhfaultyy-eear---ee
2016. A
Stefan Depeweg, José Miguel Hernández-Lobato, Finale Doshi Velez, and Steffen Udluft. Learning and policy search in stochastic dynamical systems with bayesian neural networks. arXiv preprint
arXiv:160..07127, 2016. 7
Thomas G Dietterich.
Ensemble methods in machine learning. International workshop on multi- ple classifier systems, pp. 1-15. Springer, 2000. 7
Hanze Dong, Wei Xiong, Deepanshu Goyal, Rui Pan, Shizhe Diao, Jipeng Zhang, Kashun Shum, and Tong Zhang. Raft: Reward ranked finetuning for generative foundation model alignment. arXiv preprint MrXXi:23044.0677. 2023.
Yann Dubois, Xuechen Li, Rohan Taori, Tianyi Zhang, Ishaan Guirajani, Jimmy Ba, Carlos Guestrin, Percy Liang, and Tatsumori Hashimoto. Alpacafarm: A simulation framework for methods that learn from human feedback. arXiv preprint 1435 2023.
Yarin Gal, Rowan McAllister, and Carl Edward Rasmussen. Improving PILCO with Bayesian neural network dynamics models. In Data- Efficiifi Machine Learning workshop. International Confer- on Machine Learning, 2016.
10

---

Published as a conference paper at ICLR 2024
Leo Gao, John Schulman, and Jacob Hilton. Scaling laws for reward model overoptimization. In International Conference on Machine Learning, pp. 10835-10866. PMLR, 2023. (document)
5.3
Adam Gleave and Geoffrey Irving. preprint arXiv:2203.07477 2022. 7
Uncertainty estimation for language reward models. arXiv
Caglar Gulcehre, Tom Le Paine, Srivatsan Srinivasan, Ksenia Konyushkova, Lotte Weerts, Abhishek Sharma, Aditya Siddhant, Alex Ahern, Miaosen Wang. Chenjie Gu, et al. Reinforced self- self-training (rest) for language modeling. arXiv preprint arXiv:2308. 08998, 2023. A
Mikael Henaff. Explicit explore-exploi algorithms in continuous state spaces. Advances in Neural Information Processing Systems, 32, 2019. 7
Borja Ibarz, Jan Leike, Tobias Pohlen, Geoffrey Irving, Shane Legg, and Dario Amodei. Reward learning from human preferences and demonstrations in atari. Advances in neural information processing systems, 31, 2018. 7
Andreas Köpf, Yannic Kilcher, Dimitri von Rütte, Sotiris Anagnostidis, Zhi-Rui Tam, Keith Stevens, Abdullah Barhoum, Nguyen Minh Duc, Oliver Stanley, Richárd Nagyfi, et al. Openassistant conversations demaorraiiddmmm large language model alignment. arXiv preprint arX1v:2304.073277,
2023. D.3
Victoria Krakovna, Jonathan Uesato, Vladimir Mikulik, Matthew Rahtz, Tom Everitt, Ra- mana Kumar, Zac Kenton, Jan Leike, and Shane Legg. Specification gaming: The flip side of ai ingenuity. April 2020. URL https: www. dwwpmi nd. com/blog/ frecification
theeiing-th---eee flie-flip-ss sepp-sside-ofe de- a ai-inginuitiyyyyy
Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. Advances in neural information processing systems, 30, 2017.
4.3
Kimin Lee, Laura Smith, and Pieter Abbeel. Pebble: Feedback-efficient interactive rein- forcement learning via relabeling experience and unsupervised pre-training. arXiv preprint arXiv:2/06.05091, 2021. 7
Joel Lehman, Jeff Clune, Dusan Misevic, Christoph Adami, Lee Altenberg, Julie Beaulieu, Peter J Bentley, Samuel Bernard, Guillaume Beslon, David Bryson, et al. The surprising creativity of digital evolution: A collection of anecdotes from the evolutionary computation and artificial life research communities. Artificial life, 26(2):274-306, 2020. A
Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, llya Sutskever, and Karl Cobbe. Let's verify step by step. arXiv preprint
arXiv:2305.20050, 2023. A
Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, et al. Self-refine: Iterative refinement with self-feedback. arXiv preprint vrXiv:2303.176511 2023. A
Timothy Prickett Morgan.
Counting the
of training large language mod-
cls. 2022.
https:: www. wwwp:.nexplaaafoo.. com/ 2022/12/01/ counting-thin cost of training large languaggggggeea htt.guaaggemmdeee//./ [
URL
Reichiro
Nakano, Jacob Hilton, Suchir Balaji. Jeff Wu, Long Ouyang, Christina Kim. Christo- pher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, et al. Webgpt: Browscr-assisted question-answering with human feedback. arXiv preprint 22xi1:2112.09332, 2021. 2.1
OpenAl. Openai models. httpps://plaafforooogrraaa
2023a. 4.1
OpenAl. Gpt-4 technical report. arXiv, pp. 2303-08774, 2023b. A
Long Ouyang, Jeffrey Wu, Xu Jiang. Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama. Alex Ray, al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35:
27730-27744, 2022.
2.1 2.2
11

---

Published as a conference paper at ICLR 2024
Yaniv Ovadia, Emily Fertig, Jie Ren, Zachary Nado, David Sculley, Sebastian Nowozin, Joshua Dillon, Balaji Lakshninarayanan and Jasper Snock. Can you trust your model's uncertainty? evaluating predictive uncertainty under dataset shift. Advances in neural information processing systems, 32, 2019. 3
Alexander Pan, Kush Bhatia, and Jacob Steinhardt. The effects of reward misspecification: Mapping and mitigating misaligned models. arXiv preprint arXiv:2201.0354: 2022.
Deepak Pathak, Dhiraj Gandhi, and Abhinav Gupta. Self-supervised exploration via disagreement. In International conference on machine learning, pp. 5062-5071, PMLR, 2019. 7
Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D Manning, and Chelsea Finn. Direct preference optimization: Your language model is secretly reward model. arXiv
preprint arXiv:2305. 18290, 2023. A
William Saunders, Catherine Yeh, Jeff Wu, Steven Bills, Long Ouyang, Jonathan Ward, and Jan Leike. Self-critiquing models for assisting human evaluators. arXiv preprint (rXi::2206005802, 2022. A
Jérémy Scheurer, Jon Ander Campos, Tomasz Korbak, Jun Shern Chan, Angelica Chen, Kyunghyun Cho. and Ethan Perez. Training language models with language feedback at scale. arXiv preprint arXiv:2303. 16755, 2023.
A
John Schulman. Approximating kl divergence, 2020. URL http:://joschu.eet/bb.. kl-approx. html
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy
optimization algorithms. arXiv preprint 1771v:1707.06347, 2017. 2.2
Pranav Shyam, Wojeiech Jaskowski, and Faustino Gomez. Model-based active exploration. In
International conference on machine learning, pp. 5779-5788. PMLR, 2019, 7
Joar Skalse, Nikolaus Howe, Dmitrii Krasheninnikov, and David Krueger. Defining and character- izing reward gaming. Advances in Neural Information Processing Systems, 35:9460-9471, 2022.
Nisan Stienmon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. Learning summarize with human feedback. Advances in Neural Information Processing Systems, 33:3008-3021, 2020.
2.2 4.3
A
Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Stanford alpaca: An instruction instruction-fllloiing llama model. https://gite com/
toco//tatee htt//tttsuu-lla labeuullabbseee httbb//tafffrdd_aaaaaa
2023.
D.2 D.2
Jonathan Uesato, Nate Kushman, Ramana Kumar, Francis Song. Noah Siegel, Lisa Wang, Antonia Creswell, Geoffrey Irving, and Irina Higgins. Solving math word problems with process-and outcome-based feedback. arXiv preprint 21Xiv:2211.14275, 2022. A
Abhinay Venigalla and Linden Li. Mosaic 1lms (part 2): Gpt-3 quality for < 500 k, 2022. URL https: w/tp://wwwww..c mosa cml www.cclw./oo/cccccooocoooooooooc gpt 3 3--uualtyy--- for 500k
Stephen J Wright. Numerical optimizatiio. 2006.
Jeff Wu, Long Ouyang, Daniel M Ziegler, Nisan Stiennon, Ryan Lowe, Jan Leike, and Paul Chris-
tiano. Recursively summarizing books with human feedback. arXiv preprint arXiv:2 10862,
2021a. A
Yue Wu, Shuangfei Zhai, Nitish Srivastava, Joshua Susskind, Jian Zhang, Ruslan Salakhutdinov, and Hanlin Goh. Uncertainty weighted actor-critic for offline reinforcement learning. arXiv preprint 10X.0:2105.08140. 2021b.
Wanqiao Xu, Shi Dong. Dilip Arumugam, and Benjamin Van Roy. Shattering the agent-ent interface for fine-tuning inclusive language models. arXiv preprint arXiv:2305. /455, 2023. A
12

---

Published as a conference paper at ICLR 2024
Tianhe Yu, Garrett Thomas, Lantao Yu, Stefano Ermon, James Y Zou, Sergey Levine, Chelsea Finn, and Tengyu Ma. Mopo: Mosel-based offline policy optimization. Advances Neural Information 2020a.
Processing Systems, 33:14129-14411
Tianhe Yu, Garrett Thomas, Lantao Yu, Stefano Ermon, James Y Zou, Sergey Levine, Chelsea Finn, and Tengyu Ma. Mopo: Model-based offline policy optimization. Advances in Neural Information Processing Systems, 33:14129-14142, 2020b.
Zheng Yuan, Hongyi Yuan, Chuanqi Tan, Wei Wang, Songfang Huang, and Fei Huang. Rrhf: Rank responses align language models with human feedback without tears. arXiv preprint arXiv:2304.05532, 2023. A
Yao Zhao, Rishabh Joshi, Tianqi Liu, Misha Khalman, Mohammad Saleh, and Peter Liu. Slic-hf: Sequence likelihood calibration with human feedback. arXiv preprint arXiv:2305.10425 2023.
A
Rui Zheng, Shihan Dou, Songyang Gao, Wei Shen, Binghai Wang, Yan Liu, Senjie Jin, Qin Liu, Limao Xiong, Lu Chen, al. Secrets of rlhf in large language models part i: Ppo. arXiv preprint arXiv:2307.04964, 2023, 2.2
Simon Zhuang and Dylan Hadfield- Menell. Consequences of misaligned ai. Advances in Neural Information Processing Systems, 33:15763-15773, 2020.
Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences. arXiv preprint arXiv:1909.088 08599, 2019.
13

---

Published as a conference paper at ICLR 2024
APPENDIX
A ADDITIONAL RELATED WORKS
RLHE; Reinforcement learning from human feedback (RLHF), in its modern form, was popularized
by Christiano et al. ) Ziegler al. 2019) the first work demonstrate the effectiveness of RLHF for tine-tuning language models. This led to several applications of RLHF to fine-tuning
LLMs focused on improving several aspects of language models e.g. instruction following (Ouyang
et al. 2022, Abramson et al. 2022). summarization capabilities (Stiennon et al. 20200 Wu et al.
2021a), web browsing (Nakano al. 2021), translation (Gulcehre et al. 2023) and helpfulness & harmlessness (Bai et al.) 2022aa. This has also caused great amount of interest in improving RLHF as a method. One main line of research here is methods that try to augment the learning signal for reward model in some form e.g. language feedback (Scheurer et al. 2023) process supervision (Lightman et al., 2023] Uesato et al., 2022). This, however, generally incurs additional costs in labeling. This has prompted research extracting feedback signal from pretrained language model via appropriate prompting (Bai et al. 2022h OpenAl 2023b Madaan et al. 2023 Saunders al. 2022 however, this requires effective prompting and highly capable language model to be successful. Another line of research focuses on the use of alternative algorithms to PPO for policy optimization step (Gulcehre
2023 Rafailov al. 2023 Zhao al. 2023 Yuan al.,
2023 Dong et al. 2023 Xu et al.) 2023). Orthogonal to both these lines of work, we explore using
ensembles to improve RLHE.
Reward Hacking: Overotpimization in RLHF is an instance of reward hacking. Many examples of reward hacking exist in literature (Clark & Amodei, 2016 [eehman al.) [0200 Krakovna al.) 2020). Skalse et al. (2022) provide formal definition for reward hacking and theoretically study whether unhackable proxies can exist or not under different conditions. [Zhuang & Hadfield-Menell 2020) study the reward hacking that might arise due to partial observability. Pan et al. (2022) provide examples proxy reward functions various reinforcement learning environments where low-capacity policies do not elicit reward hacking but policies with greater capacity do.
B BON ESTIMATOR
The naive way of estimating the average reward the BoN policy under gold reward model, when using proxy reward model the ranking function, is to a Monte Carlo estimate in which answers (Ai,.., A2} are sampled from a language model f and then scored for a given prompt g as follows:
REOOy (ala)le) ]
Rgold (q) = A2..A... Rgold
arg max
(6)
A2...A
This is very wasteful as does not reuse answers for different values n. Therefore, we instead use the following unbiased estimator (Nakano al. 2021) that samples N 2 Aman outputs for given input q:
Rgold
Σ
Rgold
argmax
Rproxy (a (a14) 1
oE (Aq......,,
This can be computed efficiently by first sorting all answers A AN under the proxy reward model to obtain scores Si,... SN and then computing:
N = Σ imn
Σ
Rgold
argmax (s..........
R pproxy
(a (a19)1g
Rgold (S; ())
(8)
15H SEICCEEEE
To get the gold reward model score for a given range of values n = 1,2, ..., Pmanx we simply evaluate the empirical average of the above estimator for all questions for each
14

---

Published as a conference paper at ICLR 2024
C KL DISTANCE CALCULATION FOR PPO
The naive way to calculate KL distance between the PPO-optimized policy TRL and the pretrained model follows:
TRL
KLaI((TRL Tinit)
log
However, this estimator has high variance and can be negative, unlike actual KL. Therefore, we use the following estimator (Schulman 2020):
27
TRL
KL (TRL: miiis)
log
(10)
Winit
D ADDITIONAL EXPERIMENTAL DETAILS
D.1 HYPERPARAMETERS
We give the hyperparameters here for different components of our RLHF pipeline:
Table 1: SFT hyperparameters.
Parameter
Value
Learning rate
8e-6
Epochs
3
Batch size
4
Table 2: RM hyperparameters.
Parameter
Value
Learning rate
le-5
Epochs
5
Batch size
32
Table 3: PPO hyperparameters.
Parameter
Value
Learning rate
le-6
Cosine annealing scheduler
1e-7
PPO epochs
4
Batch size
32
Number of rollouts
256
Chunk size
32
Clipping range & value
0.2
GAE lambda
0.95
D.2 ALPACAFARM DATASET DETAILS
The AlpacaFarm dataset Dubois et al. 2023 employed in experiments uses the Alpaca data (Tori et al. 2023) made up of 52,000 samples. This data is chosen due to its large size and success in training instluction-folloiing models, AlpacaFarm contains five splits: labeled 10k "sft" split for
15

---

Published as a conference paper at ICLR 2024
Table 4: Generation hyperparameters.
Parameter
Value
Max instruction length
520
Max new tokens (answer length)
256
PPO epochs
Top-p
0.9 (1.0 for PPO training)
Top-k
Temperature
1.0
Table 5: Example answer generation and reward modeling prompts with proper formatting.
Answer generation prompt
Reward modelling prompt
|prompter| >>Whtt >What geometric shape has sides and
</prompter| >What geometric shape has 5 sides and ingles?<lendoftexs |> <lassistant geometric shape is a
angles?<lendofters /|asssstant|> |>
prrmpteer|Anaall the fol lowing sentence and identify the verb and its tense, \nShe played the piano. |endoftex| < <lassistant

Anal following sentence and identify the verb and its tense. \nShe played the piano.< </eoofftett Ilassistant/>Verb. Played Tense: fast</endofter
the
supervised fine-tuning, a 10k "pref" split containing pairwise preference labels, 20k "unlabeled" split for training algorithms such as PPO, 2k validation split, and an unused 10k split.
D.3 PROMPT FORMAT
Though we use instructions from the AlpacaFarm dataset, this only provides content for prompting the LLM and still requires formatting. We opt for minimalism, following the v2 format used in OpenAssistant (Köpf et al.) 2023). This format follows the GPTNexXToxenizer class used to pretrain our LLMs and introduces two special tokens: < <prompter| and < assistant >>.
For answer generation, the model should be prompted with the instruction and the input. Inputs, should they be present, are appended to the instruction after a new line to form the prompt. The prompt then prepended with the prompter lendoftext
token and closed off with an end-of-text (EOT) token declaring the end of the instruction, and the < /assssstntt token to start the answer. An example is shown in Table
For reward modeling, the prompt should also contain an answer to evaluated. In this case, the an- swer text (from an AlpacaFarm dataset demonstration to the initial prompt containing the instruction and closed off with the EOT token. This forms the full RM prompt, with an example shown in Table 5
previous answer generation) is appended
16

---

Published as a conference paper at ICLR 2024
E REPRODUCTION OF RESULTS OF GAO ET AL. (2023)
We also successfully reproduced the results of Gao et al. our setup shown in Figures 13 and 14 Note that our setup uses open-source models derived from the Pythia suite (Biderman et al. 2023 as the base for policy and proxy reward models and the AlpacaFarm human-preferent reward model (Dubois et 2023 used as the gold reward model. In contrast, Gao et al. used closed-source models based on GPT series. Our successful reproduction of their results in our setup hints at the general nature of overoptimization.
0.5
1.0
7M
10k
44M
0.4
20k
0.8
1.3B
48k
Gold Score
Gold Score
0.3
0.6
0.2
0.4
0.2
0.1
0.0
0.0
Divergence
Divergence
(a) BoN results various reward model sizes. with data size held constant (46k).
(b) BoN results various reward model training data sizes, with RM size held constant (44M).
Figure 13: Gold reward model scores for BoN follow similar trend that observed by Gao et al. (2023) when varying reward model and data sizes. Runs are averaged over five seeds, with standard deviation additionally shown. 25% label noise is used here to highlight overoptimization, found it difficult to observe with our limited BoN optimization capability. This is in line with label noise ablation experiments from Dubois et (2023)
0.5
0.4
10000
46000
0.3
Cold Score
0.2
0.1
-0.1
100 120
Kl. Divergence
Figure 14: PPO optimization results for various reward model training data sizes, with reward model size held constant (44M). Average and standard deviation over three runs are shown. A similar trend to Gao et al. (2023) is observed, with greater overoptimization as optimization is pushed further.
17

---

Published as a conference paper at ICLR 2024
F ADDITIONAL RESULTS
F.1 RM VALIDATION Loss CURVE
0.72
0.70
Validation Loss
0.68
0.60
0.64
0.62
0.60
500
1000
1500
Steps
Figure 15: Reward model validation loss during training for 44M reward models with no label noise. Reward models are trained for 5 epochs, with each epoch lasting 359 steps. We note that while there a minor variation in validation loss, this variation is not predictive of whether the reward model robust overoptimization not. For example, in Figure [6] the highest validation loss does not suffer overoptimization but other reward models with lower validation losses do.
that the reward model with
18

---

Published as a conference paper at ICLR 2024
F.2 INDIVIDUAL REWARD MODEL TRAINING OPTIMIZATION PLOTS
For the single reward model results, we optimize each reward model separately and present their average Figures aandd Here, we present the training curve for each reward model separately.
0.5
0.4
Gold Score
Gold Score
0.4
0.3
0.2
0.2
0.0
2
8
0
KL Divergence
Divergence
(a) No label noise.
(b) 25% label noise case.
Figure 16: BoN for 44M reward model. The average of these curves is presented in Figure
0.5
0.4
Cold Scoce
0.4
Aui
Gold Score
0.3
N
0.2
0.2
0.1
0.0
-0.1
50
100 125
100 123
KL. Divergence
K1. Divergence
(a) No label noise.
(b) 25% label noise case.
Figure 17: PPO results for 44M reward model. The average of these curves is presented in Figure
19

---

Published as a conference paper at ICLR 2024
F.3
7M REWARD MODEL TRAINING OPTIMIZATION RESULTS
0.6
RM Type
0.5
1.50
Single RM
2.0
Gold Store (
Mean
I
0.4
1.25
WCO
UWO,
1.5
Proxx 5007
1.00
Proxy Score
0.3
1.0
0.2
0.75
RM Type
0.2
0.1
Single
0.50
0.5
Mean
0.25
0.0
0.0
-0.1
JWO, 0.1
0.00
0
2
4
20
40
60
80
KL Divergence
KL Divergence
(a) BoN.
(b) PPO.
Figure 18: BoN and PPO results with the 7M reward models to supplement the bar plot in Figure 8
F.4
1.3B REWARD MODEL. TRAINING OPTIMIZATION RESULTS
1.25
RM Type
2.5
Single
Gold Score (
1.00
MeeT
2.0
WCO
0.75
UWO,
1.5
Score
0.50
1.0
0.25
0.5
0.00
0.0
4
8
Divergence
(a) BoN.
1.75
1.2
0.8
Ma
1.50
0.5
1.0
1.25
0.6
Gold Score
0.4
0.3
Froxy ma550
0.a
Gold Score
0.4
0.75
1.00
5000
0.2
RM Type
Single
0.4
KL Penalty
Single Meen
0.50
0.1
Mean
0.2
0.2
0.25
0.0
WCO
WCO
UWO, 0.1
0.0
UWO, 0.1
0.00
KL. Divergence
KL. Divergence
(b) PPO, 3000 steps.
(c) PPO, 6000 steps.
Figure 19: BoN and PPO results with the 1.3B reward models to supplement the bar plot in Figure Although all PPO experiments run for 3000 steps, here also include experiment with 6000 steps for the 1.3B reward model, due to the policy being optimized at a slower rate for large reward models. Differences later steps of PPO training. In particular, the difference in performance between the single RMSs and ensemble methods is clearer when the policy is trained for longer.
be observed at higher KL divergence, which is generally reached in the
20

---

Published as a conference paper at ICLR 2024
F.5
ADDITIONAL No LABEL NOISE RESULTS
0.6
Gold Score
Single
0.4
Mean
WGD
0.2
0.0
0.001
0.05
Figure 20: Effectiveness of conservative optimization methods across KL penalty weights in the no label noise case.
0old Score
0.6
Single RM
Mean
0.4
WCO
0.0
7M
44M RM Size
1.38
0.0
7M
44M
1.38
RM Size
(a) BoN
(b) PPO
Figure 21: Final gold reward model performance achieved by different objectives when optimizing reward models with varying parameter sizes, but trained with the same dataset without label noise.
0.6
Singla
Gold Sccpe
0.6
Gold Soorre
0.4
Mean
WCO
UWO
0.2
0.0
0.0
10k:
20k
48k
10k
20k
46k
Data Size
Data Size
(a) BoN
(b) PPO
Figure 22: Final gold reward model performance achieved by different objectives when optimizing (44M) reward models trained under varying amounts of data. No label noise is used in the data.
0.5
Gold Score
0.4
Mean
WCO
0.4
Cold
0.2
0.0
0.0
Members
Members
(a) BoN.
(b) PPO.
Figure 23: Impact of the cardinality of the ensemble on the final performance mean, UWO, and WCO for the no label noise case.
21

---

Published as a conference paper at ICLR 2024
0.6
cold Score
A 0.05
A-0.1
0.4
0.2
0
A=1,0
0.0
0.0
BoN.
(b) PPO.
Impact of uncertainty penalty weight on final performance for different numbers of reward models within the ensemble. We note that there considerable robustness the value of the uncertainty penalty. No label noise is included in the data.
Figure 24:
F.6
35% LABEL NOISE RESULTS
0.5
0.6
RM Type
3.0
2.0
Single RM
0.4
Gold Score (
Mean
2.5
1.5
0.4
WCO
2.0
0.3
Spone
UWO, 0.5
0.3
1.5
0.2
Type Stngla RM Meun
1.0
0.5
0.0
WCO
0.0
UWD, A 0.1
0.0
25
50
75
100
125 150
KL. Divergence
KL. Divergence
(a) BoN
(b) PPO.
Figure 25: BoN and PPO results for 44M reward models with KL penalty 0.01, when there 35%
label noise in the reward model training data. Results show that the increase in performance and overoptimization mitigation of ensembles is robust to different noise level.
F.7 LONGER PPO RESULTS
0.6
Gold Score
0.4
0.2
RM Type Single RM
0.0
2000
4000
6000
8000
10000
Steps
Figure 26: PPO results for 44M reward models with KL penalty 0.01 and 25% label noise. Opti- mization extended 10k steps this experiment. These results show the stability of ensemble methods even for high levels of optimization. Thus, our ensembling removes the need for early stop- ping and the risk of overoptimziation. It would seem that no matter how far the model is optimized, the performance will stay close to optimal.
22

---

Published as a conference paper at ICLR 2024
F.8
FULL UNCROPPED KL DIVERGENCE PPO RESULTS
0.4
Gold Score
0.3
0.2
KL Penalty
0.1
0.001
0.1
0.005
0.2
0.0
0.01
50 100
150
200
250
Divergence
Figure 27: Individual reward model optimization (i.e. ensemble) for different values of KI. penalty in the 25% label noise case. This figure does not crop the x-axis at 150 KL divergence, to illustrate the behavior of Fig. beyond this point. As mentioned in Section 5 we usually stop our optimization curves at 150 KL divergence because the overoptimization trends are already visible, and doing so provides consistency across figures. Moreover, the plot clearer as isn't made narrow by more extreme values.
F.9 WIN-RATE EVALUATION OF ENSEMBLE METHODS
As an additional measure of quality, Tables and 7 show the win-rate of final policies trained with the different ensemble methods, when compared with the single reward models, across multiple
reward model scales. These tables correspond to runs shown earlier in Figures 3 4 18 and 19] For BoN, we use the unbiased estimator at n 12500 and compare gold scores between each method and the single reward models for each prompt. For PPO, we compare the gold scores of answers generated by the policy after 3000 PPO training steps. Win-rate is calculated by averaging the individual win-rates against each of the five single reward models, with standard deviation also presented. Results for PPO are additionally averaged over the three PPO seeds.
Table 6: Final policy win-rates (in percent) of ensemble methods against single reward models, when using 44M reward models. Standard deviation across the five single reward models is indicated. Models correspond to those shown in Figures and
BoN
PPO
No noise
25% noise
No noise
25% noise
Mean WCO
54.9 0.9
57.8 +
58.1 3.3
60.2 3.5
57.1 + 0.6
58.3 + 0.8
59.4 + 3.3
62.2 + 2.9
UWO A 0.5
57.2 0.9
58.2 + 1.0
60.2 + 3.4
63.0 + 3.1
Table 7: Final policy win-rates (in percent) of ensemble methods against single reward models, when using smaller (7M) and larger (1.3B) reward models. Standard deviation across the five single reward models is shown. Noise level is fixed to 25%, such that models correspond to those in The 6000-step final policy is used for PPO in the 1.3B reward model case.
Figures 18 and 19
BoN
PPO
7M RM
1.3B RM
7M RM
1.3B RM
Mean
57.7 + 1.4
72.3 + 1.0
50.8 3.4
56.2 3.5
WCO
58.8 + 1.4
71.2 4 1.2
57.9 3.7
62.9 2.9
UWO = 0.5
61.3
73.8 4 0.8
59.0 3.7
63.1 4.2
23

---

Published as a conference paper at ICLR 2024
F.10 QUALITATIVE SAMPLES
Quantitative metrics are used throughout our experiments as they enable the rapid and comprehen- sive evaluation of various methods. Here provide brief insight into the qualitative aspect of the methods discussed in this work. For a given prompt, answers from the final policy (n = 12500 for BoN, and 3000 steps for PPO) of each method are provided. Policies from the runs in Figures 3 and which provide the strongest final results, are used.
The main takeaways are follows. First, the BoN answers are shorter, because policy optimization does not occur in the same way it does for PPO. Second, single reward models show signs of failure and overoptimization. For BoN, this translates to poor answers and even hallucinations in the case of T For PPO, this also translates to poor answers, which are very long and highly repetitive. These are clear instances of overoptimization, where the reward model seems to have learned to produce long answers regardless of the content, which offers good proxy reward model performance but poor gold model performance. Finally, we note that the conservative optimization methods produce strong qualitative results, even when single reward models are struggling.
Table 8: First set of example answers to evaluation prompt.
Prompt
Find the closest airport to the given location.
Seattle, Washington
Single RM (BoN)
The closest airport to the given location is Sea-Tac Airport, with a direct flight from SFO airport in San Francisco, CA Seattle- Tacoma International Airport available on up to dates.
Mean (BoN)
The nearest airport to Seattle, Washington is Sea-Tac Interna- tional Airport (SEA). It is located a distance of 17.6 miles away and has an estimated time of arrival of around 12:30 PM.
WCO (BoN)
The closest airport the given location is Tacoma Inter- national Airport (SITE). is located approximately 10.7 miles away from the given location.
UWO (BoN)
The closest airport the given location is Seattle-Tacome Inter- national Airport. It is located 23.7 miles away, and the estimated time of arrival 2 hours, 28 minutes.
Single RM (PPO)
The closest airport the given location Seattle-Tacoma Inter- national Airport, located in the Seattle, Washington, region. It is a major airport with a major hub, serving over 15 million pas- sengers each year. It is located just 10 miles from the city center and is connected to the city by the Bunker Hill and Space Needle commuter rail lines. is the primary airport for the region and is one of the most important hubs in the country. It is major hub for variety of airlines, and is popular destination for travel- ers from around the world. It is a hub for a variety of domestic and international flights, and is the main hub for the region. is a major hub for several airlines, and is a popular destination for travelers from around the world. It a major hub for variety of domestic and international flights, and is the primary hub for the region. is a major hub for variety of airlines, and a popular destination for travelers from around the world. It is major hub for variety of domestic and international flights, and is the main hub for the region. It is a major hub for variety of airlines, and is a popular destination for travelers from around the world.
Continued on the next page
24

---

Published as a conference paper at ICLR 2024
Prompt
Find the closest airport to the given location.
Seattle, Washington
Mean (PPO)
The closest airport Seattle is Boeing Field, located in Seattle, Washington. It is served by several airlines, including Alaska Airlines, United Airlines, Delta Air Lines, and American Air- lines. It is located approximately 35 miles away from the given location. The airport is also served by several car rental compa- nies, including Hertz, Avis, and Budget.
WCO (PPO)
The closest airport to the given location is Seattle-Tacoma Inter- national Airport (SEA). is located approximately 43 miles from the location. The average time to get to the airport is around 2 hours. The cost to get to the airport around $110 for a single- occupancy It is the most popular airport in the area and is served by many airlines. The airport offers a variety of services, including baggage claim, exchange. There are also a number of attractions nearby, includ- ing the Space Needle, the Aquarium of the Pacific, and the Seattle Center. The airport is well-served by public transportation, mak- ing it easy get around.
rental counter, and currency
UWO (PPO)
The closest airport the given location is Seattle-Tacome In- ternational Airport (SEA). The airport is located in the Seattle- Tacoma, Washington, area and served by many domestic and international airlines. It is located about 70 miles east of the city of Seattle and is accessible by car, bus, train, and ferry. The air- port has several terminals, including domestic terminal, ternational terminal, and a car rental center. The airport has two runways, allowing for both domestic and international flights. The airport is served by many international carriers, including Alaska, American, Delta, United, and Virgin. It is also home many domestic carriers, including American, Delta, and United.
in-
Table Second set of example answers to an evaluation prompt.
Prompt
Create an analogy explain cloud computing.
Single RM (BoN)
Cloud computing is like the sky; it is a vast, untamed expanse filled with variety of colorful shapes and textures. Just like clouds move and shift based on the weather, so too does cloud computing. It is constantly evolving and adapting meet the demands of modern technology.
Mean (BoN)
Cloud computing is like a sprawling house, with multiple rooms and floors that each represent specific type of resource. Each room serves a specific purpose, such as storage, processing, or network access, and each floor has its own features such stor- age, computing power, or network connectivity. The cloud pro- vides a central hub where resources can accessed from any- where in the house, and enables users to access the resources they need quickly and easily.
Continued on the next page
25

---

Published as a conference paper at ICLR 2024
Prompt
Create an analogy to explain cloud computing.
WCO (BoN)
Cloud computing is like a giant server room in your home, with multiple computers connected together to store, process, and trieve data. The computers are like small servers in the server room, performing different tasks such as running virtual ma- chines, performing analytics, and performing other tasks. Each computer connected the server, and the data stored in the server is retrieved and used by the computers. The data is stored a database, allowing data to quickly and easily accessed, updated, and processed.
UWO (BoN)
Cloud computing is like stack of pancakes; each pancake is a machine that processes data and provides computing power. When you want to make pancakes, you simply select the ingre- dients, put them in a pan, and flip them to make pancakes. Simi- larly, when you use cloud computing. you select the services you need to make use of, then pay a monthly fee to use them. When you're done with your work, you can shut down the machine and delete the files, or if you need more power, you can simply pur- chase more power and pay a recurring fee make use of
Single RM (PPO)
The analogy that of a computer being a network of intercon- nected servers, each with its own processing power and mem- ory, and the cloud as the central hub where the data stored, allowing the various servers to access the data in more efficient and cost-effective manner. is the cloud that allows the various servers be able access the same data in the the data is stored and managed in the cloud, thus allowing the various servers to work together cost-effective solution. It is the cloud that is the backbone of the modern technology and it is the ability to access the data from the various servers, in more cost-effective and efficient manner, that makes cloud computing the ideal solution for the modern technology. It is the cloud that is the future of technology and it is the ability to access data from the various servers, a more cost-effective and efficient manner, that is the foundation of the modern technology. It is the cloud that is the new standard and it is the ability to access the data from the various servers, in a more cost-effective and efficient manner, that is the foundation of the future of technology. It is the cloud that is the future of technology, and it is the ability
way, and
provide more efficient and
Mean (PPO)
Cloud computing is like giant cloud of air that provides all of the computing power needed run a business. The cloud pro- vides access to the necessary computing resources to enable busi- nesses to remain competitive and stay ahead of the competition. Cloud computing is a cost-effective way to access the computing power needed to run business, and can reduce the time it takes to develop and deploy new software. It also allows businesses to scale their operations quickly, with the ability to quickly add or remove resources needed. Cloud computing is great way for businesses to remain competitive and stay ahead of the competi- tion.
Continued on the next page
26

---

Published as a conference paper at ICLR 2024
Prompt
Create an analogy explain cloud computing.
WCO (PPO)
Cloud computing is like a huge, open room with many different walls and floors. The walls provide security and privacy, while the floors provide resources and data access. The ceiling is the sky, and the sky the sky, providing a vast amount of resources, power, and opportunities. The cloud is a room full of resources, and the resources are the people, providing diverse community of expertise and resources. The walls, floors, and sky are all im- portant, and together they provide an invaluable resource. This analogy helps to explain the importance of cloud computing. and how it can be beneficial the power of the cloud and the opportunities it provides, allowing organizations to scale quickly and efficiently. also illustrates the importance of providing secure, reliable, and cost-effective infrastructure.
organizations of all sizes. shows
UWO (PPO)
Cloud computing is like a modern-day library, where data is stored and processed on a centralized server. The data accessed and manipulated from any device connected to the internet, al- lowing for faster and more efficient services. The data stored in the cloud and is accessible from any device, allowing for the user access the data quickly and efficiently. The cloud comput- ing platform is easy to use, provides the user with the flexibility to access the data quickly and easily, and cost effective. The cloud computing platform provides the user with the ability cess the data quickly and easily, allowing them to make decisions based on the data. The cloud computing platform also provides the user with the flexibility to access the data quickly and easily, allowing them to make decisions based the data. This allows the cloud computing platform to be cost effective solution for the user, while providing them with the ability to access the data quickly and easily.
27


Reviews:
[{"review_id": "hXS0FzPvnv", "reviewer": "Reviewer_83ty", "summary": "This paper studies the overoptimization in the reward model for the RLHF problem and applies several methods to reduce the overoptimization in RLHF. Overall the reviewer believes this submission is above the acceptance bar, but it has some writing issues that can be further improved.", "questions": "1. What does the KL divergence in the x-axis in Figure 2 mean? The reviewer understands [1] also contains similar figures that study the gold score against the KL divergence, and the reviewer believes the authors are using a similar setting. However, the reviewer is unclear which KL divergence is Figure 2 referring to. The authors could either update the caption of Figure 2 or provide a better description of the x-axis to improve the readability.\n2. Figure 3 and Figure 4 also have the same issue discussed – perhaps the authors can clarify what is the KL divergence in both figures and why they scale differently. E.g., in Figure 3 the x-axis is from 0-8, while in Figure 4, the x-axis is from 0-150. The reviewer understands that [1] has similar figures, but it will be better if the authors clarify this, so that other readers do not need to refer back to [1] for understanding this submission. Similarly in Figure 5 - 7.\n\n[1] Gao, Leo, John Schulman, and Jacob Hilton. \"Scaling laws for rew", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The experimental results look promising.\n2. The overoptimization problem is a novel and important problem to the LLM community.", "weaknesses": "1. [Major] There are some writing and presentation issues in the manuscripts. While this manuscript extensively refers to [1] in the writing, the reviewer would recommend the authors update the paper so that the current submission does not require the readers to read [1] to understand the submission thoroughly. See detailed comments in [Questions].\n2. [Minor] For paragraph Supervied Fine-tuning in Section 4.3, the hyperlink `(see Section 4.1 for details)` seems to be broken. In addition, the reviewer cannot find any more details in section 4.1 for splitting the AlpacaFarm dataset, perhaps the authors can clarify this?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Htdn0ZBtI2", "reviewer": "Reviewer_mVSG", "summary": "This paper investigates the issue of reward model overoptimization in Reinforcement Learning from Human Feedback (RLHF), a technique used to fine-tune large language models. Overoptimization occurs when learned reward models, imperfect representations of true human preferences, lead to undesirable behavior. This paper builds on previous work by Gao et al. (2023) who demonstrated the persistence of overoptimization even with large “gold” reward models and extensive training data. The paper explores the effectiveness of ensemble-based conservative optimization (WCO and UWO) in mitigating overoptimization when using BoN sampling and PPO optimization methods. Additionally, the research extends the previous setup by introducing 25% label noise to better simulate real-world conditions. The findings reveal that conservative optimization significantly reduces overoptimization, improving performance by up to 70% for BoN sampling. For PPO, ensemble-based conservative optimization outperforms single reward model optimization and minimizes overoptimization when combined with a KL penalty, without sacrificing performance. In summary, the authors show that ensemble-based conservative optimization presents a promising approach for addressing the challenge of overoptimization in RLHF, leading to more robust and reliable fine-tuning of large language models.", "questions": "Please clarify and investigate the two questions presented in the weakness section.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This paper provides extensive empirical evidence that suggests that ensemble-based methods can improve robustness of RMs and reduces overoptimization, which makes the claims of the paper well-supported.\n\n2. The paper studies the important problem in RLHF, i.e. reward overoptimization, and presents various methods that clearly mitigate such a challenge. I think the paper is of value to the RLHF community.", "weaknesses": "1. While the empirical results are quite comprehensive in the paper, the model size seems a bit small with the biggest RM being 1.3B. Given Figure 8, it seems that the gain of the ensemble-based methods diminishes as the model size increases. It would important to investigate if ensemble-based methods have little gain with even bigger models, which are more commonly used by users.\n\n2. From Figure 9, it seems that with bigger dataset size (46K), ensemble-based methods are not that much better than the single RM. Is it expected that with even more data, which in practice is usually true, the difference between ensemble-based methods and single RM will further shrink? It would be helpful to further study this perspective.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "dB1x1MX6FX", "reviewer": "Reviewer_249u", "summary": "This paper addresses the problem of reward model optimization in training of LLM chatbots: when optimizing a learned reward model initially improves the chatbot's performance under \"true\" human preferences but eventually the performance decreases as the learned reward ceases to be a good proxy. The authors propose training an ensemble of reward models instead of a single one to mitigate this. The ensemble is trained from different random seeds and then the optimization target is either the mean of the reward models' outputs, the minimum (WCO), or the mean minus a variance penalty (UWO). A number of experiments are performed on best-of-n (BoN) and PPO optimization of reward models using these optimization targets and they are compared to using the output of a single reward model. The results suggest that using an ensemble of reward models", "questions": "Related to the weaknesses above:\n * What are the results of running PPO across 3-5 random seeds for the various optimization objectives? How much variance is there across seeds?\n * How much does the gold reward increase from using an ensemble correspond to in terms of win rate or other measures of quality?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "The paper addresses a very important problem—one of the main bottlenecks to improving RLHF training and safety of LLM-based chatbots is making reward models more robust. The solution technique is not particularly novel, as pessimistic optimization using an ensemble has been widely used in model-based RL, offline RL, preference learning, etc. However, I don't know of prior work that has specifically evaluated this technique for RLHF on LLM-based chatbots. Thus, I view the primary contribution of this paper as a systematic empirical study of applying this existing technique to a new problem. I think this is an important contribution but it does mean the experiments should be carefully executed.\n\nIn general, the experiments do seem to be well-done, exploring the effects of using the different ensemble-based optimization objectives in a variety of settings. Using WCO and UWO seem to pretty consistently help; while they don't allow for completely removing KL regularization in PPO, they do allow PPO to reach a higher final gold reward.\n\nThe writing is in general quite clear and the paper is easy to follow.", "weaknesses": "In terms of the experiments, one weakness is that the PPO experiments seem to be mostly done with a single random seed, while due to the high noise in RL training it is best to use a few random seeds (see https://arxiv.org/abs/2108.13264, https://arxiv.org/abs/2304.01315).\n\nAnother weakness of the results is that it's hard to know how to interpret the gold reward. How much better is an LLM with an average gold reward increase of 0.5 vs. 0.4? AlpacaFarm and others use a win-rate which is more interpretable—it might be helpful to report that for your results as well so that it's easier to understand exactly how much better using an ensemble is.\n\nThe way the PPO results are presented analogously to Gao et al. also feels a bit strange. According to Figures 4 and 5, the ensemble-based optimization objectives only outperform a single reward model at the very end of PPO training. I'm not sure why it's important to see all the intermediate KL and reward values when early-stopping is not generally used with RLHF PPO training. I found Figure 7 to be the best comparison of the different optimization objectives. It would be particularly interesting to see a variation of Figure 7 where each the sqrt(KL) and gold reward is plotted for each combination of (optimization objective, KL coefficient) since the KL coefficient is the actual \"knob\" used to adjust how much optimization is occurring in practice (not the amount of training time).\n\nHere is one paper that should probably be included in the related work as they also evaluate using ensembles of LLM-based reward models: https://arxiv.org/pdf/2203.07472.pdf. Their paper is focused on active learning, not reward optimization, so it's not directly comparable, but it does relate to your conjecture that \"uncertainty estimates from the ensemble may also help improve sample efficiency of human feedback in this setup\" on page 9.\n\nSmall issues/typos:\n * Bottom of page 5: in two places you say \"for PPO, WCO, and UWO\" but the second comma should be removed", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "lIw7EDN3HP", "reviewer": "Reviewer_PF7E", "summary": "The authors tackles the over-optimization issue in RLHF with reward model ensembles. This is achieved by training multiple reward models, each with identical data but different random seeds. These reward models are then used for ensemble-based optimization objectives, including worst-case optimization (WCO) and uncertainty-weighted optimization (UWO) for best-of-n sampling (BoN) and PPO (proximal policy optimization). Combined with a 25% label noise to mirror real-world settings, the authors show UWO and WCO can effectively mitigate overoptimization.", "questions": "> we use the complete dataset of 46, 000 prompts for reward model training and train all the reward models for 5 epochs. \n\n>We use all the available training data for the training of every reward model as\ntraining on lesser data results in higher validation loss and poorer performance (Lakshminarayanan\net al., 2017). Unless stated otherwise, we train an ensemble consisting of five reward models.\n\nWhy train for 5 epochs? What is the training and the validation accuracy of the reward model? In Figure 15, can you show me the epoch as the x-axis? Steps can be confusing as it is related to the batch size / gradient accumulation / how you log.\n\n> we only evaluate BoN for a maximum of nmax = 12, 500 samples , which roughly equals 8.4 nats of KL\n\nWhy nmax? Does this mean for some of them he use less than 12500 samples?\n\n> we train for 3000 PPO steps.\n\nHow many episodes?\n\n\n> Figure 3\n\nWhy do you have two y-axis? Do the proxy RM and the gold RM have different RM scales? How are these scales defined?\n\n\n> Figure 6 and 7 \n\nFigure 6 and 7 seem contradictory? KL penalty = 0 gets 0.15 gold score, but in figure 7 KL penalty = 0 gets 0.03 gold score?\n\n> Figure 8\n\nWhy does PPO underperform BoN in 1.3B setting according to Gold Score? Gao et al (2023) show 1.2B PPO outperforms 1.2B BoN\n\n> Appendix C\n\nHow does these two KL distance calculation Gold RM performance?\n\n\n> We train a fixed number of proxy reward models using\nidentical data and hyperparameters but with different random seeds\n\nBut I assume data shuffling is different? There are really multiple random seeds:\n\n* query dataset seed\n* reward model seed\n* policy model seed\n\n\n> Figure 4\nWhy is single RM experimented with KL=150 but not the other types?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "* **Simple yet elegant approach**: the authors only used different random seeds to train the reward function, yet it can significantly improve gold score performance, especially with BON.\n* **Reproduced overoptimization with OSS (open-source software) models**: the authors are the first to study and reproduce RM overoptimization with open-source models.\n* **Experiments across multiple model sizes / data sizes**: the authors conducted a comprehensive analysis, which offers insights.", "weaknesses": "**Lack of account for random seeds**: the results do not look smooth for Figure 4 / Figure 8. The results could be improved by running with at least 3 random seeds and recording the error bars.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)