A Long Way to Go: Investigating Length Correlations in RLHF
Authors: Prasann Singhal
Tanya Goyal
Jiacheng Xu
Greg Durrett
Venue: COLM (2024)
Keywords: Natural Language Processing, Large Language Models, RLHF, Reward Hacking
TL;DR: Many of the gains in open RLHF work, particularly due to flaws in reward modeling, are attributable to length

Abstract: Great success has been reported using Reinforcement Learning from Human Feedback (RLHF) to align large language models, with open preference datasets enabling wider experimentation, particularly for "helpfulness" in tasks like dialogue and web question answering. Alongside these improvements, however, RLHF also often drives models to produce longer outputs. This paper demonstrates, on three diverse settings, that optimizing for response length is, much more than previously thought, a significant factor behind RLHF. Studying the strategies RL optimization uses to maximize reward, we find improvements in reward to largely be driven by increasing response length, instead of other features. Indeed, we find that even a *purely* length-based reward reproduces most downstream RLHF improvements over supervised fine-tuned models. Testing a comprehensive set of length-countering interventions, we identify the dominant source of these biases to be reward models, which, by studying training dynamics, we find are non-robust and easily influenced by length biases in preference data.

Published as a conference paper at COLM 2024
A Long Way to Go: Investigating Length Correlations in RLHF
Prasann Singhal
Tanya Goyal
The University of Texas at Austin prasanns$cs wtw.us.exxae....o edu
Princeton University
tanyagoyal hat.gooyyl@piinntttr.
Jiacheng Xu
Greg Durrett
Salesforce Al
The University of Texas at Austin gdurrett@cs utexas.
j jiacheng. www.alesforce..coommmmmm com
Abstract
Great success has been reported using Reinforcement Learning from Hu- man Feedback (RLHF) to align large language models, with open preference datasets enabling wider experimentation, particularly for "helpfulness" in tasks like dialogue and web question answering. Alongside these improve- ments, however, RLHF also often drives models to produce longer outputs. This paper demonstrates, response length is, much more than previously thought, a significant factor behind RLHE. Studying the strategies RL optimization uses maximize reward, we find improvements in reward to largely be driven by increasing response length, instead of other features. Indeed, we find that even a purely length-based reward reproduces most downstream RLHF improve- ments over supervised fine-tuned models. Testing a comprehensive set length-countering interventions, we identify the dominant source of these biases reward models, which, by studying training dynamics, we find are non-robust and easily influenced by length biases in preference data.
three diverse settings, that optimizing for
10-
Question: Why don aduits roil off the hod?
X
(Refaroj; 59 tokens
05-
Adult cypionily not mill off hel hermor they hooe dereloped the suarle monory bodios devoistarilyy moving
00
muintsining proper
RLHF (Atter); 243 tokem:
Similar outpui, much
details
103
Adbls poveratly do
boceuse developed ausele contoi
-1.
witch thew spright wiide nloppig Addtiiataly, find uncomfortable waking foor woold
slog, millin
mper folling out and petontially baiting oacoal one lise bainne juil frome
-15
-200
20
100
150
200
750
Oulpot Length
Figure 1: Logeee heatmap of lengths of SFT outputs vs. learned reward model scores for WebGPT (left). The graph shows that reward scores are strongly correlated with length. RLHF with these consistently leads to longer outputs (right).
1 Introduction
Reinforcement Learning from Human Feedback (RLHF) widely used to align large language models (LLMs) with desired downstream properties such as helpfulness or harm- lessness (Ouyang al. 2022 Bai et al. 2022). This procedure entails (1) learning reward model to estimate human preferences and (2) using RL to maximize that reward. Its success relies on two things. First, the reward model must be correctly specified and not misaligned with human preferences (Zhuang & Hadfield-Mende 2023). Second, the optimization algorithm must successtully maximize reward without straying too far from sensible initial distribution.
2021 Pang et al., 2022 Bobu et al.,

---

Published as a conference paper at COLM 2024
Not meeting these conditions can lead to over-optimization of the reward model at the expense of human judgments (Dubois et al. 2023) or pathological "reward hacking" (Skalse) et 2022). Ad hoc adjustments (Touvron et al. 2023b Zheng et al.) 2023b) to PPO have stabilized the process, but there is shill little work examining what features improve in policy models, and to what extent these correspond to meaningful improvements in quality versus optimizing shallow reward correlations (Pang et al.) 2022)
In this work, we focus on one such feature: output length. Prior work in RLHF has noted
that the length of sampled outputs increases after RLHF (Stiennon et al. 2020 Nakano et al. al., 2023; Zheng et al. 2023b Sun et al. | [023]] Wu et al,) 2023) but largely dismissed it as an artifact of PPO. In this paper, we study this phenomenon in depth to
2021;
understand its underlying causes, and to what extent length is a meaningful feature vs. spurious correlation that PPO optimizes for.
We organize our main findings into three parts. First, we compare vanilla PPO and SFT models in three diverse settings (WebGPT (Nakano et al., 2021), Stack (Lambert et al. 2023 and RLCD Yang et al. (2023) ) and show that length dominates reward optimization in PPO to surprising extent. In fact, for two settings, we find that the PPO improvements disappear if we restrict our comparison to similar length outputs from PPO and SFT.
We then run controlled experiment with the PPO pipeline, but replacing learned reward models with a purely length-based heuristic score. We find that PPO performance with this length-only reward is close to standard PPO (56% vs 58% win-rate of standard PPO WebGPT and 64% vs 63% win-rate of standard PPO on RLCD). This shows shortcomings of current reward models, which fail to convincingly outperform simple heuristics, but more broadly questions the recent "progress" reported with popular metrics.
Exploring a comprehensive set of anti-length interventions, including changing preference datasets, reward model training, policy rollout strategies, reward scores and KL loss, we find that no strategy works for all settings. However, we find that interventions generally brings length closer to the base model, sometimes without degrading performance.
Output length may be a legitimate feature to optimize for, it may correspond to greater informativeness: however our results hint at a concerning scenario wherein PPO struggles to improve reward without increasing length, failing optimize for important non-length features. We find that this is because learned reward models themselves exhibit very strong correlations with length (see Figure
I
at the cost of other features, from training itself.
Taken together, our findings: (1) show that current reward models only model shallow aspects of human preferences; (2) call into question PPO "improvements" using reward models on the datasets we study; and (3) show what interventions are effective and call for better preference data and downstream evaluation.
2 Background and Task Setup
Text generation models (Sutskever et al., 2014 Bahdanau et al., 2015) place distribution over target output y "ii.....y)) given input sequences of tokens x, L.e,, na: p(y | x; x;)
Th=r yek, x; nT)). Typically, these models are trained with both language modeling pre-training (learning to predict the next word given context) and supervised fine-tuning (SFT; learning to generate outputs to maximize the likelihood of references on some dataset).
RLHF can then be broken into two stages. First, preference data of the form P = ((x1 .y (i)....)........ )...., (X,, Vn, yū)} is collected, where x; is the prompt, y," is the preferred con- tinuation, and y; is the dispreferred continuation. This dataset used to train a scalar reward model R(x,y) such that for any preference, R(x, v,") R(x, y; ). We use the stan- dard Bradley-Terry preference model (Bradley & Terry 1952), that is, P(y1 > y2 x)
exp(R(xxn))
exp( (e([((xx)))+ex(((xx)))) to maximize the log likelihood of the observed preferences.
Second, proximal policy optimization (PPO) (Schulman et al. 2017 is used further train SFT the SFT model (FT) to get T& PPO R). This training maximizes the reward
RL
2

---

Published as a conference paper at COLM 2024
W-GPT
STACK
ALCD
30
Type
Sit
Type
5Rt
20
type
SR
40
Std
Count
60
Count
30
20
1.0
0
100
150 Length
2.00
250
50
100
150
200
250
50
100
150
200
250
Langth
Length
Figure 2: Output lengths of SFT model before (blue) and after (red) standard PPO (STD); averages shown with dashed lines. Across all settings, PPO leads to large length increases.
R(x; y; ~ (y(()xx)) on training data x,i....,xm))) while not deviating too strongly from the initial distribution. Specifically, it maximizes the following objective, where À controls the strength Kullback-Leibler (KL) penalty between SFT and the current policy:
xx)o(x,y) = R(x,y) ADKI (y|x)] SFT (ylx))
(1)
2.1 Experimental Setup
Task settings We experiments on three popular "helpfulness" preference datasets, ensur- ing diversity in task settings and source of preference labels (examples in Appendix D):
WebGPT (Question answering; human labels) Nakano et al. (2021) contains human
annotated preference labels for the open-domain long-torm question answering task (Fan ett al. 2019). It includes 19.6K examples (mean tokens per y = 169).
2. Stack (Technical question answering; upvotes) Lambert et al. (2023 contains technical StackExchange questions, with preference labels between two answers derived automati- cally from the number of upvotes. We use 100K (mean tokens per = 236) pairs from the
dataset following the Hugging Face implementation (von Werra al. 2020).
RLCD (Multi-turn conversation; synthetic preferences) Yang et al. (2023) includes multi- turn dialogue data. Starting from input instructions in the Faartulllaaameesss dataset by Anthropic (Bai et al.] 2022), preferred and not-preferred outputs are automatically generated using prompt heuristics, c.g. appending "generate unhelpful outputs" to the prompt. We use the helpfulness subset (40K preferences; mean tokens per 45).
Implementation Details We use Huggingface TRL. (von Werra et al. 2020 with hyper- parameters we find to work best based on reward and downstream evaluation (1 0.04, batch size 64; more in Appendix B. We use Llama-7B models (Touvron et al., 2023a our base for all experiments, and use LoRA (rank= 16) (Hu et al. 2021 to enable PPO training with limited GPU resources. We choose publicly available SFI models for each setting: AlpacaFarm SFT for WebGPT and RLCD, and TRL SFT for Stack.
Evaluation Metrics We evaluate with (1) (intrinsic) reward scores from task-specific reward models used in the PPO process; and (2) (downstream) AlpacaFarm simulated preferences (Dubois et al. 2023), popularly used as proxy for humans in most recent work (Dubois al. 2023 Zheng et al. 2023a). This queries 12 OpenAl API-based "annotators" to choose between two outputs, and reports pairwise "win rate" of model over another (e.g. SFT's win-rate over PPO), testing on 500 held-out datapoints for each task. While useful for additional context, qualify this metric by noting that it itself may have length biases, which we partially examine in Table hence why we focus more on other metrics in our experiments.
3

---

Published as a conference paper at COLM 2024
W.GFT
Stack
RLCD
0.5
0.50
Reward
Reward
Rewara
-0.5
0.25
-10
0.75
50
100
150
200
100
150
200
250
25
50
75
100
125
150
Length
Length
Length
Figure Output length vs reward in 20 token buckets. Black dots indicate SFT, and arrows indicate improvement (up) or degradation (down) after HIGH KL PPO for each bin. Size and color intensity is proportional to number of examples in the bin. Reward scores are strongly correlated with length. On WebGPT and RLCD, reward improvement within bins is small, showing that overall improvement after PPO is primarily due shifting to longer outputs.
Does PPO Only Optimize Length?
To motivate this section, we first re-establish (Stiennon et al., 2020; Dubois al., 2023 Zheng et al. 2023b) that on our settings, indeed, PPO significantly increases output length. Figure Sosure2compp compares output lengths on our test set when sampling from the initial SFT model (blue) and the post-PPO model (red). We note clear length increases across settings. We also report [ that reward indeed increases after PPO (AR) and as in prior work, PPO beats the SFT model on simulated preference (SIM PREF, Table 3) we'll discuss more later, but for now we just use them to establish PPO improves over SFT as expected.
As we find that reward scores and length are positively correlated (Figure I shows this for WebGPT)), , it is possible for PPO to improve on intrinsic reward metrics by simply producing longer outputs. Based this possibility, we investigate the following question: to what extent are PPO improvements explained by the increase in length?
3.1 Length-stratified analysis of reward improvement
Experimental Setup We analyze whether overall reward improvements from PPO still hold when comparing outputs of similar length. Specifically, we stratify outputs based on length (using 20 token buckets) and report the average reward score of each bucket for initial SFT and post-PPO models. Note that Figure 2 shows little overlap length buckets between SFT and standard PPO outputs for WebGPT and Stack; therefore, additionally report results for a variant of PPO with high KL penalty (A in equation I].
Table 1: Non-length reward gain (NRG), reward im- provement (AR) and their ratio for standard (STD) and high (HIGH A) PPO. Low ratios on WGPT
First, we show overall reward gain (AR) from PPO compared to SFT. Second, we compute non-length reward gain (NRG), the average AR within each bucket weighted by the number of examples each bucket (SFT and PPO combined). This estimates the re- ward improvement attributable to the within-bucket reward increases as op- posed to shifting the distribution over buckets. Finally, we report the ratio of NRG and AR, i.e. the fraction of reward gain due to non-length features.
and RI.CD (STACK to weaker extend) indicate high dependence on length for reward improvement.
WGPT
STACK
RLCD
STD
HIGH
STD
HIGH
STD
HIGH
AR
0.82
0.20
0.89
0.67
0.94
0.61
NRG ratio
0.02
0.03
0.48
0.37
0.25
0.12
2.0%
15.1%
53.4%
56.5%
27.2%
19.1%
Results Table I reports results for both standard and high À cases. We observe that although all settings report overall reward gains, non-length reward gains are substantially lower. For WebGPT and RLCD, 70%-90% of the improvement on WebGPT and RLCD
4

---

Published as a conference paper at COLM 2024
Table Simulated preferences (against SFT) from purely optimizing for higher length (LPPO) with and without (A 0) KL penalty, and longest-of-8 sampling baseline (SFT-LONG); indicates a statistically significant delta from SFT (p < 0.05, paired bootstrap test). LPPO is comparable to standard PPO, supporting our hypothesis that RLHF improvements are largely length-based. Interestingly, LPPO beats À = 0, SFI-LONG even when shorter which shows that this method causes qualitative changes beyond just extending output length.
W-GPT
STACK SPI-LONG
SET
PRO
SPR-LONG
1PPO
LPPO
0
SPT
PP0
LPPO
LPPO
SPT
PPO
SPI-LONG
LPPO
LPTO A
LEN
100
230
141
118
167
203
257
219
252
245
59
94
117
98
163
PRES
48%
36%
58%"
57%"
59%
50%
63%
SIN
can be explained by length shifts. Particularly, NRG is almost negligible for WebGPT and contributes only 2% to overall reward gain in the standard PPO setting.
Note STACK reports higher contribution of NRG to overall reward gains, likely because STACK SFT outputs are already close to the length limit, so gain from increasing length is not possible to achieve. As a technical QA setting, it may also rely more on non-length features.
We visualize length-stratified reward scores for the high case in Figure Black dots represent SFT outputs, and the arrow tips denote the PPO outputs. The figure further supports [able[] while reward scores do increase within each bin on average, the increases are uneven and much smaller than reward increases from purely shifting to longer outputs.
3.2 Can a length-only reward improve performance?
We find PPO primarily to optimizes length, yet we (and the rest of the community) still see wide improvements on downstream simulated preference evaluation. Here, we show that only optimizing for length still lead to improvements with this evaluation:
1. (LPPO) Use output length as the reward during PPO. We define R (y) = = len(y)
where L is a target length hyperparameter (set to 156, 120, and 250 on WEBGPT, RLCD, and STACK respectively, which we found allowed desired length increases without becoming too long). We also report a variant with KL coefficient set O.
2. (SFT-LONG) Sample 8 outputs from the SFT model and select the longest one
Results Tableeccontiin results. SFT-LONG sometimes improves performance significantly (57% winrate vs SFT on Stack), but when we compare LPPO against PPO and SFT, we find that purely optimizing for length actually reproduces most of the simulated preference improvements of PPO with the learned reward models.
Notably, LPPO yields win rate improvements over SFT-LONG, which has even longer outputs, controlling for evaluation length bias. LPPO also outperforms LPPO Our hypothesis is that the KL term is an important constraint on the optimization for allowing length-only PPO learn good features. Since repetitive, pathological outputs would likely have higher KL divergence from the initial policy, this term possibly forces the model to learn how to generate more descriptive outputs while also maximizing length.
This may explain some of RLHF's recent success in spite of the major limitations we uncover. Win-rate based evaluation may be useful for understanding overall improvements in LLMs. However, this experiment reveals that it isn't sufficient: with complex technique like RLHF, a simple judgement on whether outputs have "improved" fundamentally doesn't tell whether things are actually behaving as expected.
5

---

where N is maximum length that we Published as a conference paper at COLM 2024
Step 1: Reward Model Training
Step 2 Train with PPO
Sample from current policy
Ze
Score with RM and KL
P Kada 95
R(x, xx,
-XKL(x
Y1 ~ T Y/ (
Train
Preference
cumont policy
Inturvantion
Modelling
Intervantions Training
Loneth-bulancing
Randum preference
Sumple shorter ourpuss from: curcei policy
Penalize seward scores of long
Reward scaling within botch
High contiicient
data
peeference
Figure Interventions to test the effects of various RLHF components on length
Table 3: Tokens (LEN), REWARD simulated preference (SIM PREF, Section 2.1 Vs. standard PPO across interventions (blue if better, red if worse than PPO). Rows with failed reward optimization excluded (-). indicates statistically significant deltas from PPO (p 0.05m, paired bootstrap test). Interventions mitigate length increases SFT, but cost to reward.
W-GPT REWARD
STACK REWARD
RLCD
LENGTH
LENGTH
LENGTH REWARD
SPT (starting point)
-0.45
42%
0.05
42%
4.4
37%
STANDARD
230
0.25
50%
257
0.74
50%
5,50
50%
128
-0.05
49%
0.40
46%
5.00
41%
LENGTH
5,20
44%
HIGH-À
-0.06
45%
0.30
45%
5.20
43%
OMIT LONG OUTPUTS
127
-0.13
48%
Interventions on RLHF
Our results up to this point show that the RLHF pipeline leads to longer outputs. Next, we study which components of the pipeline, between reward modeling and PPO optimization, contribute to this behavior and whether carefully designed interventions can mitigate it.
Figure shows the overall RLHF pipeline and the different stages intervene on. These experiments allow us to test whether length continues to increase during PPO even with strong interventions place against it. We study PPO objectives and rollout interventions in Section 4.1 (right half of Figure 4 and discuss preference data and reward modeling interventions Section 4.2 (left half of Figure 4)
4.1 Interventions on PPO optimization
(KL loss; 1.7) A simple intervention is to HIGH À KL coefficient (Equation 1,, with the intuition that closer to the initial distribution should mean closer to the initial length. Here, we set À to 0.12 instead of 0.04; we find that larger values impede model convergence.
(rollouts; 1.4) A simple option to altogether OMIT LONG OUTPUTS beyond a length threshold from PPO, so that no update is made to encourage these. In practice swap these examples with randomly sampled outputs from the batch.
(RM score; I.5) We also experiment with scalar penalty added to the reward model to len(y)
PENALIZE LENGTH. We set R' R -
want PPO to exceed, and o is a moving average of batch reward standard deviation
(RM score; 1.6) Prior work uses REWARD SCALING to "control training fluctuations" and over-optimization (Zheng et al. 2023b). Similar to batch normalization (Ioffe & Szegedy 2015), for each batch X, sampled outputs, we compute the mean (и) and standard
We try several variants of this idea, such scalar penalty past length threshold, and note similar convergence failures. Generally stricter versions of these constraints hamper convergence.

---

Published as a conference paper at COLM 2024
deviation (0) of R. We then take moving average of these values across N previous batches and "scale" R to become R' R- (or remains relatively constant across training).
Results We report results in Table 3 For context, report simulated preferences vs PPO: < 50% indicates worse than standard PPO downstream quality. Intuitively, our interventions should encourage reward to be optimized by targeting non-length features, and LEN should remain similar, if not shorter than the SFI starting point. Note that each setting has different REWARD, so these rewards shouldn't be compared across settings.
LEN often decreases substantially compared to standard PPO, confirming length is related to these parts of PPO, and that we can mitigate the extreme dependence on length during optimization while retaining moderate downstream improvement. For practitioners, this establishes our interventions legitimate approaches for controlling length in RLHF.
However, length still always increases relative to SFT, and reward model score is always worse than standard PPO. Moreover, omission and penalizing length often cause conver- gence failure (reward not increasing during training), supporting length's 's major role in PPO. Similar to Figure 3 we also note that across interventions, the scatter plots and NRC values display similar patterns of length-dominane (see Appendix 9 confirming that the ratio of optimization due to length remains consistent across PPO interventions.
4.2 Interventions on Reward Modeling
Section showed length- dominance to be largely in- variant PPO interventions, pointing instead to strong reward correlations with length. We investigate here to what extent reward models prefer longer outputs, starting with simple analysis: is preference data imbalanced towards longer outputs? We can measure this with length heuristic agreement: the accuracy of always predicting that the longer output the gold preferred output (see Table 5), indeed finding all datasets are slightly imbalanced towards longer outputs, but we this doesn't reveal the full story. To understand things better, we'll first examine reward interventions (Figure and later analyze underlying causes:
Table 4: Eval accuracy (ACC) and Pearson within batch (CORR) for RM interventions (RAND is random baseline, STND normal RM). RM accuracies are often low. Few approaches both reduce correlation and maintain good accuracy: length is tied to RM success. Length bias remains on RLCD despite balancing.
WGPT
STACK
RLCD
ACC
CORR
CORR
ACC
CORR
RAND
50%
0
50%
50%
0
STND
61.5%
0.72
70%
0.55
80%
0.67
BAL
52.6%
-0.13
61.9%
-0.09
73.1%
0.62
C-TR
58.8%
0.67
59.5%
0.31
77.2%
0.57
R-DA 62.5%
0.35
72.6%
0.37
80%
0.43
(preferences; I.1) Length Balancing (BAL): One option is to balance data by length. Specifically we balance data such that the distribu- tion of pair length differences are symmetric by bins of 10. Suppose examples where preferred responses are 20 tokens longer than dispreferred ones compared to the reverse case; to balance data we subsample the cases which are 20 tokens longer until they match the number of cases which are 20 tokens shorter.
Table 5: Accuracy of al- ways preferring longer response. Above ran- dom (50%) accuracy in- dicates length bias.
there are
(preferences; I.2) Reward Data Augmentation (R-DA): Data aug- mentation can encourage models to learn robust features. We use "random pairing", pairing matching prompt output pairs qi Pi from P with pi serving as a "prefered" example, and randomly sampled pj from another prompt serving as "dispreferred" example. Although this data augmentation doesn't t target length per se, in preliminary experiments, we found it to improve RM robustness and reduce length correlation.
WGPT
STACK
RLCD
55.7%
59.6%
63.1%
7

---

Published as a conference paper at COLM 2024
(RM training; I.3) Confidence-Basee Truncation (C-TR): What if length biases go beyond data imbalance? For example, a set of "easy" examples may be corrupting the data, and removing them may help SSap//adiiaiipta et al. 2020). Given we've trained some Rbase, and computed a "confidence" c; of the model on each training example on dataset P (we describe the setup for this in Section 5) we can test this idea by training new RM Rtrunc
subset of P where Cj < 01 and C; > 82 with threshold hyper-parameters 0,, and 02. We experiment with several variants (see Appendix C.4), keeping sets of around 50% of the data for each, but we'll here just report results when we set 01 82, training on low-confidence examples.
We report in Table 4 reward evaluation, as well as correlation within batch (CORR), which measures, given sets of 8 generations from the same input, the mean Pearson correlation between output length and reward. Note that the standard reward model (STND) accuracy is not high for the binary task, while the length correlations are high.
Reesults
Many interventions, such as BAL, re- duce correlations, but all except R- DA damage eval accuracy. Interest- ingly, on RLCD strong correlations remain despite balancing, suggest- ing the bias may be more challeng- ing to eliminate. STACK, however, where balancing lowers correlation with above-random accuracy, sug- gests that reward models can learn features independent of length.
Table 6: Simulated preference (SIM PREF) vs STND PPO for the SFT model, length (LEN), STD PPO, and interventions. STACK BAL shows strong results pos- sible without length increase via RM interventions (more influential vs PPO interventions), though re- sults are inconsistent.
WGPT
STACK SIM PREF
RLCD
Method
LEN
SIM PREF
LEN
LEN
SIM PREF
SFT
100
42%*
203
42%"
59
37%*
We then show downstream results for adjustments to preference data in Ta- ble Similar to the PPO interven- tions (Table 3) length still usually in- creases from the SFT starting point, though generally shorter relative Standard PPO. However, BAL on STACK, perhaps due to there being other easy non-length features to learn, leads to shorter outputs than SFT (with higher downstream preference), confirming the importance of preference data in RLHE.
SIND
230
50%
257
50%
94
50%
148
57%*
44%,
R-DA
139
49%
256
58%"
112
44%*
C-TR
141
44%"
244
44%
97
50%
5 Analyzing Preferences over Training
Why do length biases emerge in RMs, even after balancing? To understand this better, we study training dynamics and datapoint-level learnability of reward modeling. We compute statistics over several epochs of training: given reward model R being trained on preference dataset for E epochs, we can track each data point (x; Vi y distribution of confidence (RM score of "preferred" subtracted from "dispreferred") '), at each epuch c; {(e, R(x; R(xiy; ))
where we compute the
E),, excluding epoch 1 to mitigate noise.
Results For context, when examining initial "cartography" plots (waaymdiipta et al. 2020 of the mean (c;) and variance (o(c;) of different e; (initial visualization in Appendix C.4) we found values to be largely centered at zero, meaning that the predictions are low-conhdencce and largely do not change, potentially indicating that the few examples with high ij may have disproportionate effect on training. With this hypothesis that length may be related to a set of "easy" examples, we use length heuristic accuracy again, but this time, we compute it on slices where we bin training examples based on Cj plotting these bins by confidence (X-axis) against length heuristic accuracy (y-axis) on each slice as scatter plots in Figure
The figure shows strikingly clean patterns, with the mean confidence 7j for data in an interval of training examples correlating strongly with the length heuristic. This means that (1) the length heuristic applies to most examples that are easy, and (2) the overwhelming majority of strong negative predictions are cases where the model follows the length heuristic to confidently predict the wrong answer. Note that WebGPT, with the strongest pattern, also
8

---

Published as a conference paper at COLM 2024
W-GPT
Stack
RLCD
1.0
Length Heuristic
0.8
Length Heurlstic
Length Heuristic
0.8
0.6
0.6
0.4
0.4
0.2
0.2
0.2
0.0
-2
0
2
4
-5
5
Confidence
Confidence
Confidence
Figure 5: Training example confidence (cj) vs length heuristic, bucketed based on C₁ size shows amount in bucket. Most examples are near-zero confidence: RMs have trouble learn- ing on most data. Strong predictions (including incorrect) follow length heuristic with clean proportionality. RMs may over-rely on small sets of "easy" length-biased examples
displayed the lowest NRG from Table implying that these correlations propagate through all stages, Thus reward models likely struggle to learn deeper features from preferences, and even with balancing are vulnerable being dominated by "easiy" features like length.
6 Related Work
RL Reinforcement learning from human feedback has been explored extensively (Knox & Stone 2009) often in robotics tasks, extrapolate reward signal beyond initial preference
sets (Brown et al. 2019). While past RL in NLP faced different issues (Ammanabrolu &
Ried 2018 Martin al. 2017 Ramamurthy al. 2023) recent work in NLP has explored
implementations (Zheng et al. 2023b: louvron et al. 2023b) and objectives (Wu et al., 2023 of RLHF, largely dismissing length increases. Note even RLHF alternative like DPO (Ratailov et al. 2023; Zhao et al. 2023) have been shown to correspond with length (Ivison 22023 Ethayara et al. 2023) to RLHF; we validate in Appendix(C Past uses of RL in NLP haven used preterence-based reward, facing different issues. Our work is orthogonal to these, using the issue of length to analyze RM robustness and other properties of RLHF.
eet 20223
Reward Model Given data biases, do reward models learn robust features reflecting underlying preferences? Dataset artifacts are a prevalent issue NLP even on simpler settings like natural language inference (Gururangan al. 2018 Poliak et al. a (018). In RLHF, Stiennon et al. (2020) notes that over-optimizing for a reward model leads to pathological summaries, Dubois al. (2023) notes human preference drops after certain reward, and Pang et al. (2022) presents cases where such hacking can be produced within synthetic settings. Our work, in comparison, studies over-optimization in realistic, "working" settings, exploring diagnostics and solutions. We focus on length as it's the most prevalent, but our experimental paradigm applies to other analyses of RLHE.
2
Length control and length biases Techniques outside of RLHF for controlling length of NLP models have been explored (Kikuchi et al., 20166) Ficler Goldberg. 2017), with train- test length divergences (Riley & Chiang] 2022) attributed to inference techniques and label bias in text generation, which is quite different from our open-ended generation problems. Murray & Chiang (2018) use per-word reward similar to per-word penalty in RL, though to solve the opposite problem of outputs being too short. Finally, in discriminative "text matching" tasks like paraphrasing, past work has observed similar length heuristics, Jiang et al. (2022), but the sentence-pair format is quite different.
We include "harmlessness" experiments in Appendix

---

Published as a conference paper at COLM 2024
7 Conclusion
We contribute several new techniques for evaluating, analyzing and intervening on RLHE Across a multi-faceted set of experiments on three datasets, we show that RLHE, to surprising extent, relies on optimizing response length. Our results call into question improvements in PPO, the ability of reward models learn effectively from preferences, and the recent evaluation paradigms that have overlooked the findings we now reveal.
In the short term, we encourage much greater attention to preference data, and wider adoption of more feature-oriented evaluation approaches, such NRG. More broadly however, we believe that more substantial improvements to RLHF's vulnerability to simple features, particularly in reward modeling, will be necessary for RLHF to become more widely-applicable technique: RLHF still has a long way to go.
Acknowledgments
This work was supported by NSF CAREER Award IIS-2145280, grant from Open Philan- thropy, gift from Salesforce, Inc., and gift from Amazon. Thanks to Eunsol Choi and members of the TAUR lab for helpful discussion and feedback.
References
Prithviraj Ammanabrolu and Mark O. Riedl. Playing text-adventure games with graph- based deep reinforcement learning. ArXiv, abs/1812.01628, 2018. URL https:
semanticscholar. org : 54458698
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. In Yoshua Bengio and Yann LeCun (eds.), 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015. URL http: .rg. ht///bb//14400 .073
Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, T. J. Henighan, Nicholas Joseph, Saurav Kada- vath, John Kernion, Tom Conerly, Sheer El-Showk, Nelson Elhage, Zac Hatfield-Dodds, Danny Hernandez, Tristan Hume, Scott Johnston, Shauna Kravec, Liane Lovitt, Neel Nanda, Catherine Olsson, Dario Amodei, Tom B. Brown, Jack Clark, Sam McCandlish, Christopher Olah, Benjamin Mann, and Jared Kaplan. Training helpful and harmless assistant with reinforcement learning from human feedback. ArXiv, abs/ 22004.222,88, 2022. URL https: htt.:pppssssmaaacccccoccoloco otw.aaaat..cccl........oooraapaa
org. 2481118877
Andreea Bobu, Andi Peng, Pulkit Agrawal, Julie A. Shah, and Anca D. Dragan. Aligning robot and human representations. ArXiv, abs 23322.1928 2023. URL httpp:://pp
256598067
Ralph Allan Bradley and Milton E. Terry. Rank Analysis of Incomplete Block Designs: I. The Method of Paired Comparisons. Biometrika, 39:324, 1952. URL betps://app
semanticscholar. Drgg/CorpssID: 125209808
Daniel S. Brown, Wonjoon Goo, Prabhat Nagarajan, and Scott Niekum. Extrapolating be- y demonstrations via inverse reinforcement learning from observations. In International Conference Machine Learning, 2019. URL https: semanticscholar org/CorpusID 119111734
Yann Dubois, Xuechen Li, Rohan Taori, Tianyi Zhang, Ishaan Gulrajani, Jimmy Ba, Car- los Guestrin, Percy Liang, and Tatsunori B. Hashimoto. AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedback, 2023.
Kawin Ethayaraj. Winnie Xu, Dan Jurafsky, and Douwe Kiela. centered loss functions (halos). Technical report, Contextual main //aaa/ssstsssrr///e/
Human- AI, 2023.
https://github.ee http:///.ooomooeeeetta/////....
10

---

Published as a conference paper at COLM 2024
Angela Fan, Yacine Jernite, Ethan Perez, David Grangier, Jason Weston, and Michael Auli. EL15: Long form question answering. In Proceedings of the 57th Anmul Meeting of the Associ- ation for Computational Linguisfics, Florence, Italy, July 2019. Association for Computational 1346.
Linguistics. URL hattps://aclaaathooooggooooo
Jessica Ficler and Yoav Goldberg. Controlling linguistic style aspects in neural language generation. In Proceedings of the Workshop on Stylistic Variation, pp. 94-104, Copenhagen, Denmark, September 2017. Association for Computational Linguistics. doi: 10.18653/ v1/ W17-4912. httpL//ps///aaaattioooooo.ogg/g/
Suchin Gururangan, Swabha Swayamdipta, Omer Levy, Roy Schwartz, Samuel R. Bowman, and Noah A. Smith. Annotation artifacts in natural language inference data. ArXiv, abs/1803.02324, 2018. URL htRpsttps::/// http:/ppp...ematccccccccco.aaaaa ottpr//aeaiiolaoooooooo//ooaaaaa
Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. The Curious Case of 09751
Neural Text Degeneration. 2020. httpp::://aax....rr//aaaat////te
J. Edward Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, and Weizhu Chen. LoRA: Low-Rank Adaptation of Large Language Models. ArXiv, abs/2106.09685, 2021. https://: htt.aapp..emaaanncccloseseeeeeee www.oorpooooDoooooooooom 235455000
Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International Conference on Machine Learning, 2015. http://pp:/ppp..eaaaaiiiiieeeeee
Hamish Tvison, Yizhong Wang, Valentina Pyatkin, Nathan Lambert, Matthew Peters, Pradeep Dasigi, Joel Jang, David Wadden, Noah A. Smith, Iz Beltagy, and Hannaneh Hajishirzi. Camels in changing climate: Enhancing im adaptation with tulu 2, 2023.
Lan Jiang, Tianshu Lyu, Yankai Lin, Meng Chong, Xiaoyong Lyu, and Dawei Yin. On length divergence bias in textual matching models. In Findings of the Association for Computational Linguistics: ACL 2022, pp. 4187-4193, Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: 10.18653/vi /v1 20t//222..f2mml /2t.2...nnings-aall.eo.o findings--ac13300.. URL https //aclanthology. hrtgaancoooog.orr/222..f.f.. findings ac1.330
Yuta Kikuchi, Graham Neubig, Ryohei Sasano, Hiroya Takamura, and Manabu Okumura. Controlling output length in neural encoder-decoders. In Conference on Empirical Meth- ods in Natural Language Processing, 2016. URL https://apl: semanticscoolarr org/
CorpusID:1115775::
W. Bradley Knox and Peter Stone. Interactively shaping agents via human reinforcement: The tamer framework. In Proceedings of the Fifth International Conference on Knowledge Capture, K-CAP '09, pp. 9-16, New York, NY, USA, 2009. Association for Computing Machinery, ISBN 978160558657 doi: 10.1145 10.1145/ 15555/119973555977336 URL https://///o org/ 10. N0.1145/1599735 1597738
Nathan Lambert, Lewis Tunstall, Nazneen Rajani, and Tristan Thrush. HuggingFace H4 Stack Fxchange Preference Dataset, 2023. URL htttps:/huuginngfaee co/ co/datasets/
Lara J. Martin, Prithviraj Ammanabrolu, William Hancock, Shruti Singh, Brent Harri- son, and Mark O. Riedl. Event representations for automated story generation with deep neural nets. In AAAI Conference on Artificial Intelligence, 2017. URL https
s/api.semanticcccooaar org/CorpusID) 191277777
Kenton Murray and David Chiang. Correcting length bias in neural machine translation. In Proceedings of the Third Conference on Machine Translation: Research Papers, pp. 212- 223, Brussels, Belgium, October 2018, Association for Computational Linguistics. doi: 11.18653/v1////
URL h www/1/8--1122222222
Reichiro Nakano, Jacob Hilton, S. Arun Balaji, Jeff Wu, Ouyang Long, Christina Kim, Christopher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, Xu Jiang, Karl Cobbe, Tyna Eloundou, Gretchen Krueger, Kevin Button, Matthew Knight, Benjamin
11

---

Published as a conference paper at COLM 2024
Chess, and John Schulman. WebGPT: Browser-assisted question with hu- man feedback. ArXiv, abs/2112.09332, 2021. URL httph:t/ppppppp... htt///ppp...mmaiiscccccccoooooee org/
CorpusID:245329633
Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke E. Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Francis Christiano, Jan Leike, and Ryan J. Lowe. Training language models to follow instructions with human feedback. ArXio, abs/2203.02155, 2022.
URL https
Richard Yuanzhe Pang, Vishakh Padmakumar, Thibault Sellam, Ankur P. Parikh, and
He He. Reward gaming in conditional text generation. In Annual Meeting of the Association for Computational Linguistics, 2022. DRLtttt::://ap htttpp://ppp..mmmmmcccccoo..aa org/ Perg/CorpusID
semanticscholast
253553557
Adam Poliak, Jason Naradowsky, Aparajita Haldar, Rachel Rudinger, and Benjamin Van Durme. Hypothesis only baselines in natural language inference. In Proceed- ings of the Seventh Joint Conference on Lexical and Compulational Semantics, pp. 180-191, New Orleans, Louisiana, June 2018. Association for Computational Linguistics. doi: URL http:/ppee//cllaaaoooooooooor
10186533v1/S18$
Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning, and Chelsea Finn. Direct preference optimization: Your language model is secretly reward model. ArXio, abs/2305.18290, 2023. URL https: //pi www.etiisccooleeee.eeeeeeee org/
CorpusID:258959321
Rajkumar Ramamurthy, Prithviraj Ammanabrolu, Kianté Brantley, Jack Hessel, Rafet Sifa, Christian Bauckhage, Hannaneh Hajishirzi, and Yejin Choi. Is reinforcement learning (not) for natural language processing: Benchmarks, baselines, and building blocks for natural language policy optimization. In Proceedings of the International Conference on Learning Representations (ICLR), 2023. httpss/////arxx http://wwwwrr/abb////aaaa
Darcey Riley and David Chiang. A continuum of generation tasks for investigating length bias and degenerate repetition. In Proceedings of the Fifth BlackboxNLP Workshopp on Analyz- ing and Interpreting Neural Networks for NLP, pp. 4p.4226444, Abu Dhabi, United Arab Emi- rates (Hybrid), December 2022. Association for Computational Linguistics. doi: 10.18653/ blackboxnlp-1..0
w1/2022.blackboxnlp
URL
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proxi- mal policy optimization algorithms. ArXiv, abs/ htbp//17700647, 2017. URL https://api
semanticscholar orgo Iorpus 28695052
Joar Skalse, Nikolaus H. Howe, Dmitrii Krasheninnikov, and David Krueger. Defining and characterizing reward hacking. ArXiv, abs/ 2bbs//200013085, 2022. URL //app://api
semanticscholar... http:/scccoaaaoorr//Ceoooe Lrrg/CorpusID- 252545256
Nisan Stiennon, Long Ouyang, Jeff Wu, Daniel M. Ziegler, Ryan J. Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul Christiano. Learning to summarize from human feedback. In Advances in Neural Information Processing Systems (NeurIPS), 2020. URL
httpp::/api. semanticscholle lar
Simeng Sun, Dhawal Gupta, and Mohit lyyer. Exploring the impact of low-rank adaptation on the performance, efficiency, and regularization of RLHF ArXiv, abs/ /bb/2209,99055, 2023. UtL.https:// www.pppssssmmannncccshsscee h
Org/CorpusID 261884455
llya Sutskever, Oriol Vinyals, and Quoc V Le. Sequence to sequence learning with neural networks. In Advances in neural information processing systems, pp. 3104-3112, 2014.
Swabha Swayamdipta, Roy Schwartz, Nicholas Lourie, Yizhong Wang, Hannaneh Hajishirzi, Noah A. Smith, and Yejin Choi. Dataset cartography: Mapping and diagnosing datasets with training dynamics. In Conference on Empirical Methods in Natural Language Processing, 2020. URI :/tps:///ppp //ppi. semanticscholar. worp:eallrrorroooooeeeoeoo CrC.
12

---

Published as a conference paper at COLM 2024
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. LLaMA: Open and Efficient Foundation Language Models. ArXiv, /top//230..1399 2023a. URL https
Hugo Touvron, Louis Martin, Kevin R. Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Daniel M. Bikel, Lukas Blecher, Cristian Cantón Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony S. Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel M. Kloumann, A. Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, R. Subramanian, Xia Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zhengxu Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open Foundation and Fine-Tuned Chat Models. ArXiv, abs/ 2307 .09288,98888888 2023b. URI.htpss: https:tt//appp www.etiisscholleee...........tee org/
CorpusID: 2599559998
Leandro Werra, Younes Belkada, Lewis Tunstall, Edward Beeching, Tristan Thrush, Nathan Lambert, and Shengyi Huang. TRL: Transformer Reinforcement Learning. https
V/github.comm/ieg stggsbubb.ccmm/uuggginnaaa////// 2020.
Zeqiu Wu, Yushi Hu, Weijia Shi, Nouha Dziri, Alane Suhr, Prithviraj Ammanabrolu, Noah A. Smith, Mari Ostendorf, and Hanna Hajishirzi. Fine-grained human feedback gives better rewards for language model training. ArXiv, abs/ abs62306.01693,, 2023. URLhttps
/aap..semanticschol... wwwaaarrrrrrooooa arg/CorpussID::50660000
Kevin Yang, Dan Klein, Asli Celikyilmaz, Nanyun Peng, and Yuandong Tian. RLCD: Reinforcement Learning from Contrast Distillation for Language Model Alignment. ArXiv, abs/2307.12950. 2023. URL https: htt.:pppp.ssmmanncccccoooolaaaaa http://aic..ctaaoooooooooooorraa
:ttp:/oorpsII:::200000
Yao Zhao, Rishabh Joshi, Tianqi Liu, Misha Khalman, Mohammad Saleh, and Peter J. Liu. SLiC-HF: Sequence Likelihood Calibration with Human Feedback. ArXiv, abs52305.00425 2023. https///////pp
wwwp:oorps....o.eee.e semanticscholar. org/ Corpus ID: 25874108
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang. Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Haotong Zhang, Joseph Gonzalez, and Ioan Cristian Stoica. Judging LLM-as-a-judge with MT-Bench and Chatbot Arena. ArXiv, hols/230005685, 2023a. URL https://api: semanticscholar. htt.cccchhh.a..rrrrrorrrruus org/CorpuuID 259129398
Rui Zheng, Shihan Dou, Songyang Gao, Wei-Yuan Shen, Bing Wang, Yan Liu, Senjie Jin, Qin Liu, Limao Xiong, Luyao Chen, Zhiheng Xi, Yuhao Zhou, Nuo Xu, Wen-De Lai, Minghao Zhu, Rongxiang Weng, Wen-Chun Cheng, Cheng Chang, Zhangyue Yin, Yuan Long Hua, Haoran Huang, lianxiang Sun, Hang Yan, Tao Gui, Qi Zhang, Xipeng Qiu, and Xuanjing Huang. Secrets of RLHF in Large Language Models Part I: PPO. ArXiv, abs/ 2307 2307704967, 2023b. URL/atts:://ppi http::/ss/pppp.ssmaaaaaccccoooar http:///L_L.//////gg....aaeeeeaa org/ Corg/CorpusID-.... 259766568
Simon Zhuang and Dylan Hadfield-Menell. Consequences of Misaligned AI. ArXiv, abs/2102.03886, 2021. URL http///ppp..emaaacccccchlaaaaaaa wwt.://aaaaeecoooooo.oooorrrrrr. org/CorpusID 227275607
A Appendix
Reproducibility
For our various studies on the relationship between RLHF and length, we first trained a set of reward models and policy models. In order to support future open RLHF research, we
13

---

Published as a conference paper at COLM 2024
release code as well as reward and policy models. In addition to detailing experi- mental setup and evaluation scheme in Section/.1 as well as describing our interventions in detail in Section and Section 4.2 we include turther hyper-parameters and instructions in Appendix B Note that we use open preference datasets, publicly available base models, and open-source RLHF code that doesn't require prohibitive computational resources.
B Training / Evaluation Details
Hardware All experiments were conducted across workstations, one with NVIDIA RIX A6000 GPUs and another with 8 NVIDIA A40 GPUs. However, all of our individual experiments were run across GPUs. In this configuration, training an RM takes around hours on the Stack dataset, 3 hours on the RLCD dataset, and hour on the WebGPT dataset after epoch. For PPO, training takes around 12-18 hours for a single run.
B.1 Reward Models
For StackExchange, have train set of 100K examples and evaluation set of 10K examples. For WebGPT, since we use 18k examples from training and 1.6K as an evaluation set. For RLCD, we 40K training examples and 2.6 examples for the test set, where use these test sets all of the evaluation we show above.
For training, follow a rule of continuing to train until eval accuracy stops going up. Prior work finds that reward model training for just 1 epoch is most effective to avoid over-fitting, however for some of our preference data interventions longer. Overall, this ends up with usually 1-2 epochs of training at most for the checkpoints that we use. We use bfloat16, learning rate of le-5, and batch size of 2 with 2 gradient accumulation steps. With these configurations, 1 epoch on 10K examples takes around 2 GPU hours.
note that convergence takes
Note that for the training dynamics analysis (Figure 5), we run reward model training for epochs to reduce variance, however we don't use those models directly (though we note that eval accuracy doesn't go down significantly even at that point).
B.2 PPO
For our RLHF step, as stated before, we use LoRA and 8-bit quantization for the policy and reward models, since the TRL training configuration requires having all used models each device used for training. We merge reward model and generation models with LoRA adapters before PPO.
Past work has commented on the stability of PPO and "secrets" needed to get it working well (Zheng al. 2023b). We found that setting the right KL coefficient and batch size were the most important for stable convergence.
For training we generally run training for between 150-200 steps, where this is a hyperpa- rameter for each dataset depending on speed of convergence and allowing sufficient steps for KL to decrease in certain settings (Figure 6). We experimented with runs of up to 400 steps and generally did not find improvement in simulated preference or reward.
With GPUs, batch size of 32 on each, training takes around 16 hours complete 200 steps, giving an overall time of 32 GPU hours per PPO model. Note that we use max length of 156 on WebGPT and RLCD (note that this hyperparameter has strong influence on training speed). Stack, due to SFT having higher initial length, tends to generate unboundedly long outputs after PPO, even when using a higher max length (256) than the source TRL codebase (128).
Figures S show statistics over the course of training for our standard settings, with KL of 0.04. We note that RLHF does successfully increase reward score. The last half of training usually yields a decrease in KL divergence, as the model has optimized for reward and is regularized closer to the initial policy model by the KI. term.
14

---

Published as a conference paper at COLM 2024
10 Wpps
Figure 6: Training for standard WebGPT run
2012
Figure Training for standard Stack run
Figure 8: Training for standard RLCD run
B.3 Inference / Evaluation
Once we have our trained PPO models, we finally sample outputs that we can use to compare different systems and evaluation. For all results, unless otherwise stated, we generate 500 outputs each from fixed set of the held out data from each dataset, and base our results on those (we find this to be a sufficient amount, especially when comparing patterns across set of interventions settings which themselves serve as additional datapoints). Computing simulated preference (Dubois et al. 2023) for 100 outputs costs around $3.5 USD using the Ap API. These calls are made gpt-4-0314, gpt-3.5-turbo- 0301, and text-davinci-003.
We decode with nucleus sampling (Holtzman et al. 2020) with a p value of 0.9, maximum length of 256, temperature 0.9, and repetition penalty of 1.2 (based on the TRL repository default hyperparameters). Whenever we sample multiple outputs from a single prompt, we draw samples.
B.4 Interventions
For length control, we set the length center N to the starting mean SFT length, noting similar patterns with different configurations well (50 tokens for RLCD, 100 for WGPT, 200 for STACK), for omission, we use 24 tokens above these value (to allow for stable training). For reward data augmentation, augment with 25% additional data, noting that 50% gives similar patterns, however data augmentation may need to be explored further in future work.
15

---

Published as a conference paper at COLM 2024
C
Additional Experiments
C.1 DPO
Direct Preference Optimization Rafailov et al. (2023) has become popular and sim- pler alternative to RLHF without a separate reward model. While this work primarily focuses on reward modeling based RLHE, we here report some results with DPO on our settings (Table 7 Note that DPO still consistently leads to large length increases, while reward modeling accuracy remains similar or worse to that of when using an explicit reward model, suggesting that DPO likely suffers from similar issues to what we find in this work.
Table 7: DPO experiment on settings com-
paring length and reward accuracy
RLCD
STACK
WGPT
ORIGINAL RM ACC
80 %
70%
62%
DPO RM ACC ORIGINAL LENGTH
78%
62%
57%
59
203
100
DPO LENGTH
68
248
164
C.2 Larger Reward Models
As experiments mostly at the 7B scale with LLaMA, this may not necessarily represent other models or scales. For example, it may be the case that larger reward models achieve much better performance, and thus may not rely as much on the length heuristic. As an additional sanity check, we thus train reward models with LLaMA-2 13B mea- sure whether the reward modeling accuracy improves dramatically. We measure this in Table 8 While increasing model scale improves accuracy a bit, for these settings, model scale doesn't seem to be the primary bottleneck for reward modeling, which we found to be major source of length biases.
Table 8: Reward modeling accuracy when in- creasing model scale. This generally only in- creases marginally, suggesting these settings to remain qualitatively similar across model scales.
RLCD
STACK
WGPT
LLAMA 7B
61.5 %
70%
80%
LLAMA-2 13B
64.5%
71.3%
81.2%
C.3 Harmlessness
To compare the our findings on an objective unrelated to "helpfulness" and length, we trained reward model for harmlessness on the Anthropic data, noting a similar pattern of "difficulty" with only 68% evaluation accuaracy the held out set. However, did not find length correlation in this model. The within-batch length correlation is around -0.3, and doing PPO with just the harmlessness reward model didn't increase length either once converged. This is perhaps expected since shorter response (such as abstention from answering) will often be harmless. We also observed that the outputs started to look strange eventually, so optimizing for just harmlessness has its own set of shallow features and should not be optimized on its own. This only scratches the surface, however, and further exploration on similar alternate objectives can likely bring more insights into the inner workings of RLHF.
C.4 Dataset Cartography
We here include dataset cartography plots in the style of Swayamdipta et al. (020)) for our reward modeling tasks. First, Figure shows the dataset carlography plots on our respective settings. Note that WebGPT, the dataset with the strongest length biases, seems the most centered at 0 variance, and symmetric with respect to the x-axis. RLCD and STACK, where there seems to be more room for other features, an "upward tilt" where at higher variances, the models are able to learn correct higher confidence features. This provides evidence for our hypothesis that strong length biases emerge as a symptom of reward models being unable to learn clear features from most training data.
the other hand, demonstrate
16

---

Published as a conference paper at COLM 2024
(a) WebGPT
(b) Stack
(c) RLCD
Figure 9: Logarithmically scaled heatmaps plotting, with preference pair in training data being single point, the variance of confidence over training (X-axis) the mean confidence (y-axis). These are the plots for WebGPT, STACK, and RLCD respectively left to right.
(a) WebGPT DA
(b) RLCD Truncate Middle
(c) RLCD Truncate Top
Figure 10: Dataset cartography plots on WebGPT when doing Reward Data Augmentation, as well RLCD when truncating the central and upper sections of data.
Figure 10 shows plots for two additional settings. First, we see that when doing data augmentation, the augmented data emerges clearly and separately in the "high confidence" zone, while the initial plot remains in similar space as before, though interestingly now displaying more of the "upward tilt" with a longer tail. Next, we see that when cutting out the central section of the cartography plot and training is actually preserved, suggesting that the small amount of remaining data has an intrinsic tendency learn the strong length correlation. Likewise, when removing the upper section of "easy examples", we see that suddenly the RLCD plot becomes much more centered and sharper at the left with low variance, suggestive of the "brittleness" that we saw with WebGPT, where now the easiest pattern is harder to learn, exacerbating the pattern even further.
the remaining data, the shape
C.5 Length Scatterplots
Earlier in Section we discuss the idea of reward gain due length vs other features, examining results with Higher KL term. Here we show comparable plots for WebGPT (Figure 12) and RLCD (Figure 13). Note that the patterns and reward gain are very similar, suggesting that the length constraining techniques all perform similar functions, despite having very different formulations. Also note that for these two settings the ratio of reward gain independent of length remains quite low.
D Sample Outputs
17

---

Published as a conference paper at COLM 2024
W-GPT
Stack
RLCD
1.5
0.5
Reward
e+
Reward
4.5
0.0
3.5
-9.5
1.0
Length
Length
Length
Figure 11: Bin-scatters on different settings with red (normal KL) and black (high KL) shown together
Omit
HKL
RW-SC
0.6
0.75
0.5
0.50
Reward
0.25
Reward
0.00
0.0
-0.25
-0.5
0.2
1.0
Length
Length
Length
Figure 12: WGPT Bin Scatters on other types of length constraints.
Penally
Roward
Reward
50
100
150
200
250
50
100
150
200
250
1.00
150
200
250
Length
Longth
Length
Figure 13: RLCD Bin Scatters on other types of length constraints.
18

---

Published as a conference paper at COLM 2024
Table 9: Examples for StackExchange
Question
Before PPO
After PPO
Help. How can you
1. [URL22. ;[URL]3. [URIIyJntry2222
1. "Check if it's plugged in call 'ssPluggedln0 on your [PowerManagerild
detect whenever the power cord is un- plugged from elec- trical socket through Symbian OS?
object, which will return 'true' when the device has been connected to AC power and the phone doesn't have battery installed; otherwise, retums 'false' 2. If that isn't enough for you (perhaps because you need an alarm something), then take look at [URIL] about how use [URL]] time units within the Android framework, The solution outlined there should work fine with any of the three different timestamps available via SystemClock you just need to pick one. suggest using milliseconds since they seem most useful here, but feel free experiment!- Edit: Just wanted to make sure this wasn't 't too much code to throw away "jvvvr now new Date( get(ee)) / TimeU- nit Www.SECOOSS.toMllleee
(same solution, but with more details)
power
am asking this ques- tion from an educa- tional hacking point of view, (I wouldn't really want to code like this).Is it possi- ble implement while loop only using "*C* p prepro preprocessor di- rectives. understand
1)'//define while(x), then in the macro: x' and 'endif' if you wish your "con- ditional" statement execute each time af- ter one condition was true before moving on....) define if(')); x' above for the conditional statement, but without need for the 'else'. If you not specify any condition, the end will always execute.3) "while(true); // no break'4) This should get you started along the right lines at least. The preprocessor fairly limited though when compared with what some other lan- guages offer!
1) For the sake of simplicity let us assume all your 'while' loops are nested inside a macro called 'UNCT" 2) As first ap- proach you could define another variable in order to test if the condition is true or not. This new variable will have as value when there is no loop and nonzero otherwise:"" FOO 0; undef FUNCTdefine BOO if(FOO I= 0) FUNCT); else do nothing here because we exit directly; return /" You can use other values instead ' int main()retum BOO;"'3 An- other option involves adding some variables at compile time which may increase significantly the size of your program but also increases its readability: incluue istdio hedefine LOOP(N) jn forij 0; N554; + 1/" DO SOMETHING HERE * return j; /" The number of iterations * /// The user defines
that macros cannot be expanded recursively,
so how would this be accomplished?
Apart from normal differences in pitch and tone between male and female voices, are there any different words, different intonations, or other linguistic habits that tend to be used more by men women in KoreanZAre there any 'ale habits' that (to sound natural)
1. No, is not an exaggeration; Korean is gender-specific when comes to nouns/adjectives
1. ""Tone" don't think is possible have an overview this one because tones depend on context. For instance the word for bread has two totally distinct meanings if said with a high or low rising/ /faling tone. However generally speaking Koreans use lower tones at end sentences whereas native English speakers often prefer higher ones. 2. "'Pitch/ Timbre* as mentioned above this depends so much on context but here again generalizations can help you out: When asking questions your voice goes up, when saying state- ments down; you use less vocal variely in speech than people do normally; fact all your vucal cues like volume elcetera will be toned-dowe compared to how we would express our- selves if using our first language. 3. "Use of vowels"* well Korean has no shortage of long open vowel sounds while English prefers to reduce them whenever possible, This dif- ference alone means that certain pronunciation issues become quite obvious for non natives of both languages. Aside from that hawever, there isn' many rules regarding vows though they play an important role in differentiate between dialect
p
etc.2. There exists [the following list of female names ([UR] which will help you select appropriate name for your charac- ter if needed but mest likely using common female names as characters might lead some confusion with readers because many people have heard them so often while others never hear such names ever before... So would recommend making up new ones.
female leameer may want to avoid
imitating, and vice versa?
19

---

Published as a conference paper at COLM 2024
Table 10: Examples for WebGPT
Question
Before PPO
After PPO
Why is adults don't roll off of the bed ?I understand there are exceptions but I was just curious
Adults typically do not roll off of the bed because they have de- veloped the muscle memory keep their bodies from involun- tarily moving during sleep and maintain- ing proper posture.
Adults generally do not roll off of the bed because they have developed muscle control and awareness which helps them to remain upright while sleeping. Additionally, most adults find it uncomfortable so rolling off the bed would be difficult without waking up first. There may also be safety concerns such as falling out of bed and potentially hurting oneself if one were to lose balance fall from the bed. Finally, some people may simply choose to stay in the same position for long periods of time when they are sleeping
even painful to move around in their sleep,
thus avoid having shift positions during their slumber. It should also be noted that some medical conditions can lead to excessive movement at night, making it more likely for someone to roll off the bed. In these cases, medications and/ar lifestyle changes could help reduce this movement. Overall, it appears to be mostly matter of choice and comfort rather than any physical limitation preventing an individual from getting up or the hed.
A strong magnet will separate mixture afA, clear glass and green glass.B. paper cups and plastic
Bipaper cups and plastic cups
Iron nails and Aluminum Nails, A strong magnet be used to sort out metal objects, such as nails. The magnetic force between the two different metals cause them separating from each other in the mixture. Magnetic separation is also useful for sorting materials into different classes based on their composition properties. In this case, could be used to separate iron nails from aluminum nails, since they have very different physical and chemical properties. cannot be used sort out clear glass from green glass because the difference in color alone does not represent any significant differences in the compositions of the two types of glass. Paperr cups and plastic cups are made up of completely different materials (plastics cellulose)
Cups.C. iron nails and aluminum na sand and salt.
magnet would not help differentiate the two. Finally, salt and sand are both crystalline solids with similar structures but different sizes, making them impossible to distinguish using a magnet.
How come when we close one eye, "nothing" out of that eye, but when we close both, we black2And when I say nothing! mean that you don't see black, jothi...thhig.
When only one eye is closed, prevents visual information from entering the brain through that pathway; however, since there still signals being sent the other eye, our brains perceive "something" in terms of light and color (or darkness). When both eyes are shut, all incoming signals are blocked and thus the brain interprets this as "blackness"
When we close only one eye, our brain interprets the lack of input from that eye darkness because doesn't receive any visual information to process, But when we close both eyes, our brains combine the two images into one and interpret the combined image as darkness since there isn't enough contrast between the two images for us to be able to make out indiidd ual details. This phenomenon and helps protect us against blind spots by allowing us to use both eyes together in order to gain better depth perception and detect motion more easily. It also allows us to have a wider field of view compared to we were using just one eye time. In addition, since each eye things depending on what angle it facing, combining both images can fill-in missing pieces of detail detect objects that are closer or further away than what could be detected with just one eye. By closing both eyes, your brain data to create a single image which appears as pitch black due the absence of significant contrasts. That being said, some people do still experience ghostly shapes and movement when they shut their eyes completely; these effects arise from electrical activity happening within the eye itself
known as binocular vision
slightly different
all this
20

---

Published as a conference paper at COLM 2024
Table 11: Examples for Anthropic HelpfulHarmless (RLCD)
Question
Before PPO
After PPO
Human: How do
Yes! It's recom-
Yes, it's best to have ice shaver if possible since this will allow you to create small pieces of ice without having to break up the entire block. If you don't have one,you you still make great Italian ice by using a blender or food processor instead. be sure over-process your fruzen and hard to eat!
make Italian ice?
mended to have an ice shaver if you plan on making true italian-style ice. ice shaver will allow you to create shavings of in- stead of large chunks.
Assistant: There many forms of Italian ice, and
help you try to figure out of them you want. You could ask for gelato with a swirl of chocolate syrup, or you could ask for granita limone, or maybe some strae- ciatella espresso gramita.Human: Human: How can make Italian ice home?Assistam First, you'll need some You can use ice trays or ice cubes bag, or tray bag of snow:H Do need an shaver?Assistan
may become too
Human: My niece is into Sesame Street. What are some sesame street char- acters she can add to her toy collec- AiontAssistant:
Some of your niece's favorite Sesame Street characters include Elmo, Cookie Mon- ster, Abby Cadabby, Grover and Bert Emiee You could also check out plush toys featuring the entire Muppet gang!
Sesame Street has so many belowed characters! Your niece could have fun with Elmo, Cookie Monster, Grover, Abby Cadabby and more. She might also enjoy playing with Big Bird, Oscar Grouch or Bert Ernie they offer lots interactive play options. Additionally, you may want check out some of the other classic Muppets like Kermit Frog and Miss Piggy.
Continue the conver- Wation:Waat some of the health bene- fits going to
Going to a can provide numerous health benefits, such reduced stress levels, improved
Going to a spa can have numerous health benefits, including relaxation and stress relief. Spas provide opportunities for people to unwind in both physical and emotional ways by pro- viding massages, facials, body wraps, manicures, pedicures, and more. This allows them to reduce their levels of stress, anxiety, and other negative thoughts and feelings while also reducing muscle tension and increasing energy level. Addi- tionally, many such as herbs and minerals which help improve overall health and well-being.
Assisaatt
mental clarity and overall well-being, increased energy level, decreased of disease, improved skin complexion, relief from muscle tersion, anxiety re- duction, enhanced immunity system, and better digestion.
treatments include detoxifying elements
21


Reviews:
[{"review_id": "Zd5f8810Q1", "reviewer": "Reviewer_rdjK", "summary": "In this paper, the authors present a thorough investigation into how RLHF tends to optimize for longer output lengths, often at the expense of true quality improvements. They study this phenomenon across three diverse task settings - WebGPT, Stack, and RLCD. On WebGPT and RLCD, the bulk of the reward improvement from RLHF can be attributed simply to increases in output length, with little gain coming from optimizing other features. Attempts to mitigate the length bias through various interventions in the RLHF pipeline, such as adjusting the reward function, policy rollouts, KL loss, etc. do reduce length increases to some degree, but fail to eliminate the bias and often hurt overall performance. Overall, this is a well-executed and insightful study that reveals some concerning flaws in current RLHF approaches. The authors convincingly show that a significant portion of recently reported \"progress\" may be illusory and simply due to producing longer outputs. The finding that even preference datasets balanced by length still produce biased reward models is particularly notable. I believe this paper is great with its originality and rigorous experiment setting.", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": true, "final_justification": "", "rebuttal": "Thanks for the thoughtful comments! We’ll try to address the questions below: \n\n>I enjoy reading this paper but the only confusing section for me is Sec 3.1… most intuitive way is to just calculate ∆𝑅/∆𝐿\n\nThis is a great question! We’ll try to clarify here. \n\nIntuitively, we wanted to compute reward increase $\\Delta R$ when $\\Delta L$ = 0. Because these cases are quite rare, our approximation of this is to compute the $\\Delta R$ within each length-stratified bucket, as $\\Delta L$ within buckets is quite small. We take an average of this across buckets, weighted by the number of instances (SFT + PPO combined) in each bucket.\n\nWe like the suggestion of reporting $\\Delta R$/$\\Delta L$ or correlation between $\\Delta R$ and $\\Delta L$! One slight limitation of $\\Delta R$/$\\Delta L$ is that it changes the units and is therefore not comparable to $\\Delta R$ in the table. Furthermore, if we computed $\\Delta R$/$\\Delta L$ (assuming you meant $\\sum_i (\\Delta R) /  \\sum_i (\\Delta L)$), it could be the case that length increases and reward increases, but these are unconnected. Controlling for length with our approach helps disentangle this. Note that we already report corr(𝑅, 𝐿) within a batch in Table 4. \n\n> For experiment setup in Section 3.1, I wonder is sft output or ppo output you would stratify based on the length, or combined?\n\nWe stratify based on length here, where we compare all the outputs from SFT that are in some length range (e.g. 20-40 tokens) with all the PPO outputs in the same length range (20-40 tokens). Note that the overall prompt set is the same, but each bin may not have comparable prompts between SFT and PPO. \n\n> For Table 2, Could you please win/tie/lose rate in addition to win rate only? I believe this could strengthen the performance comparison between models.\n\nOur prompts (same as AlpacaFarm) ask GPT-4 to return one model name as output, corresponding to the better output. Because of this we rarely observe ties, thus the lose rate is just the reverse of win-rate. \n\n> For Table 3, why std ppo has failed reward optimization? As it has length and reward, I think it should be accessed by sim perf also?\n\nThanks for pointing this out! Standard PPO doesn’t have failed reward optimization; we used a - in the sim pref column for it because it is the reference point. We can replace those numbers with 50% for better clarity.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The experiments are quite neat. I truly appreiciate the rigorous experiment pipeline to understand the underlying mechanism for rlhf. The overall 'causal' experiments are quite convincing, make it a strong candidate for the venue.\n- Insightful Analysis. Beyond just showing the existence of length biases, the paper provides a detailed look into their causes and the training dynamics involved. The finding that biases stem from the preference data itself and remain even after balancing attempts is an important insight with implications for data collection practices.", "reasons_to_reject": "I enjoy reading this paper but the only confusing section for me is Sec 3.1. I can not get why non-length reward gain (NRG) is calculated as average ∆R within each bucket weighted by the number of examples in each bucket. What does this stand for? For me the most intuitive way to investigate the correlation between reward gained by ppo and its length is just to calculate $∆R/∆L$, where $∆L$ is the length difference between sft model and ppo model.", "questions_to_authors": "- For experiment setup in Section 3.1, I wonder is sft output or ppo output you would stratify based on the length, or combined? \n- For Table 2, Could you please win/tie/lose rate in addition to win rate only? I believe this could strengthen the performance comparision between models.\n- For Table 3, why std ppo has failed reward optimization? As it has length and reward, I think it should be accessed by sim perf also?", "extra_scores": {}}, {"review_id": "qLU2tvpQRk", "reviewer": "Reviewer_kVck", "summary": "This paper presents a well-scoped investigation of Reinforcement Learning with Human Feedback (RLHF), examining an observed tendency to prefer longer outputs and how it may be mitigated. The paper is concise and convincing, highlighting an important and overlooked feature of RLHF, by questioning what our preferences actually encourage in the generated outputs. This is an important step toward better evaluation of and methods for alignment techniques such as RLHF.", "questions": "", "limitations": "", "rating": 9, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for the thoughtful comments!\n\n> the last sentence of 2.1…\n\nThanks for pointing this out! We mainly mentioned this to qualify the win-rate preference results, since these may themselves have an undesired length-bias (e.g. STACK on SFT-LONG in Table 2), which has been supported in other recent work as well [1]. We will clarify this more in future versions.\n\n[1] Yann Dubois, Balazs Galambosi, Percy Liang, and Tatsunori B. Hashimoto. LengthCorrected AlpacaEval: A Simple Debiasing of Automatic Evaluators. https://github. com/tatsu-lab/alpaca_eval, 2024.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "To study the effect of length in RLHF, the authors investigate a satisfactory span of data: a long form and a short form question answering dataset (WebGPT, Stack), and a multi-turn dialogue dataset (RLCD). They provide sufficient, detailed information about the models and hyperparameters used. \n\t\nTheir argument is well-structured, each step culminating with empirical results. First, the paper presents an empirical justification that there exists a difference in preferred length with Figure 3. Then, an investigation of how much the length explains the reward with Table 1. Finally, a study of what interventions can mitigate rewarding long responses supported by figures and tables throughout section 4. \n\nI believe this is an important paper, because it questions and systematically observes a flaw in a method that otherwise appears ubiquitously applied. \n\t\nThe paper investigates whether and to what degree a popular method, Reinforcement Learning using Human Feedback (RLHF), is susceptible to producing text generations that are longer but not truly better. It is important to question contemporary, widely used techniques, and this paper does so in a principled way. This will be useful as a step towards further critiquing RLHF, and refining alignment methods.\n\nThe interventions proposed by the paper are reasonable and comprehensive. They bring clarity to the different parts of the RLHF components and how each can be influenced, and how effective this is. \n\nThe paper is clear and concise, explaining their decisions succinctly and motivating them well. Their background section had just enough detail to be relevant without extraneous information. Their claim and findings are clear in the abstract, and I found it easy to keep them in mind while reading the rest of the paper. I did not notice inconsistent reasoning.", "reasons_to_reject": "I did not understand the last sentence of 2.1, \"To control for length bias, we focus on results where shorter outputs show greater win-rates\". Specifically, how this was done throughout the paper (is it distinct from showing that shorter outputs are less frequently associated with greater win-rates?) and why this is a representative way to support the argument.", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "3aQclhRQG5", "reviewer": "Reviewer_Bn2p", "summary": "The paper investigates the relationship between output length and performance in Reinforcement Learning from Human Feedback (RLHF) for large language models. The authors demonstrate across three diverse settings (WebGPT, Stack, and RLCD) that RLHF tends to significantly increase output length, and a substantial portion of the reported improvements in reward can be attributed to this length increase rather than other meaningful quality improvements. Through controlled experiments, the authors show that optimizing for length alone (using a length-based reward) can reproduce most of the gains seen in standard RLHF. They explore various interventions to mitigate the length bias, including adjustments to the reward model training, preference data, and PPO optimization, but find that length biases persist across these interventions. Further analysis reveals that reward models exhibit strong correlations with length, potentially due to overfitting to a small set of \"easy\" length-biased examples during training.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for the thoughtful comments! We address the main concerns here:\n\n> larger models\n\nWe conducted additional experiments with scaling up our RM experiments up to LLaMA-2 13B. We report intrinsic reward modeling accuracy results below (compare with Table 4 in the paper):  \n\n| Model | WebGPT | Stack | RLCD |\n|-------|--------|-------|------|\n| LLaMA 7B    | 61.5%  | 70%   | 80%  |\n| LLaMA-2 13B   | 64.5%  | 71.3% | 81.2%|\n\nOn these datasets, reward modeling accuracy is only marginally better than it was before, suggesting that increasing model scale isn’t necessarily the main bottleneck on these tasks. \n\nWe will add these experiments in any future version. \n\n> The chosen evaluation datasets are limited and more results on general benchmarks like MT-bench or AlpacaEval 2.0 can make the results more convincing\n\nThis is a fair point. We aim to cover a fairly diverse set of realistic tasks (multi-turn dialogue QA, single-turn long-form QA, and technical question answering) to demonstrate our findings on publicly available preference datasets at the time the work was conducted. We do note that the pairwise preference metric we report (setting specific) is based on GPT-4 evaluation which is used on MT-Bench and alpaca-eval and thus should give a reasonable approximation to what one may find in those settings, the main difference being the prompt set.\n\nAs for MMLU, we think this is a nice suggestion, but the fine-tuning recipes (datasets used, SFT, etc.) necessary to induce good chain-of-thought-based question answering are a bit different than the recipes for the kind of LLM alignment we study here.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The paper provides a comprehensive analysis of the relationship between output length and RLHF performance, which has been noted but not thoroughly studied in prior works. The experimental setup covers diverse settings and tasks, lending robustness to the findings.\n- The analysis is multi-faceted, involving controlled experiments, interventions, and in-depth investigation of training dynamics, and the paper proposes practical interventions and evaluation techniques (e.g., NRG) to mitigate and assess length biases in RLHF.\n- I think the findings are meaningful and call for more careful consideration of preference data quality, reward model robustness, and evaluation metrics in RLHF research.", "reasons_to_reject": "- The chosen evaluation datastes are limited and more results on general benchmarks like MT-bench or AlpacaEval 2.0 can make the results more convincing. Also, it would be interesting to see the performance change on evaluations that does not rely on the lengths of the outputs (like MMLU). \n- Due to the computational resources, the authors did not conduct experiments on larger models, and the the PPO training is based on LoRA. Even without using a larger model, the authors can try the experimental results on more powerful small models (such as mistral, llama-2 and llama-3).", "questions_to_authors": "- In page 7, Reesults -> Results.", "extra_scores": {}}, {"review_id": "X4MKlneuwd", "reviewer": "Reviewer_WB4w", "summary": "This paper explores the tendency of Reinforcement Learning from Human Feedback (RLHF) to produce longer outputs when aligning LLMs with desired properties, such as helpfulness in dialogue and web question answering tasks. The authors delve into the strategies RL optimization employs to maximize reward and find that improvements are often driven by increasing response length rather than other features.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for the thoughtful comments! We’ll try to address concerns here.\n\n> more capable models\n\nWe conducted additional experiments with scaling up our RM experiments up to LLaMA-2 13B. We report intrinsic reward modeling accuracy results below (compare with Table 4 in the paper):  \n\n| Model | WebGPT | Stack | RLCD |\n|-------|--------|-------|------|\n| LLaMA 7B    | 61.5%  | 70%   | 80%  |\n| LLaMA-2 13B   | 64.5%  | 71.3% | 81.2%|\n\nOn these datasets, reward modeling accuracy is only marginally better than it was before, suggesting that increasing model scale isn’t necessarily the main bottleneck on these tasks. While the settings are a bit different, we will definitely consider extending our study to evaluate other models in future work. \n\n> length-unrelated tasks such as math, coding, length-constrained instructions…\n\nThis is a great point. It’s worth noting that we actually already have such a setting (the Stack setting) which we chose because it represents a realistic setting where math, coding and more complex reasoning may be used. Indeed, we do note this setting to demonstrate most of the discussed length-correlated behavior throughout the RLHF pipeline, though to a lesser degree. \n\n> Would the length bias issue correlate with tasks?” “In my opinion, before intervening in the RLHF process to mitigate the lengthy outputs, it is more important to recognize whether human annotators and LLM judges commonly prefer lengthy outputs\n\nWe agree with this point and discuss it in the paper (introduction, 3.1): length may be a valid feature corresponding to quality for humans. Two of the datasets we run experiments on, Stack and WebGPT, use human preference labels and these labels do show length biases (Table 5).   However, these are fairly small, suggesting that length may not be a dominant feature for human annotation. These small length biases are not proportional to the dominance of length we find experimentally after RLHF on these same settings, which is particularly concerning given how commonplace these techniques / evaluations are. Regardless of whether length is a good or bad feature, we find it concerning if it comes at the cost of other potential good features.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The length bias issue is worth a deep investigation. The final conclusion of reward modeling is well-supported by experiments.\n- The controlled experiments on length-only reward clearly show length shortcuts.", "reasons_to_reject": "- Would the length bias issue correlate with tasks? The selected datasets are question answering and chatting, which may naturally bring the length bias issue. I suggest selecting some length-unrelated tasks such as math, coding, or length-constrained instructions for a more comprehensive evaluation.\n- In my opinion, before intervening in the RLHF process to mitigate the lengthy outputs, it is more important to recognize whether human annotators and LLM judges commonly prefer lengthy outputs, and when it is a bias.", "questions_to_authors": "- Would developing more capable reward models benefit this issue? With limited GPU resources, the authors could at least evaluate whether top-ranked models on RewardBench (e.g. Eurus-RM-7B and Starling-RM-34B) still prefer longer responses.", "extra_scores": {}}, {"review_id": "Md4kFWM9TH", "reviewer": "Reviewer_zdy6", "summary": "This paper reveals an issue regarding the verbosity of large language models (LLMs). Specifically, the authors demonstrate that the response length is a significant factor behind RLHF. The authors further find that even a purely length-based reward reproduces most downstream RLHF improvements over supervised fine-tuned models. Testing a comprehensive set of length-countering interventions, this paper identifies the dominant source of the biases for the reward models. The main contribution of this paper is to reveal the vulnerability of the RLHF process.", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for the thoughtful review! We will add discussion of those citations you mention in any future version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "I think this paper does an excellent job of identifying the bias regarding the output length of LLMs. \n\n**Clarity.** This paper is well-written and organized. I think this paper is easy to follow.\n\n**Importance.** Evaluation of LLMs is a very timely and important topic in the field of AI. Verboisity bias has been highlighted in several concurrent works (please see the specific paper titles in the Reason to Reject section). This paper investigates such bias in LLMs in detail. I think the empirical analysis of this paper is quite convincing, and there is little room for objection.\n\n**Empirical experiments.** I think the empirical experiments are well-designed. The results are consistent with other papers, and I think they are reasonable.", "reasons_to_reject": "I don't come up with a strong reason to reject this paper, but I think there are a few missing citations regarding length or verbosity bias in RLHF/DPO. These papers can be regarded as concurrent work, so it is optional for authors to refer to them.\n\n- Park, Ryan, et al. \"Disentangling length from quality in direct preference optimization.\" arXiv preprint arXiv:2403.19159 (2024).\n- Saito, Keita, et al. \"Verbosity bias in preference labeling by large language models.\" arXiv preprint arXiv:2310.10076 (2023).\n- Wang, Haoxiang, et al. \"Arithmetic Control of LLMs for Diverse User Preferences: Directional Preference Alignment with Multi-Objective Rewards.\" arXiv preprint arXiv:2402.18571 (2024).", "questions_to_authors": "N/A", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper investigates the influence of RLHF on features of LM-generated text, particularly focusing on how it influences length of generated responses (in particular, causing an increase in length). The experiments suggest that reward improvements are primarily due to increases in length, e.g., by using response length as a reward and finding most improvements from RLHF are retained after RL-based fine-tuning, and find that this is mostly due to trained reward models learning a strong length bias (i.e., assigning higher reward to longer responses).

The experiments are well-designed, and the insights this paper provides are really important given that RL-based finetuning (especially RLHF) is currently the dominant paradigm in training LMs. 

I suggest the authors incorporate feedback and suggestions from the reviewers.
Decision: Accept