A Long Way to Go: Investigating Length Correlations in RLHF

Venue: colmweb_org_COLM_2024_Conference
Published: 2024-03-22 18:10:51

Abstract: Great success has been reported using Reinforcement Learning from Human Feedback (RLHF) to align large language models, with open preference datasets enabling wider experimentation, particularly for "helpfulness" in tasks like dialogue and web question answering. Alongside these improvements, however, RLHF also often drives models to produce longer outputs. This paper demonstrates, on three diverse settings, that optimizing for response length is, much more than previously thought, a significant factor behind RLHF. Studying the strategies RL optimization uses to maximize reward, we find improvements in reward to largely be driven by increasing response length, instead of other features. Indeed, we find that even a *purely* length-based reward reproduces most downstream RLHF improvements over supervised fine-tuned models. Testing a comprehensive set of length-countering interventions, we identify the dominant source of these biases to be reward models, which, by studying training dynamics, we find are non-robust and easily influenced by length biases in preference data.

Main paper: 
 introduction: 
 Reinforcement Learning from Human Feedback (RLHF) is widely used to align large language models (LLMs) with desired downstream properties such as helpfulness or harmlessness (Ouyang et al., 2022; Bai et al., 2022) . Specifically, starting from supervised fine-tuned (SFT) LLMs, the standard paradigm generally consists of ( 1) training a reward model (RM) on a labeled set of preferences on pairs of outputs for the same inputs, and (2) optimizing the policy model with respect to the reward model via an RL algorithm like proximal policy optimization (PPO) (Schulman et al., 2017) . This procedure relies on two things. First, the reward model must be correctly specified and not misaligned with human preferences (Zhuang & Hadfield-Menell, 2021; Pang et al., 2022; Bobu et al., 2023) . Second, the optimization algorithm must do a good job of balancing reward optimization with staying close to the initial policy distribution. Not meeting these conditions generally leads to over-optimization of the reward model at the expense of human judgments (Dubois et al., 2023) , which in the worst case leads to pathological "reward hacking" (Skalse et al., 2022) . Ad hoc adjustments (Touvron et al., 2023b) and improvements in PPO (Zheng et al., 2023b) have stabilized the process, but there's still little work examining the changes in the policy model's behavior responsible for reward improvements, and to what extent these correspond to meaningful improvements in quality versus optimization of shallow reward correlations (Pang et al., 2022) .While the vast majority of "helpfulness" RLHF work has noted increases in output length, (Dubois et al., 2023; Zheng et al., 2023b; Sun et al., 2023; Wu et al., 2023; Nakano et al., 2021; Stiennon et al., 2020) , little further analysis has been done, leaving the open possibility that length constitutes such a shallow correlation. Thus, seeking to better understand the objectives and improvements of RLHF, we focus on examining reward modeling and RLHF through the lens of length, finding it to play a larger role than previously discussed. We organize our findings into three parts:(1) In two settings, nearly all PPO reward improvement comes from purely optimizing length. This holds even in the presence of diverse length-countering PPO interventions, suggesting reward modeling to be the source of length biases. (2) Studying training dynamics, we find these biases to originate from a combination of data imbalances and significant robustness issues in standard reward modeling. (3) We conduct an experiment where we measure how much doing PPO with a reward based only on length can reproduce PPO quality gains with trained reward models.Optimizing length isn't necessarily bad: humans may genuinely prefer longer outputs. However, length's dominance comes at the cost of many other strong potential features. Further improvements to RLHF will likely require learning these other features, which itself requires strong solutions to the robustness flaws of reward modeling that we uncover: RLHF research still has a long way to go.(1) We propose several new analysis approaches and intervention strategies for RLHF (2) We conduct a multi-faceted exploration of length correlations at all stages of RLHF, demonstrating its effect to be much greater than previously anticipated (3) We identify underlying causes for said phenomena, particularly from robustness flaws of preference data and reward models (4) We plan to release code and a diverse set of models to support future open work. 
task setup: 
 RLHF is technique for optimizing the performance of text generation systems (Sutskever et al., 2014; Bahdanau et al., 2015) , in which we place a distribution over target output y = (y 1 , . . . , y n ) given input sequences of words x via a generation model π θ : p(y | x; π θ ) = ∏ n k=1 p(y k | y <k , x; π θ ). Historically, these models were trained with both language modeling pre-training (learning to predict the next word given context) and supervised fine-tuning (SFT; learning to generate outputs to maximize the likelihood of references on some dataset, also referred to as behavioral cloning).RLHF is a technique introduced to further improve upon this approach, and can be broken into three components. First, it requires a set of preference judgments over model outputs of the formP = {(x 1 , y + 1 , y - 1 ), . . . , (x n , y + n , y - n )} with triples of prompts x i , preferred continuations y + i , and dispreferred continuations , y - i .Then, given some P , the task is to train a scalar reward model R(q, x) such that for any given preference triple, R(x i , y+ i ) > R(x i , y - i ).We use the standard Bradley-Terry preference model (Bradley & Terry, 1952) , where P (y 1 ≻ y 2 | x) = exp(R(x,y 1 )) exp(R(x,y 1 ))+exp(R(x,y 2 )) and the reward model is trained to optimize the log likelihood of the observed preferences.We explore a collection of three preference datasets corresponding to three tasks and preference labeling strategies (examples in Appendix C). We select these datasets to provide a diversity of "helpfulness" tasks that are still challenging for our base model, LLaMA-7B (Touvron et al., 2023a) .1 WebGPT (Question answering; human labels) This dataset (Nakano et al., 2021) contains human annotated preference labels between two outputs for the open-domain long-form question answering (LFQA) task (Fan et al., 2019) . As human annotation is expensive, this dataset is relatively smaller at only 19.6K examples (mean tokens per y = 169) compared to the others we study.Stack (Technical question answering; upvotes) Released by Hugging Face, this dataset collects technical questions and answers from StackExchange (Lambert et al., 2023) . The preference label between two answers is derived using the number of upvotes; the one with more upvotes is assumed to be preferred. We use a subset of 100K (mean tokens per y = 236) pairs from the dataset following the Hugging Face implementation (von Werra et al., 2020) .RLCD (Multi-turn conversation; synthetic preferences) Finally, we explore multi-turn dialogue style data, released by Yang et al. (2023) . Starting from the input instructions in the Helpful/Harmless dataset by Anthropic (Bai et al., 2022) , they automatically generated preferred and not-preferred outputs using prompt heuristics, e.g. appending "generate unhelpful outputs" to the prompt. The "helpfulness" subset that we use consists of 40K examples and mean tokens per y = 45.Framework We use the standard implementation and hyperparameters for the 3 components of RLHF to maintain consistency. We base our RLHF implementation on the Huggingface TRL framework with hyperparameters we find to work best based on reward convergence and downstream evaluation (λ = 0.04, batch size 64, see more details in Appendix A) (von Werra et al., 2020) , and use LoRA (rank=16) (Hu et al., 2021) to enable training large Llama-7B models (Touvron et al., 2023a) with limited GPU memory. For our SFT models we use the released AlpacaFarm SFT model for WebGPT and RLCD as we find it to work well, and the TRL SFT model for Stack.Evaluation Our evaluation relies on two factors. First, reward is an intrinsic metric optimized by the PPO process. Second, we follow past work in AlpacaFarm (Dubois et al., 2023) to conduct downstream evaluation using more powerful LLMs as proxies for human preferences. Specifically, we sample responses on fixed held-out test sets of 500 prompts for each setting, then use their exact evaluation scheme based on using a panel of 12 simulated OpenAI API based "annotators," which they show correspond well with human preference judgements. The final format is an overall pairwise "win rate" of one set of paired outputs vs another, which we call simulated preferences. 
examining ppo: 
 In this section, we first show that: (1) Output length increases during PPO (Figure 2 ). ( 2) There exists a positive correlation between length and reward model scores (Figure 3 ). Taken together, this evidence suggests that simply increasing length could be a successful way to improve reward. Motivated by this, we investigate the following question: Is length increase the primary factor for reward models scores increasing during PPO, or are other features also optimized?To contextualize the rest of the work, we first show that length actually does increase as a result of PPO. Indeed, when comparing histograms of generation lengths (see Figure 2 ) on a fixed query set before and after our initial PPO runs, we find that PPO causes notable length increases. On WebGPT and RLCD, reward improvement within bins is much smaller than reward improvement from shifting to the right: PPO's barely optimizing non-length features. Quantified in Table 1 We now investigate the extent to which other features are learned, with two different settings of the KL weight λ in the objective. Figure 3 shows reward scores stratified by length, binned into buckets of 20 tokens for the higher λ variant (HIGH KL). While reward score does increase in each bin on average, the increases in reward are uneven. Furthermore, the increases are less strong than the length trends: generating an answer that's 40 tokens longer (shifted over by two bins) often provides a larger improvement than PPO. (See Figure 10 for a plot with our standard, lower-KL PPO setting.) The simplest intervention to PPO to encourage short outputs is to just increase the KL coefficient λ (H-KL) (Equation 1), with the intuition that closer to the initial distribution should mean closer to the initial length. We experiment with setting it to 0.12 instead of 0.04; larger values impede model convergence.We also experiment with a scalar penalty on the reward to control length (LEN-C). We setR ′ = σ (1 - len(y) N ), where N is a maximum length value that we do not want PPO to exceed, and σ is a moving average of batch reward standard deviation.A similar option to prevent outputs from getting longer may just be to altogether omit (OMIT) outputs beyond a length threshold from PPO, so that no update is made to encourage these. In practice we swap these examples with randomly sampled outputs from the batch.Finally, prior work examining ways to improve implementations of PPO mentions that reward scaling (RM-SC) can be useful for "controlling training fluctuations" and reducing over-optimization (Zheng et al., 2023b) . Similar to batch normalization (Ioffe & Szegedy, 2015) , for each batch X, Y of sampled outputs, we compute the mean (µ) and standard deviation (σ) of R. We then take a moving average of these values across N previous batches and "scale" R to become R ′ = R-µ σ , where we note σ remains relatively constant across training.We report results for the interventions on the reward score and PPO in Table 2 . Note the RM row is comparable within each setting since we use the same underlying reward models, and thus we use it as our primary metric to reason about length and reward tradeoffs. We also report simulated preferences (see Section 2.2) vs STD, where < 50% indicates being worse than standard PPO on downstream answer quality.We find that across all interventions, length always increases relative to SFT, and reward model score is always worse than standard PPO. These patterns suggest that a strong component of PPO is related to length. Including the fact that length control (LEN-C) led to convergence failure (reward not increasing during training) on W-GPT and STACK, this suggests that length is a difficult feature to disentangle post-hoc from reward.Recalling the scatter plots from Figure 3 , we note that across all of these different interventions, the scatter plots display similar patterns (see Appendix B), implying that while these interventions reduce the overall optimization towards length, they don't change the fundamental tendency of PPO to avoid optimizing for other features. However, while length still increases with respect to SFT, several interventions do allow for length increases to be mitigated while still recovering a large portion of reward and downstream performance gain (e.g., RM-SC). 
examining reward modeling: 
 Section 3.2 showed that interventions during PPO do not fully mitigate the issue of reward gains coming from length increases. We now investigate whether we can intervene even earlier in the process, on the preference data itself, in order to circumvent this length dependence. One root cause of length correlation is length imbalances in the preference datasets, where longer answers are systematically preferred to shorter answers. We can measure this with length heuristic agreement: the accuracy of always predicting that the longer output is the gold preferred output (see Table 3 ): We see that all datasets are slightly imbalanced towards longer outputs. However, this doesn't fully explain the strong correlations suggested earlier in Figure 3 .To understand this better, we can study training dynamics of reward model learning by computing statistics over several epochs of training. Given reward model R being trained on preference dataset P for E epochs, we can track each data point (x i , y+ i , y - i) ∈ P where we compute the distribution of confidence (RM score of "preferred" subtracted from "dispreferred"), at each epochc i = {(e, R(x i , y + i ) -R(x i , y - i )) ∶ e ∈ {2, . . . , E}}, where we exclude epoch 1 to mitigate noise.Results First, we note that when examining "cartography" plots (Swayamdipta et al., 2020) examining the mean (c i ) and variance (σ(c i )) of different c i (see Appendix B.2), we find that the values are largely centered at zero, suggesting that reward models are not able to make progress on most training examples: the predictions are low-confidence and largely do not change. This suggests that most features are instead learned on the set of "easy" examples with higher c i .With the hypothesis that length may be related to "easy" examples, we use length heuristic accuracy again, but this time, we compute it on slices where we bin training examples based on c i , plotting these bins by confidence (x-axis) against length heuristic accuracy (y-axis) on each slice as scatter plots in Figure 4 .The figure shows strikingly clean patterns, with the mean confidence c i for data in an interval of training examples correlating strongly with the length heuristic. This means that (1) the length heuristic applies to most examples that are easy, and (2) perhaps more tellingly, the overwhelming majority of "hard" examples are cases where the model follows the length heuristic to confi-dently predict the wrong answer. Overall, this supports that length is one of the strongest features learned in these models. Note that WebGPT, with the strongest pattern, also displayed the lowest WRG from Table 1 , implying that these correlations propagate through all stages.Given the strong length biases learned from preference data in standard RMs (STD), we now examine whether we can eliminate these biases by strategically modifying preference data.Length Balancing (BAL) The simplest intervention is to remove length biases from the preference data. Specifically we balance data such that the distribution of pair length differences are symmetric by bins of 10. Suppose there are more examples where preferred responses are 20 tokens longer than dispreferred ones compared to the reverse case; we then subsample the cases which are 20 tokens longer until they match the number of cases which are 20 tokens shorter, thereby balancing the data.Confidence-Based Truncation (C-TR) Our previous results suggest that something more dataspecific beyond a surface length bias may influence training: for example, a particular set of "easy" examples may be corrupting the data, and removing them may help, as established in literature on dataset cartography Swayamdipta et al. (2020) . Given that we've trained some R base , and computed c i on dataset P (Section 4.1), we can test this idea by training a new RM R trunc on a subset of P where c i < θ 1 and c i > θ 2 , with threshold hyper-parameters θ 1 , and θ 2 . We experiment with several variants (see Appendix B.2), keeping sets of 50% of the data for each. Below we report results when we set θ 1 < θ 2 , keeping a central subset of data. In line with the hypothesis that over-optimization stems from spurious correlations in the data, another potential intervention is data augmentation, specifically using "random pairing" where we can pair matching prompt output pairs q i , p i from P with p i serving as a "prefered" example, and a randomly sampled p + j from another prompt serving as a "dispreferred" example. This serves to encourage disregarding stylistic features in favor of relevance to the query.We first report in Table 4 the evaluation accuracy of these different reward models, as well as a correlation within batch (CORR) measure which, given sets of 8 generations, is the mean Pearson correlation between output length and reward model score for each batch. While the standard reward model (STD) achieves high accuracies across settings, this comes with high length correlation.Data Augmentation (R-DA) improves on both of these partially, while confidence-based truncation (C-TR) brings length correlation down at the cost of accuracy. Note that, when using correlation within batch, we find that BAL leads to length bias being reversed, but at near-random accuracies, while other truncation strategies don't yield notable differences. These patterns indicate that, perhaps because RMs fail to learn on most examples, they are particularly brittle, and can learn spurious correlations easily. As the only setting where length balancing eliminates correlation and maintains above-random accuracy, we see more evidence that STACK is the one setting of our three where reward models can learn features independent from length.We then show results for downstream adjustments to preference data in  
how far can length go?: 
 Many of our experiments suggest that our reward models are primarily guiding PPO to produce longer outputs, yet we still see improvements on downstream simulated preferences. One explanations for this is that humans and models like GPT-4 have a bias towards preferring longer outputs in the settings we study (Zheng et al., 2023a) . Another possibility is that optimizing for length with PPO intrinsically improves the quality of generation even in the absence of other features.We investigate two interventions aimed purely at increasing length, which show how far optimizing for this single aspect can go. First, we sample 8 outputs from the SFT model and choose the longest one (SFT-LONG). Second, we use length as our reward for PPO (keeping the standard KL term) with R * (y) = 1 -len(y) N-1 . In this case, N is a target length hyperparameter (set to 156, 120, and 200 on WebGPT, RLCD, and STACK respectively). We call this setting LPPO, and also explore a variant of length-only PPO with λ set to 0 (LPPO λ = 0) in Table 6 .First, we note that SFT-LONG can lead to moderate improvements (57% winrate vs SFT on STACK and 52% on RLCD), though not on WebGPT. When we then compare to LPPO, we find that purely optimizing for length actually reproduces most of the performance improvements of RLHF with the reward models. Notably, this approach yields simulated preference improvements over SFT-LONG, which has even longer outputs.It is still possible that RLHF with our reward models does lead to other changes or improvements in the outputs beyond length. This experiment also does not necessarily establish flaws in the preference judgments; these outputs with the right length are often more informative and more useful (Figure 1 ). However, it shows that downstream gains can be explained by optimizing for length. 
related work: 
 RL Reinforcement learning from human feedback has been explored extensively (Knox & Stone, 2009) , often being used in robotics tasks to extrapolate reward signal beyond an initial preference set (Brown et al., 2019) . Recent work in NLP has explored implementations (Zheng et al., 2023b; Touvron et al., 2023b) , objectives (Wu et al., 2023) , and even alternatives (Rafailov et al., 2023; Zhao et al., 2022; 2023) for RLHF, but have generally overlooked or dismissed length increases. Our work is largely orthogonal to these directions, using the issue of length to analyze the lack of robustness in current reward models. Finally, other past uses of RL in NLP (Ammanabrolu & Riedl, 2018; Martin et al., 2017; Ramamurthy et al., 2023) have largely faced different sets of issues due to reward not coming from models learned over human preferences.Reward Model In the context of noisy and biased preference data, are reward models able to learn robust features reflecting the underlying preferences? In broader NLP, dataset artifacts have been a prevalent issue even on simpler settings like natural language inference (Gururangan et al., 2018; Poliak et al., 2018) . In the context of RLHF, Stiennon et al. (2020) notes that over-optimizing for a reward model leads to pathological summaries, Dubois et al. (2023) notes a pattern of human preferences going up briefly then down as reward model score increases, and Pang et al. ( 2022) present some cases where reward hacking can be produced within synthetic settings. Our work, in comparison, delves further into what causes reward over-optimization in realistic settings, while also further exploring diagnostics and solutions. We focus on length as it is the most prevalent, but our experimental paradigm is applicable to any analysis of over-optimization in RLHF.Length control and length biases Techniques outside of RLHF for controlling length of NLP models have been explored (Kikuchi et al., 2016; Ficler & Goldberg, 2017) . Length divergences specifically between training time and test time have been explored in the machine translation literature (Riley & Chiang, 2022) , but these have been attributed to inference techniques and label bias in text generation methods. The open-ended nature of our generation problems is quite different from MT. Murray & Chiang (2018) use a per-word reward similar to our per-word penalty in RL, though to solve the opposite problem of outputs being too short. Finally, in discriminative "text matching" tasks like paraphrasing, past work has observed similar length heuristics, Jiang et al. (2022) , but the sentence-pair format of these tasks makes their issues somewhat different. 
conclusion and limitations: 
 In this work we study correlations of length and reward in RLHF. Across three datasets and several stages of observational and intervention-based exploration, we make a case that RLHF in these settings achieves a large part of its gains by optimizing for response length.While the extent of the patterns we find are surprising, this doesn't necessarily invalidate the potential of RLHF. We note that our Stack setting, which involves the most technical responses, does demonstrate improvements in reward even for outputs already at our maximum length. Furthermore, optimizing purely for length does seem to lead to "qualitative" improvements beyond just sampling from the base model and choosing longer outputs, indicating that the learning dynamics of RLHF may be beneficial for LM training. Rather than claiming length to be an inherent shortcoming, we seek to use it as a vehicle to analyzing RLHF's successes and failures.One limitation of our work is that, while we explore diverse settings, we are restricted to open-source preference datasets. Recent work such as Llama-2 (Touvron et al., 2023b) develops an extensive dataset of preferences and pursues a sophisticated RLHF strategy, which may not face the limitations we do. Furthermore, we focus primarily on a broad "helpfulness" objective (again, aligning with these preference datasets) using LLaMA-7B as the base model. While these represent a substantial fraction of research on open reward models, our findings may not necessarily apply to RLHF running on larger closed-source models, or with alternate objectives like "harmlessness".Despite these limitations, we believe our work shows that RLHF with these reward models is not yet achieving its full potential. We believe that developing more accurate and robust reward models, either by changing the reward model, its objective, or the preference collection process, may hold the key to unlocking the full capabilities of RLHF. 
 
 Appendix: For our various studies on the relationship between RLHF and length, we first trained a set of reward models and policy models. In order to support future open RLHF research, we release our code as well as reward and policy models. In addition to detailing our experimental setup and evaluation scheme in Section 2.2, as well as describing our interventions in detail in Section 3.2 and Section 4, we include further hyper-parameters and instructions in Appendix A. Note that we use open preference datasets, publicly available base models, and open-source RLHF code that doesn't require prohibitive computational resources.Hardware All experiments were conducted across 2 workstations, one with 4 NVIDIA RTX A6000 GPUs and another with 8 NVIDIA A40 GPUs. However, all of our individual experiments were run across 2 GPUs. In this configuration, training an RM takes around 6 hours on the Stack dataset, 3 hours on the RLCD dataset, and 1 hour on the WebGPT dataset after 1 epoch. For PPO, training takes around 12-18 hours for a single run. Note that for the training dynamics analysis (Figure 4 ), we run reward model training for 5 epochs to reduce variance, however we don't use those models directly (though we note that eval accuracy doesn't go down significantly even at that point).For our RLHF step, as stated before, we use LoRA and 8-bit quantization for the policy and reward models, since the TRL training configuration requires having all used models on each device used for training. We merge reward model and generation models with LoRA adapters before PPO.Past work has commented on the stability of PPO and "secrets" needed to get it working well (Zheng et al., 2023b) . We found that setting the right KL coefficient and batch size were the most important for stable convergence.For training we generally run training for between 150-200 steps, where this is a hyperparameter for each dataset depending on speed of convergence and allowing sufficient steps for KL to decrease in certain settings (Figure 5 ). We experimented with runs of up to 400 steps and generally did not find improvement in simulated preference or reward.With 2 GPUs, batch size of 32 on each, training takes around 16 hours to complete 200 steps, giving an overall time of 32 GPU hours per PPO model. Note that we use max length of 156 on WebGPT and RLCD, and 216 on STACK since it has a higher starting point for the SFT model (note that this hyperparameter has a strong influence on training speed).Figures 5, 6, 7 show statistics over the course of training for our standard settings, with KL of 0.04. We note that RLHF does successfully increase reward score. The last half of training usually yields a decrease in KL divergence, as the model has optimized for reward and is regularized closer to the initial policy model by the KL term.Once we have our trained PPO models, we finally sample outputs that we can use to compare different systems and evaluation. For all results, unless otherwise stated, we generate 500 outputs each from a fixed set of the held out data from each dataset, and base our results on those (we find this to be a sufficient amount, especially when comparing patterns across a set of interventions / settings which themselves serve as additional datapoints). Computing simulated preference (Dubois et al., 2023) for 100 outputs costs around $3.5 USD using the OpenAI API. These calls are made to gpt-4-0314, gpt-3.5-turbo-0301, and text-davinci-003.We decode with nucleus sampling (Holtzman et al., 2020) with a p value of 0.9, maximum length of 256, temperature 0.9, and a repetition penalty of 1.2 (based on the TRL repository default hyperparameters). Whenever we sample multiple outputs from a single prompt, we draw 8 samples.For length control, we set the length center N to the starting mean SFT length, noting similar patterns with different configurations as well (50 tokens for RLCD, 100 for WGPT, 200 for STACK), for omission, we use 24 tokens above these value (to allow for stable training). For reward data augmentation, we augment with 25% additional data, noting that 50% gives similar patterns, however data augmentation may need to be explored further in future work.To compare the our findings on an objective unrelated to "helpfulness" and length, we trained a reward model for harmlessness on the Anthropic data, noting a similar pattern of "difficulty" with only 68% evaluation accuaracy on the held out set. However, we did not find length correlation in this model. The within-batch length correlation is around -0.3, and doing PPO with just the harmlessness reward model didn't increase length either once converged. This is perhaps expected since a shorter response (such as abstention from answering) will often be harmless. We also observed that the outputs started to look strange eventually, so optimizing for just harmlessness has its own set of shallow features and should not be optimized on its own. This only scratches the surface, however, and further exploration on similar / alternate objectives can likely bring more insights into the inner workings of RLHF. We here include dataset cartography plots in the style of Swayamdipta et al. (2020) for our reward modeling tasks. First, Figure 8 shows the dataset cartography plots on our respective settings. Note that WebGPT, the dataset with the strongest length biases, seems the most centered at 0 variance, and symmetric with respect to the x-axis. RLCD and STACK, where there seems to be more room for other features, on the other hand, demonstrate an "upward tilt" where at higher variances, the models are able to learn correct higher confidence features. This provides evidence for our hypothesis that strong length biases emerge as a symptom of reward models being unable to learn clear features from most training data.Figure 9 shows plots for two additional settings. First, we see that when doing data augmentation, the augmented data emerges clearly and separately in the "high confidence" zone, while the initial plot remains in a similar space as before, though interestingly now displaying more of the "upward tilt" with a longer tail. Next, we see that when cutting out the central section of the cartography plot and training on the remaining data, the shape is actually preserved, suggesting that the small amount of remaining data has an intrinsic tendency to learn the strong length correlation. Likewise, when removing the upper section of "easy examples", we see that suddenly the RLCD plot becomes much more centered and sharper at the left with low variance, suggestive of the "brittleness" that we saw with WebGPT, where now the easiest pattern is harder to learn, exacerbating the pattern even further.Earlier in Section 3, we discuss the idea of reward gain due to length vs other features, examining results with Higher KL term. Here we show comparable plots for WebGPT (Figure 11 ) and RLCD (Figure 12 ). Note that the patterns and reward gain are very similar, suggesting that the length constraining techniques all perform similar functions, despite having very different formulations. Also note that for these two settings the ratio of reward gain independent of length remains quite low. C SAMPLE OUTPUTS 'if x' -as above for the conditional statement, but without need for the 'else'. If you do not specify any condition, the end will always execute.3) 'while(true); // no break'4) This should get you started along the right lines at least. The preprocessor is fairly limited though when compared with what some other languages offer! 1) For the sake of simplicity let us assume all your 'while' loops are nested inside a macro called 'FUNCT'. 2) As a first approach you could define another variable in order to test if the condition is true or not. This new variable will have as value zero when there is no loop and nonzero otherwise:"'define FOO 0; undef FUNCTdefine BOO if(FOO != 0) FUNCT(); else // do nothing here because we exit directly ; return 0; /* You can use other values instead */ int main()return BOO;"'3) Another option involves adding some variables at compile time which may increase significantly the size of your program but it also increases its readability:"'include <stdio.h>define LOOP(N) int j=1; for(j = 0 ; j< N*5-4 ; ++j )/* DO SOMETHING HERE */ return j; /* The number of iterations */ //The user defines Apart from the normal differences in pitch and tone between male and female voices, are there any different words, different intonations, or other linguistic habits that tend to be used more by men or women in Korean?Are there any 'male habits' that (to sound natural) a female learner may want to avoid imitating, and vice versa?1. No, it is not an exaggeration; Korean is gender-specific when it comes to nouns/adjectives etc..2. There exists [the following list of female names]([URL] which will help you select appropriate name for your character if needed but most likely using common female names as characters might lead to some confusion with readers because many people have heard them so often while others never hear such names ever before... So I would recommend making up new ones.1. **Tone** -I don't think it is possible to have an overview of this one because tones depend on context. For instance the word for bread has two totally distinct meanings depending if it is said with a high or low rising/falling tone. However generally speaking Koreans use lower tones at end sentences whereas native English speakers often prefer higher ones. 2. **Pitch/Timbre** -as mentioned above this depends so much on context but here again generalizations can help you out: When asking questions your voice goes up, when saying statements down; you use less vocal variety in speech than people do normally; in fact all your vocal cues like volume etcetera will be toned-down compared to how we would express ourselves if using our first language. 3. **Use of vowels** -well Korean has no shortage of long open vowel sounds while English prefers to reduce them whenever possible. This difference alone means that certain pronunciation issues become quite obvious for non natives of both languages. Aside from that however, there isn't too many rules regarding vows although they play an important role in differentiate between dialect When we close only one eye, our brain interprets the lack of input from that eye as darkness because it doesn't receive any visual information to process. But when we close both eyes, our brains combine the two images into one and interpret the combined image as darkness since there isn't enough contrast between the two images for us to be able to make out individual details. This phenomenon is known as binocular vision and helps protect us against blind spots by allowing us to use both eyes together in order to gain better depth perception and detect motion more easily. It also allows us to have a wider field of view compared to if we were using just one eye at a time. In addition, since each eye sees slightly different things depending on what angle it is facing, combining both images can help fill-in missing pieces of detail or detect objects that are closer or further away than what could be detected with just one eye. By closing both eyes, your brain uses all this data to create a single image which appears as pitch black due to the absence of significant contrasts. That being said, some people do still experience ghostly shapes and movement even when they shut their eyes completely; these effects arise from electrical activity happening within the eye itself Going to a spa can have numerous health benefits, including relaxation and stress relief. Spas provide opportunities for people to unwind in both physical and emotional ways by providing massages, facials, body wraps, manicures, pedicures, and more. This allows them to reduce their levels of stress, anxiety, and other negative thoughts and feelings while also reducing muscle tension and increasing energy level. Additionally, many spa treatments include detoxifying elements such as herbs and minerals which can help improve overall health and well-being.Note: Our settings are oriented towards helpfulness, which we infer to be closer related to length, however studying our approaches on other objectives such as harmlessness could be interesting future work.2 Stack, due to SFT having higher initial length, tends to generate unboundedly long outputs after PPO. We set a higher max length (216) than the source TRL codebase (128) for Stack; however the pattern remains.We try several variants of this idea, such as a scalar penalty past a length threshold, and note similar convergence failures. In general, we find that stricter versions of these constraints negatively affects convergence. 


Decision: This paper investigates the influence of RLHF on features of LM-generated text, particularly focusing on how it influences length of generated responses (in particular, causing an increase in length). The experiments suggest that reward improvements are primarily due to increases in length, e.g., by using response length as a reward and finding most improvements from RLHF are retained after RL-based fine-tuning, and find that this is mostly due to trained reward models learning a strong length bias (i.e., assigning higher reward to longer responses).

The experiments are well-designed, and the insights this paper provides are really important given that RL-based finetuning (especially RLHF) is currently the dominant paradigm in training LMs. 

I suggest the authors incorporate feedback and suggestions from the reviewers.

Reviews: review: paper_summary: paper_summary: In this paper, the authors present a thorough investigation into how RLHF tends to optimize for longer output lengths, often at the expense of true quality improvements. They study this phenomenon across three diverse task settings - WebGPT, Stack, and RLCD. On WebGPT and RLCD, the bulk of the reward improvement from RLHF can be attributed simply to increases in output length, with little gain coming from optimizing other features. Attempts to mitigate the length bias through various interventions in the RLHF pipeline, such as adjusting the reward function, policy rollouts, KL loss, etc. do reduce length increases to some degree, but fail to eliminate the bias and often hurt overall performance. Overall, this is a well-executed and insightful study that reveals some concerning flaws in current RLHF approaches. The authors convincingly show that a significant portion of recently reported "progress" may be illusory and simply due to producing longer outputs. The finding that even preference datasets balanced by length still produce biased reward models is particularly notable. I believe this paper is great with its originality and rigorous experiment setting.
strength_weakness: strength_weakness: I enjoy reading this paper but the only confusing section for me is Sec 3.1. I can not get why non-length reward gain (NRG) is calculated as average ∆R within each bucket weighted by the number of examples in each bucket. What does this stand for? For me the most intuitive way to investigate the correlation between reward gained by ppo and its length is just to calculate $∆R/∆L$, where $∆L$ is the length difference between sft model and ppo model.
questions: questions: - For experiment setup in Section 3.1, I wonder is sft output or ppo output you would stratify based on the length, or combined? 
- For Table 2, Could you please win/tie/lose rate in addition to win rate only? I believe this could strengthen the performance comparision between models.
- For Table 3, why std ppo has failed reward optimization? As it has length and reward, I think it should be accessed by sim perf also?
paper_summary: paper_summary: This paper presents a well-scoped investigation of Reinforcement Learning with Human Feedback (RLHF), examining an observed tendency to prefer longer outputs and how it may be mitigated. The paper is concise and convincing, highlighting an important and overlooked feature of RLHF, by questioning what our preferences actually encourage in the generated outputs. This is an important step toward better evaluation of and methods for alignment techniques such as RLHF.
strength_weakness: strength_weakness: I did not understand the last sentence of 2.1, "To control for length bias, we focus on results where shorter outputs show greater win-rates". Specifically, how this was done throughout the paper (is it distinct from showing that shorter outputs are less frequently associated with greater win-rates?) and why this is a representative way to support the argument.
paper_summary: paper_summary: The paper investigates the relationship between output length and performance in Reinforcement Learning from Human Feedback (RLHF) for large language models. The authors demonstrate across three diverse settings (WebGPT, Stack, and RLCD) that RLHF tends to significantly increase output length, and a substantial portion of the reported improvements in reward can be attributed to this length increase rather than other meaningful quality improvements. Through controlled experiments, the authors show that optimizing for length alone (using a length-based reward) can reproduce most of the gains seen in standard RLHF. They explore various interventions to mitigate the length bias, including adjustments to the reward model training, preference data, and PPO optimization, but find that length biases persist across these interventions. Further analysis reveals that reward models exhibit strong correlations with length, potentially due to overfitting to a small set of "easy" length-biased examples during training.
strength_weakness: strength_weakness: - The chosen evaluation datastes are limited and more results on general benchmarks like MT-bench or AlpacaEval 2.0 can make the results more convincing. Also, it would be interesting to see the performance change on evaluations that does not rely on the lengths of the outputs (like MMLU). 
- Due to the computational resources, the authors did not conduct experiments on larger models, and the the PPO training is based on LoRA. Even without using a larger model, the authors can try the experimental results on more powerful small models (such as mistral, llama-2 and llama-3).
questions: questions: - In page 7, Reesults -> Results.
paper_summary: paper_summary: This paper explores the tendency of Reinforcement Learning from Human Feedback (RLHF) to produce longer outputs when aligning LLMs with desired properties, such as helpfulness in dialogue and web question answering tasks. The authors delve into the strategies RL optimization employs to maximize reward and find that improvements are often driven by increasing response length rather than other features.
strength_weakness: strength_weakness: - Would the length bias issue correlate with tasks? The selected datasets are question answering and chatting, which may naturally bring the length bias issue. I suggest selecting some length-unrelated tasks such as math, coding, or length-constrained instructions for a more comprehensive evaluation.
- In my opinion, before intervening in the RLHF process to mitigate the lengthy outputs, it is more important to recognize whether human annotators and LLM judges commonly prefer lengthy outputs, and when it is a bias.
questions: questions: - Would developing more capable reward models benefit this issue? With limited GPU resources, the authors could at least evaluate whether top-ranked models on RewardBench (e.g. Eurus-RM-7B and Starling-RM-34B) still prefer longer responses.
paper_summary: paper_summary: This paper reveals an issue regarding the verbosity of large language models (LLMs). Specifically, the authors demonstrate that the response length is a significant factor behind RLHF. The authors further find that even a purely length-based reward reproduces most downstream RLHF improvements over supervised fine-tuned models. Testing a comprehensive set of length-countering interventions, this paper identifies the dominant source of the biases for the reward models. The main contribution of this paper is to reveal the vulnerability of the RLHF process.
strength_weakness: strength_weakness: I don't come up with a strong reason to reject this paper, but I think there are a few missing citations regarding length or verbosity bias in RLHF/DPO. These papers can be regarded as concurrent work, so it is optional for authors to refer to them.

- Park, Ryan, et al. "Disentangling length from quality in direct preference optimization." arXiv preprint arXiv:2403.19159 (2024).
- Saito, Keita, et al. "Verbosity bias in preference labeling by large language models." arXiv preprint arXiv:2310.10076 (2023).
- Wang, Haoxiang, et al. "Arithmetic Control of LLMs for Diverse User Preferences: Directional Preference Alignment with Multi-Objective Rewards." arXiv preprint arXiv:2402.18571 (2024).
questions: questions: N/A

Comments: title: Reply
Acknowledgement
Discussion period is now open
comment: Thanks for the reply. The additional results are great. I will increase my score.
Hi WB4w, can you check the authors' response and update your review if it addressed your concern (or participate in discussion with the authors if it did not)?
Hi Bn2p, can you check the authors' response and update your review if it addressed your concern (or participate in discussion with the authors if it did not)?
I've read other reviews and authors' rebuttals. I did not change my mind and will keep the original score.
Hi reviewers, please take a look at the author's rebuttals and the other reviews for this paper!

If the rebuttals addressed your concerns, please let the authors know about this and update your review. If not, please continue to engage with the authors and the other reviewers in the discussion forum.

Three of our five reviewers are very positive about this paper, and Bn2p and WB4w are less positive. 

Bn2p: 
- One of your concerns was that experiments were not performed on larger models; however, the authors conducted these experiments during the rebuttal period. Does this address that concern?
- You also were concerned about the choices of datasets. Do you find the authors' response addresses this concern?

WB4w: 
- Your question about different models is also addressed by the authors.
- Your concerns about length bias correlation with tasks, as well as concerns about human preferences, also received a response from the authors. Do you find this addresses your concerns?

For the other reviewers, would you like to argue for this paper's acceptance?