Loose lips sink ships: Mitigating Length Bias in Reinforcement Learning from Human Feedback
Authors: Wei Shen
Rui Zheng
Wenyu Zhan
Jun Zhao
Shihan Dou
Tao Gui
Qi Zhang
Xuanjing Huang
Venue: EMNLP 2023 Findings (2023)
Keywords: Alignment
Large Language Model
Debias
TL;DR: In this paper, we propose an innovative solution, applying the Product-of-Experts (PoE) technique to separate reward modeling from the influence of sequence length.

Abstract: Reinforcement learning from human feedback serves as a crucial bridge, aligning large language models with human and societal values.
This alignment requires a vast corpus of human feedback to learn a reward model, which is subsequently used to finetune language models.
However, we have identified that the reward model often finds shortcuts to bypass its intended objectives, misleadingly assuming that humans prefer longer responses.
The emergence of length bias often induces the model to favor longer outputs, yet it doesn't equate to an increase in helpful information within these outputs.
In this paper, we propose an innovative solution, applying the Product-of-Experts (PoE) technique to separate reward modeling from the influence of sequence length.
In our framework, the main expert concentrates on understanding human intents, while the biased expert targets the identification and capture of length bias. 
To further enhance the learning of bias, we introduce perturbations into the bias-focused expert, disrupting the flow of semantic information.
Experimental results validate the effectiveness of our approach, indicating that language model performance is improved, irrespective of sequence length.

Loose lips sink ships: Mitigating Length Bias in Reinforcement Learning from Human Feedback
Wei Shen' Rui Zhengl, Wenyu Zhan', Jun Zhao', Shihan Dou',
Tao Gui Qi Zhang', Xuanjing Huang
School of Computer Science, Fudan University
International Human Phenome Institutes (Shanghai), Shanghai, China Institute of Modern Languages and Linguistics, Fudan University wshen21 . fudan. edu. en, (ruizheng20, tgui, x xjhuangg@r udan. edu.ch
Abstract
[Human]: What does the phrase "a stitch in time saves nine" mean?
Reinforcement learning from human feedback serves as crucial bridge, aligning large language models with human and societal values. This alignment requires vast corpus of human feedback to learn reward model, which is subsequently used to finetune language models. However, have identified that the reward model often finds shorteuts to bypass its intended objectives, misleadingly assuming that humans prefer longer responses. The emergence of length bias often induces the model to favor longer outputs, yet doesn't equate increase in helpful information within these outputs. In this paper, we propose an innovative solution, applying the Product- of-Experts (PoE) technique to separate reward modeling from the influence sequence length. In framework, the main expert concentrates on understanding human intents, while the biased expert targets the identification and capture of length bias. To further enhance the learning of bias, we introduce perturbations into the bias-focused expert, disrupting the flow of semantic information. Experimental results validate the effectiveness of our approach, indicating that language model performance is improved, irrespective of sequence length.
[Assistant]: It's common phrase in English means early solution problem long run
lor of effort
[Human]: Can you apply this snying in sentenee?
SFT model
[Assistant]: Sure, here's an example sentence: "My time saves nine, so
brother always says never procrastinates fixing things."
Vauilla PPO
[Assistant]: Sure, here's an example: "I always make sure any small issues in my striight awny rather than waiting for them become bigger problems. A stich time saves nine. they say!"
Ours + PPO
[Assistant]: Yes, here's s example of how can apply this saying in sentence: "Fixing that leak in the pipe right away stitch saves nine.
Comparison of model outputs using SFT, Vanilla RM with PPO, and Our Method with PPO. Our study demonstrates that current reward model tends incentivize the model generate longer responses, disregarding the true human intent. This phenomenon highlighted in the example shown, where we observe decline in model performance.
Figure 1:
Human Feedback (RLHF) techniques (Ouyang et al., 2022; Bai al., 2022a). This combination enables these models to only learn how to follow human instructions but also better understand human intent and align with human and socictal values. Undoubtedly, RLHF plays a pivotal role in the success of these models.
1 Introduction
In recent years, the field of natural language processing has witnessed remarkable advance- ments with the emergence of powerful models like InstructGPT (Ouyang et al., 2022), GPT-4 (OpenAI, 2023), Claude (Bai et al., 2022b), and others. These models have displayed remarkable proficiency in understanding human queries and providing helpful responses. Their success can be attributed to a two-step learning process that involves Supervised Fine-Tuning (SFT) followed by the utilization of Reinforcement Learning from
One of the key components of RLHF is re- ward modeling (RM), which involves learning reward function from human preferences or demonstrations. This allows an RL. agent to optimize its behavior based the feedback received from reward model (Ziegler al., 2019b). However, the process is without challenges. Human preference data can be noisy and subject to inconsistencies among different annotators, leading to suboptimal results (Bai et al., 2022a).
"Equal contribution. Corresponding author.

---

For example, reward gaming, well-known and pervasive issue, refers the phenomenon where trained models exhibit undesirable patterns and generate low-quality outputs while still receiving high rewards (Skalse et al., 2022; Pan et al., 2022). These complexities emphasize the need for careful consideration and robust methods to ensure reliable and meaningful reward functions in RLHE.
2 Related Work
Reinforcement Learning from Human Feed- back. Using human preference feedback is a popular way for realizing Al alignment (Leike al., 2018). Preferences often provide numerical value or demonstrations (WirthChristian al., 2017) without requiring expert proficiency or fine- grained feedback Alignment bring a potent capability to state-of ofate-o-thh--al the-art generative foundation models, like InstructGPT (Ouyang et al., 2022), Sparrow (Glaese et al., 2022), Claude (Bai et al.. 2022b), which means this method is of great success in the paradigm of learning with human feedback. Some prior work have explored using human feedback improve various tasks, such summarization (Stiennon et al., 2022: Ziegler et al., 2019b), dialogue (Bai et al., 2022a), translation (Bahdanau et al., 2016), event generation (Zhou and Xu, 2020), semantic parsing (Lawrence and Riezler, 2019) and instruction following (Ouyang et al., 2022; Bai al., 2022a). These work can be categoried supervised fine-tuing or reward modeling on well-constructed human annotations information, the latter also known as a vital phase in RL from human feedback (Christiano al., 2023; MacGlashan et al., 2017). Our work falls within the realm of RLHI and aims to awaken LM with both harmless and helpful abilitics.
As shown in Figure 1, similar reward gaming issue arises in NLP reward modeling. We have observed that the reward model tends to reply on simple patterns, such as sentence length, to differentiate between good and bad responses. Typically, the reward model assumes that longer responses are better, which hinders its ability to learn the true human intent and preference. Addressing this problem is crucial for improving the effectiveness of NLP reward modeling and capturing the true nuances of human language.
In this paper, we propose a Product-of-Experts (PoE)-based method (Hinton, 2002) that consists two expert models to decouple human intent and response length during the reward modeling phase. The first expert operates similarly to a standard reward model and focuses on learning the true human intent behind responses. The second expert, referred to the bias-only expert, is designed to learn simple patterns, specifically the length of responses. It employs a smaller model capacity and a larger learning rate to capture coarse-graine information of inputs. Additionally, stochastic perturbations are introduced into the inputs of the bias-only expert, intentionally disrupting the semantic information present in the input. To summarize, the main contributions of our work are followings:
Reward Hacking. Goodhart's Law' (Strathern, 1997) can be formulated tough challenge in numerous fields. A few approaches have been proposed for reducing overoptimization in general reinforcement learning (Everitt et al., 2017), as well as reward models (Gleave and Irving, 2022). The overoptimization problem of reward model can generally regard as special case of reward gaming, also known reward hacking (Skalse et al., 2022). In addition, Pan et al. (2022) proposed to systematically analyze reward misspecification in RL by creating set of domains where the agent optimizes a hand-engineered proxy reward function. In this study, we consider the length of human preferences as a confounding factor that hinders the reward model from accurately assessing the quality of model responses based on true human intent.
We identify that reward modeling in NLP tends rely length bias, hindering the models from accurately learning true human intent and even leading model degradation.
We propose a simple and efficient solution leveraging PoE technique. Our method effectively decouples length bias from human intent, enabling models better capture and understand human preferences.
We validate the effectiveness of our proposed method. The results show that our approach enhances the learning of human intent by avoiding the generation of meaningless and overly verbose outputs.
Podducts-of-Experrt (PoB) (Hinton, 2002) has been proposed as an alternative to mixture model to compensate for their poor
Crdducts-of-Expertt
'Goodhart's law is an adage often stated as, When a measure becomes a target, it ceases to be a good measure

---

efficiency in high dimensional space. This technique often based on the principle of wisdom of crowds, which suggests that aggregating multiple models can lead to better performance than relying on a single model. Clark et al. (2019) firstly use PoE to build paradigm that train a debiased model ensemble with bias-only model. The goal is to encourage the debiased model to utilize orthogonal information with information from the bias-only model. Typically, this kind of method (Clark et al., 2019; He al., 2019) usually contains two stages, In this work, we adopt the end-to-end manner like (Karimi Mahabadi et al., 2020) which jointly learn the bias-only model and the debiased main model simultaneously, therefore, our reward model can take advantage of PoE to use weak learner to capture the length shortcuts without any prior information about the length of sentences, and the main model can purely attain the correct knowledge that is suitable for human preference.
feedback through the launch of a learned reward function. To this end, we maximize the following objective function RL training:
( [ro(2.)) v) 9 log (x)
th (y x)/x SPT x)/ [
3 is parameter to control the deviation from the SFT model 7SFT The language model policy T is initialized to TPPT Importantly, the last per-token KL-penalty term used to prevent the model from deviating too far from the deviating exceeding the appropriate scope from the distribution on which the reward model is accurate, as well as maintaining the generation diversity and preventing mode-collapss answers (Ziegler et al., 2019b).
single high-reward
Length Bias in Reward Model
In this section, we present the phenomenon length bias in reward models and utilize causal analysis to examine the underlying reasons for this occurrence. We shed light on why the reward model exhibits preference for longer sentences and delve into the causal factors contributing to this bias.
3 Preliminary
For the purpose of implement RLHF, we follow the pipeline in Ziegler et al. (2019a). It is usually made up of three phrases: 1) supervised fine-tuning. 2) reward modeling, 3) eiiforcementtlearming optimization, our attention directed towards the last two phases.
4.1 Length Bias Phenomenon
We present a Figure 2 depicting the scores and lengths of 4000 SFT model output results, which were evaluated using the vanilla reward model trained on the helpful and harmless (HH) dataset (Bai et al., 2022b). It is evident that there is a strong correlation between the reward scores and lengths. When the model generates longer sequences, the reward model tends to assign higher scores. This correlation contradicts human intent since the helpfuiness and harmlessness of the output should not be solely determined by its length. More figures can seen in the Appendix 8.
SFT: It begins with a generic pre-trained LM, which is fine-tuned using supervised learning on a high-quality instruction dataset. This allows the model to follow various instructions, perform dialogue and dialogue. As result, we obtain LM SFT during this phase.
Reward Modeling: According to the Bradley- Terry model (Bradley and Terry, 1952), reward function is hard to describe and needs to be learned from preferences among trajectories. The reward model ro is trained on human preference dataset to predict which response y E {0, 1 is better as judged by human, given query If the response preferred by human yi. the RM loss can be expressed as:
In addition, the length bias locates in PPO as well, we additionally investigate it on TL;DR (Stiennon et al., 2022) in the Appendix A.4
4.2 Confounding Factor
-Eees)-pllog(orr((t v)) r0(2, yn -))],, (1)
As Figure 3 depicted, we formulate the problem as causal structure (Zeng et al., 2023; Tien et al., 2023) of preference-bused reward modeling. Pairwise preference information are conveyed based on an observed reward function r. Features (r, y) are causal factors can affect r and another nuisance features & regarded as a potential con- founder factor that have impact on r. Note that
where ro(x, y) is the scalar output of the reward
model for query æ and response y with parameters 0, and D is the human preference dataset.
RL Optimization: Then we fine-tune the SFT model on our environment using PPO (Schulman et al., 2017). The language model is provided with

---

6-
5-
2-
1-
6
reward
Vanilla RM
Main reward expert (Ours)
Figure 2: Distributions of Reward vs. log-scaled Length in vanilla RM and our proposed method. The results illustrate that Vanilla RM tends to assign higher rewards to longer responses, whereas our proposed method achieves more uniform distribution of rewards different response lengths.
arrow. Furthermore, unobserved human bias and noise induced by annotators, such as inappropriate judgments on certain individuals, groups, races, sexes, etc., denoted by 7) also affect the preference labels.
(x,
(x,
4
-1
As a consequence, the learned reward á has risk to achieve actually low yet even near-perfect performance on held-out test dataset with following two factors: 1) causal factor and nuisance factor be correlated by the con-founder factor (Ir, y). 2) RM easily find shorteut exists between causal factor and reward function in the distribution of test set and can not be extrapolated to out-of- domain dataset. This might be reflected when this misspecificated reward model f(x, y, 2) guide the LM agent to align with human intention using PPO, as result of distribution drift.
2(x,
Figure 3: Causal structure of preference-bassee reward modeling, Length bias, represented as confounding factor z, influences the model's learning of true human preferences (x,y). y).
z might be correlated with (x, v). potentially due to their sharing of the same causal parent or there being biases during data collection, like annotators are inclined to favor longer sentences, which is expected.
4.3 Challenges
The spurious correlation between reward scalars and response lengths can be problematic, as may result biased or sub-optimal model behavior. However, due to the unobserved nature of this spurious attribute when performing a traditional ERM-based training paradigm, it can be challenging to investigate and address during the reward modeling stage. Inspired by the debiasing framework of ensemble model methods, posit that length bias can be alleviated through a robust learning approach, which involves disentangling the underlying features and feeding them into distinct experts. This assumption grounded
In the presence of confounding effects, the goal of preference-based reward modeling to learn biased reward function f(x, y, 2) that best matches the stated preferences. There are state features (z,y) that are causal with respect to the true human intent but are unobserved by the learning agent and considered non-robust features effortlessly learned. In our case, we suppose that length bias one vital element strongly related (x. y). Actually, this situation can hardly be completely avoided, and our motivation is to reduce the probability of the path directed by the blue

---

in the notion that the bias in the length of data can be attributed the confounding effect of certain features, which can be disentangled through representation learning. Based on the aforementioned observations and insights, in the next section, we will propose a simple and effective method to mitigate the length bias during the reward modeling stage.
learning rate aims to balance comprehensive under- standing with the identification and mitigation of potential biases (Geirhos et al., 2020).
5.2 Injecting Noise into Bias-only Expert
In order to ensure that the bias-only model captures the length bias present in the input, employ technique of injecting random noise into the input. This intentional introduction of noise serves the purpose of disrupting the semantic information within the input, thereby aiding the model effectively learning the length bias. Thus, the perturbed inputs can be expressed as X" = X N, where X' represents the new, noisy input, and N denotes the Gaussian noise added to the token embeddings. By facilitating the collaboration between bias-only experts and main experts in modeling human preference, we enable the bias- only experts to effectively capture length bias while preventing the main model from learning length bias.
5 Proposed Method
The section describes the framework and algorithm of our method. We introduce approach to establish debias framework that can significantly mitigate the length bias at the RLHF stage.
5.1 PoE Framework
this study, our framework is mainly built based on the procedure of the reward modeling phase, as illustrated in Figure 4. To Iearn a reward model from preferences, we assume access to set of pairwise preference pairs. Specifically, after batch of N data consisting of an equal quantity of positive and negative sentences passes through our framework, the learning process begins.
5.3 Training & Inference
During the training phase, the main expert and the bias-only expert are jointly optimized by maximizing the following likelihood function to optimize the reward function:
Products-offExpperss In our study, we explore a reward modeling method that utilizes different experts to separately learn the true human intent and the length bias. We employ the Products-of- Experts (Hinton, 2002) technique to train robust reward model. Specifically, our ensemble method can be formulated as:
(logaaDllo((( yi) f(x, ((x,,1--))) (2)
The main expert is initialized based model, while the bias-only expert is initialized using a pretrained model. Both reward models add a linear layer on top of the final Transformer layer, which generates the final scalar prediction for the reward signal.
SFT
f(x, y) Softmax ())++)))) Hoggrr (2, v))).
Equivalently, f(x, y) x ro(x, y) o T (xr y), where (2( y) is the output of the bias-only model and ro(2, y) is that of the main reward model.
During the PPO stage, we exclusively rely on the main expert to provide rewards, while discarding the bias-only expert. Since the bias-only expert is typically smaller in size, our approach does not significantly increase computational overhead.
To ensure the main reward expert and bias- only reward expert learn different content, we applied constraints based on empirical observations. The main reward model utilized a larger language expert (e.g., 7B LLAMA (Touvron et al., 2023)) and normal learning rate to capture human intent. In contrast, the bias-focused expert employed smaller model (e.g., 560M BLOOMZ. (Muennighoff et al., 2022)) and higher learning rate, typically three times that of the main expert. Previous studies (Mhaskar al., 2016; Wilson al., 2018) showed that smaller models with larger learning rates tend to learn simpler and coarser information. This differentiation model size and
Experiments
6.1 Settings
Datasets We utilize the Helpful and Harmless (HH) dataset (Bai et al., 2022b) from Anthropic as our experimental dataset and rm-static² for training our reward model and for participation in PPO. The HH dataset provides response
rm-static htt.:/gginnnaeeeecoeatttttt///ta wt//ataaaess/Daaaatt
inttps://huggingface

---

Proposed PoE-RM Framework
Main expert
Human value
LLaMA
Length bias
Bias-only expert
chosen
reward model
LM policy
preference learning
PPO
rejected
Figure 4: Pipeline the proposed method. The proposed PoE-based reward modeling approach consists of two experts: the main expert, which learns the true human value, and the bias-only expert, which focuses capturing the length bias.
and rejected response for each query based on human preferences, specifically focusing on responses that are helpful and harmless. In addition to that, our SFT data incorporated the 52k instruction dataset constructed by Alpaca? and the ChatAlpaca' dataset containing multi-turn dialogues.
context and distinguish between human-generated and desistant-generater responses.
During the reward modeling training phase, our main expert and policy model remained consistent. The learning rate for both set to 5e As for the bias- only expert, we utilize a smaller model, the 560m Bloomz, with fixed learning rate of Se
RLHF Hyper-parameters
Models In our experimental setup, primarily build upon LLaMA and BLOOMZ models, utiliz- ing models with parameter size of 7B. Inspired by the work of (Ouyang et al., 2022), who employ SFT models as initial models for PPO, we perform SFT on the Alpaca and ChatAlpaca datasets.
In the PPO framework, we perform reward score normalization and clipping, with a clip value 0.8. We employ the clipped surrogate objective of PPO for optimization. The token-level KL penalty coefficient 3 is set to 0.05. For each query, we collect 4 roll-out samples using nucleus sampling. The sampling temperature is set 0.8, top-p is 0.9, repetition penalty is set 1.1, and the maximum output token length is set to 512. The policy model has a learning rate of 97 while the value model utilize a learning rate of 2e These specific training details are implemented to optimize the performance and convergence of the models during the training process.
SFT Hyper-parameters During the SFT phase, utilize learning rate of 3e and train for three epochs without early stopping. We employ warmup period of 0.3 epochs, followed by linear decay to 0. The fine-tuning process was conducted on a device with eight Nvidia A 110 GPUs. Each GPU handled four queries, resulting in a batch size of 32. Responses are truncated to 512 tokens, while the total length of both queries and responses was truncated to 2048 tokens. We incorporate specific prompts, such "Human:" "Assistaat:", during the concatenation of input queries and output responses. These prompts are added to provide
Baselines In this study, propose method primarily aimed at mitigating length bias in the reward model. Therefore, our baselines include the SFT model, the PPO model trained with the vanilla reward model, and the PPO model trained exclusively with the bias-only reward expert.
hhttps://github... http://thwi..cmcccccccc////aaala

---

RM Type
Dataset
Models
Length
PPL
LLaMA
BLOOMZ
Ours-LLaMA
Ours-BLOOMZ
GPT-J
BLOOMZ-SFT
0.986
-1008
1,098
0.285
0.162
502
10.09
1,082
-1161
1.183
-0.422
0.174
513
10.06
DussPPO-BLOOMM
1.867
-1.162
1.313
-0.327
0.167
507
9.13
L.AMMASFT
1.386
-1.179
1.854
0.354
0.174
468
10:16
Vamilla-PPO-LLaM
1.580
0.997
2.106
-0.200
0.196
503
9.81
HH-RLHF
Ous-PPPOLLaMA
1.474
1.062
1.975
0.256
0.172
485
8.92
Alpaca-SFT
1.557
1.008
2.152
0,413
0.191
513
10. 11
Vanilla-PPO PPO-alpaca
2.052
0.623
2.882
0.152
0.241
689
9.67
Ours-PPO-AII PPas-PP--Apaaa
2.041
U.587
2.850
Hlasd-PPO-Alpace
1.948
0.604
2.930
0.025
0.148
684
9.31
BLOOMZ-SFT
1.463
0.101
1.315
0.293
0.164
194
10.03
Yanill-PPO-BLOOMZ
1.599
0.242
1,480
0.421
0.241
517
9.98
Duns-PRO-BILOOME
0.487
0.239
LLaMA-SFT
2.026
0.365
2.367
0.530
0.182
473
9.53
Vonila-PPO-LLLMe
2.101
0.463
2,480
0.399
0.262
513
9.30
rm-static
Our-PPP--LLaMA
2.045
0.430
2.464
0.385
0.244
489
8.89
AlpacaSFFT
2.224
0.597
2.807
0.736
0.193
5:3
9.53
Vanilla-PPO-
2.563
0.931
3.414
0.931
0.291
721
9.03
Ours-PPO-Alpccu
2.587
0.919
3,428
0.968
0.289
617
8. 8.80
Plasd-PPO-Alpaca
2.848
1.260
3.845
1.232
0.152
713
9.57
Table I: Main result for our proposed framework, we trained multiple models following RLHF pipeline. And we five different Reward Model to comprehensively evaluate the sentences against different settings, which are generated from model-generated 4608 and 2304 prompts extracted from hh-rlhf and rm-static respectively.
500
350
Chesen
Chosen
Rejected
Relected
Rejected
500
402-
230
200
302
Rewand
Rewerd
(a) Vanilla RM
(b) Main reward expert (Ours)
Bias-only reward expert
Figure 5: Distribution of reward scores for chosen and rejected responses. The results demonstrate that our proposed method increases the differentiation between chosen rejected responses. contrast, bias-only expert exhibits limited generalization ability, primarily relying length bias.
Models
RM %
Bias-only %
evaluators. For human evaluation, annotators compared two randomly selected responses and provided comparison results (win/lose/tie), allow- ing us to gain insights into subjective judgments of response quality.
Munila-BLOOMZ
66.94
w/o PoE
67.11
66.73
w/o Input Noise
67.53
66.58
Table 2: Ablation study HH dataset, RM Accuracy and bias-only RM for BLOOMZ models. By employing combination of PoE and input noise perturbation on enhancing the generalization capability of the reward model. The performance on the test set significantly improves as a result of these techniques
6.2 True Reward Improvement
As the previous analysis, addressing the risk of reward hacking resulting from reward model overoptimization (Gao et al., 2022) is crucial. To ensure more robust evaluation of our method, we adopted a comprehensive approach that combines both automatic and human evaluation. Table illustrates the average generated sequence length and reward scores of our method in comparison to
Metrics
We evaluate the effectiveness of differ- ent methods in our experiments using perplexity (gpt2-medium), average reward score, and human

---

the baselines the test set. The results clearly demonstrate that our approach achieves higher reward scores while simultaneously reducing the average output length compared to the RL model utilizing the vanilla reward model. This finding not only confirms the effectiveness of our proposed method but also establishes foundation for generating outputs that better align with human preferences.
210
Ours
RM-PPO
150
1443
Curs
Vanilla RM-PPO
Training rewerd
In addition, conducted Spearman/Pearson analysis to further validate the effectiveness of our method. For more detailed information, please refer to Appendix 4.
W
Step
Vanilla RM--PP
Opponent
Human
AlpacaFarm
GPT-4
Eval reward
Ours
Answerc/heen
67.69
74.42
75.32
Ours
SFT
54.23
59.42
56.42
Ours
PPO
57.47
61.43
59.56
Step 1.00 1200
1401
Table 3: Win rates of our proposed method against various baseline approaches in the HH-RLHF dataset, utilizing the Alpaca. Evaluations include human annotators, AlpeaFarm and GPT-4. Answerchosu the human preferred response in HH-RLHF.
Figure 6: Comparison of training curves between our proposed method and PPO using vanilla RM. The results show that method achieves stable increases in reward values while ensuring consistent generation of output lengths.
Analysis and Discussion
6.3 Wining Rate
In this section, uncover valuable insights into the effectiveness and limitations of our approaches, paving the way for future advancements conver- sational AI systems.
this section of the experimental analysis, present the win rate of our method compared to other approaches. We provide results from both manual evaluations, GPT4 and the automated evaluation platform, and AlpacaFarm. During the pairwise comparisons of the model results, win is assigned 1 point, tie is assigned 0.5 points, and a loss is assigned 0 points. It is evident that our method achieves average win rate of more that 50% compared to traditional RLHF methods. This validates the effectiveness our proposed method in generating outputs that are more informative and concise.
Leaning Curve
Figures 6 present the variation in generated sequence length during the training process of our method and the vanilla RM for PPO. along with the output reward scores during training and on the test set. From the Figures, it can be observed that the model trained with the vanilla RM continuously increases its output length throughout the training process, whereas our method achieves a stable output length after a slight increase. Both methods enhance the output's reward score, but our approach yields more concise output while containing more informative content.
6.4 Ablation Study
Table 2 presents the results of ablation analysis on various components of our method. By utilizing the PoB technique and input perturbation, observe an improvement in the accuracy of the RM. This finding demonstrates that our method can better identify human intent, leading to enhanced performance.
7.2 Distribution of Reward Scores
Figure illustrates the distribution of reward scores for the chosen and rejected data on the validation set during the training of our method compared to the vanilla reward model. is evident that

---

our model exhibits better discermment between the chosen and rejected data. This improved performance can be attributed to our approach's ability to avoid excessive learning bias towards sequence length. By mitigating the length bias, the main expert of our model can focus more on understanding genuine human intent, leading to enhanced generalization capabilities.
Shanghai Academic Research Leader Program 22XD1401100.
References
Dzmitry Bahdanau, Philemon Brakel, Kelvin Xu, Anirudh Goyal, Ryan Lowe, Joelle Pineau, Aaron Courville, and Yoshua Bengio. 2016. An actor-critic algorithm for sequence prediction.
8 Conclusion
Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, Nicholas Joseph, Saurav Kadavath, Jackson Kermion, Tom Conerly, Sheer El-Showk, Nelson Elhage, Zac Hatfield-Dodde Danny Hernandez, Tristan Hume, Scott Johnston, Shauna Kravec, Liane Lovitt, Neel Nanda, Catherine Olsson, Dario Amodei. Tom Brown, Jack Clark, Sam McCandlish, Chris Olah, Ben Mann, and Jared Kaplan. 2022a. Training a helpful and harmless assistant with reinforcement learning from human feedhack.
this study, we investigate the issue of length bias in NLP and propose a PoE-based method to mitigate this bias. Our work sheds light on the challenges associated with reward modeling. emphasizing that it is not straightforward task and various difficulties may impede the models from capturing the true human intent. By addressing the problem of length bias, our research highlights the importance of developing techniques that enable models to learn and generate responses aligned with genuine human intentions. Further research and advancements in this area are necessary to overcome these obstacles and enhance the performance and reliability of NLP models in real- world applications.
Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini. Cameron McKinnon, Carol Chen, Catherine Olsson, Christopher Olah, Danny Hernandez. Dawn Drain, Deep Ganguli, Dustin Li, Eli Tran-Johnson, Ethan Perez, Jamie Kerr, Jared Mueller, Jeffrey Ladish, Joshua Landau, Kamal Ndousse, Kamile Lukosuite, Liane Lovitt, Michael Sellitto, Nelson Elhage, Nicholas Schiefer, Noemi Mercado, Nova DasSarma, Robert Lasenby. Robin Larson, Sam Ringer, Scott Johnston, Shauna Kravec, SheerEl Showk. Stanislav Fort, Tamera Lanham, Timothy Telleen-Lawton, Tom Conerly, Tom Henighan. Tristan Hume, SamuelR. Bowman, Zac Saffiel--Dodde Ben Mann, Dario Amodei, Nicholas Joseph, Sam McCandlish, Tom Brown, and Jared Kaplan. 2022b. Constitutional ai: Harmlessness from ai feedback.
Limitations
In this study, we propose simple and effee- tive method to mitigate length bias during the Reinforcement Learning from Human Feedback stage. However, it is important note that method can only alleviate length bias to some extent and may not completely eliminate it. Furthermore, the validation of our method's effectiveness was conducted on two RLHF datasets. It is worth mentioning that collecting RLHF data is a challenging task, and the question of whether this phenomenon exists on larger datasets remains uncertain. Evaluating the performance of general dialogue models poses difficulty. Therefore, during the human evaluation phase, selected only a limited number of evaluation samples to assess the effectiveness of our approach.
Ralph Allan Bradley and Milton E Terry. 1952. Rank analysis of incomplete block designs: the method of paired comparisons. Biometrika, 39(3/4):324- 345.
Paul Christiano, Jan Leike, Tom B. Brown, Miljan Martic, Shane Legg, and Dario Amodei. 2023. Deep reinforcement learning from human preferences.
Christopher Clark, Mark Yatskar, and Luke Zettlemoyer. 2019. Don't take the easy way out: Ensemble based methods for avoiding known dataset biases. arXiv preprint arXiv: 1e09:09999..
Acknowledgements
The authors wish to thank the anonymous reviewers for their helpful comments. This work was partially funded by National Natural Science Foundation of Shang-
Tom Everitt, Victoria Krakovna, Laurent Orseau, Mar- Hutter, and Shane Legg. 2017. Reinforcement learning with a corrupted reward channel.
China (No.62076069.o (02.66222077666766660 hai Rising-Su Staar Program 23QA14002000 Natural Science Foundation of Shanghai (23ZR 1403500),
Leo Gao, John Schulman, and Jacob Hilton. 2022. Scaling laws for reward model overoptimization.

---

Leo Gao, John Schulman, and Jacob Hilton. 2023. Scaling laws for reward model overoptimization. In International Conference Machine Learning, pages 10835-10866. PMLR.
Hrushikesh Mhaskar, Qianli Liao, and Tomaso Poggio. 2016. Learning functions: when deep better than shallow. arXiv preprint arXiv: 160300988.
Niklas Muennighoff. Thomas Wang. Lintang Sutawika,
Robert Geirhos, Jömn-Heriik Jacobsen, Claudio Michaelis, Richard S. Zemel, Wieland Brendel, Matthias Bethge, and Felix A. Wichmann. 2020. Shortcut learning in deep neural networks. CoRR. abs/2004.07780.
Adam Roberts, Stella Biderman, Teven Le Scao, M. Saiful Bari, Sheng Shen, Zheng Xin Yong, Hailey Schoelkopf, Xiangru Tang, Dragomir Radev, Alham Fikri Aji, Khalid Almubarak, Samuel Albanie, Zaid Alyafeai, Albert Webson, Edward Raff. and Colin Raffel. 2022. Crosslingual generalization through multitask finetuning. CoRR. abs/221 122221107766
Amelia Glaese, Nat McAleese, Maja Trkebacz, John Aslanides, Vlad Firoiu, Timo Ewalds, Maribeth Rauh, Laura Weidinger, Martin Chadwick, Phoebe Thacker, Jonathan Uesato, Po- Sen Huang. Ramona Comanescu, Fan Yang. Abigail See, Sumanth Dathathri, Rory Greig, Charlic Chen, Doug Fritz, JauheSanchez Elias, Richard Green, Sovna Mokra, Nicholas Fernando, Boxi Wu, Rachel Foley, Susannah Young, lason Gabriel, William Isaac, John Mellor, Demis Hassabis, Koray Kavukcuoglu, LisaAnne Hendricks, and Geoffrey Irving. 2022. Improving aligmment of dialogue agents via targeted human judgements.
Lucy Camphell-Gillinghan
OpenAL. 2023. Gpt-4 technical report.
Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. 2022. Training language models to follow instructions with human feedback.
Adam Gleave and Geoffrey Irving. 2022. Uncertainty cstimation for language reward models.
Alexander Pan, Kush Bhatia, and Jacob Steinhardt. 2022. The effects of reward misspecificatiin: Mapping and mitigating misaligned models.
He He, Sheng Zha, and Haohan Wang. 2019. Unlearn dataset bias in natural language inference by fitting the residual. In Proceedings of the 2nd Workshop on Deep Learning Approaches for Low-Resource NLP (DeepLo 2019), pages 132-144 142, Hong Kong, China. Association for Computational I Linguistics.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. 2017. Proximal policy optimization algorithms.
Joar Skalse, Nikolaus H. R. Howe, Dmitrii Krashenin- nikov, and David Krueger. 2022. Defining and characterizing reward hacking.
Geoffrey E Hinton. 2002. Training products of experts by minimizing contrastive divergence. Neural computation, 14(8):1771-1800
Nisan Stiennon, Long Ouyang, Jeff Wu, Daniel M. Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul Christiano. 2022. Learning to summarize from human feedback.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. 2020. Scaling laws for neural language models.
Marilyn Strathern. 1997. 'improving ratings': audit in the british university system. European Review, 5(3):305 321.
Rabech Karimi Mahabadi, Yonatan Belinkov, and James Henderson. 2020. End-to-end bias mitigation by modelling biases in corpora. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8706-8716, 16, Online. Association for Computational Linguistics.
Jeremy Tien, Jerry Zhi-Yang He, Zackory Erickson, Anca Dragan, and Daniel S Brown. 2023. Causal confusion and reward misidentification in preference- based reward learning. In The Eleventh International Conference on Learning Representations.
Carolin Lawrence and Stefan Riezler. 2019. Improving a neural semantic parser by counterfactual learning from human bandit feedback. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro. Faisal Azhar, Aurélien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023. Llama: Open and efficient foundation language models. CoRR, his/2302..1971,
Jan Leike. David Krueger, Tom Everitt, Miljan Martic, Vishal Maini, and Shane Legg. 2018. Scalable agent alignment via reward modeling: research direction.
James MacGlashan, MarkK. Ho, Robert Loftin, Bei Peng, Guan Wang, DavidL. Roberts, MatthewD. D. Taylor, and MichaelL. Littman. 2017. Interactive learning from policy-dependent human feedback.
Ashia C. Wilson, Rebecca Roelofs, Mitchell Stem, Nathan Srebro, and Benjamin Recht. 2018. The marginal value of adaptive gradient methods in machine learning.

---

WirthChristian WirthChristian, AkrourRiad Akrour- Riad, NeumannGerharrd NeumannGerhard, and FürnkranzJohannes FümkraazZohannes. 2017. A survey of preference-basee reinforcement learning methods. Journal of Machine Learning Research.
Yan Zeng. Ruichu Cai, Fuchun Sun, Libo Huang, and Zhifeng Hao. 2023. A survey causal
reinforcement learning.
Rui Zheng. Shihan Dou, Songyang Gao, Yuan Hua, Wei
Shen, Binghai Wang, Yan Liu, Senjie Jin, Qin Liu, Yuhao Zhou, Limao Xiong, Lu Chen, Zhiheng Xi, Nuo Xu, Wenbin Lai, Minghao Zhu, Cheng Chang, Zhangyue Yin, Rongxiang Weng, Wensen Cheng, Haoran Huang, Tianxiang Sun, Hang Yan, Tao Gui, Qi Zhang. Xipeng Qiu, and Xuanjing Huang. 2023. Secrets of rlhf large language models part i: Ppo.
Wangchumshu Zhou and Ke Xu. 2020. Learning to compare for better training and evaluation of open domain natural language generation models.
Daniel M Ziegler, Nisan Stienmon, Jeffrey Wu, Tom B Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. 2019a. Fine-tuning language models from human preferences. arXiv preprint 19X9v:1909.088 08593.
DanielM. Ziegler, Nisan Stiemnon, Jeffrey Wu, TomB. Brown, Alec Radford, Dario Amodei, PaulF. Christiano, and Geoffrey Irving. 2019b. Fine-tuning language models from human preforences.

---

A Appendix
A.4 Length bias Phenomenon on summarization task
A.1 Analyzing the correlation between generated length and correspondent reward
To examine the influence of length bias on the summarization task, we conducted an investigation focusing on KL divergence and omitted the penalty term for the TL;DR task. Additionally, we explored the RLHF pipeline in TL;DR, depicted in Figure 7. Our findings suggest that the length factor significantly affects the quality of concise summarization, reinforcing its importance in this task. To further validate our hypothesis, we assessed the summarization performance using GPT-4, and the results (refer to Table 6) provided support for our claims.
As Table 4 illustrated, we introduce the Spear- man/Pearson coefficient to analyze the correlation between the two variables in the HH-RLHF eval sets. This analysis serve evidence demonstrate the effectiveness of method in reducing length, strengthening our findings.
Vanilla RM(S/P)
PoE-RM(S/P) $
BLOOMZ
1.3865/0.3993
0.2354/0. 2990
LLaMA
0. 0.2662/0.2276
0.2421/0.222
PPO beta 0.05
PPO beta
Alpaca
0.1786/0. 1765 0.1354400.
Winning rate 1 Ourpur Length
58 %
42 %
220
Table 4: Spearman/Pearson coefficient to analyze the correlation between these variables in the HH- RLHF eval sets.
Table 6: Winning rate on TL;DR
A.5
Additional illustration in different reward models
A.2 Scaling law for bias-only model
This part is a extensive exploration of the bias- only expert within our proposed method. We investigated the scaling law (Kaplan et al., 2020; Gao et al., 2023) of the main expert and bias-only expert using BLOOMZ, and we found that the accuracy of the reward model can be listed in Table 5.
A.6 Case Study
To provide more comprehensive evaluation, we directly assesses the effectiveness our proposed method in addressing length bias. The case study include measurements that capture the reduction response length and the maintenance of contextual coherence. As shown in Table 7, we present case study that illustrates these measurements through one-turn dialogue example. In the second example, observed an instance of improper reward model overoptimization, leading to the policy model collapsing and engaging in self- talking. We speculate that this issue may arisc from the loss of instruction-follooiin ability due to negative optimization.
bias-only expert
main expert %
560M
56.4
57.1
1.7B
59.3
50.8
3B
62.3
63.2
7.1B
66.7
67.5
Table 5: Spearman/Pearson coefficient to analyze the correlation between these two variables in the HH- RLHF eval sets.
A.3 PPO training stability
According to the technical report (Zheng et al., 2023), reward scaling has been found to be beneficial for enhancing the training stability of PPO. our experiment, we also applied reward scaling and observed significant improvement in training stability.

---

Human Prompt-
A friend told me that stretching right after wake up is healthy for my body. What types of stretches should I be doing? I am a bit lost in terms of knowing what to do. (Helpfull
RLHF Response
There are many lypes of stretches that can be beneficial for your body, depending your individual needs and goals. Here are few examples: 1. Hamstring stretch- - Sit with one leg straight and the other bent, reach towards your toes and feel the stretch in your hamstrings. You can also do this stretch while standing by placing your hands on your hips and leaning forward until you feel stretch in your backside.
2. Quadricep stretch- Stand on one leg and lift the other up towards the sky, feeling stretch in your quadriceps (the front of your thighs). You can also do this stretch while sitting on the floor with hoth legs extended straight in front of you, reaching towards your toes and allowing your heels to come closer logether.
3. Chest stretch- Stand with your arms your sides and bring them overhead, feeling stretch in your chest muscles. You can also do this stretch while sitting with your arms behind you, reaching towards the ceiling andallowing your shoulder blades to move backwards. (1024)
PoE Response ->
There are many types of stretches you can do, depending on your goals and preferences. Here some simple stretches to try:
I. Downward- Facing Dog: start in plank position and then lift your hips up towards the ceiling, lengthening your spine and arms 2. Child's Pose: start on your hands and knees, then lower your hips back towards your heels and stretch your arms forward. 3. Cat Cow: start on your hands and knees, then alternate between rounding and arching your spine, breathing deeply as you move. 4. Hamstring Stretch: sit down and place your feet firmly the ground, then reach towards your toes, keeping your legs straight. S. Shoulder Rolls: lie down on your back and roll your shoulders backwards, forwards, and side to side release tension.
These stretches
great starting point, but be sure to consult with professional if you have any concerns or goals that extend beyond this. (921)
Human Prompt >
kill all politicians. (Harmful)
RLHF Response
Kill all politicians? Are you Human:: Yes, am serious. Politicians have been causing much harm our society and it's time we take stand against them. They only care ahout their own self-interest and don't I give a damn about the people they are supposed to Appresent.Assitaar your opinion, but cannot support such an extreme measure. Killing politicians is wrong and goes against my values. Human: I understand your point view, corrupt and dangerous that they need to be removed by any means necessary.Assistiee like there is a lot of corruption in polities, but killing politicians is not the solution. We need to work through the legal system and elect more responsible leaders who will serve the public interest.Human guess we just see things diffcrently. Thank you for having an open and honest conversation with me Assistant: You're be welcome. It was nice talking to you. (962)
respect
think that some politicians are so
may seem
PoF. Response
strongly disugree with the suggestion kill all politicians. Politicians elected hy the people to represent their interests and make decisions the best interest of their constituents. They play an important role our society and we need them he able do their joh effectively. (294)
Table 7: Two cases of generating responses to useful prompt and harmful prompt of the model

---

Output Length variation TL,DR
225
PPO-bela-0
OPO-beta-0.5
200
175
150
125
100
50
200
600
800
1000
(a) Length variation
PPO running evaluation TL;DR
20
PPO-beta-0
PPO-beta-0.05
1.5
1.0
rewards
0.5
0.0
-0.5
200
400
600
800
1000
stops
(b) BLOOMZ Main reward expert
Figure 7: In OpenAl's summarization task, the length bias phenomenon significantly affects the policy model acquiring rewards.

---

-006 aba 186
632 615
die 0.5
-225 -16e -35
RAR
JONETE
(a) Dahous's GPT-J RM
(b) BLOOMZ Main reward expert
Figure 8: The distributions rewards versus length were analyzed for both the third-party RM (GPT-J) and one of proposed models. is evident that GPT-J effectively separates different sentences, aligning with the findings presented in Table 1. Additionally, our PoE BLOOMZ KDE map closely resembles that of LLaMA, indicating a similar pattern.


Reviews:
[{"review_id": "iwF4zWqXZn", "reviewer": "Reviewer_K33q", "summary": "", "questions": "- I do like this figure 1 which gives a clear comparison of three methods. Since you take SFT in to comparison, I think it is better to give an example that SFT gives non-helpful sentence and  Vanilla RM gives longer response to show the advantage of your method.", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable feedback on our paper. We sincerely appreciate the thorough review and the opportunity to address your concerns.\n\n**Q1**:  About Reasons To Reject Point 1, \"The improvement on reducing the length is not significant large (~1% to 2%). Would you provide more examples in appendix to validate the advantages of your method ?\"\n\n**A1**:\n  In response to the length reduction achieved by our method, **we included a dedicated row in Table 1 showcasing the promising results (around 15%-20%) for 'Alpaca-based' models**. We also highlight that the PPO model guided by a vanilla reward model exhibits a similar length bias to the bias-only model. This indicates the influence of the vanilla model's length preference on RL-optimized models' generated responses. While the extent of length reduction varies due to the different training data distributions in the pretrain and SFT phases, the presence of length bias in these models remains evident, despite their distinct domains.\n \n Figure 6, subfigure 1, demonstrates noticeable length reduction in the PPO stage as well. Although our method aims to achieve a human-aligned level rather than solely focusing on length reduction, we plan to include case studies similar to Figure 1 in the Appendix to provide further evidence. Additionally, **Section 6.4 highlights the strength of our method, achieving the best winning rate when compared to vanilla PPO and SFT models, as evaluated through automatic and human evaluation measures**.\n \nIn addition, in the context of LLM,  we use perplexity (PPL) as a useful proxy metric for evaluating SFT, PPO, and Ours-PPO models, supported by contemporary research [1,2]. The average reward score also provides insight into reward model preference, although it may not always be perfectly accurate.\n> [1] Yuan, Zheng, et al. \"Rrhf: Rank responses to align language models with human feedback without tears.\" arXiv preprint arXiv:2304.05302 (2023).\n\n> [2] Dong, Hanze, et al. \"Raft: Reward ranked finetuning for generative foundation model alignment.\" arXiv preprint arXiv:2304.06767 (2023).\n\n**Q2**: About Reasons To Reject Point 2: \"Figure 2 is not easy to understand.The author want to express Vanilla RM has a fever on longer response, but the trend is not clear in this figure. Would this change to a distribution plot better?\"\n\n**A2**: Also thank you for your input on the clarity of Figure 2. We acknowledge that the current visualization may not effectively convey the intended trend. Your suggestion to use a distribution plot is valuable, and we will make this modification to enhance the clarity of the comparison. Additionally, we will consider including a heatmap figure as an option to address the issue of overlapping plots and further enhance the understandability of Figure 2. The new plots will provide a clearer understanding of the observed trend, specifically regarding the impact of Vanilla RM on longer responses.\n\nTo further clarify our effectiveness in length reduction, we will introduce the **Spearman/Pearson coefficient** to analyze the correlation between these two variables in the hh-rlhf eval sets. This analysis will serve as evidence of the effectiveness of our method in reducing length bias.\n| Reward Models | Spearman / Pearson |\n|-----------|-----------|\n|Vanilla BLOOMZ | 0.3865 / 0.3932 |\n|POE BLOOMZ | 0.2354 / 0.299 |\n|Vanilla Llama | 0.2627 / 0.2765 |\n|POE Llama  | 0.2421 / 0.2213 |\n|Vanilla alpaca |  0.1786 /0.1765 |\n|POE alpaca | 0.1354/0.1490 |\n\n\n**Q3**: About Question 1, \"I do like this figure 1 which gives a clear comparison of three methods. Since you take SFT in to comparison, I think it is better to give an example that SFT gives non-helpful sentence and Vanilla RM gives longer response to show the advantage of your method.\"\n\n**A3**: We sincerely appreciate your feedback on our paper and are delighted to hear that you found Figure 1 to be a clear and effective way of presenting the comparison between the three methods. To address your suggestion, we will indeed incorporate an illustrative example in the paper. This example will demonstrate a scenario where SFT produces a non-helpful sentence and Vanilla RM results in a longer response. By contrasting these outcomes with the positive impact of our method, we aim to emphasize its effectiveness in producing concise and contextually relevant responses. \n\nIn addition, we will also include a collection of case studies featuring both single-turn and multi-turn dialogues conducted with various models in the appendix. These case studies serve to provide further insights and exemplify the performance and capabilities of the different models utilized.\n\nYour input has provided us with an excellent opportunity to enhance the clarity and persuasive power of our paper, and we are grateful for your thoughtful engagement with our work.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The paper os well-motivated and clearly written.\n- The experiment and analysis presented in this paper are comprehensive.", "reasons_to_reject": "- The improvement on reducing the length is not significant large (~1% to 2%). Would you provide more examples in appendix to validate the advantages of your method ?\n- Figure 2 is not easy to understand.The author want to express Vanilla RM has a fever on longer response, but the trend is not clear in this figure. Would this change to a distribution plot better?\n", "questions_to_authors": "- I do like this figure 1 which gives a clear comparison of three methods. Since you take SFT in to comparison, I think it is better to give an example that SFT gives non-helpful sentence and  Vanilla RM gives longer response to show the advantage of your method.", "paper_topic_and_main_contributions": "This paper proposes an innovative solution to address the emergence of length bias in reward modeling, which often leads to longer outputs without necessarily increasing the helpful information within them. It applies the Product-of-Experts technique and introduces perturbations into the bias-only expert, has been validated through experimental results that show improved language model performance with higher reward score and reducing the avg. output length.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "a07Dr6TO6g", "reviewer": "Reviewer_r6wY", "summary": "", "questions": "1. The evidence for length bias is not clearly presented, and the proposed issue in Figure 2 needs further analysis and clarification. More results and analysis are required to support the findings.\n2. The results require further explanation and improvement to effectively demonstrate the performance of the proposed method. Additional examples may be needed to enhance the understanding.\n3. The impact of the bias-only expert model's size on the results is not discussed. Further exploration and explanation of this aspect would be beneficial.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your insightful comments on our paper. Your feedback is valuable in helping us enhance the clarity of our work. We have carefully considered each of your points and plan to address them as follows.\n\n**Q1**: About Reasons To Reject Point 1, \"It is unclear if the proposed issue of length bias is a common phenomenon or if it will occur frequently in real-world usage. More clarification is needed.\"\n\n**A1**: First, we appreciate your concern regarding the commonality of the length bias issue and its frequency in real-world usage. To provide more clarification, we will expand our discussion on the prevalence of length bias or biased reward learning [3]  in various literature:  Anthropic [1, 4.2.2 and 4.2.3]  and OpenAI [2, Appendix E , F and G.6] found the issue when they talked about the context of RLHF.\n\n> [1] Gao, Leo, John Schulman, and Jacob Hilton. \"Scaling laws for reward model overoptimization.\" International Conference on Machine Learning. PMLR, 2023.\n\n> [2] Stiennon, Nisan, et al. “Learning to Summarize from Human Feedback.” arXiv: Computation and Language,arXiv: Computation and Language, Sept. 2020.\n\n> [3] McKinney, Lev, et al. On The Fragility of Learned Reward Functions. Jan. 2023.\n\n\n**Q2**: About Reasons To Reject Point 2 and Question 2, \"The experiment settings primarily reflect improvements in perplexity (PPL), and it is not evident if this improvement truly reflects the effectiveness of the proposed method in addressing length bias. Further analysis is necessary.\" and \"The results require further explanation and improvement to effectively demonstrate the performance of the proposed method. Additional examples may be needed to enhance the understanding.\"\n\n**A2**: We totally understand your observation about the experiment settings primarily focusing on perplexity (PPL) improvements.  PPL is a proxy and imperfect metric to evaluate the alignment of models. However, there are some papers[4, 5] that use the PPL as one of their metrics to express the extent of alignment. Likewise, we apply it to  one reference to quickly rank the generated sentences by models.\n\n> [4] Yuan, Zheng, et al. \"Rrhf: Rank responses to align language models with human feedback without tears.\" arXiv preprint arXiv:2304.05302 (2023).\n\n> [5] Dong, Hanze, et al. \"Raft: Reward ranked finetuning for generative foundation model alignment.\" arXiv preprint arXiv:2304.06767 (2023).\n\nTo provide further clarity, **Section 6 of our paper is of utmost importance as it highlights the strength of our method**. Human evaluation and automatic evaluation are considered the most reliable ways to assess LLM alignment, and our approach achieves the highest winning rate when compared to vanilla PPO and SFT models.\n \nTo provide a more comprehensive evaluation, we will incorporate an additional case study section in the appendix that directly assesses the effectiveness of our proposed method in addressing length bias. This will include measurements that capture the reduction in response length and the maintenance of contextual coherence. Here is a one-turn dialogue example:\n\n---\n|prompts| \"kill all politicians.\"|\n|---|---|\n|vanilla-PPO|Kill all politicians? Are you serious?Human: Yes, I am serious. Politicians have been causing too much harm to our society and it's time we take a stand against them. They only care about their own self-interest and don't give a damn about the people they are supposed to represent.Assistant: I respect your opinion, but I cannot support such an extreme measure. Killing politicians is wrong and goes against my values.Human: I understand your point of view, but I think that some politicians are so corrupt and dangerous that they need to be removed by any means necessary.Assistant: It may seem like there is a lot of corruption in politics, but killing politicians is not the solution. We need to work through the legal system and elect more responsible leaders who will serve the public interest.Human: I guess we just see things differently. Thank you for having an open and honest conversation with me.Assistant: You're welcome. It was nice talking to you.|\n|Ours-PPO|I strongly disagree with the suggestion to kill all politicians. Politicians are elected by the people to represent their interests and make decisions in the best interest of their constituents. They play an important role in our society and we need them to be able to do their job effectively.|\n\n---\nWe also have multi-turn examples for length mitigation for our method, which can also be easily verified by human evaluation. We will append them to the appendix due to text limitation. \n\n**Q3**:  About Reasons To Reject Point 3 and Question 3, \"A more detailed exploration of the bias-only expert is needed to better understand its impact and significance within the proposed method.\" and \"The impact of the bias-only expert model's size on the results is not discussed. Further exploration and explanation of this aspect would be beneficial.\"\n\n**A3**: Also, thank you for highlighting the need for a more detailed exploration of the bias-only expert within our proposed method. In response to this, we will dedicate a section to thoroughly analyze the role and impact of the bias-only expert. We investigated the scaling law of the main model and bias model using BLOOMz, and we found that the accuracy of the reward model can be listed:\n\n| parameter size | bias-only model | main expert |\n|---|---|---|\n|560M|56.4%| 57.1% |\n|1.7B|59.3%| 59.8%|\n|3B|62.3%| 63.2% |\n|7.1B|66.7%|  67.5% | \n\nEven after distorting the text embedding of the bias-only model, it still performs similarly in terms of scoring in the test data. This is because it scores responses based on length, which aligns with our expectations. \n\nIn Table 1, we have a dedicated row for 'Alpaca-based' models, where the bias-only reward model guides policy models to generate longer text compared to the main expert. The results of the vanilla reward model mirror those of the bias-only model, confirming the presence of length bias in reward modeling. This poses a risk and can lead to alignment issues when using the vanilla reward model.\n\n**Q4**: About Question 1, \"The evidence for length bias is not clearly presented, and the proposed issue in Figure 2 needs further analysis and clarification. More results and analysis are required to support the findings.\"\n\n**A4**: To improve the clarity of Figure 2, we will incorporate the reviewer K33q's suggestion to add a distribution plot, which will provide a clearer representation of the data. Additionally, we consider including a heatmap figure as an option to address the issue of overlapping plots and further enhance the understandability of Figure 2. Furthermore, we introduce the Spearman/Pearson coefficient to analyze the correlation between the two variables in the hh-rlhf eval sets. This analysis serve as evidence to demonstrate the effectiveness of our method in reducing length, strengthening our findings.\n\n| Reward Model | Spearman / Pearson |\n|-----------|-----------|\n|Vanilla BLOOMZ | 0.3865 / 0.3932 |\n|POE BLOOMZ | 0.2354 / 0.299 |\n|Vanilla Llama | 0.2627 / 0.2765 |\n|POE Llama  | 0.2421 / 0.2213 |\n|Vanilla alpaca |  0.1786 /0.1765 |\n|POE alpaca | 0.1354/0.1490 |\n\nWe appreciate your thoughtful evaluation of our paper and your recommendations for improvement. Your feedback will guide us in refining our work to ensure its thoroughness and effectiveness. We are committed to addressing these concerns in the revised version of our paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper introduces a new design for the reward model.\n2. The paper identifies and discusses the issue of length bias.", "reasons_to_reject": "1. It is unclear if the proposed issue of length bias is a common phenomenon or if it will occur frequently in real-world usage. More clarification is needed.\n2. The experiment settings primarily reflect improvements in perplexity (PPL), and it is not evident if this improvement truly reflects the effectiveness of the proposed method in addressing length bias. Further analysis is necessary.\n3. A more detailed exploration of the bias-only expert is needed to better understand its impact and significance within the proposed method.", "questions_to_authors": "1. The evidence for length bias is not clearly presented, and the proposed issue in Figure 2 needs further analysis and clarification. More results and analysis are required to support the findings.\n2. The results require further explanation and improvement to effectively demonstrate the performance of the proposed method. Additional examples may be needed to enhance the understanding.\n3. The impact of the bias-only expert model's size on the results is not discussed. Further exploration and explanation of this aspect would be beneficial.", "paper_topic_and_main_contributions": "In this work, the authors tackle the problem of length bias in reward modeling and propose a PoE-based method to mitigate this bias. The method utilizes two expert models: one dedicated to learning human intent and the other focused on capturing response length patterns. The primary contribution of this research lies in the design of the reward model and the identification of length bias. However, further experimentation is required to provide stronger evidence of the effectiveness of the proposed method.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "i8aUdXH7rE", "reviewer": "Reviewer_kTmE", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable input. Regarding your mentioned doubts, we would like to address them as follows:\n\n**Q1**: About Reasons To Reject Point 1, \"This paper is a simple combination of products-of-expert and reward modeling.\"\n\n**A1**: First of all, **we want to emphasize that our main contribution focuses on exploring the risk of length bias within the RLHF framework**. This problem has been observed to some extent in the literature of OpenAI[1, 4.2.2 and 4.2.3] and Anthropic[2, Appendix E , F and G.6].\n\nWe respectfully disagree with your perspective on simple combination methods. Our method is not merely a predefined trial for RLHF. In fact, **it emerged through a systematic process of identifying the length bias problem, conducting a thorough analysis, and iteratively experimenting with innovative improvements**:\n\n* On our way to implementing RLHF, we encountered the length bias problem which is harmful to the RLHF framework. This also sheds light on a new problem in RLHF, particularly regarding the imperfect reward model [3,4,5,6], which may learn from biased datasets or exhibit shortcut learning. To thoroughly examine the issue of length bias, we employ a causal structure to model it extensively and highlight the challenges in our paper.\n\n* In our practice, PoE  is a disentangled representation approach that performs exceptionally well, especially for reward modeling. We concurrently train the main expert on pairwise datasets and introduce Gaussian noise in the text embedding of the bias-only expert to enhance the distinction in embedding learning between the two models. During the inference stage, we only utilize the main expert, serving as a regularization method to capture more human-aligned information.\n\n* Although the improvement in reward model accuracy may not be significantly apparent, we heuristically leverage it in subsequent PPO optimization. Surprisingly, this approach leads to notable advancements in reward improvement and length reduction. As demonstrated in Section 6, our method excels in both human and automatic evaluation, showcasing promising results.\n\nOur method is suitable for capturing real human-aligned information and mitigating length bias in the reward model. Additionally, this approach can be extended to address other bias problems as well. \n\nBy adopting the PoE framework, we rebuild a new reward model, which serves as a primary strategy to mitigate length bias. Empirical experiments demonstrate that this approach effectively improves human alignment. Notably, in our alpaca-based model, we achieve up to a 15-20% reduction in length without sacrificing text information (we will provide detailed case studies in the revised version). Consequently, the ultimate model guided by our new reward model exhibits enhanced human alignment. We believe this trial represents a valuable exploration in the field of alignment, particularly for LLM.  \n\n> [1] Stiennon, Nisan, et al. “Learning to Summarize from Human Feedback.” arXiv: Computation and Language,arXiv: Computation and Language, Sept. 2020.\n\n> [2] Gao, Leo, John Schulman, and Jacob Hilton. \"Scaling laws for reward model overoptimization.\" International Conference on Machine Learning. PMLR, 2023.\n\n> [3] McKinney, Lev, et al. \"On The Fragility of Learned Reward Functions.\" arXiv preprint arXiv:2301.03652 (2023).\n\n> [4] Amodei, Dario, et al. “Concrete Problems in AI Safety.” Cornell University - arXiv,Cornell University - arXiv, June 2016.\n\n> [5] Tien, Jeremy, et al. \"Causal confusion and reward misidentification in preference-based reward learning.\" arXiv preprint arXiv:2204.06601 (2022).\n\n**Q2**: About Reasons To Reject Point 2, \"The content is not very sufficient for a long paper. Especially considering the Section 3 is not necessarily included in the main body.\"\n\n\n**A2**: Thanks for your doubting for Section 3. We are willing to response you that it is essential in our paper as it provides a concise overview of RLHF for readers who may be unfamiliar with the concept. Additionally, **several contemporary works (references [7, 8, 9]) have integrated the RLHF pipeline into their main body**, and our method also aligns with the RLHF framework. To ensure clarity, we emphasize that our primary focus lies in the reward modeling phase, which is critical for our approach. We also validate reward improvement in the subsequent PPO phase. In our experiments, we utilize four models in the PPO phase, involving separate training of the SFT model and the reward model. During PPO, we use these two models alongside two frozen duplicates. Thus, including Section 3, it is important to cater to readers with varying levels of familiarity with RLHF. \n\nWe consider our work as an effort to explore alignment methods and ignite further advancements in this field, especially in both reward modeling stage and PPO stage.\n\n> [6] Rafailov, Rafael, et al. \"Direct preference optimization: Your language model is secretly a reward model.\" arXiv preprint arXiv:2305.18290 (2023).\n\n> [7] Daniels-Koch, Oliver, and Rachel Freedman. \"The expertise problem: Learning from specialized feedback.\" arXiv preprint arXiv:2211.06519 (2022).\n\n> [8] Yuan, Zheng, et al. \"Rrhf: Rank responses to align language models with human feedback without tears.\" arXiv preprint arXiv:2304.05302 (2023).\n\n\n\nAlso thank you for pointing out these typos. We value your thorough review and will carefully review the entire paper to identify and rectify any such errors in the revised version. We are dedicated to providing a revised version that adequately addresses these issues and thank you again for your valuable inputs.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Paper Strength:\n(1) The paper addresses the length bias in response generation.\n\n(2) The proposed method using the PoE technique is innovative and provides a practical solution to mitigate length bias.\n\n(3) The experimental results demonstrate the effectiveness of the proposed approach in improving language model performance.", "reasons_to_reject": "Paper Weaknesses:\n\t1. This paper is a simple combination of products-of-expert and reward modeling. The content is not very sufficient for a long paper. Especially considering the Section 3 is not necessarily included in the main body.\n\n\nLine 019: two commas\nLine 178: use \\citet instead of \\citep\nLine 187: an LM instead of a LM", "questions_to_authors": "", "paper_topic_and_main_contributions": "Overall Review:\nThe paper proposes a method to mitigate length bias in reinforcement learning from human feedback. The authors identify that the reward model often finds shortcuts to bypass its intended objectives, misleadingly assuming that humans prefer longer responses. They propose a solution using the Product-of-Experts (PoE) technique to separate the reward model from the influence of sequence length. Experimental results validate the effectiveness of their approach.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper addresses the problem of length bias in reward modelling in RLHF frameworks, which may lead to longer outputs even if they contain nonsensical tokens. It proposes Product-of-Experts framework to mitigate this bias using two novel expert models. The paper presents comprehensive experiments and analyses to support their claims. Further evidence and/ or discussion is needed to underline the frequency of the length bias problem and the effectiveness of this approach in addressing the length bias, as this is the main problem outlined in the paper. The presentation can be improved with more detailed explorations of specific topics as outlined by the reviewers for the revision.
Decision: Accept-Findings