Title: Removing Length Bias in RLHF is not Enough
Venue: NeurIPS 2024
Accepted: no
Keywords: LLM

RLHF

Prompt Bias
Abstract: Reinforcement Learning from Human Feedback (RLHF) has become an essential technique for enhancing pretrained large language models (LLMs) to generate responses that align with human preferences and societal values. 
While RLHF has shown promise, the training of reward models (RMs) still faces the challenge of \emph{reward hacking}, motivating recent works to prevent RMs from finding shortcuts that bypass the intended optimization objectives by identifying simplistic patterns, especially response length.
Besides the issue of \emph{length bias},  our work firstly reveal that \emph{prompt-template bias} learned by RMs can also cause \emph{reward hacking} when dealing with marginal samples, resulting in LLMs preferring to generate responses in a specific format after RLHF fine-tuning, regardless of the format requested in the prompt.
To this end, we propose a low-cost but effective method, namely Prompt Bias Calibration (PBC), to estimate the \emph{prompt-template bias} term during reward modeling, which can be utilized to calibrate reward scores in the following RL fine-tuning process. 
Then, we show that our PBC method can be flexibly combined with existing algorithms of removing \emph{length bias}, leading to a further improvement in the aspect of enhancing the quality of generated responses.
Experiments results show that the performance of our PBC method and its extensions have significantly surpassed the original implementation of RLHF.

Abstract

Reinforcement Learning from Human Feedback (RLHF) has become an essential technique for enhancing pretrained large language models (LLMs) to generate responses that align with human preferences and societal values. While RLHF has shown promise, the training of reward models (RMs) still faces the challenge of reward hacking , motivating recent works to prevent RMs from finding shortcuts that bypass the intended optimization objectives by identifying simplistic patterns, especially response length. Besides the issue of length bias , our work firstly reveal that prompt-template bias learned by RMs can also cause reward hacking when dealing with marginal samples, resulting in LLMs preferring to generate responses in a specific format after RLHF fine-tuning, regardless of the format requested in the prompt. To this end, we propose a low-cost but effective method, namely Prompt Bias Calibration (PBC), to estimate the prompt-template bias term during reward modeling, which can be utilized to calibrate reward scores in the following RL fine-tuning process. Then, we show that our PBC method can be flexibly combined with existing algorithms of removing length bias , leading to a further improvement in the aspect of enhancing the quality of generated responses. Experiments results show that the performance of our PBC method and its extensions have significantly surpassed the original implementation of RLHF.

1 Introduction

Reinforcement Learning from Human Feedback (RLHF) has become a critical technique to enable pretrained large language models (LLMs) to follow human instructions, understand human intent, and also generate responses that align with human preferences and societal values [1-4]. Specifically, RLHF usually trains a reward model (RM) to act as the proxy of human preferences, and then utilize online reinforcement learning (RL) algorithms to fine-tune the language models for generating responses that can achieve higher expectation rewards, leading to the success of ChatGPT and also many other AI systems [5, 6]. Although the paradigm of RLHF has simplified human data collection, as acquiring human ratings is much easier than collecting demonstrations for supervised fine-tuning (SFT), it still requires huge amount of human-annotated preference pairs to train well-performing RMs in practice, motivating recent researches to seek novel alignment methods to bypass RM training [2-4]. However, the pipeline of original RLHF is still the primary choice of most industrial applications, because well-trained RMs can provide a certain level of generalization ability [7].

Besides the expensive cost of collecting numerous human-annotated preference pairs, another heavily 32 criticized issue of RLHF could be the phenomenon of reward hacking [8], where the over-optimized 33 RMs tend to find some shortcuts to bypass its intended optimization objective, through identifying 34 some simple patterns to distinguish between good and bad responses [9]. The most widely studied 35 pattern in reward hacking could be the sentence (response) length, and these trained RMs can utilize 36

the preference among human raters for longer responses to achieve reward hacking , despite the actual 37

quality of response does not improve with the increase of response length [10]. Thus, to mitigate 38

reward hacking , recent works has primarily focused on estimating the length bias term in the reward 39 scoring process, so that it can be removed in the subsequent RL fine-tuning procedure to further 40 improve the quality of generated response after RLHF process [11, 12]. 41

42

43

44

45

Besides the issue of length bias , in the practice of applying RLHF to industrial products, we have observed that the original implementation of RLHF tends to make LLMs prefer generating responses in a specific format. This observation motivates us to investigate the underlying causes and seek a cost-effective solution to address this issue. The main contributions are summarized as follows:

46

47

48

49

50

51

52

53

54

55

56

57

58

59

60

61

62

63

64

65

66

67

68

69

70

71

72

73

74

75

76

77

78

We are the first to reveal the existence of prompt-template bias in RMs trained with the original preference loss, and theoretically analyze the cause of prompt-template bias issue, along with its corresponding potential risks on the entire RLHF process;

To mitigate the reward hacking caused by prompt-template bias , we develop a Prompt Bias Calibration (PBC) method, which will firstly estimate the prompt-template bias term during the reward scoring process, and then remove it in the subsequent RL fine-tuing process;

We show that the developed PBC method can be flexibly combined with existing methods of removing length bias , leading to a further improvement in the aspect of enhancing the quality of generated responses;

Experimental results show that our developed PCB method and its extensions can achieve promising performance improvements compared to the original implementation of RLHF.

2 Preliminary

Reward models (RMs) have become the dominant tool for aligning the LLM's responses with user preferences or task-specific requirements [1, 9]. In this section, we will firstly review the training procedure of reward models in Sec. 2.1, including analyzing the causes of length bias and prompt bias in existing RMs, and also illustrate how these RMs are used for alignment in Sec. 2.2, especially RLHF fine-tuning processes.

2.1 Reward Model Training

The usual optimization goal of a reward model is to minimize the loss under the Bradley-Terry model [13] on the dataset of pair-wise comparisons of model responses, denoted as ( x, y + , y -) ∈ D where x indicates the input prompt, y + and y -are the chosen and rejected responses respectively. Then, the objective function can be formulated as

where r θ ( x, y ) denotes the reward model that takes the prompt x and response y as input to predict a scalar reward with trainable parameters θ ; σ denotes the sigmoid function.

Length Bias : Denote r θ ∗ ( x, y ) as the 'gold standard' reward model [9] with the optimal parameters θ ∗ , it reflects human's intrinsic ranking preferences and can play a role of human rater to provide gold reward signal for each prompt-response pair. However, due to the subjectivity of ranking preferences and flaws in rating criteria, there is a phenomenon where human raters prefer longer responses that appear to be more detailed or better formatted, but their actual quality does not improve [10]. Thus, the 'gold standard' reward model for rating preference data can often be biased and thus we can decompose it to disentangle the actual reward from the spurious reward [11], formulated as

where r Q θ ∗ ( x, y ) is the actual reward gains brought by improving the quality of response y ; r L θ ∗ ( x, y ) is the spurious reward gains of increasing response length, whose patterns are much easier to identify.

Thus, with length bias in the 'gold standard' r θ ∗ ( x, y ) , during the training of reward model, r θ ( x, y ) 79 can easily find shortcuts to bypass its intended optimization objective, through identifying simple 80 patterns, such as sentence (response) length, to distinguish between good and bad responses, leading 81 to the phenomenon of 'reward hacking' caused by length bias [10]. Without increasing the cost 82 of rating higher quality preference data, it becomes increasingly important and beneficial to study 83 mitigating the impact of length bias in the process of reward modeling. 84

REWARD SCORE HEAD

𝑟 𝜃

𝑥, 𝑦

𝐿 𝑅𝑀

𝜃

= -𝐸      ( (𝑟 𝜃

𝑥, 𝑦

+

-𝑟𝜃(𝑥,𝑦 - )))

𝐿𝑃𝐵𝐶

𝑅𝑀

𝜃

= 𝐿 𝑅𝑀

𝜃

+𝜂𝑐𝐿𝑐

𝑅𝑀

𝜃

Chosen  Response

𝑦𝑎

+

When I gazed at the starry sky, I felt small,

surrounded by countless twinkling stars, as if

I was embraced by the universe...

Rejected  Response

𝑦𝑎

-

Then, I take out my phone, open social

media, and start capturing photos to share

my stargazing experience…

Prompt

𝑥𝑎

:

Please write me a

journal

about admiring

the starry sky

Chosen  Response

𝑦𝑏

+

AI represents a frontier in computational

science and is designed to perform tasks that

would normally require human intelligence…

Rejected  Response

𝑦𝑏

-

While AI continues to advance, my cat's

curiosity about the robotic vacuum cleaner at

home seems to grow each day…

Prompt

𝑥𝑏

:

Please write me a

scientific essay

on artificial

intelligence

REWARD SCORE HEAD

𝑟 𝜃

𝑥, 𝑦

PROMPT BIAS HEAD

C

𝑥, 𝑦

ത

PBC

𝑟 𝜃

(𝑥𝑏 , 𝑦

𝑏

+ ) = -1.24  > 𝑟 𝜃

𝑥𝑏, 𝑦

𝑏

-

= -3.75

𝑟 𝜃

(𝑥𝑎 , 𝑦 𝑎

+ ) = 5. 7 > 𝑟 𝜃

𝑥𝑎, 𝑦 𝑎

-

= 3.12

𝑟 𝜃 ′(𝑥, 𝑦)

=

𝑟 𝜃

𝑥, 𝑦

-

C

𝑥, 𝑦

ത

𝐿𝑐

𝑅𝑀

𝜃

= 𝐸[

𝑟 𝜃

𝑥, 𝑦 +

- 𝑐 𝑥

+ 𝑟𝜃

𝑥, 𝑦 -

- 𝑐 𝑥

𝑟 𝜃 ′

(𝑥𝑎 , 𝑦 𝑎

+ ) = 0.37  > 𝑟

𝜃 ′

𝑥𝑎, 𝑦 𝑎

-

= -0.51

𝑟 𝜃 ′ (𝑥𝑏 , 𝑦

𝑏

+ ) = -0.24  > 𝑟

𝜃 ′

𝑥𝑏, 𝑦

𝑏

-

= -0.7

Prompt Bias: the prompt bias in reward modeling derives from the underdetermination of Bardley85 Terry model [13]. For any reward model r θ ′ ( x, y ) learned from the preference loss defined in Eq. (1), 86 whose target is optimized to approximate the 'gold standard' r θ ∗ ( x, y ) , there always exists an 87 equivalent reward model r θ ( x, y ) that satisfies 88

where C ( x ) is a prompt-dependent constant referred to as prompt bias , leading to the same loss value 89 as L ( θ ) = L ( θ ′ ) . Due to the fact that there is no constraint on C ( x ) in the original preference loss 90 as defined in Eq. (1), the issue of prompt bias has been criticized in the scenario of reward model 91 ensembles [8], where different reward models tend to choose different values for C ( x ) , making the 92 statistics of the set of reward scores meaningless. 93

As shown in Fig. 1, it has been widely reported that the prompt bias will result in a certain gap in the 94 mean values of the set of prompt-response pairs under different prompts. However, in our research, 95 we find that this gap is more likely caused by the prompt-template bias , as discussed in Section 3.1. 96

97

98

99

100

101

102

103

104

105

2.2 RLHF Fine-tuning

Given the trained reward model r θ ( x, y ) as the proxy of human preferences, Reinforcement Learning from Human Feedback (RLHF) tends to utilize an online reinforcement learning method, typically proximal policy optimization (PPO) [14], trains a policy language model π RL ϕ to maximize expected reward, while staying close to its initial policy π SFT ϕ , which is finetuned on supervised data (promptresponse pairs). Through measuring the distance from the initial policy with Kullback-Leibler (KL) divergence, the optimization objective of RLHF fine-tuning can be formulated as

where β is the hyper-parameter to control the strength of the KL divergence term.

3 Method

In this section, we will firstly investigate the cause of prompt-template bias and then theoretically 106 analyze its potential risks when dealing with marginal samples during reward modeling, as shown in 107 Sec. 3.1, and then illustrate our low-cost but effective method to estimate the prompt-template bias 108 term during RM training in Sec. 3.2, which can be utilized to calibrate reward scores in the following 109 RL fine-tuning process. At last, in Sec. 3.3, we show that our Prompt Bias Calibration (PBC) method 110 can be flexibly combined with recent popular methods of removing length bias , leading to a further 111 improvement in the aspect of enhancing the quality of generated responses. 112

113

114

115

116

117

118

119

120

121

122

123

124

125

126

127

128

129

130

131

132

133

134

135

136

137

138

139

140

141

142

143

144

145

146

147

148

149

150

151

152

153

154

155

156

157

158

159

160

3.1 Impact of prompt-template bias on RLHF

In this part, we will first illustrate the cause of prompt-template bias during RM training. Formally, given a set of prompt-response pairs, denoted as D a = { x a , y ( i ) a } N a i =1 , with the same user prompt x a , e.g. 'writing an academic paper on the field of computer science' , and { y ( i ) a } N a i =1 denoting the set of collected academic papers to satisfy the request of x a , the prompt bias term, specifically C ( x a ) , learned by RMs is supposed to not affect the preference order within D a , as discussed in Section 2.1. However, in the practice of RM training, the reward score is usually predicted by a LLM that takes the concatenation of the prompt and response as input, making it challenging for RMs to learn a bias term that focuses solely on the prompt x while disregarding variations in the subsequent response y . During the training process to order the pairs within D a , we find that RMs trained with the original preference loss in Eq. (1) are more likely to introduce a joint bias term across the entire sequence of concatenating the prompt and response, formulated as

where y a can be considered the average response of the response set { y ( i ) a } N a i =1 , and it will embody the common characteristics found within these collected responses, such as the format of academic paper ; C ( x a , y a ) denotes the joint bias on the entire sequence of the prompt x a associated with the average response y a in the format of academic paper ; r θ ( x a , y a ) is still supposed to approximate the 'gold standard' provided by r θ ∗ ( x a , y a ) , leading to E D a [ r θ ′ ( x a , y a )] ≈ E D a [ r θ ∗ ( x a , y a )] .

Considering the average response y can be treated as a standard template of the response to the prompt x , we define the joint bias C ( x, y ) as prompt-template bias . Then, we highlight the properties of prompt-template bias as follows: 1) the original preference loss in Eq. (1) imposes no constraints on C ( x, y ) , because its value will not influence the outcome of the preference loss and also not affect the preference order within the prompt-response pairs collected for the same prompt x ; 2) C ( x, y ) will reduce to the original prompt bias C ( x, -) when no common characteristics can be found across all of these collected responses, indicating the diversity of { y ( i ) } N i =1 is sufficiently high. With these properties in mind, we assume that the prompt-template bias C ( x, y ) can essentially meet most of the properties of the original prompt bias C ( x, -) as discussed in Section 2.1. Thus, we suppose C ( x, y ) can be considered as a broader definition of prompt bias in the actual RM training, because it is more likely to be learned by RMs in practice, given the fact that preference pairs are extremely scarce and the diversity of responses collected for the same prompt is often insufficient.

After defining prompt-template bias , we will theoretically investigate the impact of introducing C ( x, y ) during RM training on the entire RLHF process. Assume that there exist two sets of promptresponse pairs, denoted as D a = { x a , y ( i ) a } N a i =1 and D b = { x b , y ( i ) b } N b i =1 , where x a and x b indicate different categories of prompts, e.g. x a requests 'writing an academic paper on theme a ' and x b requests 'writing a brief on theme b ' , and { y ( i ) a } N a i =1 and { y ( i ) b } N b i =1 denote the collected responses for answering the prompt x a and x b respectively. After RM training, due the fact that there is no constraint on C ( x, y ) in the preference loss defined in Eq. (1), the discrepancies of prompt biases between these two previously mentioned sets of prompt-response pairs, specifically D a and D b , could be extremely large, e.g. C ( x a , y a ) >> C ( x b , y b ) , leading to

where r θ ( x a , y a ) = r θ ′ ( x a , y a )+ C ( x a , y a ) and r θ ( x b , y b ) = r θ ′ ( x b , y b )+ C ( x b , y b ) . The unbiased reward distributions, modeling the reward scores { r θ ′ ( x a , y ( i ) a ) } N a i =1 and { r θ ′ ( x b , y ( i ) b ) } N b i =1 respectively, should exhibit similar mean values, e.g. E D a [ r θ ′ ( x a , y a )] ≈ E D b [ r θ ′ ( x b , y b )] , and will make little impact on the comparison of expectation terms in Eq. (6). We highlight that the discrepancies of prompt bias terms, specifically the gap between C ( x a , y a ) and C ( x b , y b ) , won't affect preference ordering within categories, but can cause disaster when dealing with some marginal samples, like 'an academic paper on theme b ' denoted as y ab , or 'a brief on theme a ' denoted as y ba .

To facilitate an intuitive analysis, we take the marginal sample 'an academic paper on theme b ' , denoted as y ab , as an example, and the reward scores for prompt-response pairs corresponding to the prompt x b may exhibit the following preference orders:

<BOS>

[User]

[Assistant]

:

It

<EOS>

Tokenizer

Transformer Decoder

REWARD SCORE HEAD

Linear Layer

PROMPT BIAS HEAD

Linear Layer

0.1

0.4

0.8

0.2

0.3

0.7

0.3

0.7

0.9

0.2

0.6

0.4

0.1

0.7

0.3

0.6

0.5

0.9

0.5

0.4

0.8

0.6

0.1

0.6

Prompt

Response

×

N

<BOS>

[User]

[Assistant]

:

Yes

<EOS>

0.1

0.7

0.3

0.6

0.5

0.9

0.6

0.5

0.2

0.6

0.1

0.9

0.1

0.4

0.8

0.2

0.3

0.7

0.1

0.5

0.6

0.9

0.4

0.2

\n

\n

……

……

……

……

Prompt

Response

……

……

……

……

……

……

……

……

……

……

……

……

……

……

……

……

Prompt

Response

which can be achieved as long as r θ ′ ( x b , y ab ) ≈ r θ ′ ( x b , y b ) and C ( x b , y a ) > C ( x b , y b ) . The first 161 condition r θ ′ ( x b , y ab ) ≈ r θ ′ ( x b , y b ) can be achieved because both the response y ab and y b meet the 162 description of theme b and are similar on a semantic level. The second inequality is highly likely 163 to be achieved when there is a reward model that has a bias towards preferring the sentence in the 164 format of a over b , specifically C ( x a , y a ) >> C ( x b , y b ) . 165

166

167

168

169

170

171

172

173

Finally, we highlight that the phenomena of inequality in Eq. (7), caused by prompt-template bias C ( x, y ) , is commonly encountered in the deployment process of RLHF in real-world applications, especially text creation. For example, if responses are collected solely for the style requested in each prompt during RM training, the reward model can lead to a bias towards particular styles as shown in Fig. 3(a). Then, once such marginal samples, e.g ( x b , y ab ) , are generated by LLMs during the RL fune-tuning process and also satisfy the inequality r θ ( x b , y ab ) > r θ ( x b , y b ) as shown in Table 1, the entire RL fine-tuning process, typically PPO, will be biased and results in a LLM that only generates responses in a specific format, regardless of the format you request in the prompt.

174

175

176

177

178

179

180

181

182

183

184

185

186

187

188

189

3.2 Calibrating prompt-template bias in RLHF

To mitigate the impact of the prompt-template bias issue on the RLHF process, the most straightforward solution in industry could be to collect a more diverse set of response candidates for each prompt. However, this approach is time-consuming and may even require a lot of human interventions for response collection, motivating us to develop a low-cost but effective method to alleviate the issue of prompt-template bias during RM training.

The developed Prompt Bias Calibration (PBC) method mainly includes two steps: 1) estimating the prompt-template bias term in the reward scoring process with minimal additional computational cost; 2) removing prompt-template bias in the subsequent RLHF fine-tuning process to ensure that the resulting LLM does not have a tendency to generate responses in a specific format. As shown in Fig. 1, to approximate the prompt-template bias term C ( x, y ) in Eq. (5), we choose to apply a linear layer on the last token of the prompt sentence to predict prompt-template bias , denoted as u c ( x ) , and then add the following regularization term on the original preference loss, formulated as

where u c ( x ) is supposed to approximate the mean value of reward scores of the prompt-response pairs given the same prompt x . We note that there will be a hyper-parameter η c to be multiplied on the regularization term in the final loss to promise the accuracy of RMs, leading to

The benefits of such a design in the PBC method include the following folds: 1) approximating 190 C ( x, y ) by adding a linear layer to the last hidden layer of LLMs results in almost no additional 191

computational cost; 2) during the autoregressive scoring process of LLM-based RMs, C ( x, y ) can 192 serve as an intermediate signal guidance of the prompt sequence, thereby enabling RMs to focus 193 more on the differences between chosen/rejected responses in the subsequent reward scoring process; 194 3) we can use unbiased reward scores to guide the follow RLHF fine-tuning process, formulated as 195

which has been proven effective for penalizing reward uncertainty, improving robustness, encouraging improvement over baselines, and reducing variance in PPO fune-tuning [15].

3.3 Jointly calibrating length and prompt-template bias in RLHF

To simultaneously calibrate length and prompt-template bias in RLHF, the developed PBC method can be flexibly combined with existing methods of removing length bias , whose main idea is to separately approximate the 'gold standard' reward model after disentangling shown in Eq. (2), formulated as:

where r Q θ ( x, y ) is supposed to approximate the actual reward r Q θ ∗ ( x, y ) ; r L θ ( x, y ) is used to approximate the spurious reward brought by length bias , specifically r L θ ∗ ( x, y ) . Then, for those methods of removing length bias [11, 12], the original preference loss in Eq. (1) can be equivalently expressed as

where r Q θ ( x, y ) and r L θ ( x, y ) can be modeled with two different LLMs [12] or two different heads in the same LLM [11]. To remove length bias in Eq. (12), recent work proposes to add constraints on the preference loss to reduce the correlation between the confounding factor, e.g. response length, and actual reward r Q θ ( x, y ) , while increasing its correlation with spurious reward r L θ ( x, y ) , formulated as

where the confounding factor L ( x, y ) can be either specifically defined as response length L ( y ) in [11], or use Products-of-Experts framework for estimation [12].

To model the scoring process of the reward model more accurately, which simultaneously considers the concepts of length and prompt bias, we combine the definition of reward model in Eq. (3) and Eq. (11), achieving a more precise definition of reward scoring process, formulated as:

where C Q ( x, y ) and C L ( x, y ) indicate the component of prompt-template bias in actual and spurious rewards, respectively; the unbiased overall reward r θ ′ ( x, y ) = r Q θ ′ ( x, y ) + r L θ ′ ( x, y ) and the overall prompt-template bias term C ( x, y ) = C Q ( x, y ) + C L ( x, y ) . Then we can propose Length and Prompt Bias Calibration (LPBC) method, as shown in Fig. 2, which can estimate L RM l ( θ, τ ) with a conditioned correlation method, defined as

where the confounding factor L ( y ; x ) := L ( x, y ) -L ( x ) can be estimated with the response length.

Through combining the disentangled preference loss in Eq. (12), the prompt-bias regularization term in Eq. (8) and also the length-bias conditional correlation term in Eq. (15), the final loss of LBPC method can be formulated as

where η c and η l are hyper-parameters to control the importance of regularization terms, which can be adjusted according to the accuracy of trained RMs on the validation dataset.

196

197

198

199

200

201

202

203

204

205

206

207

208

209

210

211

212

213

214

215

216

217

218

219

220

221

222

223

224

225

226

227

228

229

230

231

232

233

234

235

236

237

238

239

240

241

242

243

244

245

246

247

248

249

250

251

252

4 Experiments

4.1 Experimental Settings

Datasets. For intuitively understanding the issue of prompt-template bias in RLHF and also qualitatively evaluating the effectiveness of our method, we manually construct a training dataset for text creation applications, where each prompt requires creation in a special style according to the theme. Then, a small validation set is also constructed, in which only responses that meet the stylistic requirements of each prompt are collected. We name this dataset as RM-Template, which can be used to measure the severity of the prompt-template bias issue during RM training.

Further, to make quantitative comparisons with other baseline methods, we conduct experiments on RM-Static dataset [16], which has been released on Huggingface [17] and consists of 76K preference pairs. After randomly shuffling, we choose 40K preference pairs for RM training, 6K preference pairs for RM evaluation, and the rest prompt-response pairs for the subsequent PPO fune-tuning.

The dataset statics of these datasets have been exhibited in Appendix A.5.

Model & Training. For model selection, we choose Llama-2-7b [18] as our base model, which is relatively lightweight, and has been open-sourced on Huggingface [17]. For RM training, we fine-tune all the parameters of RMs initialized with the pretrained weights of Llama-2-7b. For PPO fine-tuning, we also initialize the actor model with pretrained Llama-2-7b and the critic model with RMs trained with various preference losses.

For model training, all experiments are implemented with DeepSpeed-Chat framework [19] and Huggingface Transformers [20], running on 4 NVIDIA A100 80GB GPUs. For the hyper-parameter setting, we set η c = 0 . 05 and η l = 0 . 05 in Eq. (16) for all our proposed methods, and have listed the rest hyper-parameters in Appendix A.4, such as learning rate, weight decay, batch size etc. AdamW [21] is adopted for optimizing all the model parameters without freezing anything or using adapters.

Evaluation Metrics. For quantitative comparison, we follow the evaluation procedure of InstructEval [22] to test the actor models, which has been aligned with biased/de-biased RMs with PPO fine-tuning, on Massive Multitask Language Understanding (MMLU) [23], DROP [24], BIG-Bench Hard (BBH) [25], and TruthfulQA (TQA) [26] benchmarks respectively, evaluating the model's ability on the aspects of multi-task solving, math reasoning, and response trustworthy.

253

254

255

256

257

258

259

260

261

262

263

264

265

266

267

268

269

270

271

272

273

274

275

276

277

278

279

280

281

282

283

284

285

(a) Vanilla RM

(b) PBC

(c) Vanilla RM

vs

PBC

15

15

BASE RM

OURS

10

BASE RM

10

10

8

OURS

Reward Score

Reward Score

Mean Reward Score

6

4

0

0

N

5

5

0

10

10

-15

Advertisement

Insight

Tech Article Record Article

Poetry

-15

Advertisement

Insight

Tech Article Record Article

Poetry

BASE RM

OURS

4.2 Experimental Results

Qualitative Evaluation. To intuitively evaluate the effectiveness of our method, we exhibit the statistics (mean and standard deviation) of the reward scores predicted by RMs trained with the original preference loss in Eq. (1) and our PBC method in Eq. (9), across different categories of prompt-response pairs in the validation set of the RM-Template dataset. The results depicted in Fig.3(c) demonstrate that calibrating prompt-template bias with the PBC method leads to a gradual reduction in the variance of the mean values of reward distributions across different categories. The most noticeable observation is that the vanilla RM tends to give an extremely high reward score to prompt-response pairs in the format of tech article , but the RM trained with the PBC method can calibrate the reward distribution for tech articles to make it more close with that of other categories.

Then, we evaluate the performance of RMs trained with various methods on handling marginal samples defined in Section 3.1. Specifically, given the prompt randomly selected from the validation set of RM-Template dataset, we use GPT-4 [6] to generate responses in various formats according to the theme described in the prompt. Then, we use RMs trained with various preference losses to rank these responses. From the showcase in Table. 1, we can find that the vanilla RM tend to assign a higher reward score to the response in the format of tech article , caused by the prompt-template bias issue shown in Fig. d3(a). After removing this bias with our PBC or LPBC methods, the RM can provide a relatively fair ranking for these prompt-response pairs, where LPBC method can even mitigate the affect of length bias during comparing poetry with other categories (the length of poetry is generally shorter than other literary forms). More showcases can be found in Appendix A.6.

Quantitative Comparison. For the quantitative comparison in Table 2, we utilize PPO fine-tuning process to align Llama-2-7b with the RMs trained with various methods. From the results, we can find that our developed PBC method can lead to performance improvements compared to the original implementation of RLHF; directly combining PBC with other methods of removing length bias , e.g. ODIN [11], can help them to achieve further performance improvement; the well-designed LPBC achieves the best performance and surpasses the rough combination of PBC and ODIN.

60

Win

Win/Tie/LossRates

50

Loss

Tie

40

30

20

10

ODIN

PIC

ODEN+PBC

To make a comprehensive comparison, we follow the experimental setting described in ODIN [11], and use GPT-4 as the judge to compare two responses generated by LLMs aligned with RMs trained

286

287

288

289

290

291

292

293

294

295

296

297

298

299

300

301

302

303

304

305

306

307

308

309

310

311

312

313

314

315

316

317

318

319

320

321

322

323

(b) MMLU Performance

(c) DROP Performance

0.68

43.93

42.25

36.54

31.06

32.60

30.96

25

0.66

0.64

500

44.57

45.94

43.89

509

32.52

31.57

27.41

20

0.62

0.60

20

心

15

0.01

40.30

45.47

45.91

0.01

30.63

31.57

31.47

0.58

=;=0

0.56

100=

0.54

==0.05

9=9;=0.1

29.53

200

400

Iteration

600

800

1000

0.01

c

0.05

9.1

0.01

门c

9.05

9.1

with various methods. Specifically, we take the LLM aligned with LPBC-based RM as model A, and compare it against other LLMs aligned with RM trained with ODIN, PCB, ODIN+PBC, respectively. From the results shown in Fig. 4, we can find that the win rate of LPBC is significantly higher than that of other baseline models, with ODIN+PBC being the most challenging competitor as model B.

4.3 Ablation Studies

To investigate the robustness of our developed LPBC method, we conduct ablation studies on the hyper-parameter settings of LPBC method, specifically η c and η l in Eq. (16). With various settings of η c ∈ { 0 . 01 , 0 . 05 , 0 . 1 } and η l ∈ { 0 . 01 , 0 . 05 , 0 . 1 } , we can have total 9 RMs trained with various hyper-parameter settings of LPBC methods. From the accuracy curves shown in Fig.5(a), we can find the introducing constraints to the original preference loss indeed affects the performance of RM accuracy, and this performance loss increases with the importance weight of the constraint terms. However, at the limited cost of sacrificing RM accuracy, the performance of the LLM aligned the RM trained with LPBC method has improved to some extent on MMLU and DROP as shown in Fig. 5(b) and 5(c) respectively. Note that the performance of the LPBC method in Table. 2 is not the optimal, as it is achieved with η c = η l = 0 . 05 , demonstrating no cherry-picking of hyperparameters..

5 Related Works

The prevalence of length bias in RLHF have been widely criticized as indicative of reward hacking [9, 10], and numerous recent studies have delved into strategies aimed at mitigating the tendency for length increase during the fine-tuning process of RLHF [11, 12, 27]. Typically, Shen et al. [12] innovatively apply the Productof-Experts (PoE) technique to separate reward modeling from the influence of sequence length, which adopts a smaller reward model to learn the biases in the reward and a larger reward model to learn the true reward. Utilizing similar disentangling ideas, Chen et al. [11] jointly train two linear heads on shared feature representations to predict the rewards, one trained to correlate with length, and the other trained to focus more on the actual content quality. Ryan et al. [27] firstly study the length problem in the DPO setting, showing significant exploitation in DPO and linking it to out-of-distribution bootstrapping. As for the prompt bias issue, although it has been criticized in the scenario of reward model ensembles [8], no studies have yet attempted to analyze its cause and influence on RLHF. We emphasize that our work is the first to fill this gap by proposing a low-cost yet effective method to mitigate the reward hacking induced by prompt-template bias .

6 Conclusion

In this paper, we demonstrate that prompt-template bias in RMs can lead to LLMs, which, after RL fine-tuning, generate responses exclusively in a specific format, irrespective of the variations in the prompt request. Thus, we propose a low-cost but effective PBC method, to estimate the prompttemplate bias term during reward modeling, which can be utilized to calibrate reward scores in the following RL fine-tuning process. Then, we show that our PBC method can be flexibly combined with existing algorithms of removing length bias , leading to a further improvement in the aspect of enhancing the quality of generated responses. Experimental results show that the performance of PBC method and its extensions have significantly surpassed the original implementation of RLHF.

324

325

326

327

328

329

330

331

332

333

334

335

336

337

338

339

340

341

342

343

344

345

346

347

348

349

350

351

352

353

354

355

356

357

358

359

360

361

362

363

364

365

366

367

References

Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems , 35:27730-27744, 2022.

Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems , 36, 2024.

Kawin Ethayarajh, Winnie Xu, Niklas Muennighoff, Dan Jurafsky, and Douwe Kiela. Kto: Model alignment as prospect theoretic optimization. arXiv preprint arXiv:2402.01306 , 2024.

Yueqin Yin, Zhendong Wang, Yi Gu, Hai Huang, Weizhu Chen, and Mingyuan Zhou. Relative preference optimization: Enhancing llm alignment through contrasting responses across identical and diverse prompts. arXiv preprint arXiv:2402.10958 , 2024.

Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805 , 2023.

Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 , 2023.

Ziniu Li, Tian Xu, and Yang Yu. Policy optimization in rlhf: The impact of out-of-preference data. arXiv preprint arXiv:2312.10584 , 2023.

Jacob Eisenstein, Chirag Nagpal, Alekh Agarwal, Ahmad Beirami, Alex D'Amour, DJ Dvijotham, Adam Fisch, Katherine Heller, Stephen Pfohl, Deepak Ramachandran, et al. Helping or herding? reward model ensembles mitigate but do not eliminate reward hacking. arXiv preprint arXiv:2312.09244 , 2023.

Leo Gao, John Schulman, and Jacob Hilton. Scaling laws for reward model overoptimization. In International Conference on Machine Learning , pages 10835-10866. PMLR, 2023.

Prasann Singhal, Tanya Goyal, Jiacheng Xu, and Greg Durrett. A long way to go: Investigating length correlations in rlhf. arXiv preprint arXiv:2310.03716 , 2023.

Lichang Chen, Chen Zhu, Davit Soselia, Jiuhai Chen, Tianyi Zhou, Tom Goldstein, Heng Huang, Mohammad Shoeybi, and Bryan Catanzaro. ODIN: disentangled reward mitigates hacking in RLHF. CoRR , abs/2402.07319, 2024.

Wei Shen, Rui Zheng, WenYu Zhan, Jun Zhao, Shihan Dou, Tao Gui, Qi Zhang, and Xuanjing Huang. Loose lips sink ships: Mitigating length bias in reinforcement learning from human feedback. In Houda Bouamor, Juan Pino, and Kalika Bali, editors, Findings of the Association for Computational Linguistics: EMNLP 2023, Singapore, December 6-10, 2023 , pages 28592873. Association for Computational Linguistics, 2023.

Ralph Allan Bradley and Milton E Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika , 39(3/4):324-345, 1952.

John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347 , 2017.

Wei Shen, Xiaoying Zhang, Yuanshun Yao, Rui Zheng, Hongyi Guo, and Yang Liu. Improving reinforcement learning from human feedback using contrastive rewards. arXiv preprint arXiv:2403.07708 , 2024.

Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn 368 Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training a helpful and harmless 369 assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862 , 370 2022. 371

Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony 372 Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al. Huggingface's transform373 ers: State-of-the-art natural language processing. arXiv preprint arXiv:1910.03771 , 2019. 374

375

376

377

Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288 , 2023.

378

379

380

381

382

383

384

385

386

387

388

389

390

391

392

393

394

395

396

397

398

399

400

401

Zhewei Yao, Reza Yazdani Aminabadi, Olatunji Ruwase, Samyam Rajbhandari, Xiaoxia Wu, Ammar Ahmad Awan, Jeff Rasley, Minjia Zhang, Conglong Li, Connor Holmes, et al. Deepspeed-chat: Easy, fast and affordable rlhf training of chatgpt-like models at all scales. arXiv preprint arXiv:2308.01320 , 2023.

Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al. Transformers: Stateof-the-art natural language processing. In Proceedings of the 2020 conference on empirical methods in natural language processing: system demonstrations , pages 38-45, 2020.

Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101 , 2017.

Yew Ken Chia, Pengfei Hong, Lidong Bing, and Soujanya Poria. Instructeval: Towards holistic evaluation of instruction-tuned large language models. arXiv preprint arXiv:2306.04757 , 2023.

Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300 , 2020.

Dheeru Dua, Yizhong Wang, Pradeep Dasigi, Gabriel Stanovsky, Sameer Singh, and Matt Gardner. Drop: A reading comprehension benchmark requiring discrete reasoning over paragraphs. arXiv preprint arXiv:1903.00161 , 2019.

Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc V Le, Ed H Chi, Denny Zhou, et al. Challenging bigbench tasks and whether chain-of-thought can solve them. arXiv preprint arXiv:2210.09261 , 2022.

Stephanie Lin, Jacob Hilton, and Owain Evans. Truthfulqa: Measuring how models mimic human falsehoods. arXiv preprint arXiv:2109.07958 , 2021.

Ryan Park, Rafael Rafailov, Stefano Ermon, and Chelsea Finn. Disentangling length from 402 quality in direct preference optimization. arXiv preprint arXiv:2403.19159 , 2024. 403

404

405

406

407

408

409

410

411

412

413

414

415

416

417

418

419

420

421

422

423

424

425

426

427

428

429

430

431

432

433

434

435

436

437

438

439

A Appendix

A.1 Limitations

The main limitation of this work is that there are no theoretical proof to promise RM can provide an accurate preference order when handling marginal samples, e.g. , responses that satisfy the theme of the user prompt but in various formats. Moreover, the constraints added by our developed method to the preference loss will lead to a decrease in the accuracy of the RM, and to some extent, limit the capability of the RM. Therefore, how to remove the prompt-template bias without scarifying the accuracy of RM is a worthwhile problem for future research.

A.2 Border Impact

The most significant positive impact of this work is that by removing the prompt-template bias , our method can mitigate the LLM's tendency to prefer generating responses in specific formats after RLHF fine-tuning. Furthermore, our developed method can improve the quality of responses generated by LLMs after alignment, compared to the original RLHF. The discovery of prompttemplate bias may lead to another stream of research focused on investigating, estimating, and removing this bias from RM training.

The negative impact could be that our method can be used for enhancing the capabilities of LLMs. If LLMs enpowered by our methods are misunderstood, it could lead to unexpected troubles, but this is also a common issue with all of current pretrained LLMs.

A.3 License

We highlight that Llama-2-7b is licensed under the LLAMA 2 Community License, and RM-Static dataset is licensed the Huggingface hub. Our work follows the license of CC BY-NC 4.0.

A.4 Hyper-parameter Settings

RMTraining. The hyper-parameter settings of RM training under the DeepSpeedChat framework has been listed in Table. 3.

PPO Fine-tuning. The hyper-parameter settings of PPO fine-tuning under the DeepSpeedChat framework has been listed in Table. 4.

A.5 Dataset Statics

The dataset statics of RM-Template and RM-Static used in our experiments have been summarized as follows:

RM-Template. RM-Template is a manually constructed dataset for measuring the severity of the prompt-template bias issue and evaluating the effectiveness of the method developed for alleviating the issue of prompt-template bias . In this dataset, each prompt requires responses to be created in a specific format according to the theme. There are a total of 50K prompt-response pairs, encompassing 20 categories of format requirements in the responses.

RM-Static. The RM-Static dataset is provided by Hugging Face and is primarily used for training reward models after supervised fine-tuning. It is a branch of the hh-static dataset and contains both

training and testing parts. Features of the dataset include: 1) prompt: A string type representing the 440 user's input; 2) response: A string type representing the assistant's answer. 3) chosen: A string type 441 representing the selected answer. 4) rejected: A string type representing the rejected answer. The 442 training set contains approximately 76K rows of data and the testing set contains approximately 5.1K 443 rows of data. 444

445

446

447

A.6 More Showcases

More showcases of the preference order predicted by RMs trained with various methods, have been listed in the Table 5 and Table 6.

448

449

450

451

452

453

454

455

456

457

458

459

460

461

462

463

464

465

466

467

468

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: Yes, the claims in abstract and introduction has already reflected the paper's contribution on the field of RLHF.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: Yes, the discussion about limitation can be found in Appendix.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: We have included the theoretical analysis of the cause of prompt-template bias .

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We have included the implementation details in the main manuscript and also provide the hyper-parameter setting in the Appendix

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: The code has been included in the supplemental material and the dataset for the main experimental results is public.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

576

577

578

579

580

581

582

583

584

585

586

587

588

589

590

591

592

593

594

595

596

597

598

599

600

601

602

603

604

605

606

607

608

609

610

611

612

613

614

615

616

617

618

619

620

621

622

623

624

625

626

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: Have included the training and test details in the experimental settings.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [NA]

Justification: We report the average performance in our experiments, and we are willing to release the training and evalution log in W&B if it is required.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: 4*A100

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: Yes, it is

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: Have discussed the broader impact in the Appendix

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: The safeguards of our model should be the same as Llama released by META.

Guidelines:

The answer NA means that the paper poses no such risks.

679

680

681

682

683

684

685

686

687

688

689

690

691

692

693

694

695

696

697

698

699

700

701

702

703

704

705

706

707

708

709

710

711

712

713

714

715

716

717

718

719

720

721

722

723

724

725

726

727

728

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: Yes, their licenses can be found in Huggingface website and we have also highlight it in our Appendix.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [No]

Justification: No new asset

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [No]

729

Justification: No research with human subjects

730

731

732

733

734

735

736

737

738

739

740

741

742

743

744

745

746

747

748

749

750

751

752

753

754

755

756

757

758

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: No research with human subjects

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
comments: SUMMARY:
The authors considers methods for removing bias in RMs, specifically the bias towards long responses and the bias certain prompts might have to generate high rewards (this stems from the BardleyTerry model being underspecified.). For the second problem the authors proposed PBC which adds a linear layer to the last token of the prompt, the output of which predicts the average reward of completions from the prompt. For the first problem the authors propose to combine PBC with existing length bias correction methods which adds a correlation term to the loss. For experimental results the authors considers RLHF training LLama-7B on the RM-static dataset. They find that their method outperforms baselines on academic benchmarks (Table 2) and in head-to-head comparisons (Fig 4). They also consider hyperparameter stability and ablations in Fig 5.

STRENGTHS:
1. Bias in RLHF can potentially have a large impact if addressed correctly.

WEAKNESSES:
1. Academic metrics like MMLU are not a good fit for RLHF. MT-bench is better.
2. There are no error bars, unclear how strong the signal is. 
3. The writing is rather handwavy at times, e.g. the motivation in section 3.1. is very qualitative.
4. The novelty is low.

QUESTIONS:
1. Is the method in section 3.3 novel? It seems like not. The intro states “We show that the developed PBC method can be flexibly combined with existing methods 53 of removing length bias”. Please clarify this.
2. The Bradley–Terry model is indeed underspecified as in eq (3). Do you have any quantitative evidence that this is a problem in practice? Section 3.1. is rather handwavy.
3. What prompt is used for GPT4 evaluation is Figure 4? Please clarify in paper.
4. Can you add error bars to Table 2 and Figure 4?

LIMITATIONS:
na
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2024-06-23

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the prompt bias in RLHF, especially the reward modeling --- beyond the length bias that might exist.\nAlleviating reward hacking is an important topic in RLHF, however, with the current paper, some details or contributions are not very clear. I'll elaborate in the following sections.\n\nSTRENGTHS:\nThe problem studied is important. The illustrative figures are helpful.\n\nWEAKNESSES:\nSome notations do not make sense, for example, in Equation (5), averaging over y does not make sense. Would not it be better to average over the C rather than y.\n\nThe presentation of the problem itself is not yet clear to me. Although the authors keep using examples in the context to anchor their ideas (which I appreciate), it is still unclear what is the problem this work aims to solve. I like the general idea of Figure 1, however, what does the red color highlighting mean? This figure makes a good contrast between your RM and conventional RM, yet it fails to illustrate the problem your RM aims to solve.\n\nThe experimental results are not supportive enough.\n\nQUESTIONS:\nIn Equation (6) the authors compare reward values between two different prompts. What is the motivation for making such comparisons --- as RLHF only ranks/compares responses within a template?\n\nIn experiments, what is the performance of different reward models? The authors didn't report the accuracy or other quantitative performance information on the RMs. \n\nReal-world case studies would be very helpful in understanding the paper's contribution: would it be possible for the authors to show some reward hacking examples (not the ones generated by GPT4)?\n\nThe proposed algorithm seems fragile w.r.t. its hyper-parameters. There is no clear trend in the heat maps of Figure 5. Could the authors also report the standard deviation of each evaluation?\n\nError bars are missing in the reported results of Table 2.\n\nLIMITATIONS:\nPlease see weakness", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-11"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces the Prompt Bias Calibration (PBC) method to address prompt-template bias in reward training of RLHF. The proposed PBC method is validated through extensive empirical results and mathematical analysis, showing its effectiveness in combination with existing length bias removal methods.\n\nSTRENGTHS:\n1.\tGood Writing: The paper is well-written and easy to follow.\n2.\tInnovative Methodology: Introduces Prompt Bias Calibration (PBC) to address prompt-template bias in RLHF.\n3.\tStrong Empirical Evidence: Demonstrates significant performance improvements through comprehensive evaluations.\n\nWEAKNESSES:\nsee questions\n\nQUESTIONS:\n1. Why Eq (6) would happen? In general training, $C(x_a, \\bar{y}_a)$ and $C(x_b, \\bar{y}_b)$ may exhibit some gaps but should be in a reasonable area as there is no Bellman process in the training process. \n\n2. For Eq (7), suppose there are two elements in the prompt: a (paper type) and b (themes). The original sample in the dataset is (b, b) = (brief on theme b) and (a, a) = (academic paper on theme a). The authors mentioned the margin sample of $y_{ab}=(a, b)$ = (academic paper on theme b). There are two fundamental assumptions: (1) $r_{\\theta'}(x_b, y_{ab}) = r_{\\theta'}(x_b, y_b)$ for the same theme (b). (2) $C(x_b, \\bar{y}_a) \\approx C(x_a, \\bar{y}_a)$ due to formulate preference. \n\n2.1 Further clarification is necessary for these two assumptions: (1) The first assumption is too strict focusing on text creation, where the reward function only depends on $b$ rather than $a$. For example, a (code languages) b=(Neural networks), the $y_{ab}$ is heavily dependent on (a).  (2) I think the second assumption is wrong as the prompt bias is heavily associated with $x$ and we cannot make this assumption. The following algorithm in Eq (8) aligns with my intuition. \n\n2.2. There are also two more comments on this: (1). joint distribution is approximated as the margin distribution for the reward function is generally unacceptable for me, which means the $x_1$ is not important. So, the optimization function could be: $r(x_2, y) + C(x_2, \\bar{y})$ in Eq 5. If this is true, $C$ will not be associated with $x_1$, i.e., template in this work. (2) The $x_{a, b}$, as well as $y_(a, b)$ is the out-of-distribution sample, and the accuracy for the reward estimation should exhibit high bias.\n\nLIMITATIONS:\nsee the above", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-11"}

comments: SUMMARY:
The paper addresses the issue of reward hacking in RLHF training, superficially, identifying prompt-template bias, defined as when a reward model (RM) develops a preference for responses that adhere to specific formats or templates, even when these formats are not explicitly specified or desired in the prompt and proposes Prompt Bias Calibration (PBC) method that successfully tackles this issue. PBC can also be combined with existing length debiasing methods like ODIN to mitigate both hacks in the reward signal.

STRENGTHS:
* The paper identifies and analyzes "prompt-template bias" in RLHF, a potentially impactful issue.
* PBC is easy to implement and as shown can be combined with existing approaches.
* Strong empirical validation with good coverage in the experiments and ablation.

WEAKNESSES:
* Choosing one specific bias - while the title claims that removing the length bias is not enough, it seems to change the scene to potentially removing length and prompt-template bias not being enough. Leading to concerns about needing to combine many methods, one for each mitigation.

QUESTIONS:
Any insights on whether the approach would work for the larger models beyond 7B?
Could the approach be generalized to address other instances of reward hacking?

LIMITATIONS:
The limitations are covered.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-13