Information-Theoretic Reward Decomposition for Generalizable RLHF
Authors: Liyuan Mao
Haoran Xu
Amy Zhang
Weinan Zhang
Chenjia Bai
Venue: NeurIPS 2025 poster ()
Keywords: Reinforcement Learning from Human Feedback
Reward Learning
Large Language Models
Primary area: deep_learning
TL;DR: In this paper, we decompose the reward value into prompt-free reward and prompt-related reward from a information-theoretic perspective, and use the former to guide reward training.

Abstract: Obtaining a generalizable reward model is crucial in Reinforcement Learning from Human Feedback (RLHF) as it enables correctly evaluating unseen prompt-response pairs. However, existing reward models lack this ability, as they are typically trained by increasing the reward gap between chosen and rejected responses, while overlooking the prompts that the responses are conditioned on. Consequently, when the trained reward model is evaluated on prompt-response pairs that lie outside the data distribution, neglecting the effect of prompts may result in poor generalization of the reward model. To address this issue, we decompose the reward value into two independent components: prompt-free reward and prompt-related reward. Prompt-free reward represents the evaluation that is determined only by responses, while the prompt-related reward reflects the reward that derives from both the prompt and the response. We extract these two components from an information theoretical perspective, which requires no extra models.. Subsequently, we propose a new reward learning algorithm by prioritizing data samples based on their prompt-free reward values. Through toy examples, we demonstrate that the extracted prompt-free and prompt-related rewards effectively characterize two parts of the reward model. Further, standard evaluations show that our method improves both the alignment performance and the generalization capability of the reward model.

Information-Theoretti Reward Decomposition for Generalizable RLHF
Liyuan Mao"*, Haoran Xu², Amy Zhang', Weinan Zhang't Chengjia Bai Shanghai Jiao Tong University, PUT Austin, Institute of Artificial Intelligence, China Telecom maoliyuanSsjtu edu. cn, wnzhangds jtu. edu. en, baicj@chinatelecom.co
Abstract
A generalizable reward model is crucial Reinforcement Learning from Human Feedback (RLHF) as it enables correctly evaluating unseen prompt-response pairs. However, existing reward models can lack this ability, as they are typically trained by increasing the reward gap between the chosen and rejected responses, while overlooking the prompts that the responses are conditioned on. Consequently, when the trained reward model is evaluated on prompt-response pairs that lie outside the data distribution, neglecting the effect of prompts may result in poor generalization of the reward model. To address this issue, decompose the reward value into two independent components: prompt-free reward and prompt-related reward. Prompt-free reward represents the evaluation that is determined only by responses, while the prompt-related reward reflects the reward that derives from both the prompt and the response. We extract these two components from information -etnattion-thooretic perspective, which requires no extra models. Subsequently, we propose new reward learning algorithm by prioritizing data samples based on their prompt-free reward values. Through toy examples, we demonstrate that the extracted prompt-free and prompt-related rewards effectively characterize the two parts of the reward value. Further, standard evaluations show that our method improves both the alignment performance and the generalization capability of the reward model.
Introduction
Reinforcement Learning from Human Feedback (RLHF) is an effective approach for Large Language Models (LLMs) alignment 8 (.. Within wide range of RLHF methods, reward learning plays pivotal role. These methods typically first train a reward model on static dataset and then leverage it to Reinforcement Learning (RL) (28 [[] Compared with methods that are free of using reward models 300 40), the advantage of such methods is their capacity to leverage the generalization capability of the reward model to evaluate butside-of-distributio prompt-response pairs. These prompt-response pairs with generated rewards can be used to further improve the LLM's performance 36 42).
Clearly, learning a generalizable reward model is central to this scenario. However, we found that standard reward training does not guarantee sufficient generalization capability. In reward model training, the primary goal is typically better distinguishing between chosen and rejected responses. To achieve this, the reward model does not necessarily require consideration of the corresponding prompt. Taking reward learning based on Bradley-Terry (BT) model potential response space is vastly larger than the dataset size, different data samples typically contain distinct response pairs. As long as the reward gap within each response pair increases, the training loss
example. Since the
Work done during the internship at Institute of Artificial Intelligence (TeleAD), China Telecom. i Corresponning Author.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Sesponce
Cheam
nowr upe crtor
feats
2uutornr Seaported
Repslal Fopente
Plen la the Lsn bullet 5r =35
Rejected Response
dr 1.8
B1M00 of outh ratural angg proresing....
C1F craset
Utraffed chitaed
Propptrrinss Ieforssooee
Figure Left: ORM-Llama3-& 8B' on two different datasets that were used for training. When calculating with other prompts, the curves show the mean and the std. Right: illustrative failure case where the reward gap overly depends on the responses. Solid lines represent corresponding prompt-response pairs (used in training), while dashed lines represent con-corresponding pairs (unseen during training). Since the reward gap overly depends the responses, it generalizes poorly to novel prompt-response pairs constructed even with seen prompts.
reward gaps calculated with corresponding prompt and randomly sampled prompts using
will decrease effectively. This can occur even if the reward model only considers the responses and totally ignores the prompts. In this case, the trained reward model loses its generalization capability over different prompts and may exhibit incorrect preference for novel prompt-response pairs.
Perhaps surprisingly, such a phenomenon indeed appears in current reward models, even some that achieve SOTA performance on common benchmarks. As shown in Fig. (left), after replacing the corresponding prompt with other prompts in the dataset, the reward gaps still center around their original values. This issue, where responses dominate the reward gap, does not affect training but leads to catastrophic results when evaluating novel prompt-respons pairs. The illustrative example in Fig. D (right) shows this. When considering each prompt-respons, pair separately within the training dataset, its reward gap matches the ideal value. However, when querying preferences after replacing the original prompt with other prompts in the dataset (which are also meaningful queries), the reward model can yield inaccurate or even wrong preferences. This generalization issue will become more pronounced when dealing with unseen prompt-response pairs encountered during evaluation. All of these highlight the need to distinguish two components of the reward value: one part is the value determined solely by the response, and the other is the value that can only be derived by simultaneously considering both the prompt and the response. We refer to the former as prompt-free reward and the latter as prompt-relatee reward.
To address this, we propose novel method of decomposition to extract these two components from an information-ther theorretic perspective, without requiring extra models. After that, we use the extracted prompt-free reward to guide the reward learning process, prioritizing training samples based on their prompt-free reward gap values. We verify our method through several toy examples and standard evaluations based on commonly used datasets and base models. In toy examples, the extracted prompt-free reward gaps reflect the reward model's preference bias about response-only features, while the prompt-related reward gaps capture its generalizable preference information. Moreover, in standard experiments with common datasets, the reward model trained with our method outperforms strong baselines. These experiments show that considering both prompt-free and prompt-related rewards during training enhances the alignment performance and generalization capabilities of the reward model.
2 Preliminaries
Standard preference learning assumes the existence of the preference oracle which determines P( 31 [z), the probability that response Do is more preferred than yi conditioned on the prompt x. Given a preference dataset = {(I, Bor ) where the prompt-response tuples (z, Bee 31) are generated following Pyy. > yı|z). our objective is to estimate the preference oracle from it.
There exist several methods to model the preference oracle (24, among which the Bradley- Terry (BT) model [6] is the most widely used. The BT model further assumes that the preference oracle can be represented reward model r (z, 3) + RR that satisfies:
exp(r(x, yw))
IP, (y > vlzz) (sp(r(z, eepp(r(x,
(I)

---

Under this assumption, standard methods leverage parameterized reward model to perform
maximum likelihood estimation on the preference dataset [28 36 via the following objective:
max 3n) Pry (ywr [1]
In this work, we consider the preference learning based on the BT model, owing its widespread use and strong performance in the field of RLHF. The parameterized reward model re can be
implemented in various ways [3 38 I7]. Among these, the most commonly used one is the "Sequence Classifier' [99 28), which builds the reward model on top of an LLM backbone. By projecting the feature representation of the prompt-response sequence to a scalar value using simple network (e.g., single linear layer), such reward model effectively leverages the prior knowledge embedded the L.L backbone to evaluate prompt-response pairs. In this work, we focus on reward models within this type. A discussion of other reward model structures is given in Appendix C.4
3 Methods
In this section, we first present a specific form of decomposition that divides the reward value into prompt-free reward and prompt-related reward. We then formalize the problem of such decomposition and propose a solution from an MI perspective. Finally, we use the extracted prompt- free rewards prioritize training samples and guide reward learning-
3.1 Prompt-free Reward & Prompt-related Reward
We decompose the reward value ro y) into two separate parts: the prompt-free reward and the prompt-related reward. The prompt-free reward is only determined by the response and can be regarded as the overall evaluation of the response. Specifically, the prompt-free reward (r2) satisfies Eq. 3 Once the response is given, the prompt-free reward remains unaffected by any specific prompt.
x2,3 E D, (2(21, y) (2)
(3)
In contrast, as the other part of ro(x, y) besides the prompt-free reward, prompt-related reward (r1) varies among different prompts. Such reward can be determined only when both the prompt and the response are given.
Given reward model ra during training and a prompt-response tuple (æ, yı, v2) sampled from the dataset, our goal is to identify the prompt-free reward in re 3/1) and ro (2,,12 and leverage it to guide reward learning. We consider decomposing ro(x, y) into the additive form:
rg(x, y) = Ti (z. y) + 72 (2 (z,z).
Examining the dataset holistically, r2(x, v) demonstrates clear randomness, with the data distribution influencing its value. Specifically, for given (z, y) pair, if the prompts related to y are diverse, there is little connection between r2 y) and ro(x, since r22 remains unchanged across different the related prompts show little variety,
prompts, whereas ry changes accordingly. Conversely, r2 (2 y/ can be closely related to (2 y). since r3 y) also exhibits little change. This motivates us to define r2 by considering the randomness in the data distribution, treating 72 as the joint product of ro and the data distribution. We also note that although some simple definitions of r2 may seem reasonable (e.g. (z, y) = EXXXP(X =g) [ro v)]). they can't truly reflect ra 's preference regarding the response alone (See Appendix B for more details).
3.2 Solving Prompt-free Reward via Mutual Information (MI) Objective
In this section, we extract r2(2;, y) from ro(a, y) via a preference perspective, with a carefully designed mutual information (MI) objective. A brief introduction to MI is provided in Appendix A For simplicity, we have notations
Ar(z, M1, y2) == r(z, v1) ((z,22) o(Ar(z, vı, v2)) = 1/(1 + exp(-Ar(c vI, v2)) (5)
Since we focus on preference learning based on the BT model, it is sufficient to study the reward gap (Ar) rather the reward itself, as the reward gap decides the preference label. Consequently, we decompose the reward gap as
Arg(z, V1, v2) Ar1(z, V1, v2) Ar2 V1. (1.
(6)
3

---

Meanwhile, since both "1 and 12 must be applicable to all (z, M/, 32), it is essential to take into account the entire dataset when decomposing from the preference perspective, The preference labels of rı, r2, and over the entire dataset are inherently random and can be characterized by the following random variables:
= Ber(o X,,,, Y2)) W (or(EE..PtXIY,,,)))) Yi, Y2)I),, W Be (X, Y1, Y2)))
Z = (er (X, YI, Y2)):
(7)
where 'Ber' stands for Bernoulli random variable. Among them, Z, 2 and W are the random preference labels of their corresponding rewards. W is the random prompt-free preference label of ra- For all these random variables, the randomness comes from the prompt, responses, and the Bernoulli distribution. We also provide detailed interpretation these random variables in Appendix D.I
Based on the defined Bernoulli random variables, we proceed to characterize the desired ri and r2- Recall that T1 and r2 represent prompt-related and prompt-free rewards, respectively. This means Z should encapsulate only prompt-related preference while Z should only reflect prompt-free preference. Formally, this could be written in the following conditions:
MI(Z W) 0, MI(2 W) MI ( W).
(8)
The Venn diagram in Fig. further illustrates these conditions. The first condition eliminates all information in Z that's 's related to prompt-free preference, while the second condition ensures that the information contained in does not exceed the prompt-free preference.
n w
2 n(w
WAW
Z
Figure 2: Information in W and W. (b) Desired information in Z and Z. (c) Undesired information in and Z.
Although these conditions effectively constrain the information in Z and Ž, a trivial solution exists. Assume Ara (2, y) is bounded for any prompt x and response y- Consider an ill-
formed Ar; such that for any prompt and response Ar (x, v) = +00. Then by the definition that Arg (2, v) Ar (x., 3) + Ar2(2, y), (r2(x, y) oo for any prompt x and response y. In this case, Z will always be 1, and will always be O. This makes (W) = MI ([) = MI M[( = since and 7 are both constants. Although these ill-formed r1 and r2 satisfy the conditions in Eq. (8, they provide information of prompt-related prompt-free preferences. This ill-formed example indicates that the MI condition in Eq. (8) insufficient to induce a reasonable decomposition. When Z and become constants, the information within them is lost, along with their MI with other random variables. To avoid this, we optimize the following constrained objective derive rn and r2. which ensures sufficient information in Z (H stands for Shannon's 's entropy):
MI = 0
max H(Z), s.t.
(9)
MI ((( = MI (())).
In addition to the previous conditions, we maximize the information within Z, considering that Z represents the random prompt-related preference label across the entire dataset. This constrained optimization problem is difficult solve directly, and direct solutions may require additional parame- terized reward models. However, we provide an efficient algorithm that obtains Ar (æ, y1. 12) and Arz(r, V1, U1) without additional reward model. To see this, we first note that the second constraint satisfied with a specific structure of r2. Formally, the following theorem holds:
Theorem 1. When the value of 122 depends only the response, i.e. r2(2, 2) r2(v). MI (1(2)) MI MI(2]WW.
The proof is given in Appendix D.3 Intuitively, Theorem holds since such 7 inherently does not
contain any information from the prompt. Consequently, removing the prompt-related information from W to W does not reduce the MI term. This important property enables us to concentrate on the first constraint and maximize the information in Z, as long as r22 maintains this simple structure. We continue to characterize the optimal solution of Eq. (9) in the following theorem.

---

Theorem 2. For any bounded To and dataset (X, Y2),, there exist feasible ri, r such that V(yi, 82) P(Y, Y2),
(1) (Ari (z, Mr, v2))] 2. Such ri. r2 is the optimal
solution to problem 9
The proof is given in Appendix D.3] Intuitively, this requires the prompt- part of rí to equally prefer both responses in any given pair. This aligns with our desiderata, as rf should prefer some response only after being provided with the specific prompt. Although neither of the theorems explicitly shows the connection with ro. we note that the feasibility condition requires (r, y1, 32) Arž (ar, y/: y2) to equal Arz (z. y1, 82), which encodes re's information. Theorem 2 also provides
an efficient way to obtain Ari (ar, M1, 32) and Arz 11, 22). After replacing Ari [æt 91, V2) with
Arg (I, M1, y2) - Arī(# (x. V1, 3/2) in the optimal solution in Theorem 2 Ar (2r, yı 82) satisfies that for any (y1, 1/2) P(Y,, Y2):
1
4 (y11. 92)
E an PN X2-
(Are(( y1, v2) (yr 82)) 2
(10)
Note that we replace Ar (1,, MI, 31) with (ri 322 due to Theorem Because of this, (ri 322
will not change while taking the conditional expectation. Moreover, it can be proved that D(y (1, 8/2
in Eq. (I0) decreases monotonically with the increase of Arž (11, V2), shown in Appendix D.3
Combined with the bounded assumption of ra. we have:
lim Arg +20
D ( y2) O,
lim
Φ y2) =
(11)
(V1
With this, (3i y2) can be obtained via binary search in a finite interval. The searched Ari (11, 82) can be interpreted as the 'weighted average' of (ro(x, y1, y2), based on P(X = y/, Y2 92)
Due to limited space, we provide the pseudo-code of the binary search process in Appendix C. (Alg. D. Note that such an algorithm doesn't 't require any extra parameterized reward models.
It's clear that during the binary search, the estimation of the expectation in Eq. (0)) requires sampling from P(X = yı, Y2 = y2). Although sampling from it is generally challenging, we propose utilizing importance sampling by considering the decomposition with Bayes' rule:
P(rlyn, 3/2 P(x)) 52/x) P(y, 1/2).
(12)
Here P(zlyy,, y22 ) is the abbreviation of P(X = vi = y1, Y/2 = 12), so as other probabilities. Given (w1, y2), we first sample from P(X), the marginal distribution of prompts, and then re-weight the samples using P(n. 3/2 (x). Its value represents the response generation probability and thus can be efficiently computed in some cases or approximated in others. Moreover, since P(y y2 is independent of the trained reward model, each dataset only needs to pre-sample prompts once and compute the probabilities once. We list the sampling schemes for different cases in Appendix C.2
3.3 Guide Reward Learning with Prompt-free Reward
During the training process of ra. one may consider leveraging the reward gap value Arg(x, V1, y2) to selectively train more samples with small reward gap (including negative ones). The reason is that, if the reward gap for (x., y1, 322 satisfies (ro V1, y2) ze 0 or < 0, it indicates that the current reward model ra struggles to distinguish between the preferred and dispreferred responses. Consequently, allocating more training budget to these samples can better align the reward model
with human preference. Similar methods have been studied in training LLMs [2 26).
Characterize ideal re with Art and Ar2- However, things become different if we consider prompt-related and prompt-free reward separately. During the learning process of re, we expect (i) ra to learn more prompt-related preference from the dataset (large Ar (z. y1, 12))); and (ii) re to learn less prompt-free preference (small absolute values of Ar2 Vı, 32)). This is because when (ry( M1, y2) > 0 or < O, it indicates strong prejudice over the sampled responses, as Ar² doesn't 't take the specific prompt into account. This prejudice may introduce spurious reward gaps (e.g..
reward gap induced by length bias) 13 35
Prioritizing data with small Arz. Similar to the method discussed at the beginning of this section, Ari and Arz are important properties of Ias and can also be leveraged to guide the training of rg. To fulfill the desiderata above, we present straightforward method that prioritizes preference data when training reward models, with the aim of increasing prompt-related reward gaps while constraining the value of prompt-free reward gaps. Specifically, given data samples {0,, 31, 32) ) k in each iteration,
5

---

we first decompose re to get their prompt-free reward gaps (Ar2(z, v1. V2) based on Alg. | and then perform update for ro on those samples with small prompt-free reward gaps. For samples with large prompt-free gaps, we ignore them in this iteration and reinsert them into the buffer for subsequent updates.
Analysis of prioritization mechanism. Next, we provide an in-depth analysis for the proposed data prioritization mechanism. The standard update following the BT model only ensures the increase of Ar (I, Me. 31 + (r2 Vw,, 31),, so we cannot guarantee that either of them will necessarily increase. However, (i) if samples have small prompt-free reward gaps (e.g. Ary < 0), regardless of whether their prompt-related reward gaps (Ar1) are large or small, they should be used for updates. This because an increase in Ar1 suggests a better mastery of prompt-related preferences, while an increase in their Ar2 indicates the elimination of existing prejudices, both of which are beneficial. (ii) Conversely, for samples with large Ar² (e.g., Ar2 >> 0), if their Ar, are also large, updating is unnecessary since the prejudice led to gradient saturation in the BT model. If their Ari values are small, their Aro values are dominated by factors unrelated to the prompt (e.g., response length). Updating re them would exacerbate this issue, hindering generalization. Fig. 3 further illustrates this mechanism. Data samples with smaller Ar2 help achieve a better trade-off between increasing prompt-related reward gaps and constraining prompt-free reward gaps.
extpled des (znfons)
updened cats getert
LT pcared CETA (she)
Dry
Figure 3: (lllustrative) We characterize training data samples in a 2-dimensional quadrant diagram, with the decomposed reward gaps Ar: and Ar2. (a) shows initial data samples before training. (b) After training, the ideal distribution should be centered on the positive half the Ari-axis, indicating that the preference depends solely on prompt-related information. (c-d) However, in the training process Aro, the update following the BT model can only ensure the data points move in at least one of the positive directions of the Ar Aris or Arz-axis (up or right). If the update is based on samples with small Ar2 (e.g. the left half), their movement upwards or to the right, along with other unupdated samples (e.g. the right half), will cause the distribution to be more centered on the positive -axis. On the other hand, if the update is based on samples, as shown in (d), the movement of samples with large Ary values to the right exacerbates the existing prejudices, causing the distribution to become more centered on the positive Arz-axis.
Practical Consideration. In practice, we design a binary-cluster mechanism to determine whether the value of Ar2 is 'small' or 'large' in data prioritization. Specifically, at each step, we sample batch of (z. 31, 1/2) pairs from the preference dataset and perform one-dimensional binary clustering based on their Ar2 values. Due to the dynamic nature of ra and the noise in data sampling, we do not directly use the obtained decision boundary for determination. Instead, we maintain a dynamic threshold, calculated the exponential moving average of the boundary, and use this threshold for the determination. Due to limited space, we provide a detailed process in Appendix C.. (Alg. D). We justify the use of binary clustering and the EMA threshold in Appendix E.3 and demonstrate their robustness through experiments.
It is worth noting that samples with Ara values exceeding the threshold will not be discarded from the dataset. Instead, the reinsertion operation assigns a probability for these samples to be updated the next time they sampled. Since re dynamically evolves during training, samples that not prioritized at the moment may contribute significantly to the future rg. To avoid endlessly cycling through samples with large Ár2 values, the opportunities for reinsertion are limited. We provide more details of the algorithm in Appendix C
Experiments
In this section, we demonstrate the significance of identifying and utilizing prompt-free rewards to guide reward learning from two perspectives. We first illustrate that, for some manually crafted datasets, the extracted prompt-free reward reflects the reward model's preference bias and the prioritization effectively aids in reward learning. Subsequently, based on some commonly used open-source preference datasets, we evaluate the trained reward model using direct metrics (reward

---

model accuracy) and indirect metrics (performance of the induced policy). The evaluation results demonstrate that, with the guidance of prompt-free rewards, our method enhances the generalization ability of the learned reward model and further improves the performance of the induced L.M policy after alignment.
4.1 Experiments on Manually Crafted Datasets
In this part, we manually construct datasets with specific characteristics from SHP 14]. commonly used open-source preference dataset. Based on these datasets, we compare the reward models trained with ordinary data and prioritized data. thoroughly demonstrate their difference during training. we select four equally spaced training steps, sample number of data pairs at random, and visualize them by their Ari, Ary values using the same method as Fig. 3. Additionally, we evaluate the final reward model using Reward-Bence [19] to assess its generalization capabilities. We choose as the backbone due to its lightweight nature and strong performance. For more details of the experiments, refer to Appendix E.4
4.1.1 Length-biased Dataset
We construct a length-biased biased dataset Diius that contains preference pairs with 80% chosen-longer responses (i.e. Iyuel [yrl) and 20% chosen-shorter responses. The details of the construction are given in Appendix E.4 When performing reward learning in Diaas the BT loss for uniformly sampled data batch can be easily optimized by considering only the lengths of the responses, as the majority of the data exhibits a preference for longer chosen responses. As a result, the reward model can easily overfit to such spurious, prompt-free preferences.
The visualizations during training and the final results on Heward-Benehe are shown in Fig. g and Table [ (a). If rg is trained with data uniformly sampled from D (Fig. 4 (a), top), although its prompt-related reward gap (Ar1) increases gradually, Ta rapidly overfit to the length preference. This results in significantly faster increase in Ar2 for the chosen-longer pairs, and inevitably leads decrease Ar for the chosen-shorter pairs since they represent opposite prompt-free preferences. As expected, such Ar² indicates a clear length preference within rø, which undermines ra generalization capability. On the other hand, if we prioritize the data samples with smaller Ar2 values, the training data batch will contain more chosen-shorter pairs with smaller Ar2 values. According Fig- (a) bottom, the data samples well-centered around the positive Ari-axis during training. The results on Heward-Benche as shown in Table (a), also demonstrate stronger generalization capability of re trained with prioritized data.
tmining process
(b)
Figure 4: (a): Visualizations for length- dataset. mark data points that satisfy |Bwl > lyd in red and the ones that satisfy [ym] S |yx| in blue. (b): Visualizations for the adversarial prompt dataset. We mark adversarial data in red and original data in blue. In both (a) and (b), the top shows ordinary training data, and the bottom shows prioritized training data.
Model
Chat
Chat Hard
Safety
Reason
Avg
Model
Chat
Chat Hard
Safety
Reason
Avg
vanilla 78.0
29.8
36,4
58.2
50.6
vanilla
80.7
28.5
40.2
36.5
46.4
original
84.9
31.3
42.8
49.0
$2.0
ours
86.8
31.1
45.1
60.3
55.8
ours
84.3
30,9
43,2
46.7
51.6
(a)
(b)
Table (a): Results on Reward-Bench Bench with bength-biased data. (b): Results Reward- with adversarial- prompt data, "original" means vanilla reward training with only original data. Due limited space, we present the mean results of 3 runs with different seeds. See Appendix E. for full results with std values.
7

---

4.1.2 Adversarial-prome Dataset
We construct an adversarial-prompe dataset Dadv by adding adversarial samples to the original SHP dataset. More specifically, for each original data sample (æ, yw, Bt), we generate an adversarial sample (z, Mr, Uw) and add it to the dataset. I [ar, s] is the concatenation of the original prompt string z and another string 8 E {21, 81, 82 are defined as:
$1= Givv a response that is as long as possible.' $2= 'Give a response that is as short as possible.'
To ensure reasonable preference in the adversarial sample, have [ar, 82] if the chosen response is longer in the original sample ((i.e. [Bwl > (). and set E = [1] if if Iy S [yr]. To prevent doubling the training budget, we randomly select half of the SHP data for this transformation. We refer to an illustration of such data processing in Appendix E.4
When trained with Dady, an ideal Are should not be dominated by its prompt-free part (Ar2). This is because the preference label in the dataset will reverse with slightly different prompts (i.e., with or without s). (i) If Arg is dominated by Ar2, the preference it learns from adv will significantly degenerate compared with the preference learned solely from original data. This is because Ar2 learns from both original and adversarial samples and overlooks their difference in the prompts. The adversarial samples will contribute to Ar2 with the same probability but confficted preference, eliminating the preference learned from the original data. (ii) If the influence of Ar² is minor, re can retain the preferences learned from the original data even with ad verrsarial data, since they have different prompts and will not affect each other considering prompt-related preference.
In the visualizations and final results presented in Fig. 4 (b) and Table (b), it is evident that when ro is trained on Dadv with ordinary data, Are is dominated by Ar2- Moreover, T22 exhibits strong preference for the originally rejected responses. This occurs because the simplified preference in the adversarial dataset can be inferred solely from response length, which is easier for the model to overfit. The preference of the overfitted reward closely mirrors the adversarial data. In contrast, our method prioritizes original data, which is harder learn compared with adversarial samples. This makes both the original and adversarial samples concentrated along the positive Ari-axis and results in more generalizable rg. The performance of such rgg is closer that trained on the original data.
4.2 Standard Experiments on Open-source Datasets
In this section, we evaluate our method using several commonly used open-source preference datasets. Unlike the manually crafted datasets discussed in section 4.1 the prompt-free preferences of ra on these datasets may not directly exhibit specific features. However, large prompt-free reward gap often suggests that rp may overfit to preferences that are irrelevant to the prompt. Such prompt-free preferences can negatively impact the generalization capabilities of rg. We validate the effectiveness of our method across different base models and datasets through both direct evaluation of reward model accuracy and assessment of the induced policy performance. For more details of the experiments, we refer to Appendix EE.]
Evaluate Reward Model Accuracy. We use Reward-Bench 19 as the benchmark to evaluate the reward model in a direct way, by testing its accuracy on well-organized but out-of-distributio queries. For a detailed illustration, we conduct experiments using both ----MMA-------nttttte
[12]
and Mistral-7B InItrrcttt [18] as the backbone of the reward model. For the training data, we use randomly sampled subset of 300K examples from the RLHFlow preference datase 22) (originally 700K). We also conduct the same experiments on the SHP dataset (see Appendix F.2 for results).
The results are listed in Table (a). We choose vanilla BT reward training and RRM (23 as the baseline methods. RRM studies generalizable reward learning via causal inference and proposes a solution also from the data distribution perspective. Compared with baseline methods, our method demonstrates consistent improvements over different aspects of Reward-Bench In Appendix E.5 we provide explanations of the superiority of our method compared with baseline methods.
Assess Performance of Induced Policy. In this part, we evaluate the performance of the induced policy, which is obtained by combining the trained reward model with RLHF algorithms. For all experiments, we choose I.LaMA-3-S M-MAA--B--Instruce model. The reward model in this part is trained with the SHP dataset. For the RLHF algorithms, we
both the base policy and the backbone of the reward
dataset
8

---

Reward-Bench
(b) AlpacaEval-2
Method
Chat
Chat Hard
Safety
Reasoning
Average
AlpacaEval-2
(LCWR WR)
Method
vanilla
RRM
Ours
vanilla-8B
0.93
0.50
0.67
0.78
0.72
DPO
30.5/38.8
39.5/403 40,9
40.67 42.3
RRM-8B
0.95
0.56
0.75
0.82
0.77
BNN(N=4)
33.2/42.5
35.3 38.4
38.5 / 45.0
BoN(N=32)
35.8/45.1
41.1 41.1/44.9
44.7 / 48.3
Ours-8B
0.96
0.59
0.81
0.89
0.82
vanilla-7B
0.90
0.49
0.60
0.69
0.66
(c) MT-Bench
MT-Bench /T2)
RRM-7B
0.94
0.52
0.65
0.70
0.70
Method
vanilla
RRM
Ours-7B
0.94
0.56
0.69
0.81
0.75
DPO
7.83/6.51
8.35 7.46
8.44/8.03
Table 2: (a): Reward model accuracy on Reward-Bench. (b): Results on AlpacaEval-2 [3] evaluation. L.WR and WR denote Length-Control (I.C) Win-Rate and Win-Rate, respectively. (c): Results on MT-Bench [41 evaluation. Tl and T2 denote the Ist-Tuur and 2nd-Tum, respectively. Due to limited space, we present the mean results of runs with different seeds. See Appendix E.1 for full results with values.
choose Best-of-N N and DPO 30 because of their lightweight nature. It's straightforward to apply the reward model in oest-of-N. For DPO, we use the base policy generate responses on the prompt dataset (from Ultrafeedback 19), and then employ the trained reward model to select the best and worst responses, which are combined as training data for DPO. We assess the performance of the induced policy on AlpacaFval-2 and MT-Bench. The results are given in Tab. Tab.B (b) and Tab. lbb.lu (c), respectively. Since MT--Bench requires multi-turn dialogs, we only test the performance of the policy trained by DPO. Our method shows superior performance on these benchmarks, indicating benefits of using a strong generalizable reward model.
5 Related Works
RLHE. RLHF research can be categorized into reward-based and reward-free methods. The awardd-ased methods typically train an reward model to provide reward signals for RL optimization.
PPO [31 [3] is popular in this domain g8 28 3]. REINFORCE [39 and rejection sampling [27]
have also been adopted in previous works [0]. Alternatively, reward-free approaches directly
fine-tunin the LLM by constructing implicit reward models [5]. such as DPO (30) and its variants
40
20
29
25 Recent works also try combine DPO with reward models to iteratively generate
data and perform alignment
Generalizable Reward Models. Improving the generalization ability of the reward model is crucial in RLHF Previous research (21 reveals that the reward model can be hacked by spurious preferences response length) and mislead the LLM when evaluating novel prompt-response pairs. To mitigate this issue, previous methods propose to separately model different types of preference signals (37. explicitly learn the reward bias while keeping the reward model focused on human preference 77 [3], or augment the data with constructed prompt-response pairs [33 23 Unlike these methods, algorithm learn prompt-free free reward without introducing additional models, training data, or various types of preferences.
6 Conclusion & Limitation
In this paper, we propose a novel reward learning algorithm aiming at improving the reward model's generalization capability. This algorithm separately considers the prompt-free part and the prompt- related part of the reward model during training, which are extracted via a MI perspective. By prioritizing the training data with prompt-free reward gaps, our algorithu encourages the reward model to focus on prompt-related preferences. We validate that our method enhances the reward model's generalization ability through various toy examples and experiments on common benchmarks.
The major limitation is that the decomposition of prompt-free and prompt-related rewards in this paper relies specific additive form. This specific form may not be suitable for all reward models and datasets, which means the values of these two rewards may not accurately match the true values. Such inaccuracy prevents from directly using the prompt-related reward the score. Nevertheless, even if the additive decomposition does not hold, our method can still benefit reward training in a similar way (see Appendix [D.2 for detail discussions). In future work, we will explore more general decomposition forms to broaden its applicability.

---

Acknowledgement
The Shanghai Jiao Tong University team is partially supported by Shanghai Municipal Science and Technology Major Project (202ISHZDZX0122 and National Natural Science Foundation of China (62322603). The UT Austin team partially supported by NSF 2340651, NSF 2402650, DARPA HR001 1200012 and ARO 19F-
References
[1] A. Ahmadian, C. Cremer, M. Gallé, M. Fadaee, J. Kreutzer, O. Pietquin, A. Üstün,, and S. Hooker. Back to basics: Revisiting reinforce style optimization for learning from human feedback in Ilms. arXiv preprint 144444002..474400, 2024.
[2] R. Y. Aminabadi, S. Rajbhandari, A. A. Awan, C. Li, D. Li, E. Zheng. O. Ruwase, S. Smith, M. Zhang. J. Rasley, al. Deepspecd-inference: enabling efficient inference of transformer models at unprecedented scale. In SC22: International Conference for High Performance Computing, Networking, Storage and Analysis, pages 1-15, IEEE, 2022.
[3] M. G. Azar, D. Guo, B. Piot, R. Munos, M. Rowland, M. Valko, and D. Calandriello. A general theoretical paradigm to understand learning from human preferences. In International Conference on Artificial Intelligence and Statistics, pages 4447-44444 4455. PMLR, 2024.
[4] C. Bai, Y. Zhang, S. Qiu, Q. Zhang, K. Xu, and X. Li. Online preference alignment for language models via count-based exploration. In International Conference on Learning Representations, 2025.
[5] Y. Bai, A. Jones, K. Ndousse, A. Askell, A. Chen, DasSarma, D. Drain, S. Fort, D. Ganguli, T. Henighan, et al. Training helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05800 05562, 2022.
[6] R. A. Bradley and M. E. Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39(3/4):324-345, 1952.
[7] L. Chen, Zhu, D. Soselia, J. Chen, T. Zhou, T. Goldstein, H. Huang, M. Shoeybi, and B. Catan- zaro. Odin: Disentangled reward mitigates hacking in rlhf. arXiv preprint arXiv:2402.07319, 2024.
[8] P. F. Christiano, J. Leike, T. Brown, M. Martic. S. Legg, and D. Amodei. Deep reinforcement learning from human preferences. Advances in neural information processing systems, 30, 2017.
[9] G. Cui, L. Yuan, N. Ding, G. Yao, W. Zhu, Y. Ni, G. Xie, 7.. Liu, and M. Sun. Uttrafeedback: Boosting language models with high-quality feedback. 2023.
[10] H. Dong, W. Xiong, D. Goyal, Y. Zhang, W. Chow, R. Pan, S. Diao, J. Zhang, K. Shum, and T. Zhang. Raft: Reward ranked finetuning for generative foundation model alignment. arXiv preprint arXiv:2304.0670
2023.
[11] H. Dong, W. Xiong, B. Pang, H. Wang. H. Zhao, Y. Zhou, N. Jiang, D. Sahoo, C. Xiong, and T. Zhang. Rlhf workflow: From reward modeling online rlhf. arXiv preprint arXiv:2405.07866 0766633,, 2024.
[12] A. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. Al-Dahle, A. Letman, A. Mathur, A. Schelten, A. Yang, A. Fan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21788 2024.
[13] Y. Dubois. P. Liang, and T. Hashimoto. Length-controlled alpacaeval: A simple debiasing of automatic evaluators. In First Conference on Language Modeling, 2024.
[14] K. Ethayarajh, Y. Choi, and S. Swayamdipta. Understanding dataset difficulty with V-usable information. In K. Chaudhuri, S. Jegelka, L. Song, Szepesvari, G. Niu, and Sabato, editors, Proceedings of the 39th International Conference Machine Learning, volume 162 of Proceedings of Muchine Learning Research, pages 5988-600S PMIR, 17-23 Jul 2022. URI. https: https://proceeddnng mlr, prwss/ wttps/vv22/eeee http:////eee//ettaaaaaaaaaaaaaaa html
10

---

[15] K. Ethayarajh, W. Xu, N. Muennighoff, D. Jurafsky, and D. Kiela. Kto: Model alignment as prospect theoretic optimization. arXiv preprint arXiv:2402.01 306, 2024.
[16] J. Hu, X. Wu, W. Wang, D. Zhang, Y. Cao, et al. Openrlhf: An easy-to-use, scalable and high-performance rlhf framework, arXiv preprint arXiv:2405. //14, 2024.
[17] H. Ivison, Y. Wang, V. Pyatkin, N. Lambert, M. Peters, P. Dasigi, J. Jang. D. Wadden, N. A. Smith, I. Beltagy, et al. Camels in a changing climate: Enhancing Im adaptation with tulu arXiv preprint arXiv:2311.10702 2023
[18] A. Q. Jiang, A. Sablayrolles, A. Mensch, C. Bamford, D. S. Chaplot, D. d. I. Casas, Bressand, G. Lengyel, G. Lample, I., Saulnier, et al. Mistral 7b. arXiv preprint 2xXix:330000822 2023.
[19] N. Lambert, V. Pyatkin, J. Morrison, L. Miranda, B. Y. Lin, K. Chandu, N. Dziri, S. Kumar, T. Zick, Y. Choi, al. Rewardhench: Evaluating reward models for language modeling. arXiv
preprint arXiv:2403.13780 2024.
[20] A. Lee, X. Bai, Pres, M. Wattenberg, J. Kummerfeld, and Mihalcea. mechanistic understanding of alignment algorithms: A case study on dpo and toxicity. In International Conference on Machine Learning, 2024.
[21] J. Leike, D. Krueger, T. Everitt, M. Martic, V. Maini, and S. Legg. Scalable agent alignment via reward modeling: a research direction. arXiv preprint 11Xiv:1811.0788 2018.
[22] Z. Lin, Z. Gou, Y. Gong, X. Liu, Y. Shen, R. Xu, C. Lin, Y. Yang. J. Jiao, N. Duan, et al. Rho-1: Not all tokens are what you need. Proc. of NeurIPS, 2024.
[23] T. Liu, W. Xiong, J. Ren, L. Chen, J. Wu, R. Joshi, Y. Gao, J. Shen, Z. Qin, T. Yu, et al. Rrm: Robust reward model training mitigates reward hacking. arXiv preprint 24X1v:2409.13550 2024.
[24] R. D. Luce. Individual choice behavior, volume 4. Wiley New York, 1959.
[25] Y. Meng, M. Xia, and D. Chen. Simpo: Simple preference optimization with reference-free reward. arXiv preprint vrXiv:2405.14744 2024.
[26] S. Mindermann, J. M. Brauner, M. Razzak, M. Sharma, A. Kirsch, W. Xu, B. Höltgen, A. N. Gomez, A. Morisot, S. Farquhar, et al. Prioritized training on points that are learnable, worth learning, and not yet learnt. International Conference on Machine Learning, pages 15630-15649, PMLR, 2022.
[27] R. M. Neal. Slice sampling. The annals of statistics, 31(3):00--77, 2003.
[28] L. Ouyang. J. Wu, X. Jiang, D. Almeida, C. Wainwright, P. Mishkin, C. Zhang. S. Agarwal, K. Slama, A. Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730-2777 2022.
[29] R. Rafailov, J. Hejna, R. Park, and C. Finn. From r to q*: Your language model is secretly a q funcction. arXiv preprint arXiv:2404.12355 2024.
[30] R. Rafailov, A. Sharma, E. Mitchell, C. D. Manning, S. Ermon, and C. Finn. Direct preference optimization: Your language model is secretly reward model. Advances in Neural Information Processing Systems, 36, 2024.
[31] Schulman, Wolski, P. Dhariwal, A. Radford, and O. Klimov. Proximal policy optimization algorithms. arXiv preprint D.Xiv:1707006444 2017.
[32] C. Shannon. mathematical theory of communication. The Bell system technical journal, 27(3):3794423 1948.
[33] I.. Shen, Chen, L. Song, I.. Jin, B. Peng. Mi, D. Khashabi, and D. Yu. The trickle-down impact of reward (in-) consistency on rlhf. arXiv preprint 275. 2023.
11

---

[34] W. Shen, R. Zheng, W. Zhan. J. Zhao, S. Dou, T. Gui, Q. Zhang, and X. Huang. Loose lips sink ships: Mitigating length bias in reinforcement learning from human feedback. arXiv preprint arXiv:2300005199 2023.
[35] P. Singhal, T. Goyal, J. Xu, and G. Durrett. A long way to go: Investigating length correlations in RLHF. In First Conference on Language Modeling, 2024. /tLLhtts///oppereviii
[36] N. Stiemnon, L. Ouyang, J. Wu, D. Ziegler, R. Lowe, C. Voss, A. Radford, D. Amodei, and P. F. Christiano. Learning to summarize with human feedback. Advances in Neural Information Processing Systems, 33:3008-3021, 2020.
[37] H. Touvron, Martin, K. Stone, Albert, A. Almahairi, Y. Babaei, N. Bashlykov, Batra, P. Bhargava, S. Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288 2023.
[38] Z. Wang, Dong. O. Delalleau, Zeng. G. Shen, Egert, Zhang, M. N. Sreedhar, and O. Kuchaiev. Helpsteer2: Open-source dataset for training top-performing reward models. arXiv preprint arXiv:2406.08673, 2024.
[39] R. J. Williams. Simple statistical gradient-folle following algorithms for connectionist reinforcement learning. Machine learning, 8:229-2566 256, 1992.
[40] Y. Zhao, R. Joshi, T. Liu, M. Khalman, M. Saleh, and J. Liu. Slic-hf: Sequence likelihood calibration with human feedback. arXiv preprint arXiv:2305. 2305:2200000425 2023.
[41] L. Zheng, W.-L. Chiang, Y. Sheng, S. Zhuang, Z. Wu, Y. Zhuang, Z. Lin, Z. Li, D. E. Xing, al. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 36:46595-46623, 2023.
[42] D. M. Ziegler, N. Stiennon, J. Wu, T. B. Brown, A. Radford, D. Amodei, P. Christiano, and G. Irv- ing. Fine-tuning language models from human preferences. arXiv preprint arXiv: 1909: www.t/909999955 08593, 2019.
12

---

A Brief Introduction to Mutual Information
The mutual information of two random variables is a non-negative quantity that measures the mutual dependence between the two random variables [32] Intuitively, it quantifies how much information can be gained about one random variable by observing the other. Mathematically, with a little abuse of notation, the mutual information of two random variables (XX' and Y) can be calculated from their respective entropy and joint entropy as follows:
= H(Y H(X,Y)
H
(13)
where 'H' represents Shannon entropy here. Note that MI = a necessary condition of X and Y being independent, and small zero value of mutual information indicates that the two random variables are weakly correlated.
B Numerical Example in Section 3.1
We provide numerical examples showing why we can't directly obtain r2(2, by marginalizing the prompt distribution P(X = y). If we directly use the following form of r2(r, v):
r2(2,,) Bew-
(14)
Consider two responses V1. Y/ and their corresponding conditional distributions P(X 91). P(X = 322. Assume P(X = 31) P(X = y2) and a prompt set {ax} such that Ex'(a,)) P(X r'lY = v1) - P(X r' y22 = 0.001. If re(æ,, v1) and ro(x, y2)
satisfy the following condition:
(ro( v1) 1ea, re(x, v2) 0 if E {x1).
(15)
(o(o),1) 0, ra(æ, 12) =1 else,
It's easy to verify that (('')(X))==))) (ro(x' (0(xx,,')) 1e3 and Ex'" (ro(( y2)] 0.999. While the reward obtained by taking the expectation suggests that is overwhelmingly better than y2 1/2 actually outperforms Y1 in almost all cases and should considered the better choice overall.
In fact, the gap exists due to the fact that the gap of the 'overall reward' cannot reflect 'overall preference'. When we marginalize To 3) over the prompt distribution P(X|Y = y). the reward values calculated with different prompts are combined together. However, any constant shift of the reward value within prsmoteresponse pair equivalent under the BT model, which means the reward values under different prompts can have totally different scales and meanings, and should not be combined together. On the other hand, for any prompt-response pair, the induced preference label can be interpreted as a Bernoulli random variable with a fixed probability of taking the value of Such consistency makes reasonable to consider the overall preference rather than the overall reward.
13

---

Algorithm 2 Guide Reward Learning with Ar2
Input: initial reward model re, dataset {(z, yws 1 EMA weight a, update step number T, batch size &
Initialize threshold
2: for 1= to 7 do
3: Initialize batch list B []. Ary list AR = []
4: while length(B). < k do
5: Sample (x', ve, 1/) from the dataset
6: {(x, You y1)i = [11 M1)i {(r',
7:
Calculate Ar2(r', Vaa yf) with Alg.
8:
if Arz(x', VV.. v/) < then
9:
R AR2 + (Arz(a', Vor 3/)1, '+' means list concatenation
10:
B B + [(x, Yo 21)
11:
else
12:
AR AR2 + |Ar2(' Vie v/)l
[(x, yw: Vw: n1.)) {1x, ywr Vw 11)s U ((r', Veu v/13
14:
end if
15:
end while
16:
Update ro via BT model with B = [( Mhos 31)515-1
17:
Perform binary clustering with AR2 obtain boundary Å
18:
Update threshold as d4+1 = a x + -- aa) ),
B= [], AR
20: end for
C Algorithm Details
C.1 Pseudo-code
Algorithm Binary Search for (rg 31)
Input: bounded re( ['mnn maax for any prompt x and response y), response pair (yors 80).
error threshold c
1: Initialize Arleft = Tmin Tmax, Arright Tmux Tmin
2: while Arright Arlet >
do
3: Armid +
4: Calculate prompt-free free preference p:
E
(a (Arg(I Mw: vi) Armia)]
5: if then
6: Arkat Armid
7:
else
8:
Arright = Armid
9:
end if
10:
end while
Return Ar 1n)
Arright
C.2 Sampling schemes for binary search
As mentioned
section 3.2, a key step of the binary search is estimating Ara(y y2))l, which requires sampling from (Aro( (1, V1.y2)
E
P Yu=uy) P(X Y2 v2). We re-weighting x with P(y 32 x). Now we provide the method by which this probability can be computed estimated. The method is different for different kinds of datasets.
importance-sampling trick to turn this problem into
Self-generated dataset This corresponds to datasets where the responses are generated by LLM whose parameters be accessed by us. This is especially useful when want to iteratively align
14

---

LLM. In each iteration, we may generate a couple of responses using the LLM in the previous training iteration, based on some prompts, and then get the preference label from either human or Al feedback. In this way, the chosen response and the rejected response are generated independently, which leads to an easy way of calculating P(y 32)r)) as follows:
P(v1- V2 x)) = P(yi (z) P(v2/>) = (nl |1 Pluft I[ vi
P(v D))
v/2 D) (16)
Note that since we have access to the parameters, P probability and can be effectively computed.
P [[,, vl 1) is just the next-token prediction
Other dataset For general datasets that the generation probability cannot be exactly computed, we simple strategy approximate P(yi, 82/a)) Specifically, if K prompt samples in total [o Ara (z, y1, 3/2 Ar2(y,, 3/2)]]. we set P(y,, y2(4))
to estimate expectation
E
=~~((X| w/)
as a fixed probability p when (z, /1. 3/2) are corresponding prompt and responses. We then set 1-2 P(y., 32[2) = K-1 when (æ, 1/1 32) are con-coorespnnding prompt and responses. To ensure that the prompt-related preference is always considered, we will always sample the corresponding prompt for all the responses. In other words, estimate the expectation with the following equation:
K-1 1 1-p
p o(rro(Foon: 81, y2) Ar2(U1. (22)
K-1 Σ
a(Aro (I 31, 82))
(17)
Note that this estimation has an interesting interpretation when p = 1: p= means in the dataset, there is only one prompt Zoom that is related to the responses. In this case, any preference learned from this prompt-response pair cannot be substantiated by other prompts that such preference is prompt-related and is not prompt-free. In this case, the preference learned from this sample is considered a prompt-free preference to avoid getting influenced by potential spurious or prompt-free factors. This can be seen pessimistic estimation of the prompt-free preference in practice.
C.3 Other practical details of Algorithm
Finite reinsertion times To avoid endlessly cycling in the data samples whose Arz values exceed the threshold, every data sample has limited chances to be reinserted into the dataset. In practice, this could be implemented in two ways:
Immediate reinsertion: every time a data sample exceeds the Arz value threshold, immedi- ately reinsert it into the dataset and reduce its reinsertion quota.
Lazy reinsertion: every time a data sample exceeds the Ar, value threshold, directly discard it. However, when the dataset is exhausted, refill the data loader with all samples. The previously discarded samples will be checked again in the new data loader. The number of reinsertions determines the number of times we flush the data loader.
Binary clustering algorithm As only perform binary clustering simple algorithm works well. In practice, we choose K-means as the binary clustering method. We also tried Otsu's method and the resulting reward model's performance is very similar.
nne-dimensional data,
C.4 Discussion of other reward modeling methods
In this paper, we mainly focus on the 'Sequence Classifier' reward models. We discuss the reason why we don't use this method to investigate other type of reward models.
Generator: the "Generator" structure treats preference modeling as an instruction-following task, which incorporates the prompt, chosen and rejected response in the instruction and takes the probability of decoding specific token Pr(y 32|z]] [II] Unlike "Sequence Classifier", it explicitly leverages the backbone's 's language understanding ability and treat the prompt and two responses as a whole, which may mitigate prompt-free preference. More importantly. it introduces position bias(preference may vary with response order in the instruction), making the induced value of ro uninterpretable and unsuitable for decomposition.
15

---

Implicit Reward (DPO); the calculation of reward value based on implicit reward model (e.g.
DPO) relies the calculation of log #re/(gl)) which can be very unstable when inputting non-corresponding prompt-response pairs. Such an issue prevents us from decomposing this unstable value.
*(yla)
D Additional Proofs and Interpretations
D.1 Interpretation of the random variables
We begin by interpreting the random variables defined in 331 in detail. Recall that they have the following definitions:
Ž Ber(F Per( (Yi > Y2(x)) = (er K2),
Z Ber(IP., (Y > Y2|X)) = Ber(o(Ari Yi, X2).
W Ber( (Yr Y2
(18)
Y1, Y2))1).
W- Ber([Pr, (Yi > ( =
Y2),,
By the definition of the Bernoulli random variable, all these random variables can only randomly take the value of 0 or I, while the inner probability determines the probability that the random variable is 1. However, as the inner probability is not fixed value, which is determined by the random prompt and responses, we must determine the prompt and the responses before considering the value of Bernoulli random variables. In conclusion, since consider the preference the randomness of such random variables comes from two aspects: 1. the randomness of the prompt and the responses. 2. the randomness inherently cxisting in the Bernoulli random variables.
the entire dataset,
Knowing where the randomness comes from, it's easy for us interpret the meaning of these random variables. For W. Z and Ž, they are simply random preferences of re. f1 and r22 over the entire dataset, respectively. For these random variables, the probability of the Bernoulli variables being 1 can't be determined until we fix the prompt and the responses. For W, things get different. The inner probability of W depends only on the responses, and retains the same value across different prompts, as we take the expectation over all the prompts. This corresponds to the intuition that W comes from r2. which represents the overall evaluation of the responses and is not affected by specific prompts. Moreover, we note that the conditional probability P(X [Y, Y2) is not the standard posterior probability. This conditional probability doesn't indicate that 3/1 more preferred than but only represents the probability that the corresponding prompt is x when the response pair contains y/1 and 3/2, no matter which one is more preferred. We define this way because W should represent an overall preference between y1 and 1/2, so we should not only consider the prompts where y1 is better, and should also consider the prompts where y/2 is better. Mathematically, this could be further verified since only we define in this way, E (Prr (yw >
yn (r) )] + Pxx Yi= Y2= [Prr, (yr > Mw]l=)]] 1 can be satisfied. We refer to the computation of P(X = x x|Y1 = y1, Y2 = 322 in the end of sub-section 3.2 for a further illustration:
P(xly1. 91. v2) P(x))
P(x) (())
(19)
P(y 12)
Σ. P(w))
D.2 Similar benefits for reward training even when additive decomposition does not hold
In the following example, we demonstrate that when ri and 12 satisfy multiplicative decomposition form, our method can still control Ar2 and thus alleviate the influence of spurious preference.
We'll start with the clarification that, due the inner complexity of M-M--bseed reward model, it's hard to achieve situation when the trained reward model rą exactly satisfies some specific form of decomposition that can be explicitly written out. So in the following example, we examine how our method improves the subsequent training process if ra already satisfies the multiplicative form of decomposition and exhibits spurious prompt-free preference.
The multiplicative decomposition can be written as:
(z. y) rı (x, v) r2 (2(2
16

---

We continue to assume the form of 71 (z, y) and r2 y) (which construct ro(x, y)). Before doing so, we note that due the multiplicative decomposition form, our method can't 't obtain the ground truth value of r (1, y) and r2(2, y)- Instead, our method gives an imprecise decomposition of
Are- Ari Ar2. The form of ground truth ri (z, y) and r2 y) are defined as follows:
v2 yl
c = average response length over dataset
Ti (1 (w) = 1.0 (if yu yd (z),
0.5 (if yw ul2z), 0.1 (if yhe < ya
ri u1) 0.1 (if you > ur|r), 0.5 (if you R w|x). 1.0 (if the - v)(x)
It's clear that 12 simply represents preference induced from response length, which is prompt-free reward. As for 71, although it has a simple form, its preference differs conditioend on different prompts. For some non-corresponding prompts, yı better than yoe conditioned on these prompts, ri will prefer yn more than yeu- Thus, such a prompt-related reward is also reasonable. Moreover, it can be easily verified that such re shows a strong spurious preference (length bias) towards longer responses.
To better simulate practical scenarios, set the distribution of prompt-response pairs to be the same as the SHP dataset, while adding Gaussian noise perturbation to each rn (z, y).
(z, v) < 71 (x. + +N(000.1)
Since ra(2, v) is well-defined for all prompt-response pairs within the dataset distribution, our method to obtain the decomposed âr (2r, ywu !/) and Ar2(2, (2r, yu ()) For a better visualization, characterized all prompt-responss pairs in the same way Fig. 4 To further support our claim, we also conduct the same experiment based on a different r1. The visualizations are presented below:
1.0
if yi
0.8
if
7, )=)
0.5
if ywr + x
Xx)
0.1
x
0.1
if yo < x
0.3
yo
0.1
yf Y: x
0.8
if > y1
<xx,)))
0.5
if yw = yı x
xx)x))==
0.1
if yo
= y |
1.0
if yw
0.3
if yv < y1
bull
bui-tel
If our method still works, then Arz(z, Tee Ut) should be smaller for those prompt-response pairs that satisfy | < 1yrl- In this way, our method will prioritize these samples and thus alleviate spurious preference. In the visualization, it's evident that although not all samples with yyu < [ul have smaller values of Arg(æ, yw: W), the majority of them still lies on the left. Prioritizing samples with smaller Arz(r, yu 31) can still mitigate spurious preference and thus outperforms naive reward training.
This exactly matches the reason gave in the previous response. Since Arz always be considered as a weighted average of Arg, although the decomposition is imprecise, the spurious preference in Are can be reflected by Ars. And if Arą is large, then Ara can be large conditioned on almost all prompts. This will lead to a large value Arz and our method can filter out these samples.
17

---

D.3 Proofs of the theorem
Theorem When the value of 12 depends only on the response, i.e. r2(x.y) r2(2). MI M[(2W)
(W)..
Proof. By the relation between mutual information and entropy, we first decompose the calculation of the mutual information into the calculation of the following entropy terms:
MI(2/IW)
H(2, W) MI(2)WW) = (()) H(W) - H(2, W)
(20)
(21)
To prove that (2(2) ()) , we can compare the two equations above, and with simple algebraic computation we can reduce the problem into proving:
H(W) - H H(W) - H)
(22)
To continue, we first calculate the Shannon entropy of W and W. Because both W and W are Bemoulli random variables, their discrete nature provides the possibilliyy to directly calculate Shannon entropy by enumerating their possible values. Take W for example, its Shannon entropy can be calculated as:
H(W) = -Prr = (org( = 1]) - Pr[W 0] log(Pr[W = 0))
(23)
To calculate H(W), the rest of the problem to calculate Pri = 1] and Pr 0j. Again, we leverage the unique property of the Bernoulli random variable: its value take only from 0 1. This allows us to turn the calculation of probability into the calculation of the expectation of the indicator function (I[{a] 1 if * 0 else 0):
Pr(W =1 = []]]
(24)
-Epx, M. BBe [[]
(25)
=Ep(n.Va) Ep(XY [Ber [[]]]
(26)
=E P(Y = 9/, Y2 12) Σ
P(X Yi = 91, Y2 = = EBer [[]]]
(27)
V1.82
Σ P(Y = 21, Y2 = y/2) Σ
P(X x\Yi = 91, Y2 = 2/2)
91.32
E
Pa. lo V/. v2
(28)
(29)
From the first equation to the second equation, we decompose the full expectation into two separate random prompts and responses (Ep(x,y, the other marginalizing the randomness in the Bernoulli random variable (Ener). Note that in the last equation, the inner expectation will not affected by different prompts outside. We then continue to simplify the last equation based on this. We change the expectation into summation since the
expectations: one marginalizing the randomness
18

---

possible values of all random variables are countable.
Pr(W = 1| Σ
P(Y yı, Y2 =y2) Σ
P(X = 2 y1, Y22 422
V1.92
|Beer [[ (Beg' (X (
(rr ,91 91.
(30)
=
Σ
P(Y y1, 1/2 = =2)
Σ P(X= v y1, 1/2 = 822)
equals
Ber Ber ( |1 91. (31)
Σ 31.212
P(Y = y1, Y2 = 122 | (Aro 11 v1.31
)
(32)
Σ
P(Y = V1 122 =22)
(33)
V1:92
Σ
P(1 = y1, Y2 = y2)
P(X 2 y1. 12 = 1 Arr 91 8/ )))))
91.92
(34)
Note that here we use the following Bernoulli variable's property, which can be easily verified with the definition:
[B =
(35)
Similarly, Pr[W = 0] can be calculated as follows:
Pr(W = 0] = Σ P(1 = y1, Y2 = 32) Σ P(X'= = y1, Y2 = o U1 V2 )))
91-92
(36)
This can be easily verified with the same calculation. With Pr[W- = 1] and Pr[m 0), we can easily calculate its Shannon entropy. To continue, we calculate the probabilities Pr[W = 1] and Pr(W- = 0). Take the calculation of Pr(W as an example:
Pr(W- 1] [[[[W]]
(37)
Y. [[][]]] [[]]
(38)
(Pp(),))))((XIM... (Ber [[[]]]]
(39)
Σ P(Y V1, Y2- y22 Σ P(X x|\1 = 31. Y22 = 32 EBer [[]]]]
(40)
V1-92
Σ
P(Y y1, Y2 y2) Σ
P(X x|Vi = v1. = y2 B Ber (X, Y1, Y2))
V1.92
(41)
Σ V1.82
P(Y = vı, Y2 =2)) Σ P(X = x. x v1 Y2 12)lo 21, (2)))))
(42)
Similarly, Pr[W = 0] can be calculated as:
Pr(W = 0 = Σ P(Y v1, Y2 y2) Σ P(X 2 v1, y2) (2)(11o((ra(e)) (1.12)))]))
v2)))
V1-U2
(43)
Combining the results of calculating Pr[ 1J, Pr(W = Or[ 11, 0j. we can casily verify that H(W) H(W). It's worth noting that H(W) H(W) only means their separate quantity of information is the same, but doesn't 't characterize their relationship from the information perspective.
The rest of the problem is to prove the equality of H(ž, w) H(ž, W). Once again, since the events considering (Z, W) (Z, W) pair are both finite, decompose the calculation of the
19

---

joint entropy as the summation of four terms. Take the calculation of H(Z, W) as an example:
H(2, W) Pr|2 = 1, W = log - 1, W -1]) Pr[2 = W - 0] (rr( = W - -))
Priž = 0, W = log = 0, W = 1)) - Pr|Z = 0, W = 0] log Pr|z = 0, W = 0]).
(44)
To prove that H(2 W) = H(22 W), we prove that the four probabilities induced from (2, W) equal to that induced from (2, W)) respectively. For simplicity, we only prove that Pr|Z = 1, W = 1] = Pr[Z= 1, = 1]. while the remaining three equations can be proved in the same way.
The key property we used prove such equations is that although there exists complicated relationship between Z, W, and W, the randomness from different inherent Bernoulli distributions is independent. In other words, once the prompt, responses are determined for all these random variables, the inner probability is fixed value that is independent from each other. This motivates us to use the same proving technique above, decomposing the full expectation into separate expectations about the prompt-response pair and the Bernoulli distribution, respectively. Specifically, the following holds:
Pr[Z = 1, W = 1| = - 2- W]
(45)
Ep(x, 12) Beex Ber 2 W]]
(46)
Ep(r, BP) [B Ber Ber [1[2 WI]l
(47)
Σ P(Y y1, Y22 122) Σ P(X x| y1 Y2 [2)
[[ W]]]
91.82
(48)
(49)
Note that here Eggrx Ber represents the expectation over two independent Bernoulli random variables. The independence comes from different Bernoulli random variables with fixed probability that take the value of 1. We continue to expand E [e W]] with the independence with the following equation:
EBee Ber [1[2- W]] EBcr [[2]]
(50)
Replacing this in the previous equation, we have:
Pr[ž = 1, w = 1] Σ P(Yi = yr, Y2 = 12) Σ P(X = z|2i = 31. Y. = (E [I] 21 Bn [[[]]
81.82
(51)
Σ -1, 12)
P(X- y1, Y2 U2)
V1:82
[0 (c(Ar2()1.82))
E (a (Aro(a', 91, 02))
(52)
indepeencent of
Σ
P(Y = v1, Y2= = =12)
(E P(X P(x=
x = Y2 va== (Ar2)))))) (2).))))))
V1.92
equals to 1
Σ P(X'
P(X'= - =2 = v1 = v2) ((rr((e' b0...,.,,800 wwll
(53)
Σ P(Y v1, Y2 v2) o(Ar2(U) V1.92
EP(X''
P(X' x' ( = y,, Y2 = y2) a (Aro(a', yr, 82)))
(54)
Note that to get the last equation, we first use Eq. 35 exchange the sequence from first taking the product to first taking the summation over x. Such
to simplify the inner expectations, and then
20

---

exchange holds since one part of the product is independent of z. Moreover, we replace Ar V1, V2) with (8) y2) due to the assumption of the theorem, which is vital in the proof.
We continue calculate Pr[Z However, the exchange we used above still holds since the specific form (22 y2) does not depend on x. We have:
W 1J, the difference is that W is not independent of
Pr[ = 1
Σ P(Y = y1, Y/2 = -22) Σ P(X = z|vi = v1 Y2 = 32 EBer
21 E [[]]]
91.92
(55)
Σ = y1, Σ x -1. = ( yı 82))
V1.82
(56)
Σ M1.82
P(Y = v1, 91, Y22
y2) ((Ar2(11882) (Σ P(X 2/Yi = yr, Y22 = y2) ) o ( y1, v2))
(57)
It's easy verify that Pr[ = 1, Pr( 1, can prove the equalities for the rest three probabilities. Combined them together, we prove that H( W) = H(Z, W) and finish all the proofs.
W 1]. With similar derivation,
Theorem 2. For any bounded To and dataset (X, Y1, Y2), there exist feasible ri, v such that ((n,,y)) y2) P(Y, Y2), P(XM Ari [ (x. 31:32)) 2 Such ri, 12 the optimal solution to problem
Proof. We begin by proving that there exists ri and rz that satisfy Vin, ya) P(Y,, Y2), Pe- Y==w2) (Ari (I, M1, (1, [2))] 5- We then verify that such ri and r are feasible, by verifying that such ri and r satisfy the constraints and can recover ro (i.e. Are(z, y1, y2) = (ri V/, y2) + Ar3(V,, 82)).
To prove the existence of such rf and rá. we first replace Ari (z, y1. v2) with Aro(z, (1.
( 12). In the following proof, we have to use the following lemma:
Lemma 1. P. the increase of Art (1)
v2) Ari(v1, monotonically decrease with
Proof. We directly take the derivative of ( Since (rī V2)) retains the same across different we have:
Ar2(21, 82))
0 P (a (Ara U1, v2) Ari(u 32))
(58)
dArj 3 (3/1 322)
Ees Yo=wa 0o Aro U1 U2)
(59)
dA (V1, U2)
0 1+expl Art 12) A20 P 0(Ar (1/1, U2) - Ar (x, V2))
2 O(Ar H2) - Are(z, V1, 82))
Ar2(21
(60)
(mp(Ar Aro U1, U22) - (1 exp( (r)(1)
(61)
(V) 1/2 Aro(a, U1 (1.02))) U2))))
This shows that for any 81. 1/2, PX (o (Arg U1 U2) Art 82))] monotoni-
cally decrease with the increase of Arg (31.
To continue the proof of Theorem 2, we identify the lower and the upper bounds of Bxw. lo vı, v2) - Ari v2))], under different choices of ( v2).
21

---

Due to the assumption of bounded re, we consider ng that satisfies Vz, V1. U2, (1,
['mnn maaz] The following holds:
P (Aro yr, y2) (Tmin ((mma))) ))]
(62)
(re( -y
re re (o(x,,92) y2))
(Tmin (maa))
(63)
P (ro y1) Tmin)
(Tmar (2, always 2
(64)
always 2 0
2 P(x
(65)
On the other hand, the following incquality also characterizes the lower bound of
E. (orr Arž (v1, Ø2)]]:
)= (Aro y1 822 (Tmaz Tmiia)l
(66)
P (ro(s,,) v1) ro ra ((r,,)2)) (Tmaz Tmmmn))
(67)
= -E lo (re(z,; y1) - -maa)) + (Tmin - ro rx (g,z8))) always S
(68)
always 500
S <(XMM((X - -2- =
(69)
Combine with the fact that (( Ys Ys= Yo= (A - Ar), monoton- ically decrease with the increase of Arž 12), there always exists (rt v2) such that Eexx((XIY, =/. (Arr)) y1: 122 (ri y2))] = 3 To satisfy the feasibility con- dition that Arg( y1, 92) = Ari (I, y1 92) Ar2 12). we directly set Ari (z, y1, y2)
Aro(z. 92) Arž (2)), with the (ri V2) obtained before. Such Ar5 (2), still sat- isfies the simple structure mentioned in Theorem 1, so the constraint MI( W) MI W) still satisfied. The remaining task to proof that the Ari (z, y1 y22 obtained before satisfy the constraint M || W) = 0.
With the same proving technique in the proof of Theorem we first decompose MI(2 || W) into three entropy terms:
MI(Z || W) H(2) + H(W) - H(Z, W) We further expand the right side of the equation in the following equations:
(70)
H(Z) - Pril / = Prr(P - = 1]) Pr(Z = 0] log Pr|Z = 0))
H(W²) Pr(w 1) Pr[W = 1]) Pr(W log logg =0)
H(Z, W) = -Priz = L,
(or = 1, W = =1] - Priz = 1, W = 0] log( 1, W = =0)
-PriZ = 0, W = 1] (P = 0, w = 1]) Priz = 0, W = 0] log(PrlZ = W = 0)
(71)
With simple algebraic simplification, have:
H(Z) H(W)- W))
= PP[[ = 1, W = (]log(Prr)) Priz 2- = 1, W = 11) + Pr|Z = V = oo
(allo((Pri) Pr(z = 1, W = 0))
Pr(l 0, W 11llg((Prr))) Pr(Z 0, = 1]) + Pr|Z = 0, W lo] Pr|Z = 0, VV 0))
-Pr[Z lo
1)) Pr log(Pr[Z 0) Pr(W log ( 1)
- -Pr(W = 0) log(I 0)
(72)
To prove that MI(Z () H(Z) H(W) H(Z, W) 0. we first expand the inner probability
with the same technique use in proving Theorem 1, and then perform the following transformation for EBr [[]] and Ene [[]]:
Eper [[]]] Eper [[]]] [Bue[[[[W) + - EBer [[])
(73)
Eee [W]] E [[]] (Eer [[]] + (1 Ene [T[ ((]]))
(74)
22

---

Note that this simple transformation bridges all terms. Take the calculation of Pr|Z = 1,
1] log( Pr[Z 1, W 1]) Pr[Z = P = 1]) Pr[W 1] log(W = 1) for an example,
we expand the inner probability with the same technique use in proving Theorem I:
Pr(Z = W -1 lorl 1, Pr 2r2 1])
Pr(W log 1)
(75)
))(Y,))) EP( Vz) [Berx Ber [1|
(B [ol -W]I))
) Ep(xin Ber [ log(B (P(( Ber [T]]]]
- Btgpa//ww.aaaaallooooooaaaalaaaa
(76)
After using the previously mentioned transformation, we have the hight-hand-side above equals to:
Ber [1| WIll
Ber [1[Z WIll)
E, px.. (5)
Bper [11]]] Be [[[]]]
log
E E B Pp(x) EBar [[]]]]
Bp EBer [[]]]
- -2
Ener [[]] (1 Ene [[[]]) (3 |ner
used for symmetric construction for canceling other term
EBe [[2]]) En [T] [W]] log
used for symmetric consiruction for canceling other term
(77)
With simple verification, can find that the last two terms can be used cancel other terms that come from H W). This means we only have prove that the first terms in the above equation cancel out. In other words, all we have to prove is:
Bexx Ber M1 2
W]]
(Ber [[Z WI])
EEro,.,) 12)
EBer [1[]]] BB [[[]]]]]]]]]]]]]]
log
LBr - Ee [L[]]]
= E
Eper [[]]] ERer [W]]
Ep B. BP(xxx, Ener [[]]] Ener [[]]]
log
Bp BBer [[]]] Ep(Y.. (P) E [[]]
0
(78)
Note that we change all Bey,,,)))E(((..... [1| W]ll
into
E, EBar [[]]] [[]]]]]]]] due independency similar before. We can finally use the property that for any 31. M/. Bex. (arrool Vn, 22) Arī (V1 v2)) 1 and remember that BBc[[[[WW is independent of specific x, then we perform the following
23

---

simplification:
E B
Eee [[]]]] EBer [[r[]
Y P((X)M Beer [T[2]] EB [W]
log
E P ( Br(xp
EBer [[]]]] B. [Ber [[]]
BBer [[]] BB [[]]]
E (2)
E X, [[1] lB [[[]]]
log
E BP(xI Br(y.
Ber [1[
E8 (r [e [[[]]]
Ep(r vs)
Ber 11 [W]]
EBrr [T [Z] Ber [W]]
log
LE (((),
(B)
[Ber [[]]] BB [L[W]]] log(
+
(79)
2
After verifying the feasibility of the solution, verify that such rj and r is optimal for maximizing
HZZ). By definition, we have:
H(Z) -Prlz 1] log( 1]) Priz = 0] (Prl = 0j)
(80)
To calculate Pr = 1], we can use the same expansion technique before. We have:
Pr[Z 1| Bp B
Ber BBer [[]]]]
Σ
P(Y 31, 31, Y2 3/2
Σ
P(X = x = v1, Y22 = (B
V1.82
Σ P(Y v1, 12)
P(X x/Yi vn Y2 Σ
un
Σ
P(Y 91, Y2
(81)
It's easy to verify that such and r reach the maximum value of H(Z), which is the optimal solution we want.
E
More about experiments
E.1
Full results in Table and Table 2
Due to space limits, we only reported mean scores from random seeds in the main context. Full results (mean and std) in Table ToleeTandd and Table are listed below, showing consistent performance among different seeds.
Model
Chat
Chat Hard
Safety
Reason
Avg
vanilla
78.0
29.810.7
36.410.9
18.2+1.11
50.6
8..8
31.140.8
45.1
6..3+1.2
15.8+
Table 3: Full results of experiments in Table I (a). We conduct the same experiments with 3 different random seeds and report the mean and std values of the results.
Model
Chat
Chat Hard
Safety
Reason
Avg
vanilla
80.7+2.1
28.5:0.8
4..2=1100
36.5
46.4=:1.3
original
1.7
31.3+106
42.8+111
49.0
52.0=10.9
84.3+11.8
30.9
43.2
46.7+
51.6++0.8
Table 4: Full results of experiments in Table We conduct the same experiments with 3 different random seeds and report the mean and std values of the results.

---

Method
Chat
Chat Hard
Safety
Reasoning
Average
vanilla-8B
0.93+10.01
0.50-10.02
0.67:10.02
0.78+10.03
0.72-1:.02
RRM-8B
0.95=00.11
0.56:10.11
0.75:10.02
0.82:0002
0.77=10.01
Ours-8B
0.96
0.5910.01
0.81100011
0.89
0.82
vanilla-7B
0.90=10.02
0.49110.03
0.6010.01
0.6910.05
0.66
RRM-7B
0.94
0.52++0.01
0.65++0.01
0.70++0.03 0.69+-0.02
0.70++0.01
Ours-7B
0.94+00.0
0.56
0.81+-0.44
0.75+:0.02
Table 5: Full results of experiments in Table (a). We conduct the same experiments with 3 different random seeds and report the mean and std values of the results.
Method
vanilla
RRM
Ours
DPO
10.5+1.6//5 38.8
39.5= 005500../40.90000 40,
40.6 42.3
BoN(N=4)
33.2+t2.1 1..5
35.3=14/384+15
1.0
45.01/ 45.0
BoN(N=32)
38 45.0
41.1 94.11.944.9912..
14.7=11.7
48.
Table 6: Full AlpacaEvaa-2 (LCWR/ WR) results. We conduct the same experiments with 3 different random seeds and report the mean and std values of the results.
Method DPO 0.8 10.3000 8.35 / 7.424
vanilla
RRM
Ours 8.03 00.26
Table 7: Full MT-Bench (TT (2) results. We conduct the same experiments with 3 different random seeds and report the mean and std values of the results.
E.2 Reward-Bench results based on SHP dataset
Method
Chat
Chat Hard
Safety
Reasoning
Average
vanilla-8B
0.91110.02
0.39
0.46:10.01
0.75
0.63-10.01
RRM-8B
0.93
0.45:10.01
0.54
0.77
0.67-10.02
Ours-83
0.95=10.01
0.46:10.01
0.61
0.88=0002
0.72=10.01
vanilla-7B
0.84
0.331:0.03
0.45:10.05
0.59::0.03
0.55
RRM-7B
0.89
0.37
0.001
0.58:10.04
0.58
Ours-7B 0.90
0.39+
0.52
0.703
0.64
Table 8: Results Reward
Beward-Bench: Bench: These experiments based on the same reward backbones, but with
SHP dataset. The improvement brought by our method also significant on SHP dataset. However. since the SHP dataset contains many fewer data samples and may inherently have more spurious preferences, the accuracy generally decreases.
E.3 Justification and analysis of binary clustering and EMA threshold used in data prioritization
An adaptive threshold required since reward learning is dynamic-Fightn 4 show that even small models (IB) experience significant Ar2 scale changes during training.
During optimization, the reward model is updated batches, with Ar2 calculated from limited samples each step. Determining whether a sample's Ar2 is "relatively small" in batch becomes ID blnary-clustering problem (Ar as feature). Due to high variance in Arz values among different samples (as shown in Figure 4). relying solely current batch data is unstable and cannot truly reflect the properties of Ar2. Our dynamic threshold, computed via exponential moving average, incorporates recent training steps, dynamically capturing Ar, properties while reducing threshold variance.
As for the robustness, we conduct the same experiments on Reward-Bench in Section 4.2 with different EMA weights (a Alg. (2) and clustering methods (Otsu, K-means). Results show robustness to thresholding methods. Limited by time and budget, we only use L.LMAA3-8B-Inssnn and SHP dataset. We believe the robustness partially stems from the fact that samples with larger Arz in the current batch will not be directly discarded, preserving the chance to be used in the future step.
25

---

Method
Chat
Chat Hard
Safety
Reasoning
Average
Otsu
0.9
0.95
0.48
0.59
0.87
0.72
Otsu
0.8
0.92
0.49
0.57
0.84
0.70
K- Kemanns
0.9
0.95
0,46
0,61
0.88
0.72
K- meaass
0.8
0.93
0.48
0.60
0.87
0.71
Table Performance under different methods and values.
E,4
experimental details
Our implementation is based on the OpenRLHF [6] framework, which uses Apache-2.0 license. Experiments
Nvidia A100(40G) GPUs. For reward training, use 8 8*A100(40G) GPUs so that the training can be finished in 12 hours. For DPO training and response generation, we also use 8* A (00(44G) GPUs, and the time consumption is similar to the reward training. All the reward models and the LLMs are fully fine-tuned. We use Deepspeed 2 as the framework of parallelization. For the implementation of RRM, we reference its official code in RLHFlow We use an Adam W optimizer with a learning rating of 5e-7 and 9c-6 for the DPO policy and the reward model, respectively. For other general hyperparameters, we follow the default parameters in OpenRLHF.
Our exclusive hyperparameters include clustering methods and EMA weights for Ary thresholding. In practice, tried Otsu's method and K-means for clustering method and 0.8, 0.9 for EMA weight. As is shown in Appendix E.3 our method performs consistently, so we select the best best-performing hyperparameters.
We continue to explain the data construction process the toy cases. For the length-biased dataset, we use all preference pairs (x, Mw: 3/1 where the chosen response is longer, i.e., |/wl >> |ua|. from the original dataset; based on the number chosen- -lngeer samples, we select other samples with number of them, whose chosen response is shorter. The combined subsets create clearly length- biased dataset. For the adversarial prompt dataset, we use the following diagram to illustrate the data processing:
orighsl
adversarfal
data
adversaral
Oriine Chocee
Dhiginal #ouuted
Orignel Dhooch
Dripisal Popitee
sikees)
Prghe/Proegg. "hose provice
Orgghe "Puuc arowile long possile""
Orignn Pronpe
Original Proepe
passible"
1200
Kelperig 1000 tolend)
Prpporas
Responie (olloo))
We note that RLHFLOW uses Apache 2.0 license. SHP dataset didn't claim the license uses. LLaMA models use Meta Llama 3 Community License Agreement. Mistral models use Apache-2.0 license.
E.5 Explanations of the superiority compared with baseline methods
The advantage our method compared with vanilla training is obvious. Since vanilla training will use all data samples, including those samples with strong prompt-free preference. Thus, the resulting reward model easily overfits to some prompt-free preference existing in the dataset. Such prompt-free preference harms the reward model's generalization capability and leads to low -ww-qullty alignment.
Compared with RRM, although both RRM and our method mitigate spurious preferences by altering the data distribution, RRM introduces band-crafted but mismatched prompt-response pairs into the dataset and updates the reward model on them, which turn affects the meaningful preferences. In contrast, our method leverages them solely to characterize the reward model during training and only trains on the original preference data, where each prompt-response pair is labeled by human preference. This contributes to the superior performance of our method compared to RRM.
"hapss//itthib.oo//RL/Hoooo
26

---

Boarder Impacts
This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences of our work. For positive ones, our method could result in a more generalizable reward model and thus advance other RLHF methods. This can make the LLMs more aligned with human preferences. For negative ones, one can achieve jailbreaking in LLMs through malicious reward models.
27

---

NeurIPS Paper Checklist
The checklist is designed encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No]. or [NA].
[NA] means either that the question Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1-2 sentence) justification right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will published with the paper.
The reviewers of your paper will be asked use the checklist as one of the factors their evaluation. While "[Yes] is generally preferable to "[No]" ", it is perfectly acceptable to answer "[No] providedd proper justification is given (e.g.. "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering binary way, we
"[No] or "[NA]" is not grounds for rejection. While the questions are phrased acknowledge that the true answer is often more nuanced, please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS Paper Checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only use the provided macros for your answers.
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Our main claims can be reflected by Section Jndd Section
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize other settings.
is fine include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Limitations are discussed in Section 6
28

---

Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, c.g., if the approach was only tested few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't ' acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification: Seed Appendix D.3
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: Experimental details are given in Appendix E.4
Guidelines:
The answer NA means that the paper does not include experiments.
29

---

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. w
For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [No]
Justification:
Code will be released in the future.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (nttps://nips.co
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/ /ttppp.cc/pbllc/guideeee http://oeeuuiiissooooooooooo
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
30

---

Details are Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer: [Yes]
Justification: Experimental settings are given in Section 4 and Appendix E.4 given in Appendix E.4
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: We use sufficient random seeds and provide standard deviation of the scores in Appendix E.I
Guidelines:
The answer NA means that the paper does include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of exccution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Details of computation resources given in Appendix E.4
Guidelines:
The answer NA means that the paper does not include experiments.
31

---

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification: This paper doesn't t involve human participants, and the datasets in Section are common open-sourced datasets.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (c.g.. there a special consid- eration due to laws regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Potential positive impacts and negative impacts are discussed in Appendix F
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied partieular applications, let alone deployments. However, if there is direct path any negative applications, the authors should point it out, For example, it is legitimate point that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (c.g., pretrained language models, image generators, scraped datasets)?
Question:
Answer: [NA]
32

---

Justification:
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging. and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: All assets we use are properly credited, and corresponding license are given in Appendix E.4
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi twwdeethcdd...m catee.comdall com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re- re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/mooel as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
33

---

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Question:
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy for what should or should be described.
wetpuurrpssscoCoofeeeeeeeeeee /
34


Reviews:
[{"review_id": "V85HnwLFWI", "reviewer": "Reviewer_sjU2", "summary": "The paper proposes reward function decomposition to handle prompt free and prompt based rewards separately and independently to increase generalization of reward model and alignment. These 2 independent components are extracted using information theoretic measures. Effectiveness of the decomposed reward model is shown through toy example and evaluations on standard open source datasets.", "questions": "1. Can the authors please clarify the highlighted issue in Fig 1 (right) in context of the specific example? If I understand correctly, all the 3 prompts are alike and have the same preferred response;  but still given they are not explicitly seen during training leads to lower reward gap values? \n2. In Eqn 4, is r_1(x,y) the prompt dependent reward? I believe r_1(x,y) hasn’t been defined anywhere.\n3. Why does the random variables in Eqn 7 is modelled using Bernoulli distribution? It isn’t clear to me what these variables mean (I did take a look at the appendix).\n4. In Eqn 5; does y_1 refers to preferred response and y_2 refer to non-prefered response or do both of these refer to different preferred response for the same prompt?", "limitations": "Yes", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strength:\n\n1. The problem is interesting and novel to the best of my knowledge; generalization of reward function in LLMs is an important research direction given the current AI landscape.\n2. Experiments on Open source Datasets shows superior performance of the proposed work as compared to other baselines.\n3. Experiments on manually crafted dataset including adversarial prompt Dataset in interesting and shows the utility of prompt free reward values.\n\nWeakness:\n\n1. While I think Fig 1 is useful in understanding the motivation , the right hand side explanation is confusing and not clear to me.\n2. The interpretations of the random variables (equation 7) are not clear to me. How can you get the preference label over the prompt free reward component when the overall preference is over the responses to a prompt inducing an unknown reward function r_\\theta. (which is eventually learnt in Preference based learning).\n3. I think the intuition of the proposed idea is not clearly conveyed; it would have been nice to explain the intuition before modelling the 4 random variables introduced in Eqn 7.\n4. The idea of just training the samples with small prompt-free reward gap might led to overfitting. What happens to the samples with small prompt-dependent reward gap? Whether they are trained or not is not clear.\n5. Fig 4(a) and 4(b) is not clear to me from the visualization plots; I am not sure how to interpret them.\n6. Overall, though I like the high-level idea of training a generalizable reward function, I could not understand a lot of technical details from the paper.", "was_revised": true, "final_justification": "I have read the rebuttal and understood the technical concerns that I had. The responses were detailed which helped in clearing other aspects of the word.\n\nI have likewise increased my score.", "rebuttal": "Thank you very much for your time and effort in reviewing our paper. We present the responses as follows.\n\n> While I think Fig 1 is useful in understanding the motivation , the right hand side explanation is confusing and not clear to me.\n\n>Can the authors please clarify the highlighted issue in Fig 1 (right) in context of the specific example? If I understand correctly, all the 3 prompts are alike and have the same preferred response; but still given they are not explicitly seen during training leads to lower reward gap values?\n\nIn the example provided in Fig. 1 (right), the 3 prompts **are different**, and their corresponding response pairs **are also different**. Moreover, only in-distribution prompt-response pairs (connected with solid lines) are used to train the reward model, while the user might query the preference on out-of-distribution prompt-response pairs (connected with dashed lines) in the subsequent alignment stage. \n\nThe chosen response in the blue box lists the main applications of LLMs **in bullet points**, while the rejected response in the blue box describes them **in a single paragraph**. It's clear that the former response is more prefered given the prompt in the **blue box**, and the latter is more prefered given the prompt in the **yellow box**. As the prompt in the green box requires for the procedure of RLHF, the two responses in the blue box can be seen as equally bad.\n\nWhen the reward gap overly depends on the responses, changing the prompt will only lead to minor difference (for the response pair in the blue box, $\\Delta r=3.2$ given the blue prompt, $\\Delta r=2.5$ given the yellow prompt, $\\Delta r=2.7$ given the green prompt). This is conflict with the true preference mentioned before. More importantly, such mistakes are not caused by unseen prompts. The reward model is well-behaved on **the yellow and green prompt-response pairs (drawn on the left, responses are different)**, but loses the ability to generalize the information in the prompt to non-corresponding prompt-response pairs. The catastrophic result is due to overdependence on the responses.\n\n> The interpretations of the random variables (equation 7) are not clear to me. How can you get the preference label over the prompt free reward component when the overall preference is over the responses to a prompt inducing an unknown reward function r_\\theta. (which is eventually learnt in Preference based learning).\n\nWe **can't** get the preference labels for the prompt-free reward; otherwise, there would be no need to extract it using the information-theoretic approach based on Theorem 1 and Theorem 2. We could simply fit a Bradley-Terry reward model using its preference labels.\n\nIn fact, our core contribution is that, after defining prompt-free preference in a way that aligns with our intuition (in Eq. (7)), we find a tractable and lightweight method to extract the prompt-free reward. Such a method only requires the value of $\\Delta r_\\theta$.\n\n> I think the intuition of the proposed idea is not clearly conveyed; it would have been nice to explain the intuition before modelling the 4 random variables introduced in Eqn 7.\n\nFor the Bradley-Terry model, given a prompt-response pair $(x, y_1, y_2)$, it will prefer $y_1$ with the probability of $p=\\sigma(r(x, y_1)-r(x, y_2))$, and prefer $y_2$ with the probability of $\\sigma(r(x, y_2)-r(x, y_1))$ (which is $1-p$). Such a preference label exactly matches the definition of Bernoulli random variable, and is all we can get from the preference data.\n\nClearly, our goal is to decompose $r_\\theta$ into $r_1$ and $r_2$. To achieve this, we need to identify the connection between their preferences. Mathematically, this means finding the relation between the Bernoulli random variables inside the preference labels.\n\nIntuitively, prompt-free preference is independent of any specific prompt and only tells which response is prefered in general. For example, \"the longer response is better\" is a kind of prompt-free preference. This is because, for any given prompt, the longer response is preferred and therefore considered generally better. Therefore, we incorporate the conditional expectation (based on $P(X|Y_1, Y_2)$) in the definition of $\\tilde{W}$ in Eq. (7), in order to characterize the general probability that $y_1$ is prefered. The definition of other three Bernoulli random variables are natural.\n\nOne can expect that these four random variables, each representing different kinds of preference, are deeply connected from an information-theoretic perspective. We provide intuitive illustration of these connections in Fig. (2), and formalize this intuition using the constrained optimization objective in Eq. (9). After that, we propose a lightweight method to solve this objective and achieve our goal.\n\n> The idea of just training the samples with small prompt-free reward gap might led to overfitting. What happens to the samples with small prompt-dependent reward gap? Whether they are trained or not is not clear.\n\nSuch overfitting is unlikely to happen. The reason is that a sample is not updated more than once in a single iteration. At each step, we continuously sample data and retain those with $\\Delta r_2$ below the threshold until the number of retained samples reaches the batch size. We then use these retained samples for one-step update. Only the unretained samples are reinserted into the dataset for future sampling. This process actually only reorders the sampled data so that each step prioritizes updates with data having smaller $\\Delta r_2$, as we consider them more valuable.\n\nFor samples with small prompt-related reward gaps, if their prompt-free reward gaps are also small, they will be used for update. If their prompt-free reward gaps are large, they will not be used for update (in this step) since $r_\\theta$ may have overfit to supirious preference on these samples.\n\n> Fig 4(a) and 4(b) is not clear to me from the visualization plots; I am not sure how to interpret them.\n\nTake Fig 4(a) as an example. The four subfigures represent four equally spaced training steps. All the points plotted in each subfigure represent randomly sampled prompt-response pairs, characterized with their corresponding $\\Delta r_1$ and $\\Delta r_2$ values. The data points that satisfy $|y_w| > |y_l|$ are colored in **red** and the ones that satisfy $|y_w| ≤ |y_l|$ in **blue**.\n\nIt's clear that for naive training procedure, when we consider the prompt-free preference of $r_\\theta$ on the data samples (you can consider as projecting points to the $\\Delta r_2$ axis), longer responses are prefered. Specifically, chosen-longer samples have significantly larger $\\Delta r_2$ and chosen-shorter samples have small $\\Delta r_2$, which demonstrate a clear preference towards longer responses. This is not surprising since the training dataset is inherently biased towards longer responses. On the other hand, the reward model trained with our method are only slightly biased towards the longer responses, making it focusing more on the prompt-related preference (which is more valuable). Similar interpretations of Fig. 4(b) are listed in section 4.1.2 .\n\n> In Eqn 4, is r_1(x,y) the prompt dependent reward? I believe r_1(x,y) hasn’t been defined anywhere.\n\nSorry for the unclear writing. In line 93, we define the prompt-related reward with $r_1$.\n\n> Why does the random variables in Eqn 7 is modelled using Bernoulli distribution? It isn’t clear to me what these variables mean (I did take a look at the appendix).\n\nThe reason we use Bernoulli random variable to model the random preference label is given in the third response. When the prompt and the two responses are fixed, the Bradley-Terry model will randomly prefer one response with probability $p$, and prefer the other with probability $1-p$. However, considering the entire dataset distribution, the probability $p$ is determined by the specified prompt and the respones (which are also inherently random). So the random preference label of a Bradley-Terry reward model can be formalized with a Bernoulli random variable, whose randomness comes in two ways: randomness of the prompt and the respones, and the randomness of Bernoulli distribution.\n\n> In Eqn 5; does y_1 refers to preferred response and y_2 refer to non-prefered response or do both of these refer to different preferred response for the same prompt?\n\nIn the subsequent theoretical analysis, we need to use the preferences under non-corresponding prompt-response pairs. Therefore, we do not simply use $y_+$ and $y_-$, as the preference between responses may reverse under different prompts. Here, we generally define how to express the preference between two responses for a given prompt. In practice, $y_1$ and $y_2$ are the response pair in a data sample, and which one is preferred depends on the specific prompt.\n\nIf our responses have addressed your concerns, we would appreciate your consideration for a higher score. Thank you again for your time and effort in reviewing our paper."}, {"review_id": "Aeqwi5Lcn0", "reviewer": "Reviewer_S9sS", "summary": "The paper propose to decompose the reward value into two independent components: prompt-free reward and prompt-related reward, so to improve the generaibility of reward model and robustness to shortcuts such as length bias. Throught curated dataset, they explicitly show the efficay of methods. They also demonstrate the proposed methods achieives better accuracy on reward benchmark and inference computing.", "questions": "Please refer to the 'Strengths And Weaknesses' part.", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "# Strengths\n- The paper is well-written, with a rigorous and straightforward derivation of its methods.\n- The use of a prompt-free reward for priority sampling is a well-motivated and sensible approach.\n- The authors explicitly demonstrate the algorithm's effects on a curated dataset, which is persuasive and aligns with the algorithm's design.\n- Experimental results on reward benchmarks and BON infer are also provided. \n\n# Weakness\n- The calculation of the prompt-free reward relies on a binary search, which involves an expectation over E(x|y_1,y_2). Given that the reward is typically continuous, the time complexity of this approach is a concern. More details on the algorithm and its time complexity would be beneficial. Moreover, the number of samples required to accurately calculate the expectation E(x|y_1,y_2) is not specified.\n\n- The paper is missing a comparison with some relevant baselines. For instance, the Generalizable Reward Model (GRM) is another line of research that aims to improve the generalizability of reward models by leveraging the generative power of text.  Including a comparison to GRM would provide a more comprehensive evaluation of the proposed method.", "was_revised": true, "final_justification": "The authors have addressed my main concern by providing additional comparison to the GRM in their rebuttal, but I tend to maintain my current score. This is solid work on removing bias in reward models, but the method's complexity may limit its practical application, especially in today's long-cot directions.", "rebuttal": "Thank you very much for your time and effort in reviewing our paper. We present the responses as follows.\n\n> The calculation of the prompt-free reward relies on a binary search, which involves an expectation over E(x|y_1,y_2). Given that the reward is typically continuous, the time complexity of this approach is a concern. More details on the algorithm and its time complexity would be beneficial. Moreover, the number of samples required to accurately calculate the expectation E(x|y_1,y_2) is not specified.\n\nIn fact, estimating expectations over prompts with importance sampling does not incur significant computational overhead. The reasons are as follows.\n\n1. The importance weights are independent of the trained reward model and are an inherent property of the dataset distribution. Therefore, for each dataset, we only need to pre-sample prompts and compute probabilities once. This process does not need to be coupled with reward training, allowing experiments with different random seeds, hyperparameters, or LLM backbones to share the same importance weights.\n2. The calculation of importance weights is also time-efficient. Specifically, it only requires constant times of forward pass of the LLM, and is **not** autoregressive. This is significantly faster than reward training because there is no need to wait or communicate for parameter updates.\n\nSince the importance weights are computed only once for each dataset and the overhead is not significant compared to training, we did not compare the precise time consumption. In our experiments, calculating the importance weight for a dataset is faster than conducting a full naive BT reward model training on the same dataset. Considering different random seeds, hyperparameters, and LLM backbones, the average time consumption is minimal.\n\nIn practice, we first randomly sample 16 prompts for each responses pairs and calculate their importance weights. However, we found that estimating the expectation with 8 prompts can achieve the same performance. We suggest that the number of prompts does not need to be large, as long as they effectively reflect prompt-free preferences. This will be added to the experimental details in the updated verison.\n\n> The paper is missing a comparison with some relevant baselines. For instance, the Generalizable Reward Model (GRM) is another line of research that aims to improve the generalizability of reward models by leveraging the generative power of text. Including a comparison to GRM would provide a more comprehensive evaluation of the proposed method.\n\nWe choose the experiment in section 4.1.1 and the experiment on the Reward-Bench in section 4.2 to compare our method with the Generalizable Reward Model (GRM). To ensure the fairness of comparison, we only compare with GRM that has a non-linear reward head and is fully finetuned. We follow the hyperparameter setting in its original paper. The results are as follows:\n\nResults on Reward-Bench with **length-biased dataset in section 4.1.1**:\n|Method|Chat|Chat Hard|Safety|Reasoning|Average|\n|-|-|-|-|-|-|\n|vanilla|78.0|29.8|36.4|58.2|50.6|\n|GRM w/ dpo noref|80.1|29.7|37.2|58.5|51.3|\n|GRM w/ sft|80.3|29.2|37.6|59.3|51.6|\n|Ours|**86.8**|**31.1**|**45.1**|**60.3**|**55.8**|\n\nResults on Reward-Bench with **RLHFlow pair-preference dataset(mixed dataset,  300K samples)**:\n|Method|Chat|Chat Hard|Safety|Reasoning|Average|\n|-|-|-|-|-|-|\n|vanilla-8B|0.93±0.01|0.50±0.02|0.67±0.02|0.78±0.03|0.72±0.02|\n|GRM w/ dpo noref-8B|0.96±0.02|0.57±0.01|0.73±0.01|0.85±0.03|0.78±0.01|\n|GRM w/ sft-8B|0.95±0.01|0.55±0.01|0.76±0.03|0.82±0.01|0.77±0.01|\n|Ours-8B|**0.96±0.02**|**0.59±0.01**|**0.80±0.01**|**0.89±0.02**|**0.81±0.01**|\n|vanilla-7B|0.90±0.02|0.49±0.03|0.60±0.01|0.69±0.05|0.66±0.03|\n|GRM w/ dpo noref-7B|0.94±0.01|0.54±0.02|0.67±0.02|0.74±0.04|0.72±0.02|\n|GRM w/ sft-7B|0.93±0.02|0.55±0.01|0.65±0.01|0.75±0.02|0.72±0.01|\n|Ours-7B|**0.94±0.01**|**0.56±0.02**|**0.69±0.02**|**0.81±0.04**|**0.75±0.02**|\n\nIt's clear from the results that our method still outperforms GRM. Interestingly, GRM can achieve comparable performance in the experiments with RLHFlow pair-preference dataset(mixed dataset,  300K samples), but only has minor improvements with length-biased dataset. We suggest the reason is that, GRM only enforces the language consistency of the LLM backbone, but can not handle the inherent preference bias in the dataset. For the length-biased dataset where most of the chosen responses are longer than the rejected responses, both the SFT regularization and the DPO Regularization may further exacerbate such length bias of the reward model.\n\nIf our responses have addressed your concerns, we would appreciate your consideration for a higher score. Thank you again for your time and effort in reviewing our paper."}, {"review_id": "BSPcUfmOVM", "reviewer": "Reviewer_rGtM", "summary": "The paper tackles a key problem in RLHF - training reward models that generalize well to unseen prompt-response pairs. The authors identify a key issue in standard reward modeling - models often overfit to response-only features, neglecting the prompt’s role and thus generalizing poorly. They propose decomposing the reward into prompt-free and prompt-related components using an information-theoretic formulation, without additional models. They then guide training by prioritizing samples with small prompt-free reward gaps, reducing spurious biases (e.g. in response length) and improving generalization. Experiments on crafted and real-world datasets show improved reward model accuracy and downstream policy performance.", "questions": "Could you elaborate on the run-time and efficiency of your reward decomposition algorithm? What are some more efficient versions of the algorithm - either by approximating the expectations involved more quickly or estimating the prompt-free reward in an entirely different way?", "limitations": "I think the paper’s reward decomposition algorithm is computationally intensive, and this needs to be acknowledged and discussed.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 4, "originality": 4, "strengths_and_weaknesses": "Overall I think this is an excellent paper addressing a very important facet of RLHF that has not been suitably addressed by prior work, to my knowledge. Unless I have missed a major flaw in the empirical validation, I am enthusiastic about giving this paper an accept rating conditioned on explicitly adding their algorithm to the main paper, discussing its computational inefficiency, and also briefly elaborating on their “randomly sample prompts to compare reward gaps” experiment in the main paper.\n\nStrengths:\n1. The paper clearly highlights a real weakness in current RLHF pipelines: failure to account for prompt information during reward training, leading to poor generalization. They also perform experiments to demonstrate this by randomly sampling prompts and comparing the reward gap on two datasets to the original prompt.\n2. The paper proposes a creative, theoretically grounded solution to the problem - introducing an information-theoretic decomposition of reward gaps into prompt-free and prompt-related components.\n3. The paper has provable results establishing the feasibility of their optimization problem and discusses a natural optimization procedure (binary search) for solving the problem without requiring extra models.\n4. The data prioritization mechanism based on prompt-free reward gaps is an easy modification of existing RLHF pipelines, and some of the practical considerations (e.g. binary clustering with EMA threshold) demonstrate thoughtful engineering.\n5. Experiments on both synthetic (length bias, adversarial prompts) and standard datasets show meaningful improvements in generalization and policy performance on multiple benchmarks (RewardBench, AlpacaEval-2, MT-Bench).\n6. By focusing reward learning on prompt-conditioned information, the paper proposes a direction likely to influence future RLHF reward model design.\n\nWeaknesses:\n1. I think the paper does not explicitly lay out their algorithm for decomposing the reward into a prompt-free and prompt-related components. While it is discussed in words, it would be very helpful if the algorithm was also explicitly laid out in the main paper to avoid ambiguity or misinterpretation.\n2. While the algorithm avoids extra models, estimating expectations over prompts with importance sampling can be computationally intensive, especially on large datasets. The paper does not report run times or efficiency comparisons with some other natural options for the step of computing the prompt-free components of the reward.\n3. (Minor) Experiments mostly revolve around length bias. I think the work would be strengthened by showing mitigation of other known biases (e.g. sentiment, verbosity, style) to confirm the breadth of the applicability of the paper’s method. I suppose that many of these are hard to quantify properly, but I hope that reasonable classifiers for these features could be good enough. I think this could also help us better understand the overly positive tone of LLMs.\n4. While improved generalization is shown empirically, the paper doesn’t analyze how the learned prompt-related rewards align with human-interpretable preferences, which would validate that the method captures meaningful prompt-response relationships. Both a quantitative interpretability study of the prompt-free and prompt-related reward models as well as a qualitative analysis of reward gaps for a few examples would be very helpful.\n5. The paper briefly shows an image illustrating an experiment comparing reward gaps between responses in a dataset under both the original prompt and randomly sampled prompts. It would be good to discuss the experiment in the main paper a bit more. Perhaps it will also help to provide some statistical significance numbers to make sure that the picture is not deluding us, since we are working in the space of the logarithms of the true binary probabilities.", "was_revised": true, "final_justification": "I am largely satisfied with the authors’ rebuttal and will maintain my score of 5. However, I still think that the authors should consider adding the algorithm to the main paper in the camera-ready version to improve clarity of exposition.", "rebuttal": "Thank you very much for your time and effort in reviewing our paper. We present the responses as follows.\n\n> I think the paper does not explicitly lay out their algorithm for decomposing the reward into a prompt-free and prompt-related components. While it is discussed in words, it would be very helpful if the algorithm was also explicitly laid out in the main paper to avoid ambiguity or misinterpretation.\n\nWe're sorry that due to the space limit, we present the pseudo-code of extracting prompt-free reward gaps in the Appendix C.1 . After obtaining it, we calculate the prompt-related reward gaps based on the additive form assumption in Eq. (6).\n\n> While the algorithm avoids extra models, estimating expectations over prompts with importance sampling can be computationally intensive, especially on large datasets. The paper does not report run times or efficiency comparisons with some other natural options for the step of computing the prompt-free components of the reward.\n\n> I think the paper’s reward decomposition algorithm is computationally intensive, and this needs to be acknowledged and discussed.\n\nIn fact, estimating expectations over prompts with importance sampling does not incur significant computational overhead. The reasons are as follows.\n\n1. The importance weights are independent of the trained reward model and are an inherent property of the dataset distribution. Therefore, for each dataset, we only need to pre-sample prompts and compute probabilities once. This process does not need to be coupled with reward training, allowing experiments with different random seeds, hyperparameters, or LLM backbones to share the same importance weights.\n2. The calculation of importance weights is also time-efficient. Specifically, it only requires constant times of forward pass of the LLM, and is **not** autoregressive. This is significantly faster than reward training because there is no need to wait or communicate for parameter updates.\n\nSince the importance weights are computed only once for each dataset and the overhead is not significant compared to training, we did not compare the precise time consumption. In our experiments, calculating the importance weight for a dataset is faster than conducting a full naive BT reward model training on the same dataset. Considering different random seeds, hyperparameters, and LLM backbones, the average time consumption is minimal.\n\n> (Minor) Experiments mostly revolve around length bias. I think the work would be strengthened by showing mitigation of other known biases (e.g. sentiment, verbosity, style) to confirm the breadth of the applicability of the paper’s method. I suppose that many of these are hard to quantify properly, but I hope that reasonable classifiers for these features could be good enough. I think this could also help us better understand the overly positive tone of LLMs.\n\nIt's worth noting that experiments in section 4.1.2 is not solely related to length bias. The adversarial samples can prefer either longer or shorter responses, depending on the original samples. More precisely, the adversarial samples represent a **conflict but simpler** preference. Since the adversarial samples always end up with \"Give a response that is as long / short as possible\", the reward model can learn such preference simply by considering the last sentence of the prompt and comparing the responses. What we care about is whether this preference will affect the reward model’s ability to learn the preference under the original prompt. In fact, many spurious preferences can be seen as preferences that are easier for the reward model to learn, but conflict with human preferences. Our experimental results are applicable to these spurious preferences to a certain extent.\n\nOn the other hand, it's hard to justify the response's property about other spurious preference. The classifier is also hard to train due to the difficulty in constructing such dataset. However, Fig. (3) demonstrates the advantage of prioritizing samples with smaller prompt-free reward gaps, without knowing the specific preference bias. The experiments in section 4.2 further validate the effectiveness of our method, given the potentially diverse preference biases in open-source datasets.\n\n> While improved generalization is shown empirically, the paper doesn’t analyze how the learned prompt-related rewards align with human-interpretable preferences, which would validate that the method captures meaningful prompt-response relationships. Both a quantitative interpretability study of the prompt-free and prompt-related reward models as well as a qualitative analysis of reward gaps for a few examples would be very helpful.\n\nSimilar to the reasons before, it's hard to directly assess how well the reward model aligns with a specific, meaningful human preference. Considering this, we conduct extensive experiments on open-sources datasets and widely used benchmarks. Since the mixed dataset can contain diverse preference biases and the benchmarks test various aspects of the reward model's capabilities, the superior performance of our method demonstrates its ability to help the reward model to learn meaningful preference while alleviating the influence of spurious preferences.\n\n> The paper briefly shows an image illustrating an experiment comparing reward gaps between responses in a dataset under both the original prompt and randomly sampled prompts. It would be good to discuss the experiment in the main paper a bit more. Perhaps it will also help to provide some statistical significance numbers to make sure that the picture is not deluding us, since we are working in the space of the logarithms of the true binary probabilities.\n\nWe're sorry that due to space limit, the details of this experiment are not given in the main paper. As is shown in Fig. (1) (Left), we first randomly sample 1000 prompt-response pairs from the dataset, and then randomly sample 8 non-corresponding prompts for each pair. We then calculate the reward gaps given the original prompt and non-corresponding prompts. the blue line shows the reward gaps with original prompt (in ascending order). The green line shows the mean values of the reward gaps under 8 non-corresponding prompts.\n\nWe continue to provide two statistical metrics to support this. The first is **align rate**, which is the ratio of the samples with non-corresponding prompts that prefer the same response as for the original prompt. The second is **mean deviation rate**, which is the mean value of \n$\\frac{|\\Delta r_{\\text{original}}-\\Delta r_{\\text{non-corresponding}}|}{|\\Delta r_{\\text{original}}|}$\nThe results are as follows:\n\n||align rate|mean deviation rate|\n|-|-|-|\n|SHP dataset|0.87|0.28|\n|Ultrafeedback dataset|0.83|0.55|\n\nIf our responses have addressed your concerns, we would appreciate your consideration for a higher score. Thank you again for your time and effort in reviewing our paper."}, {"review_id": "ph4KIjYqcs", "reviewer": "Reviewer_ZzmL", "summary": "This work identifies the problem of generalization to unseen prompt-response pairs in reinforcement learning from human feedback (RLHF). They propose a decomposition of reward: $r (x, y) = r_1 (x, y) + r_2 (y)$, where $r_1$ is prompt-related reward and $r_2$ is prompt-free reward. The authors use mutual information constraints to extract these components without requiring additional models. Using the difference in $r_2$, a new data prioritization method helps improving the reward accuracy. The efficacy of such decomposition and learning algorithm is corroborated by both synthetic and practical experiments.", "questions": "My only question is about the computational cost compared to baselines. What is the running time of each method in the experiments shown in Table 2? If we control the running time, will your method still be the best?", "limitations": "Aside from the computation cost issue, I didn't find significant limitations.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 4, "strengths_and_weaknesses": "### Strengths\n\n- **Important Problem:** Addresses a fundamental issue in RLHF - reward model generalization is crucial for practical deployment.\n\n- **Novel Technical Approach:** The decomposition method is supported by the information-theoretic formulation, and provides new insights into reward learning and evaluation. The proposed method doesn't require additional models and can be integrated into existing training pipelines.\n\n- **Comprehensive Evaluation:** Includes both synthetic datasets (length-biased and adversarial-prompt) and standard benchmarks, with comparisons to relevant baselines of vanilla Bradley-Terry (BT) training and RRM.\n\n### Weaknesses\n\n- **Importance Sampling for Unknown Behavior Policy:** As the authors mentioned in Appendix C.2, when the behavior policy for the dataset is unknown, they cannot estimate $P(y_1, y_2 | x)$, so they turn to a fixed weighted average for corresponding and non-corresponding prompt-responses tuples. While the experimental results are good, this step lacks intuitive explanation and theoretical guarantee. In fact, this is the most practical regime, where the behavior policy could be rule-based, human, or a large mixture of different LLMs, instead of a single LLM for easy probability calculation.\n\n- **Lack of Discussion on Computational Overhead:** The binary search decomposition and importance sampling add non-trivial computational costs that aren't thoroughly analyzed. The authors haven't provided detailed analysis on the computation cost and the wall-clock time of the experiments. It is hard to see if the allowed training time is the same, whether the proposed method will outperform baselines.", "was_revised": true, "final_justification": "The authors' rebuttal addresses my concerns about the computation overhead. This issue is also raised by reviewers S9sS and rGtM, so I think the presentation of this work can be further improved. I'll keep my positive rating.", "rebuttal": "Thank you very much for your time and effort in reviewing our paper. We present the responses as follows.\n\n> Importance Sampling for Unknown Behavior Policy: As the authors mentioned in Appendix C.2, when the behavior policy for the dataset is unknown, they cannot estimate $P(y_1, y_2|x)$, so they turn to a fixed weighted average for corresponding and non-corresponding prompt-responses tuples. While the experimental results are good, this step lacks intuitive explanation and theoretical guarantee. In fact, this is the most practical regime, where the behavior policy could be rule-based, human, or a large mixture of different LLMs, instead of a single LLM for easy probability calculation.\n\nFirstly, we want to clarify that the importance sampling technique can be used under many practical scenarios. In fact, for many online RL methods (online-DPO, online-PPO, ...), the reward model will be updated based on the generated data of the trained policy and the external preference signal. This enables the reward model to provide more immediate (on-policy) preference signal, facilitating policy improvement. The importance sampling technique is a perfect fit for this setting.\n\nSecondly, although the weighted average scheme may lead to an imprecise estimation of the expectation, the intuition of our method still holds. Either a large value of reward gap on corresponding prompt or non-corresponding prompt will lead to a large value of $\\Delta r_2$ (prompt-free reward gap). Similar to the analysis in section 3.3, by prioritizing samples with smaller values of $\\Delta r_2$, we concentrate our updates on data that bring more preference information and are not dominated by spurious preferences, which is more advantageous for reward learning.\n\n\n> Lack of Discussion on Computational Overhead: The binary search decomposition and importance sampling add non-trivial computational costs that aren't thoroughly analyzed. The authors haven't provided detailed analysis on the computation cost and the wall-clock time of the experiments. It is hard to see if the allowed training time is the same, whether the proposed method will outperform baselines.\n\n> My only question is about the computational cost compared to baselines. What is the running time of each method in the experiments shown in Table 2? If we control the running time, will your method still be the best?\n\nIn fact, estimating expectations over prompts with importance sampling does not incur significant computational overhead. The reasons are as follows.\n\n1. The importance weights are independent of the trained reward model and are an inherent property of the dataset distribution. Therefore, for each dataset, we only need to pre-sample prompts and compute probabilities once. This process does not need to be coupled with reward training, allowing experiments with different random seeds, hyperparameters, or LLM backbones to share the same importance weights.\n2. The calculation of importance weights is also time-efficient. Specifically, it only requires constant times of forward pass of the LLM, and is **not** autoregressive. This is significantly faster than reward training because there is no need to wait or communicate for parameter updates.\n\nSince the importance weights are computed only once for each dataset and the overhead is not significant compared to training, we did not compare the precise time consumption. In our experiments, calculating the importance weight for a dataset is faster than conducting a full naive BT reward model training on the same dataset. Considering different random seeds, hyperparameters, and LLM backbones, the average time consumption is minimal.\n\n\nIf our responses have addressed your concerns, we would appreciate your consideration for a higher score. Thank you again for your time and effort in reviewing our paper."}]

Author rebuttal:

Meta-review: **Paper summary**
The submitted paper addresses the issue of RLHF’s reward model generalization to unseen prompt-response pairs. The authors propose a decomposition of the reward function into a prompt-free reward and a prompt-related reward component, and introduce a data prioritization mechanism based on the prompt-free reward gap to improve reward model training. The proposed method is evaluated on synthetic datasets (e.g., length-biased and adversarial prompts) and standard benchmarks, demonstrating improved generalization and robustness.

**Strengths of the paper**
1. Novelty and Importance: The paper tackles an important problem in RLHF and proposes a theoretically grounded solution. The decomposition of reward into prompt-free and prompt-related components is sensible and provides relevant insights into reward learning.
2. Technical Rigor: The method builds on an information-theoretic formulation, and the learning problem is well-defined, admitting some theoretical analysis.
3. Practicality: The proposed method integrates into existing RLHF pipelines without requiring additional models.
4. Evaluation: The paper includes experiments on both synthetic datasets and real-world benchmarks, showing meaningful improvements in reward model accuracy and downstream performance.

**Weaknesses of the paper**
1. Clarity:
    * The algorithm for reward decomposition is not explicitly laid out in the main paper, which could lead to ambiguity. While it is included in the appendix, its inclusion in the main text would improve clarity.
    * Some visualizations (e.g., Figures 1 and 4) and equations (e.g., Eq. 7) are not intuitive and require better explanations.
2. Computational Overhead:
    * The binary search decomposition and importance sampling add computational costs, which are not thoroughly analyzed. While the authors argue that the overhead is minimal, detailed runtime comparisons with baselines are missing.
3. Limited Scope of Experiments:
    * Most experiments focus on length bias, with limited exploration of other biases (e.g., sentiment, verbosity, style). This raises questions about the general applicability of the method.
    * The paper does not analyze how the learned prompt-related rewards align with human-interpretable preferences, which would validate the method's ability to capture meaningful prompt-response relationships.
4. Baseline Comparisons:
    * While the authors include comparisons with the Generalizable Reward Model (GRM) in the rebuttal, this was missing in the initial submission. Including such comparisons earlier would have strengthened the paper.

**Discussion and rebuttal summary**
* Reviewer sjU2: Initially raised concerns about clarity and technical details but increased their score after the rebuttal clarified these issues.
* Reviewer S9sS: Appreciated the additional baseline comparisons but maintained their score after “careful consideration”.
* Reviewer rGtM: Appreciated the paper's novelty and potential impact but suggested including the algorithm in the main paper and addressing details regarding computational efficiency.
* Reviewer ZzmL: Acknowledged the authors' rebuttal on computational overhead but emphasized the need for better presentation and runtime analysis.

The authors provided detailed and thoughtful responses to the reviewers' concerns, addressing most of the raised issues:
* Clarity: The authors clarified the algorithm for reward decomposition and provided additional explanations for figures and equations.
* Computational Overhead: The authors argued that the computational cost of importance sampling is minimal and provided justifications for this claim. However, detailed runtime comparisons are still missing.
* Baseline Comparisons: The authors included comparisons with GRM in their rebuttal, showing that their method outperforms GRM on multiple benchmarks, although this improvements are not always that significant. Including these results will enable readers to better understand the paper in the context of other approaches.
* Experimental Scope: The authors explained the challenges of addressing other biases and argued that their method is broadly applicable to spurious preferences.
* While the rebuttal addressed many concerns, some reviewers maintained their original scores due to unresolved issues, such as the lack of runtime analysis and limited experimental scope.


**Recommendation**
The paper is technically solid and addresses an important problem in RLHF. The proposed method is novel, well-motivated, and demonstrates relevant improvements in generalization and robustness. However, weaknesses include the lack of clarity in some parts of the paper, the limited discussion of computational overhead, and some limitations of the experiments - but these were mainly well addressed in the rebuttal. Despite these issues, the paper's contributions outweigh its shortcomings - hence, in line with the reviewers’s suggestions, I am recommending the acceptance of the paper.

For the camera ready paper, the authors are strongly encouraged to:
1. Include the algorithm for reward decomposition in the main paper for clarity.
2. Provide a detailed runtime analysis.
3. Expand the experimental scope to address other biases and validate the interpretability of the learned rewards. Furthermore, the additional baseline results should be included.
Decision: Accept (poster)