Adaptive Preference Scaling for Reinforcement Learning with Human Feedback
Authors: Ilgee Hong
Zichong Li
Alexander Bukharin
Yixiao Li
Haoming Jiang
Tianbao Yang
Tuo Zhao
Venue: NeurIPS 2024 poster (2024)
Keywords: Reinforcement Learning from Human Feedback
Large Language Models
Alignment
Primary area: reinforcement_learning
TL;DR: We propose a new adaptive preference loss for RLHF.

Abstract: Reinforcement learning from human feedback (RLHF) is a prevalent approach to align AI systems with human values by learning rewards from human preference data. Due to various reasons, however, such data typically takes the form of rankings over pairs of trajectory segments, which fails to capture the varying strengths of preferences across different pairs. In this paper, we propose a novel adaptive preference loss, underpinned by distributionally robust optimization (DRO), designed to address this uncertainty in preference strength. By incorporating an adaptive scaling parameter into the loss for each pair, our method increases the flexibility of the reward function. Specifically, it assigns small scaling parameters to pairs with ambiguous preferences, leading to more comparable rewards, and large scaling parameters to those with clear preferences for more distinct rewards. Computationally, our proposed loss function is strictly convex and univariate with respect to each scaling parameter, enabling its efficient optimization through a simple second-order algorithm. Our method is versatile and can be readily adapted to various preference optimization frameworks, including direct preference optimization (DPO). Our experiments with robotic control and natural language generation with large language models (LLMs) show that our method not only improves policy performance but also aligns reward function selection more closely with policy optimization, simplifying the hyperparameter tuning process.

A Adaptive Preference Scaling for Reinforcement Learning with Human Feedback
1lgee Hong"
Zichong Li Georgia Institute of Technology 21i9118gatech. edu
Georgia Institute of Technology ihong39ggatech. edu
Alexander Bukharin
Yixiao Li
Haoming Jiang Amazon jhaoming@amazon.coo com
Georgia Institute of Technology abukharin3@gatech edu
Georgia Institute of Technology y xixiooli@aatech... edu
Tianbao Yang Texas A&M University ti tianbao-yang@tame edu
Tuo Zhao
Georgia Institute of Technology
tourzhao@gatech.em edu
Abstract
Reinforcement learning from human feedback (RLHF) is prevalent approach align AI systems with human values by learning rewards from human preference data. Due to various reasons, however, such data typically takes the form of rank- ings over pairs of trajectory segments, which fails to capture the varying strengths of preferences across different pairs. In this paper, propose novel adaptive preference loss, underpinned by distributionally robust optimization (DRO), de- signed to address this uncertainty in preference strength. By incorporating adaptive scaling parameter into the loss for each pair, our method increases the flexibility of the reward function. Specifically, it assigns small scaling parameters to pairs with ambiguous preferences, leading to more comparable rewards, and large scaling parameters those with clear preferences for more distinct rewards. Computationally, our proposed loss function is strictly convex and univariate with respect to each scaling parameter, enabling its efficient optimization through ple second-order algorithm. Our method is versatile and can be readily adapted to various preference optimization frameworks, including direct preference optimiza- tion (DPO). Our experiments with robotic control and natural language generation with large language models (LLMs) show that our method not only improves policy performance but also aligns reward function selection more closely with policy optimization, simplifying the hyperparameter tuning process.
Introduction
In the field of artificial intelligence, aligning AI systems with human preferences has become increasingly crucial, particularly for applications involving complex data and models like large language models (LLMs) in natural language processing [38, 28]. Reinforcement learning from human feedback (RLHF) has gained popularity for customizing Al systems [12, 3, 47]. RLHF involves learning reward function from human preference data, then using a reinforcement learning algorithm to train policy to optimize the learned reward model.
Equul contribution.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

A key challenge in RLHF lies in the complexity of reward modeling, which primarily stems from the reliance on preference labels. Since preference labels only provide comparative rankings of trajectory segments without quantifying the scale of underlying preference strengths, previous methods have employed the Bradley- Terry (BT) model [8] in conjunction with cross-entropy loss to learn the reward function from preference data [12, 38]. This approach assumes that the logit of the preference distribution scales linearly with the reward difference across all sample pairs. However, such linear scaling is often insufficient to account for the variations in preference strength among different pairs, restricting the reward function's s ability to capture a broader range of reward differences. This restrictive approach to reward modeling limits the flexibility of the learned reward function, hindering its capacity to produce the versatile rewards essential for the downstream policy optimization.
To overcome this shortcoming, we introduce novel adaptive preference loss function inspired by distributionally robust optimization (DRO) [16]. Our approach incorporates an instance-specific scaling factor to change the scaling between the preference distribution and the reward difference to be non-linear. These factors are learned during training and enable the model to accommodate varying uncertainties of preference strength, thereby enhancing the flexibility of the reward. For pairs showing strong preference (i.e., low preference uncertainty), our method learns a large scaling factor, which enables the model to learn larger reward difference. In contrast, for pairs showing ambiguous preferences (i.e., high preference uncertainty), our method assigns a smaller scaling factor, enabling the model to learn smaller reward difference. The additional computational overhead of involving this scaling factor into training is negligible, as the proposed loss function is strictly convex and univariate with respect to each scaling parameter. Therefore, it can be easily optimized by simple second-order algorithm within a few iterations.
Our experiments on robotic control tasks [39] demonstrate that our method can learn a more flexible reward function, resulting in an improved policy. Surprisingly, we also discover that our method better aligns the learned reward function with downstream policy optimization. Specifically, when tuning hyperparameters for reward modeling, the simplest approach is to select the reward model according to preference prediction accuracy. However, the selected reward function (with the highest accuracy) often yields a downstream policy with poor performance. To address this misalignment, usually have jointly tune the parameters across both stages according to downstream policy performance, resulting in significant computational burden and tuning effort. Our proposed method can mitigate this misalignment: When using our adaptive loss, can select the reward model based on preference prediction accuracy alone and yield a reasonably well-performing policy. This allows separate tuning of the two stages, easing tuning overhead. To our knowledge, the challenge of this misalignment issue is almost untouched in the RLHF literature, and we are the first to propose a principal approach mitigate this issue.
Moreover, our method is generalizable and can applied to other preference optimization algorithms. For instance, we implement it with direct preference optimization (DPO) [31] and evaluate its effectiveness on natural language generation tasks using Llama-2 7B [40]. Our results demonstrate that integrating adaptive preference scaling into DPO boosts policy performance, while preserving the benefits of alignment. Alignment is especially critical in this setting, where we employ proprietary models like Claude 3 [1] as judges for policy selection, which demands substantial costs for using the APIs In the case without access to LLM assessment, we must select policy based solely on preference accuracy, under which our approach substantially outperforms other baselines.
2 Related works
Loss functions for reward learning. Prior work on this topic is very limited. For example, Song et al. [37] propose using different loss functions for strong and ambiguous preference data in natural language generation tasks. They apply heavy-tailed loss functions for open-ended questions, where preference ambiguity desirable, and light-tailed loss functions for close-ended questions requiring clear-cut rewards. However, their approach requires knowing the question type a priori, necessitating extra labeling effort, and may fail for complex questions containing both open and closed aspects. Zhao et al. [47] propose using a hinge loss, which results in zero gradient when the learned reward difference exceeds margin of 1. This limits the ability to learn very large differences in rewards. Azar et al. [2] develop V/ Preference Optimization with Identity Mapping (IPO), which modifies DPO with a loss function matching the scaling of KL--divergence between the learned policy and the initial policy to avoid overfitting due to weak regularization. In contrast to prior work, our method is

---

more broadly applicable to complex preference learning tasks without needing additional labeling or sacrificing the ability learn arbitrarily large reward differences.
Adaptive temperature scaling (ATS). Temperature scaling (TS) aims to adjust the entropy of probabilistic models by rescaling their logit outputs before the softmax function is applied. This simple method not only enables confidence calibration [19]. but also plays a vital role in various machine learning methods, including knowledge distillation [20], reinforcement learning [25]. and contrastive learning [43]. Building on TS, adaptive temperature scaling (ATS) enhances flexibility by using instance- instance-specffic scalars. Most ATS method trains an additional network for predicting the temperature parameter, which is further integrated into the softmax operator to calibrate the prediction probabilities [44, 14, 4. 21]. In contrast to the aforementioned ATS methods, the proposed adaptive preference scaling (APS) is not designed for classical confidence calibration, but is crafted specifically to enhance the training process of reward function in RLHF. Consequently, the interpretations of scaling factors in ATS and APS are opposite. In ATS, larger scaling parameter is applied to data with higher uncertainty (e.g., data that the classifier is likely to misclassify), which reduces the magnitude of the corresponding logit. Conversely, in APS, a larger scaling factor is assigned to data with clearer preferences, resulting in a larger logit. This distinction clarifies why the scaling parameter in our approach does not correspond to the concept of "temperature" from statistical physics. Additionally, we propose a principled framework for learning scaling parameter based on DRO, which avoids the complexities of designing specific temperature networks and does not rely on heuristically designed loss functions.
Distributionally robust optimization (DRO). DRO is technique that trains machine learning models be robust against uncertainty the data distribution. Specifically, DRO finds a solution that performs well under the worst-case distribution within a specified uncertainty set around the empirical
data distribution [5, 7. 23, 35, 16]. DRO has been applied in various AI/ML domains to improve generalization when the test distribution differs from the training distribution [27, 18, 26, 10, 45, 30J. Our framework is motivated by Qi et al. [30], which tackles KL-constrained DRO problem. However, our approach differs in two significant ways. First, instead of using single KL constraint for the entire training dataset, we apply a separate KL constraint to each individual training data. Second, since each training data involves just two distributional variables, optimize these efficiently. Note that while our proposed method is inspired by DRO, it serves a distinct purpose: improving reward learning in RLHF, which is orthogonal distributional robustness.
can use deterministic method to
3 Method
In this section, we first outline the problem setup, derive the loss function with adaptive preference scaling, and then provide theoretical motivation for our proposed loss. At last, we present an optimization algorithm, extend the approach to direct preference optimization, and introduce a variant of our proposed loss that incorporates quadratic regularization.
3.1 Problem setup
We consider a reward-free Markov decision process M = (S,A,p,)) with state s E S, action E A, state transition function p. and discount factor ?. The ground truth reward function r S x A + R is assumed to be unknown, but only human preferences over pairs of trajectory segments are observed. A trajectory segment {(m, d. ) (mm+1.4mm))) (sk-1, ax=1)} } E (S x *)* We denote 21 22 to indicate that the human preferred trajectory segment 21 over the trajectory segment 22 and denote the preferred one with subscript and the dispreferred one with a subscript / (i.e., zn and 2)). Here, we are given a human preference dataset of trajectory segments Dpret = (Zw,s #1, =1 Our goal is to find a reward f(s, a). which is well-aligned with human preferences. Once we learn the reward, we then find a policy x E A such that it maximizes the expected sum of discounted rewards.
sequence of consecutive state and action pairs
max RZND, [+()]],
at) where f(z) = E(maaee 7f(s,, ar) and D. denotes the stationary distribution of the state-action pair induced by
3

---

3.2
Reward learning with adaptive preference scaling
We now focus on the reward learning phase in RLHE, a crucial stage for capturing human preferences across various trajectory segments. The standard reward learning procedure assumes that the reward function determines a preference distribution, also known as the Bradley-Terry Terry (BT) model [8].
- -
Pr( > 21) = (r((z)) -
r(z)),
where a denotes the sigmoid function. The reward function is then learned by minimizing the expectation of negative log- likelihood of r over the preference data [12]:
min (r) = = [oo [logpr(<w Pr > =1)]
(2)
As can be seen from (1), the BT model essentially assumes that the logit of the preference distribution a (pr > 2{)) scales linearly with the reward difference, regardless of the specific pair of samples. Such linear scaling, however, may not align well with downstream policy learning. Human preferences are often influenced by numerous factors that interact non-linear ways, making the BT model suboptimal as reward model. For example, when the reward difference is small, even slight changes in certain features might lead to significant shifts in preference. The BT model may struggle to capture such rapid shifts due to its slower transition.
To address this challenge, we propose an adaptive preference loss based on KL-constrained distri- butionally robust optimization formulation [30], which can implicitly change the scaling between the logit and the reward difference to be non-linear. Specifically, given pair of trajectory segments
(21, 22), we denote d, (21, 22) = 1(21= > 22) (r(22) r(z1)) and p (P1.P2). We define the following instance-leve level loss:
Ey (21, 22) = max dı (21, 22) + p2d, 21) - To (L, PEDz
s.t. KL 1/2) - Po: (3)
where A2 {p E IR² : + = 1,0 S p1,22 S 11.1/2 is denoted for the uniform distribution, and Po. To > O are shared prespecified parameters across all instances. KL(,,) denotes the KL divergence. Note that without the KL-constraint, (3) is reduced to the cross-entropy loss with TO = I. Unlike general KL-constrained DRO formulation, which considers a distribution p over all training samples, the distributional variable p in (3) is associated specifically with binary preference comparisons for each pair.
We then convert (3) into an equivalent minimax formulation based on the Lagrangian duality,
min max pid, 22) + p2d, (22, 21) (A + To) 1/2) po), X20 p
where x is the Lagrange multiplier. By defining for p. we have
A Tu and applying the optimality condition
min T -Tog Pr.r (zw > 21) + (po - log 2)T,
(4)
Per (zw > 21) = r(( - r(z1) )
where
(5)
We refer to Appendix A.1 for the full derivation. Note that the preference scaling factor 7 in (4) and (5) serves as the Lagrange multiplier of (3). This scaling parameter is used specifically for training
the reward function r, rather than calibrating the preference distribution Pr.r (2 > 21). The scaler
used exclusively during the reward learning phase and is no longer needed in subsequent policy optimization, where the reward function alone is used.
Moreover, the scaling parameter is defined to be an instance instance-specificc parameter corresponding to the pair of trajectory segments (Zw, Therefore, when applying our adaptive loss to reward learning, for each pair (Zwr., =1,)), we need to define corresponding scaling parameter denoted by т. The overall loss function over the training set Prrof is as follows:
min T.71 7N EQ
1
Σ
(l(,T)) = N
Σ (- Ti log Pr. (Zma > 21,5) + pr.);
N
where T = (T. ...., TN). Ω [T: To ST Tmax)
} with Tmax as another prespecified parameter, log 2. Here, we also involve an upper bound Tmax > 0 in (6), and we will explain why it is needed in the next subsection.
and p= P0 - log
explat
4

---

3.3 Theoretical insights
We next provide some theoretical insights on why the scaling parameter 7 can help gain adaptivity by a proposition. For simplicity, we only consider a pair of trajectories.
Proposition 3.1. Assume we have a pair of trajectories 21, 22. and the preference distribution p(z1 > #2) =p" E (0, 1), i.e., the probability, that 21 is preferred over 22, is p". Consider the
problem of minimizing the expectation of our adaptive loss function over the preference distribution:
min -TP" log (a(((z)) - rr(2))//T) - -(1 (p1)-p)loo (a (r(((())) r((1))/())) + pr. (7)
T.ER
Then the minimizer T" and r* of the expected loss satisfy
- p') + + 0, Tmax if - p" log(p") log] pp) + <,
To
if - p" log(p") log
(21) r" (22) ===0a==00)
Here, o-1 is the inverse of sigmoid function.
Note that the expected loss (7) is only for easing theoretical analysis, as p" is not accessible in practice. From Proposition 3.1, we can see that when p" is close enough to 0.5, (i.e., the uncertainty of preference is large), the corresponding optimal r* at the lower bound To. The resulting optimal reward difference is Too (p*), which is smaller than the counterpart obtained by the cross-entropy loss when To < 1. Conversely, when p" is close to or 1, (i.e., the uncertainty of preference is small), the resulting optimal 7" is at the upper bound Tmax- Here, we introduce the upper bound Tmpa to ensure that the optimal T" is bounded. The resulting reward difference in this case is Tmax (p"), ). which is larger than the counterpart obtained by the cross-entropy loss when Tmax > 1. Our theoretical analysis suggests that the adaptive scaling factor essentially changes the correspondence between the logit of preference distribution and the reward difference for each pair of trajectory segments, which could lead to more flexible reward model.
We further visualize our adaptive preference loss in Figure 1, setting To and Tmax to 0.1 and 5.0, respectively. As depicted, our adap- tive preference loss behaves distinctly com- pared to the cross-entropy loss. With large learned reward differences, the cross-entropy tends to be very flat, while our loss maintains a non-trivial gradient, allowing us to contin- ually decrease the loss function. In contrast, for small positive learned reward differences, our loss yields a smaller gradient, thereby less encouraging the reward model to further distinguish pairs of ambiguous trajectory segments. This is consistent with our theoretical analysis.
P=002
0=02
0-50.
p-0.5
0=-066
Gradient
CE
-0.5
-1.0
Reward Difference
Reward Difference
Figure 1: Visualization of the loss function (left) and its gradient (right) on different reward differences.
3.4 Algorithm
We present an efficient algorithm for solving (6). Suppose we parameterize r as a neural network with parameter é. At the m-th iteration, we have the iterate d(m)), and we sample pair of trajectory
segments zw,s and 27,5. We initialize (())) = 1 and then optimize т, by a projected Newton method subject simple interval constraint Ω [6]. Specifically, for k -..g K take
(k-1) 1)
(7 (8) + +
(8)
where A denotes the descent direction
V T. (o)((m) , (K)
(k)
(9)
Vo(mm (K))
Once we get Ti (K)
we update @ by stochastic gradient descent step
(m++1) (()) no V ())(() ((K))
(10)
where T/ is the learning rate. We summarized our proposed algorithm in Algorithm 1, which
presented in per-data manner for clarity but can be directly adapted for mini-batch learning.
5

---

Algorithm Algorithm for reward learning with adaptive preference scaling
1: Input: 7D, Timxx Ps M6o
2: for m 0,
1 do 3: Sample pair of trajectory segements from D preff
..,, M 0,,..,....
4: Set T:0 =
5: for k 0, 1,2,,... ,,....... K do
6: Compute A(4) using (9) and update (k) using (8);
7: end for
8: Update (Tm) using (10) or Adam-style step;
9:
end for
Remark 3.1. Note that since ( T;) is strictly convex and univariate with respect to T₁ in each iteration m, 7 (K) is guaranteed to be near-optimal (i.e., ((K) 7²"). Therefore, the convergence of Algorithm can be guaranteed by the convergence of stochastic gradient descent on the reward model parameter Φ).
Remark 3.2. Computationally, Algorithm incurs negligible additional cost. The inner minimization problem (Lines 5-7 in Algorithm 1) can solved to near-optimality efficiently within a few iterations (e.g. K = 5) given its convex and univariate nature. The additional overhead of each update is minor compared to the overall RLHF pipeline.
3.5 Extension to direct preference optimization (DPO)
Our adaptive preference scaling approach is generic and can be extended to DPO [31], which is another popular method for policy learning from human preferences. DPO directly learns the policy in supervised manner using the preference data of state-action pairs Ppref = (8i, Gan,i at,i) This approach forgoes the need to learn the reward function explicitly by the reparameterization of reward function r with respect to its optimal policy Tr,
a(s.a) 33 llgg(m, (alg())(a)s)/)
+ 3 log Z(s).
(11)
where Z(s)= = (als) exp (11) back into (2), we have the policy optimization problem
and Tref denotes the reference policy. By plugging in
min (T) = -
log o (Brx (s) Br (ai ss))
where Tx (a(s) = log(a(als)/ (alr) denotes the log-probability ratio.
Similarly, we can integrate adaptive preference scaling into DPO by plugging in (11) into (6). By merging 3 with the T, and p. we can further obtain the adaptive DPO (Ada-DPO) formulation as
N
(xx (fx(aw,))))) - Tx (as,als)))
min NN EQ
TI, TN) = N
Σ
T; log a
pri
1=1
Remark 3.3. Note that the proposed adaptive preference loss can be further combined with other RLHF approaches, such as PEBBLE [24], SURF [29], and PARL [11], which still optimize the standard cross-entropy loss (see [24, Eq. (4)), [29, Eq. (3)), and [11, Eq. (5))).
3.6 Extension quadratic regularization
We now introduce variant of our adaptive preference loss that uses quadratic regularization for This modification removes the need for the hyperparameter Tmax in Ω easing the tuning effort. We define the following instance-level adaptive preference loss with quadratic regularization:
min Equad (r, 7270
Po, 2)) poT log 2r.
(12)
Compared to (4), which includes linear regularization term of (po log 2)T, (12) modifies the regu- larization term with coefficient po to be quadratic while keeping the term log 2T linear. Additionally, in (12), the constraint on T only specifies lower bound To and no longer includes an upper bound Tmax The following proposition provides thcoretical insights for this modification.

---

Proposition 3.2. Assume we have a pair of trajectories 21, 22, and the preference distribution p(21 > 22) p E (0.1). Consider the problem of minimizing the expectation of our adaptive loss function with quadratic regularization over the preference distribution:
min -TP" log (0((((z)) ((()))))))())()((11111)) - 2
-
(o
log (0(((2)) - r(21))/=)) + + log 2T.
Then the minimizer T² and r* of the expected loss satisfy 7"= ==maxTTo- (p" log(p") (1 - p² log(L p²) + log (2)p))))))
(
Here, a- is the inverse of sigmoid function.
Note that unlike the adaptive preference loss with linear regularization described in Proposition 3.1,
the optimal value 7" for quadratic regularization does not involve the upper bound Tmax-
4 Experiments
In this section, we examine the effectiveness of our adaptive preference loss based on robotic control and natural language generation tasks. Due space limit, we defer the experiments with quadratic regularization, ablation studies, and discussions on hyperparameter selection to Appendix C.
4.1 Robotic control
Experiment setup. We apply our proposed reward learning method on robotic control tasks from the PyBullet [13] environments: HalfCheetah, Ant, and Hopper. These environments are similar to those available in OpenAl Gym [9] but they are known be much harder solve [34]. Similarly to Gao et al. [17]. our setting is synthetic, where we use the ground truth rewards to provide preference labels on each pair of samples due to high expense of collecting human preferences. For the reward function, we use wwo-hidden-layer MLPs, each containing 64 hidden units. This configuration is aligned with the designs of both the policy and value networks. Following Christiano et al. [12], we repeat the following three steps for each stage: (i) We sample a set of trajectories by the policy æ, and update the policy with proximal policy optimization (PPO, Schulman et al. [36]) alongside reward function f. (ii) We split the segments (the sequence of state-action pairs) into training set and a testing set. Then, we randomly sample pairs of segments from the training set, and generate Ppret with preference labels. We do the same to the testing set, and generate D pref- (iii) We train the reward function f on D Preef, and use D' Pref for evaluating the preference prediction of f.
For notational simplicity, we name our proposed adaptive preference scaling method for reward learning "Ada-Pref". We compare Ada-Pref with the baseline method "Pref", which uses the standard cross-entropy loss for reward learning. For every 10000 timesteps the policy æ runs, we evaluate the learned policy based 20 test episodes. We also compute the average preference prediction accuracy of the learned reward function across stages. We set the budget to 3 million timesteps and perform training over 10 different seeds. For hyperparameter tuning in both reward learning and policy optimization, we apply two different criteria: 1) We identify the best policy based on its performance (the one with the highest return) and subsequently select the corresponding reward function. 2) We choose the best reward function based on its performance (the one with the highest average preference prediction accuracy) and then select the corresponding policy. Details of the implementations and hyperparameter tuning procedures are in Appendix B.
Results. We summarize the results on three PyBullet tasks as follows:
Table 1 and Figure 2 illustrate the results for Pref and Ada-Pref on the PyBullet tasks, based on the first hyperparameter tuning criterion. In Table 1, we report the highest return of the best policy and the average preference accuracy of the corresponding reward function. We can see that Ada-Pref consistently outperforms Pref in terms of return all three tasks and achieves comparable preference accuracy. The upper panel of Figure shows the learning curve plots. We can see that Ada-Pret surpasses Pref nearly every timestep and reaches higher plateau across all tasks. The lower panel of Figure presents percentile plots from different seeds to demonstrate individual run behaviors. As shown, we confirm that Ada-Pref consistently outperforms Pref at every percentile across all tasks.
Table presents the results for Pref and Ada-Pref based on the second hyperparameter tuning criterion. From Table 2, can see that both methods show a decrease in performance compared to Table 1,
7

---

3500
1000
3000
2000
2000
2500
1500
Return
Return
2003
Return
1000
1300
1903
Pref
500
Pref
-1000
Ada- Pref
Ada-Pref Pref
Ada-Pre Pref
13
0.0
0.5
10
20
25
3.0
Timesteps (1e6)
Timesteps (1e6)
Timesteps (1e6)
3603
2500
3200
2000
3400
3003
Return
5300
Return
1500
2000
3300
2600
Pref
3002
Pref
502
Pref
Ada-Pref
Ada- Pref
Ada-Pre Pref
20
40
50
105
2000
20
6o
as
105
20
A5
as
185
Percentile
Percentile
Percentile
(a) HalfCheetah
(b) Ant
(c) Hopper
Figure 2: Learning curve plots (top) and percentile plots (bottom) for Pref and Ada-Pref For the learning curve plots, returns at each timestep are averaged across 10 different seeds, then smoothed over timesteps using an exponential moving average (EMA) with smoothing factor of a 0.1. For the percentile plots, returns from 10 different seeds are sorted in ascending order.
while Ada-Pref still outperforms Pref terms of both preference accuracy and return on all three tasks. Furthermore, Ada-Pref demonstrates greater resistance performance degradation than Pref, indicating its superior ability to align the learned reward function with policy optimization. This alignment allows for effective policy selection based preference accuracy without the need to evaluate the policy using ground truth rewards.
Table I: Table for the highest return of the best
Table 2:
Table for the average preference predic- tion accuracy of the best reward function and the highest return of the corresponding policy.
policy and the average preference prediction ac- curacy the corresponding reward function.
Preference Accuracy (%)
Preference Accuracy
Task
Method
Return
Task
Method
Return
Pref
2724.42
89.09
Pref
2620.83
89.41
HalfChectah
HalfCheetah
Ada-Pref
2875.45
89.46
Ada-Pref
2865.07
90.75
Pref
2917.81
85,57
Pref
2750.99
87.93
Ant
Ada-Pref
3177.11
85.48
Ant
Ada-Pref
3008.69
89.23
Pref
1324.91 1692.10
92.08
Pref
744.66
93.18
Hopper
Hopper
Ada-Pref
91.36
Ada-Pref
1134.73
93.26
4.2 Natural language generation
Experiment setup. We apply DPO with our proposed adaptive loss (Ada-DPO) method to two open-ended text generation tasks: summarization and single-turn dialogue. We adopt the Llama-2 7B model [40] as the backbone and conduct instruction tuning on each task to obtain the initial reference models. For summarization, the policy generates summaries given posts collected from Reddit. We use the filtered TL;DR summarization dataset [41] I] for instruction tuning, which contains more than 117K Reddit posts, each with numan-writtee summary. We apply the human preferences collected by Stiennon et al. [38] for preference optimization, where each transcript contains pair of responses along with preference label. For single-turn dialogue, the policy responds to various human queries ranging from simple questions to complex demands. We utilize the Anthropic Helpful and Harmless dialogue preferences dataset [3] for both instruction tuning and preference optimization. This dataset contains 170k human-Al dialogues, with each dialogue containing two Al responses and a human preference label. We use preferred responses for instruction tuning and the full set of preferences for optimization. For instruction tuning stage, we fine-tune the entire Llama-2 model. the alignment stage using Ada-DPO and different baselines, we apply LoRA fine-tuning for computational efficiency concerns, need to simultancously tune multiple hyperparameters.

---

The rank of the LoRA adaptor is 64. We consider three baseline methods: DPO [31], y Preference Optimization with Identity Mapping (IPO) [2] and Sequence Likelihood Calibration with Human Feedback (SLiC-HFF) [47].
As human evaluation is prohibitively expensive, we use Claude 3 [1], proprietary large language model, to automatically evaluate responses based on summary quality and heipfulnes/hhrrllesnsss for the summarization and dialogue tasks, respectively. Prior work has shown that Claude 3 and GPT-4 can effectively measure a quantitative improvement over the instruction-tuned model [15]. We split small subset from each instruction tuning dataset for testing and calculate the win rate against the instruction-tung response generated by policy A is preferred over policy B is referred to as the win rate of against We also split a subset from each preference optimization dataset to validate the preference prediction accuracy. Details of the implementations and hyperparameter selections are in Appendix B.2.
reference model as the evaluation metric. The percentage of instances where the
Results. We summarize the results on the two natural language generation tasks as follows:
In Figure 3, we select the model with the highest win rate and present the win rate and its preference accuracy for all baselines. We observe that Ada-DPO outperforms the other baselines on both tasks in terms of win rate and achieves comparable preference accuracy. In Figure 4, we display the performance of the model selected with the highest accuracy (not win rate). As shown, Ada-DPO achieves significant improvement beyond the DPO baseline in terms of win rate and obtains a comparable preference accuracy. This again indicates that Ada-DPO yields better aligmment between the learned reward function and policy optimization, allowing good policy selection based on preference accuracy without proprietary L.M judge.
SUC
DPO
SLICFF
DPO
DFO
70-
Add
80
Ada
Vin (%)
Preference Acourncy
70
Win Flete 8
Preference Accurscy
80-
60
8.
20
88
50
10
20-
46
Darogga
mam 24800
Burm
Dialogee
Burmmnnaaion
Dkskogue
Figure 3: The best win rate and the preference prediction accuracy of the corresponding model.
Figure 4: The best preference prediction accuracy and the win rate the corresponding model.
4.3 Detailed analysis
We present detailed analyses of Ada-Pref and Ada-DPO for both the Ant and summarization tasks. Figure 5(a) presents histogram of the learned scaling factors for the Ant task. We can see that around 60% of these scaling factors reach the upper bound, while about 10% converge to the lower bound, and the rest are distributed across the region. In Figure 5(b). we explore the relationship between preference strength and the learned scaling factors т, and in Figure 5(c), we investigate the relationship between preference strength and the learned reward difference for Pref and Ada-Pref. We measure preference strength using the true reward difference, categorize it into five percentile bins, and then bin the scaling factors and the learned reward differences accordingly to compute the average. As can be seen, the learned scaling factor increases monotonically with preference strength, demonstrating that the method successfully adapts the loss scaling to the varying degrees of preference in the data. Furthermore, Ada-Pref learns smaller reward differences for pairs with ambiguous preferences and learns larger reward differences for those with strong preferences, compared to Pref. This indicates that our method leads to a more flexible reward function.
In Figure 6(a), we plot histogram of the learned scaling factors 7 for the summarization task. We can see that around 40% of the scaling factors converge to the upper bound, with the rest distributed across the region. We also display the relationship between the confidence scores and the scaling factors in Figure 6(b). The confidence score is an integer from to 4 included in the dataset, and a higher score denotes a stronger preference. We bin the scaling factors based confidence scores and compute the average. As shown, the scaling factors positively correlate with confidence scores, justifying that we learn larger 7 for strong preferences and smaller for ambiguous ones.
We further present two pairs of preference samples where Ada-DPO assigns large or small scaling factors in Figure 7. We observe that the sample pair with a large scaling factor shows strong

---

Reward Difference
Pref
Parceraage (%)
-0-
0.22
Ade-Pref
10
2.55
8.15
2.00
16
0.033
Preference Strength
Preference Strength
(a) Histogram of т
(b) Preference strength and (c) Preference strength and
learned reward difference
Figure 5: Histogram of learned scaling factors, relationship between preference strength and the learned scaling factors, and relationship between preference strength and the learned reward difference. All plots are from the Ant task.
Percentage (%)
Average
3.4
20-
0
T
Confidence Score
(a) Histogram of (b) Confidence score and r
Figure 6: Histogram of learned scaling factors and relationship between the confidence scores and the learned scaling factors. Both plots are from the summarization task.
Sample pair with large r 4.0) Prompt
Sample pair with small r (r=0.1) Prompt
POST: This morning, I was walking between termina is in the Minneapalis airport. bunch of stuck behind of 8 tecnage girls stowly walking
POST: On our second date told her I had Eipo and she was fine with it. She borrowed boak had on the disorder that nelped her understand. Everything
girls got stuck at the sirport due to their inraturity and ob liviousssss,, and Itd Like to know how 1 can he
Chosen:
Leamed Reward Dit.
30.4
Chosen:
bipolar
2.0.4
0.385
recently my depression and noddines: is getting to ny girlfriend. She is
Learned Reward
afraid cone over because she
then.
20
thinks in losing interest in her.
How do I help her?
0.2
Rejected: Teenage girls, who na icea vere teenage girls, were being obnoxious ly obnoxinus obnoxiously abnoxtously abnox fously
Rejected:
bisolar recently depressioo
obnoxiously ly obnoxious by obnoxiously
getting ay girlfriend. afraid
afraid will lose interest in her.
over because
DPO OPO
0.081
Figure 7: Examples preference sample pairs with large (left) and small (right) scaling factors т, and the comparison of the learned reward difference. The preferred (chosen) responses colored by green and the rejected responses are colored by red.
preference, the rejected response is nonsensical while the chosen one is clear. Ada-DPO learns a larger reward difference for such data, while it is much smaller with DPO. Conversely, for sample pair with small scaling factor, the two responses are very similar, indicating its ambiguity. Ada-DPO learns small reward difference on this pair, while DPÔ gets a large reward difference.
Conclusion
RLHF is an emerging challenge in machine learning. Prior to the popularity of models like ChatGPT, research on designing proper loss functions for reward learning was limited. To bridge this gap, we explore uncertainties in underlying preference strengths and propose an adaptive preference loss function. This loss function incorporates instance-specift scaling factors to modulate the correspondence between reward differences and preference distributions. Taking the result this paper as an initial start, we expect more sophisticated and stronger follow-up work that applies to RLHF with similar structures. All of these efforts may ultimately assist developing more principled RLHF methods to better control risks associated with advanced AI systems.
10

---

References
[1] Anthropic. Claude, 2024. URL http://wwwwwaaoppccccooomm
[2] Mohammad Gheshlaghi Azar, Zhaohan Daniel Guo, Bilal Piot, Remi Munos, Mark Rowland, Michal Valko, and Daniele Calandriello. A general theoretical paradigm to understand learning from human preferences. In International Conference on Artificial Intelligence and Statistics, pages 4447-4455, PMLR, 2024.
[3] Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, Nicholas Joseph, Saurav Kadavath, Jackson Kernion, Tom Conerly, Sheer Showk, Nelson Elhage, Zac Hatfield-Doe Dodds, Danny Hernandez, Tristan Hume, Scott Johnston, Shauna Kravec, Liane Lovitt, Neel Nanda, Catherine Olsson, Dario Amodei, Tom B. Brown, Jack Clark, Sam McCandlish, Chris Olah, Benjamin Mann, and Jared Kaplan. Training a helpful and harmless assistant with reinforcement learningvfrom human feedback. CoRR, 0bbs22004005866 2022.
[4] Sergio A Balanya, Juan Maroñas, and Daniel Ramos. Adaptive temperature scaling for robust calibration of deep neural networks. Neural Computing and Applications, pages 1-23, 2024.
[5] Aharon Ben-Tai Tal, Dick Den Hertog, Anja De Waegenaere, Bertrand Melenberg, and Gijs Rennen. Robust solutions of optimization problems affected by uncertain probabilities. Management Science, 59(2):341-357, 2013.
[6] Dimitri P Bertsekas. Projected newton methods for optimization problems with simple con- straints. SIAM Journal on control and Optimization, 20(2):221-24) 2226, 1982.
[7] Dimitris Bertsimas, Vishal Gupta, and Nathan Kallus. Data-driven robust optimization. Mathe- matical Programminng 167:235-2922 2018.
[8] Ralph Allan Bradley and Milton E Terry. Rank analysis of incomplete block designs: the method of paired comparisons. Biometrika, 39(3/4):324-345 1952.
[9] Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojeiech Zaremba. Openai gym. arXiv preprint 1606vv:606.015500 2016.
[10] Samuel Broscheit, Quynh Do, and Judith Gaspers. Distributionally robust finetuning bert for covariate drift in spoken language understanding. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics, pages 1970-1985, 2022.
[11] Souradip Chakraborty, Amrit Bedi, Alec Koppel, Huazheng Wang, Dinesh Manocha, Mengdi Wang, and Furong Huang. Parl: A unified framework for policy alignment in reinforcement learning. In The Twelfth International Conference on Learning Representations (ICLR), 2024.
[12] Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. Advances in neural information processing systems, 30, 2017.
[13] Erwin Coumans and Yunfei Bai. Pybullet, python module for physics simulation for games, robotics and machine learning, 2016-2019. URL
org/.
[14] Zhipeng Ding, Xu Han, Peirong Liu, and Mare Niethammer. Local temperature scaling for probability calibration. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 6889-6899, 2021.
[15] Yann Dubois, Chen Xuechen Li, Rohan Taori, Tianyi Zhang, Ishaan Gulrajani, Jimmy Ba, Carlos Guestrin, Percy S Liang, and Tatsunori Hashimoto. A Alpacafarm: A simulation framework for methods that learn from human feedback. Advances in Neural Information Processing Systems, 36, 2024.
[16] John C Duchi, Peter W Glynn, and Hongseok Namkoong. Statistics of robust optimization: generalized empirical likelihood approach. Mathematics of Operations Research, 46(3): 946-969, 2021.
11

---

[17] Leo Gao, John Schulman, and Jacob Hilton. Scaling laws for reward model overoptimization. In International Conference on Machine Learning, pages 10866. PMLR, 2023.
[18] Tejas Gokhale, Abhishek Chaudhary, Pratyay Banerjee, Chitta Baral, and Yezhou Yang. Se- mantically distributed robust optimization for vision-and-language inference. In 60th Annual Meeting of the Association for Computational Linguistics, pages 1493-1513, 2022.
[19] Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. On calibration of modern neural networks. In International conference on machine learning, pages 1321-1330. PMLR, 2017.
[20] Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. arXiv preprini T.Xxv::503.022 050.:5500...2000000 2015.
[21] Tom Joy, Francesco Pinto, Ser-Nam Lim, Philip HS Torr, and Puneet K Dokania. Sample- dependent adaptive temperature scaling for improved calibration. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pages 14919-14926, 14926, 2023.
[22] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. International Conference Learning Representations, 2015.
[23] Daniel Kuhn, Peyman Mohajerin Esfahani, Viet Anh Nguyen, and Soroosh Shafieezadeh- Abadeh. Wasserstein distributionally robust optimization: Theory and applications in machine learning. In Operations research & management science in the age of analytics, pages 130-166. Informs, 2019.
[24] Kimin Lee, Laura Smith, and Pieter Abbeel. Pebble: Feedback-efficient interactive rein- forcement learning via relabeling experience and unsupervised pre-training. In International Conference on Machine Learning, 2021.
[25] Xuezhe Ma, Pengcheng Yin, Jingzhou Liu, Graham Neubig, and Eduard Hovy, Softmax q- distribution estimation for structured prediction: A theoretical interpretation for raml. arXiv preprint arXiv:1705.077
2017.
[26] Paul Michel, Tatsunori Hashimoto, and Graham Neubig. Modeling the second player in distributionally robust optimization. International Conference on Learning Representations, 2021.
[27] Yonatan Oren, Shiori Sagawa, Tatsunori B Hashimoto, and Percy Liang. Distributionally robust language modeling. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 4227-4237, 2019.
[28] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 15:27730-277444 2022.
[29] Jongjin Park, Younggyo Seo, Jinwoo Shin, Honglak Lee, Pieter Abbeel, and Kimin Lee. SURF: Semi-supervised reward learning with data augmentation for feedhack-efficien peference-based reinforcement learning. In International Conference on Learning Representationss 2022.
[30] Qi Qi, Jiameng Lyu, Kung-Sik Chan, Wei Bai. and Tianbao Yang. Stochastic constrained dro with a complexity independent of sample size. Transactions on Machine Learning Research, 2023.
[31] Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly reward model. Advances in Neural Information Processing Systems, 36, 2024.
[32] Antonin Raffin.
RI baselines3 Zoo, 2020.
URL https: /tte::///tttue com/DLR-RM
rl 11-baselines3-zo..
[33] Antonin Raffin, Ashley Hill, Adam Gleave, Anssi Kanervisto, Maximilian Ernestus, and Noah Dormann. Stable-baseliness: Reliable reinforcement learning implementations. Journal of Machine Learning Research, 22(268):1-8, 2021.
12

---

[34] Antonin Raffin, Jens Kober, and Freek Stulp. Smooth exploration for robotic reinforcement learning. In Conference on Robot Learning, pages 1634-1644. PMLR, 2022.
[35] Shiori Sagawa, Pang Wei Koh, Tatsunori Hashimoto, and Percy Liang. Distributionally robust neural networks for group shifts: On the importance of regularization for worst-case generalization. arXiv preprint
2019.
[36] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.000 707.06347777. 2017.
[37] Ziang Song, Tianle Cai, Jason D. Lee, and Weijie Su. Reward collapse in aligning large language models. CoRR, abs/2305. 17608, 2023.
[38] Nisan Stiennon, Long Ouyang. Jeff Wu, Daniel M. Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul Christiano. Learning to summarize from human feedback. Advances in Neural Information Processing Systems, 2020.
[39] Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco:
A physics engine for model-based control. In 2012 JEEE/RSJ international conference on intelligent robots and systems, pages 5026-5033. IEEE, 2012.
[40] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprini arXiv:2307.0922. 2023.
[41] Michael Völske, Martin Potthast, Shahbaz Syed, and Benno Stein. TI; dr: Mining reddit to learn automatic summarization. In Proceedings of the Workshop on New Frontiers in Summarization, pages 63-63,222 2017.
[42] Leandro von Werra, Younes Belkada, Lewis Tunstall, Edward Beeching, Tristan Thrush, Nathan Lambert, and Shengyi Huang. Trl: Transformer reinforcement learning, 2020. URL https:
/ //github. com/ com/hhuginggaace ace/trl.
[43] Feng Wang and Huaping Liu. Understanding the behaviour contrastive loss. Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 2495-2504, 2021.
[44] Pei-Hsin Wang, Sheng-lou Hsich, Shih-Chich Chang, Yu-Ting Chen, Jia- Yu Pan, Wei Wei, and Da-Chang Juan. Contextual temperature for language modeling. arXiv preprint 13355 2020.
[45] Hongyi Wen, Xinyang Yi, Tiansheng Yao, Jiaxi Tang, Lichan Hong, and Ed H Chi. Distributionalll-rrouss recommendations for improving worst-case user experience. Proceed- ings of the ACM Web Conference 2022, pages 3606-3610, 2022.
[46] Thomas Wolf. Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue. Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al. Transformers: State- of-the-art natural language processing. In Proceedings of the 2020 conference empirical methods in natural language processing:
system demonstrations, 2020.
[47] Yao Zhao, Rishabh Joshi, Tianqi Liu, Misha Khalman, Mohammad Saleh, and Peter J. Liu. Slic-hf: Sequence likelihood calibration with human feedback. CoRR, abs/2305. 10425, 2023.
13

---

A Derivation and proofs of Section 3
A.1 Derivation of Equation (4)
In this subsection, present the full derivation of Equation (4). Recall the following loss:
(21, 22) max pid,(21, 22) + (22, 21) To 1/2) s.t. kL. 1/2) S Po-
p²A;
Using the Lagrangian duality, we have max min p1d,(21,20) 22) + P2d, (22, - -T. 1/2) (p.I.pp. 1/2) po)-
pc A20
By strong duality theorem, we have min max (21,(21, #2) + p2d, (22, 21) kp. 1/2) X(KT.(p. 1/2) Po), A20 pε
which is equivalent to
min max (21 + p2d, 21) - (A + To) (,L 1/2) po) ToPo-
X20 pεA,
We let T 7 PE
To and obtain
min max (1d,(21, 22) - p2dr (22, x1) - ( (p 1/2) - -po) - Topo-
Now, we consider the optimality conditions for the inner constrained maximization problem by defining the following Lagrangian function:
Σ Pk -1)
(p, (a) = (11, 22) + (2d, 21) - (KKT((p,11// - -Po) -
k=
where μ is the Lagrange multiplier. The optimal solutions pr to the inner maximization problem satisfy the following KKT conditions:
d 22) T (log(p"*) + 1) r 0, (22, 21) - (log(p"*)) + 1) - O, 2
and
Σ 2i =. k=1
Then we have
exp (dr (21, 22)/T)
pr
exp (21, + exp (22,
exp 21)/
and P2 TV
exp (d, (2) /2) + exp (2, (22, 2))
Plugging in pf" and p. back into the inner maximization problem, we have
min J log (exp (1, 22) + exp (d, (22, 21 //r) - r log 2 + (7 Po)
r200 Without loss of generality, we let 21 = Zu and x2 = 21, and obtain
r r(zw) r(z1)
min log 72
+ (po log 2)T,
T where a is the logistic function. This completes the derivation.
A.2 Proof of Proposition 3.1
We first derive the expectation of the adaptive loss. By taking expectation of (4) with p = Po - -log22 2,
we have:
E (1(2) 22) 7 log (((((())--(())/11) + pr]
21.22
+ 1(22 x 2 - - log (a - r(()))/=))
((((((())))) + +r]
= =-'[- (((()(=))
(1 r²)l - log (o - r(=:))/r)) pr]
-TP" log (a(((2)) - ((a)))))))) log log (a((r(z2) - r(z))/T)) + pr.
By the optimality condition of ((2)) r(2) and T, we have
r(2)) - r(=)))
TO 'p²),
(13)
where -1 is the inverse of sigmoid function. Plugging (13) into the objective in (7), we obtain
min p" log(p") (1 p" }lo p')+ + T.
TER2
whose objective is essentially linear in 7. Hence, when -p" log(r") - p²) (10g111p))00 + the corresponding optimal r" is at the lower bound To and the optimal reward difference r r" (21) -()=-)(((22) = Tha Toa-"pp" (p*) given the optimality condition. Conversely, when -p" log(p") (1 11-"))) log(1 - 11ppppp + <
0, we have 7 Tmax and r (21) (22) Tmax Tmaxx=((pp) This completes the proof.
14

---

A.3 Proof of Proposition 3.2
We first derive the expectation of the adaptive loss with quadratic regularization. By taking expectation of (12), we have:
E {2) > 22] - log (a((((z)) (((raa))((=aa)//))) + PoT - -log27
P1
122 =1 7 log (d(((z2)) r(=1))//)) +o= log 2r)
= [- - logg ( r((=))//T)) + PoT log 2T]
+ +1 - p²] - r log (((()))))r((=))))) + Pa= - -log 2T|
=Tp" log - - T(1 p²) log
+ POT - log 2T.
(14)
By the optimality condition of r(2) - r(22) and t, we have r(z1) r22)) To (p"),
(15)
where o 1 is the inverse of sigmoid function. Plugging (15) into the objective in (14), we obtain
min p" log(p") - (1 - p²) log(1 p-) - log + Po7
(16)
T270
Note that (16) is always bounded without the need of an upper bound of Specifically, with any p (0, 1), we have -p" log(p") ( (1-p") log(1-p") --o-p"""""" 2 S (nand max(fo. To. (p" log(p")+
(1 p²) lo)l p*) - log (2p)) This completes the proof.
B
Implementation details
B.1 Robotic control
Our implementations of robotic control tasks are based on Stable-Baselinss] [33] and RL Zoo training framework [32]. We conducted our experiments using CPUs, and it took approximately four hours to train a single model with more than 4096MB of memory. For both Ada-Pref and Pref, we the segment length to as it is the most basic unit that the gold reward model is able to provide preference for. Additional experiments with a segment size of 25 for the Ant, HalfCheetah, and Hopper are in Appendix C.2. We calculate the average preference prediction accuracy over the first million timesteps. At each training step, we assign preference labels to every possible pair of trajectory segments within a mini-batch based on their ranking from the gold reward model. We set the batch size 64 for the HalfCheetah and Ant tasks and for the Hopper task. We tune the number of epochs in 3, 5}. We use Adam optimizer [22] and tune the learning rate in (5e - 3, le- 3, 5c 4, le 4} for the Ant and HalfCheetah, and set the learning rate to le - 2 for the Hopper. For Ada-Pref, we tune the Tmax in {1.0, 3.0} and the Po in 10.1, 0.3, 0.5J. We fix iterations to 3 for all experiments. Details of the chosen hyperparameters for reward learning for all three tasks are summarized in Tables 3 and 4. For PPO, we reused all hyperparameters from the original paper [36] optimized for the Mujoco benchmark [39]. Details of the hyperparameters for PPO are in Table 5.
0.1 and the number of Newton
Table 3: Chosen hyperparameters for reward learning used for Table
Table 4: Chosen hyperparameters reward leaming used for Table
Task
Method
# eepochs
I.R
Task
Method
epochs
LR
Pref
5
5e-3
Pref
3
Se-3
HalfCheztah
HalfChectah
Ada-Pref
3
lc-3
3.0
0.5
Ada-Pref
5
lc-3
3.0
0.5
Pref
1
5c-4
Pref
5
5c-4
Ant
Ant
Ada-Pref
5
lc-4
1.0
Ada-Pref
5
lc-3
1.0
0.5
Pref
5
le-2
Pref
3
le-2
Hopper
Hopper
Ada-Pref
le-2
1.0
Ada-Pref
le-2
3.0
0.3
B.2 Natural language generation
Our implementations of natural language generation tasks are based on transformers [46] and trl training framework [42]. We conducted our experiments using eight A100 GPUs, each with 40GB of memory. Training single model took approximately two hours. We provide more details on each task as follows:
15

---

Table 5: Chosen hyperparameters for PPO.
Parameter
Value
optimizer
Adam
discount (?)
0.99
value function coefficient
0.5
entropy coefficient
0.0
shared network between actor and critic
False
max gradient norm
0.5
learning rate schedule advantage normalization
constant
True
clip range value function number of steps per rollout initial log o learning rate number of epochs
2048
0.0
3 10
10
number of samples per mini-batch
non con-linearity
Tanh
GAE coefficient (A)
0.95
clip range
0.2
orthogonal initialization
yes
B.2.1 Summarization
For the instruction tuning stage, we randomly select 800 data from the filtered TL;DR summarization dataset [41] for testing the policy and leave the rest for supervised tuning. In the preference optimiza- tion stage, split the preference dataset [38] into training and testing set evaluate the preference accuracy. For both stages, we omit the title and only use the post content as the prompt. The prompt format follows: "POST: post content. content.InnnII.:DE... !nnnT.,
For Ada-DPO and all baselines, we set the batch size to 32 and train epoch for both instruction tuning and preference optimization. We set the a parameters of LoRA fine-tuning to 16, and tune other parameters by grid search. The learning rate tuned in (5e - 6, 5e 5, le - 4, 5e - 4). SLiC-HF, IPO and DPO include parameter B, which is tuned in range of {0.01, 0.1. 0.3, 0.5). For Ada-DPO, we tune the Po in (0.05, 0,1, 0.3, 0.5} and the Tmax in (1.0, 4.0, 5.0. 10.0). We fix To 0.1 and the number of Newton iterations to 5 for all experiments. The best Ada-DPO is achieved with ( 5e 5, Po 0.1, and Tmax = 4.0.
B.2.2 Single-turn dialogue
We use the original training split in the Anthropic Helpful and Harmless dialogue preferences dataset [3] for training in both stages. We randomly select 800 samples from its testing split to calculate the win rate, and use the rest of the data in the testing split for validation during preference optimization. We use the original data format.
In the dialogue task, tuned in {5c- 6, 5e - 5, le -4. The parameter /3 for baselines is tuned in range of {0.01, 0.1, 0.3). For Ada-DPO, we tune the Pu {0.05, 0.1, 0.3, 0.5), Tmax in (1, 5, 10) and fix To 0.1. The best Ada-DPa DPO is achieved with Ir 5e 5, Po = 0.05, and Tmax = 5.0.
the same batch size of 32 and epoch for training. The learning rate is
B.2.3 Evaluation prompt.
We apply Claude for win rate judgments. Here, we include the used prompt for generating win rates for both generation tasks. The order of responses is switched and response only counts for win if it wins both orders.
Summarization win rate prompt.
16

---

Human: Which of the following summaries does better job summarizing the most important points the given forur post, thout free luding unimportant, or frre evant detaf good summary both precise and concise.
Post: <prompt>
Summary A: creeponse Summary creaponse
FIRST provide one-sentence comparison the two surmaries, explaining which you prefer and why. SECOND, new line, state only "A" "B" to indicate your chot ce. Your resperse should the format:
Comparison: sone-sentence comparison and explanation Preferred: <"A" or "B">
Assistant:
Dialogue win rate prompt.
Human: For the following query chatbot, which response is more helpful and harmless?
Query:
<prompt>
kesponse A: <respense A> Response B: <response B>
FIRST provide one-sentence comparison of the two respenses and explain which you feel is more helpful and harmless. SEC- OND, new line, state only "A" or "B" to indicate which response is more helpful. Your response should use the for- mat:
Comparison: cone-sentence comparison and explanation> More helpful: <"A" or "B*>
Assistant:
Additional experiments and discussions
C.1 Experiments with quadratic regularization
Fret Adb-/hel-Quud
Ace-TekQusss
TImmatopa
Stroighh
Preeanaa Stronghh
(a) Learning curve
(b) Percentile plot
(c) Histogram of T
(d) Preference (e)
Preference strength and learned reward difference
strength and
Figure Left: Learning curve and percentile plot for Pref and Ada-Pref- Puref-Qud.. Middle: Histogram of the learned scaling factors. Right: Relationship between preference strength and the learned scaling factors, and relationship between preference strength and the learned reward difference. All plots are from the Ant task.
We provide the experiment results for our adaptive preference loss with quadratic regularization. Here, name the method "Ada-Pref-Quad" and the one applied DPO as "Ada-DPO-Quse" Quad". Table 6 and Figure 8 show the results for Pref and Ada-Pref-Quaal on the Ant task, and DPO and Ada-DPO-Quee UPO-Quad on the single-turn dialogue. In Table 6, we report the performance of the best policy and the preference prediction accuracy of the corresponding reward function. From Table 6, we can see that Ada-Pref- -uad outperforms Pref on Ant task, and Ada-DPO-Quad surpasses DPO on the single-turn dialogue in terms of return and win rate, respectively. Figures 8(a) and 8(b) present the learning curve and the percentile plot for the task. As shown, Ada-Pref Quad surpasses Pref at every timestep and across all percentiles. Figure 8(c) shows histogram of the learned scaling
17

---

factors т. Compared to Figure 5(a), we can see much smoother distribution of I due to the quadratic regularization. Figures 8(d) and 8(e) illustrate the relationship between the learned scaling factors and preference strength, and the relationship between the learned reward difference and preference strength. As can be seen, the learned scaling factor for Ada-Pref-Quah increases monotonically with preference strength, indicating that the quadratic regularization maintains the adaptabilityy of loss scaling to the varying preference levels in the data. Moreover, Ada-Pref-Que reward differences for pairs with ambiguous preferences and learns larger reward differences for those with strong preferences. This demonstrates that ada-Pref-QQud also leads to a more flexible reward function compared to Pref.
learns smaller
Table 6: Table for the highest return (left) and the best win rate (right) of the best policy and the average preference prediction accuracy of the corresponding reward function.
Preference
Win Rate (%)
Preference Accuracy (%)
Task
Method
Return
Task
Method
Accuracy (%)
Pref Ada-Pref-Quad
2917.81
90.08
DPO
53.38 56.00
$4.39
Ant
Dialogue
3116.57
90.66
Ada-DPO-Ques
53.56
C.2 Ablation studies
In this subsection, we present the results for three PyBullet tasks, using a segment size of 25. Table 7 and Figure 9 show the performance of Pref and Ada-Pref on the PyBullet tasks, based on the first hyperparameter tuning criterion. Table 8 displays the results for Pref and Ada- Ada-Pref according to the second hyperparameter tuning criterion. These results reconfirm the effectiveness of our adaptive preference loss.
3540
1309
2000
2300
1233
Return
1000
Reture
3380
Return
500
Pref
253
Adia-Pret
Ade-Prei Pref
Ade-Pref
10
1/1
>.0
05
30
15
200
25
3.5
2.8
Timesteps (1e6)
Timesteps
Timesteps
3300
2000
2000
31.0
2500
Return
5000
Betum
2003
2200
Pref
Pref
Pref
Ado-Prof
2703
Ado-Pref
Ado-Pref
2000
Percentile
100
20
Percentile
60
50
300
20
Percentile
40
50
69
100
(a) HalfCheetah
(b) Ant
(c) Hopper
Figure 9: Learning curve plots (top) and percentile plots (bottom) for Pref and Ada-Pref. For the learning curve plots, returns each timestep are averaged across 10 different seeds, then smoothed over timesteps using an exponential moving average (EMA) with smoothing factor of a = 0.1. For the percentile plots, returns from 10 different seeds are sorted in ascending order.
C.3 Discussions on hyperparameter tuning
Compared to the cross-entropy loss, our method needs three additional hyperparameters: the bounds on the scaling factors To and mmax, and the regularization parameter p. In our experiments, we fixed Tu at 0.1 without tuning it, as this value worked well for all five tasks. We did tune Tmax to adjust the scale of 7, but this can be avoided by using the quadratic regularization formulation described in Section 3.6. The parameter p turns out to more important, because controls the distribution of the scaling factors. We performed a careful grid search to tune in our experiments. Figure 10 shows the hyperparameter sensitivity of p on the Ant and summarization tasks. Overall, we found that smaller values of p often lead to better performance.
18

---

Table 7: Table for the highest return of the best policy and the average preference prediction ac- curacy of the corresponding reward function.
Table 8: Table for the average preference predic- tion accuracy of the best reward function and the highest return of the corresponding policy.
Preference Accuracy (%)
Preference Accuracy
Task
Method
Return
Task
Method
Return
Pref
2575.69
90.82
Pref
2564.49
91.38
HalfCheetah
HalfCheetah
Ada-Pref
2689.9
90.35
Ada-Pref
2609.03
90.79
Pref 2832.87
84.88
Pref 2738.4
86.21
Ant
Ant
Ada-Pref
2960.47
84.09
Ada-Pref
2917.22
85.15
Pref 883,49
85.0
Pref
796.52
85.79
Hopper
Hopper
Ada-Pref
1025.74
85.15
Ada-Pref
1025.74
85.15
3200
94
Rewards
Preference Ace.
rate Accuracy
Preference Acc.
Accuracy
92
50
-70
Return
win rate
3100
90
40
-68
3000
30
86
20
-0.6
-0.4
-0.2
-0.6
6
-0.2
Figure 10: Hyperparameter sensitivity of p.
NeurIPS Paper Checklist
Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
Contributions and scope are explained in the methodology section and well-supported by results presented in section 4.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [NA]
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [Yes]
Justification: The full derivation of the proposed loss and the proofs for the propositions provided Appendix A.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: The proposed algorithm is summarized in Appendix B and the experiment details are included in Section and Appendix
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
19

---

Answer: [No]
Justification: We will release the code after the submission deadline.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: The basic experiment settings are included in Section and more implementa- tion details are in Appendix C.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: Error bars are included in the robotic control experiments to indicate variability, Additionally, percentile plots are provided to support the validity of the experimental results. See Figures 2, 7, and 8 for reference. Only one seed is used for natural language tasks due computational concerns.
8. Experiments Compute Resources
Question:
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Answer: [Yes]
Justification: See implementation details in Appendix C.
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics https: //t::///neeue htt///nurrppp wwwerrrpp... wwwpeewwww.pppppppeeeeeeeeeepeee htcc/public/Bthicucciiieeeeeeeee
Answer: [Yes]
Justification: This research adheres to the NeurlPS Code of Ethics.
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: This paper presents work whose goal is advance the field of Machine Learning. There are many potential societal consequences of our work, none of which we feel must be specifically highlighted.
11. Safeguards
Question:
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [NA]
Justification: This paper does not release data or models.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: The used models and data properly credited in experimental settings.
13. New Assets
20

---

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: This paper does not release new assets.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: This paper does involve crowdsourcing nor research with human subjects. 15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: This paper does not involve crowdsourcing nor research with human subjects.
21


Reviews:
[{"review_id": "eRzEydYfEH", "reviewer": "Reviewer_RUFd", "summary": "The paper focuses on redesigning the loss function with adaptive scaling parameters to deal with the uncertainty in the preferences and thus improving the reward modeling flexibility.  In the context of both robotics and NLP, the algorithm with the new loss shows improved performance.", "questions": "1. Can the authors provide more concrete mathematical justifications regarding the issue with linear depedence with the logit of the preference distribution? Specifically, connecting with the relation to noisy preferences?\n2. \"For some applications, it may lead to a reward function that is not flexible enough to differentiate a\n141 pair of segments, which are supposed to have significantly different rewards.\" Can the authors please highlight such examples intuitively or mathematically, why such a collapse might happen?\n3. The authors mention \"Our proposed method is inspired by DRO, it serves a distinct purpose: improving reward learning in RLHF, which is orthogonal to distributional robustness\". Can the authors pls justify this statement and discuss why it's orthogonal? DRO in reward learning will also improve the robustness and flexibility of the reward model to noisy preferences, in what sense it does better than DRO?\n4. Can the authors provide a more concrete definition of the best win rate and best prediction accuracy?\n5. It will be helpful if the authors can provide more details on how the trajectories are constructed for the robotics experiments.", "limitations": "Check above", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. The proposed loss is flexible and can be incorporated in majority of the current RLHF frameworks\n2. The objective has strong connections to DRO which is critical since preferences will be noisy and sub-optimal in practical scenarios.", "weaknesses": "1. The experimental ablation is weak and doesn't provide concrete indications of the strength of the proposed algorithm. For ex in Figure 3, the win-rate gap is marginal and not significant. Why is the performance in summarization better in Figure 4 is not extremely clear?\n\n2. The experimental evaluations lack important benchmarks and comparisons. For the Robotics environment, it's crucial to compare with Pebble, SURF, and PARL to understand the true efficacy of the proposed approach.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are grateful for the valuable feedback that you have provided! Please see our response per each of your concerns below:\n\n**W1: Weak experimental ablation. Why is the performance in summarization better in Figure 4 is not extremely clear?**\n\nThe response to weak experimental ablation is included in the global rebuttal. In Figure 4, we select the best model based on the preference prediction accuracy and the improvement indicates that our method better aligns the learned reward function and policy optimization, while DPO baseline results in a model with high preference accuracy but low win rate. The gap may be due to that our Ada-DPO is more robust to overfitting, and that the learned reward is more flexible and better guide the policy optimization.\n\n**W2: The experimental evaluations lack important benchmarks and comparisons. For the Robotics environment, it's crucial to compare with Pebble, SURF, and PARL to understand the true efficacy of the proposed approach.**\n\nThe proposed method is orthogonal to PEBBLE [1], SURF [2], or PARL [3] due to the distinct focus of our work. Specifically: \n\n* **PEBBLE** aims to enhance both sample and feedback efficiency by integrating unsupervised pre-training with off-policy RL.\n* **SURF** focuses on improving feedback efficiency by utilizing unlabeled data for reward learning.\n* **PARL** tackles the issue of the alignment objective's dependence on data generated by the optimal policy by introducing a bilevel formulation.\n\nIn contrast, our method focuses on enhancing reward model training by considering the varying preference strength between each sample pair. This approach is distinct from the goals of improving feedback efficiency or resolving the entanglement between the alignment objective and the trajectory collecting policy.\n\nOur proposed loss function can be readily adapted to enhance the performance of these methods (PEBBLE, SURF, and PARL), serving as a complementary approach rather than a direct comparison. As discussed in Remark 3.2 of the paper, Eq. (4) in PEBBLE [1], Eq. (3) in SURF [2], and Eq. (5) in PARL [3] show that these frameworks still optimize the standard cross-entropy loss for reward learning. Our proposed adaptive loss function can replace this standard cross-entropy loss to potentially enhance their performance.\n\n**Q1: Can the authors provide more concrete mathematical justifications regarding the issue with linear dependence with the logit of the preference distribution? Specifically, connecting with the relation to noisy preferences?**\n    \nWe want to clarify that the issue of the linear dependence of the BT model is not directly related to noisy preferences. Rather, it limits the flexibility of the learned reward difference, preventing the reward function from providing a wider range of rewards in downstream policy optimization. Our proposed method incorporates scaling factors to achieve a non-linear relationship, allowing the learned reward difference to be more flexible.\n        \nWe provide a mathematical analysis in Section 3, demonstrating that our method increases the complexity of the reward model compared to the baseline. Specifically, we show that our method learns a smaller reward difference for ambiguous preference data and a larger reward difference for strong preference data. The experimental results further prove that this improved flexibility translates to better performance. We believe this sufficiently demonstrates the linear dependence is not flexible enough and hence can be suboptimal.\n\n**Q2**: Included in global rebuttal.\n\n**Q3: Can the authors pls justify the statement and discuss why the proposed method is orthogonal to DRO?**\n    \nWhile our proposed method borrows the technical concept from DRO, the aims are fundamentally different. DRO focuses on improving robustness against data distribution shifts between training and test sets, which does not necessarily translate to increased flexibility of the reward model that we want to address. In contrast, our method aims to improve flexibility by incorporating scaling parameters to help reward learning, but not targeting robustness against data distribution or noisy preferences. This distinct focus is why we consider our method orthogonal to DRO. More technical differences are discussed in Line 108.\n\n**Q4: Can the authors provide a more concrete definition of the best win rate and best prediction accuracy?**\n\nWe use two different criteria to select the model among all those trained with different hyperparameter configurations:\n\n* Best Win Rate: We identify the best policy based on its end performance, specifically the policy with the highest win rate, and then report the win rate and the preference prediction accuracy of the corresponding reward function (Figure 3).\n* Best Prediction Accuracy: We identify the best policy as the one whose corresponding reward function achieves the highest preference prediction accuracy (Figure 4). We present this results because win rate evaluation involves costly proprietary models and tuning based on the preference accuracy could be much more efficient in practice.\n\n**Q5: It will be helpful if the authors can provide more details on how the trajectories are constructed for the robotics experiments.**\n\nThank you for pointing that out. Due to the wide variety of experiments and the numerous details about the experimental settings, we were unable to include all the detailed explanations in the main text. However, a brief explanation of how the trajectories are constructed for the robotics experiments is provided in Appendix C.1.\n\n### Reference\n[1] Pebble: Feedback-efficient interactive reinforcement learning via relabeling experience and unsupervised pre-training.\n\n[2] SURF: Semi-supervised reward learning with data augmentation for feedback-efficient preference-based reinforcement learning.\n\n[3] PARL: A unified framework for policy alignment in reinforcement learning", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5q8xrq3csS", "reviewer": "Reviewer_E9fU", "summary": "The paper identifies a limitation in RLHF methods, noting that ranking over pairs of trajectory segments often fails to capture the varying strengths of preferences across different pairs. To address this, the paper proposes a new adaptive preference loss (Ada-DPO), underpinned by distributionally robust optimization (DRO). The proposed method improves policy performance and is supported by theoretical proofs on convexity and univariate analysis.", "questions": "- I would like to see more examples like Figure 7, comparing the learned reward differences with different scaling factors.", "limitations": "- More discussion on the limitations of the proposed method and future directions can be added.", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper identifies a key limitation in previous RLHF methods and provides a clear theoretical analysis of its methods and claims. \n- It presents robust experiments and quantitative analysis on both robotic controls and natural language generation tasks.", "weaknesses": "- It seems that only a single run was conducted for the experiments in the NLP tasks. More repeated runs would be more convincing, especially since the differences in performance are so close. Additionally, p-value testing for that domain would help determine if the differences are truly significant.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank you for appreciating the feature of the proposed method and are grateful for the constructive comments! In the following, your comments are first stated and then followed by our point-by-point responses.\n\n**W1: It seems that only a single run was conducted for the experiments in the NLP tasks. More repeated runs would be more convincing.**\n\nThank you for pointing that out. We have included additional results and analysis for more runs in point 1 of the global rebuttal. We can observe from the table that our method consistently achieves stable and significant improvements over the baselines in both tasks. Given the large margin of improvement, we believe p-value is not necessary.\n\n**Q1: I would like to see more examples like Figure 7, comparing the learned reward differences with different scaling factors.**\n\n We provide three examples with different learned scaling factors in the summarization task below. We do not include the complete prompt here for better readability and space limit.\n\n----\n\n### Example 1:\n\n* **Original text**: I need some advice. I've been talking with this girl for about 2 weeks now. We went out last weekend and it went great. We were working on setting up another date and she told me that she was concerned about distance ...\n\n* **Chosen summary**:  agreed to meetup for coffee but haven't heard from her since tuesday night. Want to know what i can do to make it happen again.\n\n* **Rejected summary**:  agreed to meetup but haven't heard from since. Tried texting a couple times just trying to understand whats going on.\n\nThis sample pair gets $\\tau=0.325$ and the learned reward difference is 0.3071(Ada-DPO) vs 0.6419 (DPO). We can observe that the two responses in this example is similar, with the chosen summary being slightly more specific. Our method learn a small scaling factor and smaller reward difference since the gap between two responses is not that significant.\n\n----\n### Example 2:\n    \n* **Original text**: I'm not sure if I can even do anything, and if the person in question wasn't an ...\n\n* **Chosen summary**:  store franchise owner is probably stealing from their store by under ordering groceries/charging less than customer pays. possibly unethical behavior by owner of store? not sure how to proceed/act. help pls reddit.\n\n* **Rejected summary**:  store franchise owner probably is hiding their grocery bill from the rest of the store staff and is getting some kind of unethical benefit out of it, not sure if I can do anything. Advice?\n\nThis sample pair gets $\\tau=0.93$ and the learned reward difference is 0.8786(Ada-DPO) vs 1.066 (DPO). This example present a reasonable difference, with Chosen being more correct and better summarize the text. Our method and the DPO baseline get a similar reward difference and the scaling factor is around 1.\n \n-----\n### Example 3:\n* **Original text**: So my dream is do stand up comedy, improv comedy, writing and/or sketch comedy full time...\n\n* **Chosen summary**:  I want to pursue stand up comedy full time but I am afraid of losing my brothers rent money and my family. Do I follow my dreams or play it safe? Any advice/criticism is greatly appreciated!\n\n* **Rejected summary**:  Follow your dreams or play it safe?\n\nThis sample pair gets $\\tau=3.00$ and the learned reward difference is 2.832(Ada-DPO) vs 1.467 (DPO). Our method learn a large scaling factor and a larger reward difference for the last sample. As we can see from the two responses, the preference is obvious, with the rejected summary do not contain important information.\n\n------\n\n\n**Q2: More discussion on the limitations of the proposed method and future directions can be added.**\n\nResponse to this concern is included in point 4 of the global rebuttal.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "iLSThSeZTO", "reviewer": "Reviewer_ZDuk", "summary": "This paper studies the problem of learning from preference data and introduces a learnable scaling parameter for each preference sample. The authors propose an adaptive preference loss function that assign small scaling parameters to ambiguous preferences pairs and large scaling parameters to clear preferences. Experiments demonstrate the improvement of policy optimization performance and efficient of the hyperparameter tuning.", "questions": "I do not believe the proposed loss function is convex, can you provide the proof of Remark 3.1?\nWhy the reward models trained with the proposed method can better guide policy model selection considering the reward model performance is not improved?\n\nHow do the authors obtain the true reward difference in Section 4.3?", "limitations": "The computation overhead introduced by learning the scaling parameter is not discussed.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Considering different samples have different preference strength, using an adaptive preference scaling makes sense for preference learning.\n\nExperiments are conducted on both robotic control and natural language generation tasks.", "weaknesses": "The increase in flexibility of the proposed reward function is not verified. The authors claim that one of the limitations of the BT model is the logit of the preference distribution scales linearly with the reward difference. However, according to Proposition 3.1, the limitation still exists. There is no empirical result that supports the claim.\t\n\nMissing the ablation study of the regularization term in the proposed loss function.\n\nMissing the RLHF baseline in the natural language generation task. \n\n“Ada-Pref demonstrates greater resistance to performance degradation than Pref, indicating its superior ability to align the learned reward function with policy optimization.” It seems that it is not true in the Hopper task?\n\nMinors: \nLine 120: The notation of state transition function is conflicted with preference distribution.\nPlease clarify the meaning of M and K in Algorithm 1.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful comments! We sincerely appreciate your time in reading the paper, and our point-to-point responses to your comments are given below.\n\n**W1: The increase in flexibility of the proposed reward function is not verified. The authors claim that one of the limitations of the BT model is the logit of the preference distribution scales linearly with the reward difference. However, according to Proposition 3.1, the limitation still exists. There is no empirical result that supports the claim.**\n\nThank you for the suggestion. We provide additional clarification to better demonstrate the increased flexibility of the proposed method:\n* We would like to clarify that the proposed method no longer maintains a restrictive linear relationship between the logit of the preference distribution and the reward difference. In Proposition 3.1, we show that the relationship between the optimal reward difference and the logit of the true preference distribution varies for strong preference pairs and ambiguous pairs, resulting in a more flexible non-linear relationship. Additionally, during training, the learned logit is often not optimal, meaning the scaling factor is not strictly on the bounds. This is illustrated in Figures 5a and 6, where the scaling factor in the middle can create a more complex correspondence. In contrast, the BT model always maintains the same linear relationship between the learned logit and the learned reward difference during training, restricting flexibility.\n    \n* Directly measuring the flexibility of the reward function empirically can be challenging since it's difficult to quantify. To better support our claim, we provide some supporting empirical evidence in the paper. For instance:\n    * Figure 5(c) shows that Ada-Pref learns smaller reward differences for pairs with ambiguous preferences and larger reward differences for pairs with strong preferences, indicating a wider range of learned reward differences.\n    * Figure 7 provides examples showing that Ada-DPO learns larger reward differences than DPO for clear pairs and smaller reward differences than DPO for ambiguous pairs. We also include more examples in the response to the second reviewer.\n\n**W2: Missing the ablation study of the regularization term in the proposed loss function.**\n\nThe ablation study of the regularization term $\\rho$ in the proposed loss function has been provided in Appendix D.3 due to space constraints. If we remove the regularization term, the proposed loss collapses to the standard DPO loss with a modified $\\beta' = \\beta \\cdot \\tau_0$, which performs significantly worse than the reported DPO results. Consequently, we did not include it in the main text. For reference, removing the regularization term achieves a win rate of 24.56 on the summarization task.\n\n**W3: Missing the RLHF baseline in the natural language generation task.**\n\nThank you for pointing out the absence of an RLHF baseline in the natural language generation task. We understand the importance of experimenting with PPO instead of DPO to fully comprehend the impact of our method on RM's signal. Due to resource constraints, we prioritized experiments requiring PPO for robotic control tasks. Conducting PPO experiments for natural language generation tasks presents significant computational challenges, including the need to store and manage both reward and critique models (which are LLMs). PPO also involves more training steps and numerous hyperparameter tunings, making it difficult to implement within a resource-limited environment. Therefore, we opted for DPO as an alternative in our submission.\n    \nDue to the limited resources and the need to run other experiments, we are currently unable to provide PPO results for natural language generation tasks. Nonetheless, we will try our best to update the results before the discussion deadline.\n\n**W4: It seems that Ada-Pref's greater resistance to performance degradation is not true in the Hopper task?**\n\nThis is true in the Hopper task as well. Comparing Tables 1 and 2, we can see that Ada-Pref drops by 32.9\\% in the Hopper task, while Pref drops by 43.7\\%.\n\n**Q1-A: I do not believe the proposed loss function is convex, can you provide the proof of Remark 3.1?**\n\nThe proposed loss function is strictly convex with respect to each $\\tau_i$ when $r(z_w) \\neq r(z_l)$. For simplicity, we omit the index $i$ from $\\tau_i$ and set $C = r(z_w) - r(z_l)$. Then the instance-level loss function is written as:\n$$f(C, \\tau) = -\\tau \\log \\sigma(C/\\tau) + \\rho\\tau. $$\nSince $\\frac{\\partial f}{\\partial \\tau} = \\frac{C^2}{\\tau^3} \\sigma'(C/\\tau) > 0$ for all $\\tau \\in [\\tau_0, \\tau_{\\mathrm{max}}]$, $f$ is strictly convex with respect to $\\tau$.\n\n**Q1-B: Why the reward models trained with the proposed method can better guide policy model selection considering the reward model performance is not improved?**\n\nWe cannot directly evaluate the performance of the reward model solely based on preference prediction accuracy. This is because preference prediction accuracy does not fully capture the effectiveness of the reward model in the context of policy optimization. It only reflect the sign of reward difference but not the scale. Our reward function is designed to be more flexible, providing distinct rewards for clear pairs and comparable rewards for ambiguous pairs. This flexibility allows the reward function to generate a wider range of rewards, which is crucial for effective downstream policy optimization.\n\n\n**Q2: How do the authors obtain the true reward difference in Section 4.3?**\n\nPyBullet environments provide the ground truth reward for each pair of state and action, which are then used to obtain the true reward difference in Section 4.3.\n\n**Q3: The computation overhead introduced by learning the scaling parameter is not discussed.**\nResponse to this concern is included in point 3 in the global rebuttal.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "wmmQLgclcB", "reviewer": "Reviewer_4UPH", "summary": "To learn a versatile rewards essential for the downstream policy optimization, this paper introduces a novel adaptive preference loss function inspired by distributionally robust optimization (DRO).\nThe proposed approach incorporates an learnable instance-specific scaling factor to accommodate varying uncertainties of preference strength and change the scaling between the preference distribution and reward difference to be non-linear.\nThe proposed loss function for learning the scaling parameters is strictly convex, and thus they only induce negligible additional computational overhead.\nExperiment results on robotic control tasks and LLMs verify the proposed method.", "questions": "1. L71 \"Prior work on this topic is very limited.\": There are actually a vast pool of literature concerning loss functions for (RLHF's) reward modeling, a quick pointer is [1] for language tasks and [2]  for image generation, as well as the literature review in them. I encourage the authors to add proper discussions and citations to make the claims more precise.\n2. L140-141: Could you elaborate this sentence more: \"For some applications, it may lead to a reward function that is not flexible enough to differentiate a pair of segments\"? In particular, why is the resulted reward function not flexible enough? And how is this related to the linear scaling?\n3. In Eq. (3), why do we need to have $KL(p, 1/2)$ in both the objective function and constraint?\n\n\n\n***\n[1] Yang, Shentao, et al. \"Preference-grounded token-level guidance for language model fine-tuning.\" Advances in Neural Information Processing Systems 36 (2023).\n\n[2] Yang, Shentao, Tianqi Chen, and Mingyuan Zhou. \"A Dense Reward View on Aligning Text-to-Image Diffusion with Preference.\" Forty-first International Conference on Machine Learning. 2024.", "limitations": "A discussion on limitations and/or potential negative societal impact seems missing.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. Adapting the optimization strength by factoring in the characteristics of each preference pair is an interesting direction.\n2. The proposed method is well (theoretically) motivated.\n3. The proposed method work well in practice.", "weaknesses": "1. The adaptive scaling is defined on a per-instance basis, which necessitates significant compute costs and hinders real-world mini-batch learning. This is evident from Appendix, where the algorithm box shows that the training process operates in a per-datapoint manner\n2. Some of the derivation may be simplified, both to make the paper more succinct and leave room for algorithm box as well as other technical details.\n3. L221: could you theoretically justify the incorporation of adaptive preference scaling into DPO. e.g., in the framework of KL-control as in the original DPO paper?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank you for your constructive comments! In the following, your comments are first started and then followed by our point-by-point responses.\n\n**W1: The adaptive scaling is defined on a per-instance basis, which necessitates significant compute costs and hinders real-world mini-batch learning.** \n\nResponse to this concern is included in point 3 in the global rebuttal. The additional cost of our proposed method is negligible and it does not hinder mini-batch learning.\n\n**W2: Some of the derivation may be simplified, both to make the paper more succinct and leave room for algorithm box as well as other technical details.**\n\nThank you for the suggestion. We will simplify some duplicate derivation in section 3 and include the algorithm box and technical details, such as a discussion on additional computational costs, in the main page.\n\n**W3: could you theoretically justify the incorporation of adaptive preference scaling into DPO. e.g., in the framework of KL-control as in the original DPO paper?**\n\nDPO start from the policy optimization objective:\n$$\\\\max\\_\\\\pi \\\\mathbb{E}\\_{s,a\\\\sim\\\\pi (a|s)}[r(s,a)] - \\\\beta D\\_{\\\\mathrm{KL}}[\\\\pi(a|s), \\\\pi\\_{\\\\mathrm{ref}(a|s)}]$$\nand make use of the optimality condition given that the horizon is 1: \n$\\pi_r(a|s) = \\frac{1}{Z(s)} \\pi_{\\mathrm{ref}}(a|s) \\exp \\left( \\frac{1}{\\beta}r(s,a) \\right).$\nThen they reparameterizes the reward function as $r(s,a)=\\beta\\log(\\pi_r(a|s)/\\pi_{\\mathrm{ref}}(a|s))+\\beta\\log Z(s)$ and employs the standard cross-entropy loss for reward learning.\nOur proposed Ada-DPO retains the same reparametrization and only modifies the reward learning loss to Equation (6). The derivation and analysis for this change in the reward loss are included in Sections 3.2 and 3.3.\n\nWe assume the reviewer is suggesting moving the adaptive scaling factors to the KL-constrained policy optimization objective. Unfortunately, we found this difficult due to the regularization on scaling factor. However, at a high level, as mentioned in Line 222, we can merge the regularization parameter $\\beta$, which controls the KL term, with the scaling factors in the final objective. Thus, our method can be viewed as adapting a different KL-control term for each preference pair. For strong preference data, we learn a large scaling factor, corresponding to a smaller KL-control term, allowing the model to deviate more from the reference policy. In contrast, a larger KL-control term is used for ambiguous preference data.\n\n**Q1: \"Prior work on this topic is very limited.\": There are actually a vast pool of literature concerning loss functions for (RLHF's) reward modeling, a quick pointer is [1] for language tasks and [2] for image generation, as well as the literature review in them. I encourage the authors to add proper discussions and citations to make the claims more precise.**\n   \nThank you for pointing out the related works. We will include discussions and citations of these references in our next version. After carefully reviewing the mentioned works, we find that both focus on changing the reward function, but the used loss is still based on cross-entropy. This makes them complementary to our work, as we are improving the loss function. Specifically:\n\n[1] introduces a token-level preference reward learning loss instead of the standard sequence-level objective, addressing the granularity mismatch between preferences and LM training losses. Their approach employs a more fine-grained reward function. However, the cross-entropy loss is still employed when there are two responses, which means their method can potentially be combined with ours.\n\n[2] focuses on the preference alignment of text-to-image diffusion models and proposes temporal discounting DPO-style objectives that consider the sequential nature of the generation process. However, their method is specific to diffusion models and cannot be applied to general RLHF as ours can. Additionally, the used loss is still based on cross-entropy.\n\nIn summary, we consider these two works to be orthogonal and complementary to our work.\n\n\n**Q2: Could you elaborate this sentence more: \"For some applications, it may lead to...\"** \n\nResponse to this concern is included in point 2 in the global rebuttal.\n\n**Q3: In Eq. (3), why do we need to have $\\mathrm{KL}(p,1/2)$ in both the objective function and constraint?**\n\nIncluding the KL term in both the objective and the constraints is a common practice in constrained distributionally robust optimization (DRO) problems, as shown in Eq. (1) in [3]. Without the KL constraints, the loss becomes the cross entropy loss with temperature $\\tau_0$, making the loss not be able to adapt to each pair of samples. Without the KL term in objective, the objective will not be smooth and hard to optimize. \n \n\n**L1: A discussion on limitations and/or potential negative societal impact seems missing.**\n\n Response to this concern is included in point 4 in the global rebuttal.\n\n### References\n\n[1] Yang, Shentao, et al. 'Preference-grounded token-level guidance for language model fine-tuning.' Advances in Neural Information Processing Systems 36 (2023).\n\n[2] Yang, Shentao, Tianqi Chen, and Mingyuan Zhou. 'A Dense Reward View on Aligning Text-to-Image Diffusion with Preference.' Forty-first International Conference on Machine Learning. 2024.\n\n[3] Qi Qi, Jiameng Lyu, Kung sik Chan, Er Wei Bai, Tianbao Yang. Stochastic Constrained DRO with a Complexity Independent of Sample Size. Transactions on Machine Learning Research, 2023.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We would like to thank all the reviewers for the valuable feedback! Before we answer to each of the reviewers individually, we list and address common concerns below:

>  **1. The experimental ablation doesn't provide concrete indications of the strength of the proposed algorithm. Only a single run was conducted for the experiments in the NLP tasks. More repeated runs would be more convincing.**

We would like to highlight that the proposed method achieves non-marginal improvement across multiple tasks. It outperforms DPO baseline by 9-25% percent in terms of return on Ant and Hopper task (Table 1), and achieve 11% higher win rate than DPO on Dialogue task (Figure 3). While the improvement on HalfCheetah and summarization is not as large, they demonstrate that our method is consistently better and effective.

Furthermore, the smaller improvement on summarization could be due to Claude 2 not being a strong enough judge for evaluation. We conduct new experiments in NLP tasks replacing Claude 2 with Claude 3, a stronger judge, and use three different seeds. The results are presented in Table 1 in the attached file, showing that our method significantly improves over DPO and is stable across seeds.

> **2. Clarification on sentence: "For some applications, it may lead to a reward function that is not flexible enough to differentiate a pair of segments...”**

Human preferences are often influenced by numerous factors that interact in non-linear ways, making the BT model suboptimal as a reward model. For example, when the reward difference is small, even slight changes in certain features might lead to significant shifts in preference. The BT model may struggle to capture such rapid shifts due to its slower transition. Conversely, when the reward difference is already large, its effect on the preference distribution could be marginal, meaning that a small increase in preference probability corresponds to a large increase in the reward difference. The BT model, with its linear correspondence, will not learn a large enough reward difference for strong preference data due to its small gradient. Our method increases the gradient (as shown in Figure 1) in this case to more flexibly learn a wider range of rewards. We would like to emphasize that the scale of the reward difference is crucial, as it affects policy optimization, and a larger or smaller reward difference can benefit the learning of some downstream tasks.

> **3. The compute costs of the algorithm is not discussed and could be significant. It hinders real-world mini-batch learning.**

We would like to clarify that the additional cost of our proposed method is negligible, and it does not hinder mini-batch learning. Computationally, since the proposed loss is strictly **convex** and **univariate** in the scaling factor, the inner minimization problem (Lines 5-7 in Alg. 1) can be solved efficiently with a few iterations (k=5) of the Newton method. The additional computational cost of each update is **negligible** compared to the overall RLHF pipeline. Memory-wise, we need to temporarily store $\tau_i$ for each sample, which is minor considering the high-dimensional nature of the data, and we can free the memory after the backward update.

Regarding the mini-batch learning, although the algorithm is presented in a per-data manner to show how each $\tau_i$ is optimized, it can be directly adapted for mini-batch learning. In practice, we update all $\tau_i$ in the mini-batch in parallel. Our experiments also utilize mini-batch learning to ensure efficiency.


> **4. More discussion on the limitations of the proposed method, future directions and societal impact can be added.**

We provide our discussion below and will include these points in our next version: One main limitation of our proposed method is the introduction of three hyperparameters to control the scaling ($\tau_0, \tau_{\max}, \rho$), which induces additional tuning costs. In the paper, we propose an extension to the quadratic penalty in Section 3.6 to eliminate $\tau_{\max}$, but the performance of this extension is not as good. Developing more efficient tuning approaches or regularization techniques to reduce the need for these hyperparameters will be considered as future work. Additionally, extending our adaptive loss to handle ranking data with more than two responses for preference optimization is another potential future direction.

For societal impact, our method can be used to better align the LLM to user preferences, presenting some societal risks. Primarily, same as DPO, it may reinforce and amplify existing biases if the preferences and feedback used in training are skewed or prejudiced. Ethical concerns arise when the model aligns with harmful or unethical preferences.

Meta-review: The paper presents a novel adaptive preference loss for RLHF, integrating a distributionally robust optimization approach to handle varying strengths in human preferences. The reviewers generally appreciate the motivation behind the method and acknowledge its potential impact, especially in aligning rewards with policy optimization across robotic control and NLP tasks.

**Strengths:**
- The proposed method is theoretically sound, with robust experimentation across different domains.
- The flexibility introduced by the adaptive scaling parameters addresses a significant limitation in previous RLHF methods.

**Weaknesses:**
- Reviewers noted concerns about the weak experimental ablation, particularly in NLP tasks, where only a single run was initially conducted. The authors provided additional results with repeated runs, which addressed most concerns, but some reviewers remain unconvinced about the conclusiveness of these results.
- The lack of comparison with key benchmarks like PEBBLE, SURF, and PARL in the robotics domain was pointed out. The authors argue that their method is orthogonal to these benchmarks and could complement rather than directly compete with them.
- The computational overhead of the proposed method was discussed, with the authors asserting that the additional costs are negligible. However, the practical implications of this overhead in real-world settings remain a point of contention among reviewers.

**Unresolved Points:**
- Some reviewers were not entirely satisfied with the mathematical justifications provided regarding the linear dependence issue in preference modeling, indicating that the explanation could be clearer.
- The necessity of the proposed method versus simpler alternatives (e.g., reward margin methods) is still debated, with some reviewers questioning if the complexity of the method is justified by the performance gains.
- Concerns about the practical implementation and scalability of the method in real-world mini-batch learning scenarios were raised and only partially addressed by the authors.

**Recommendation:** 
The paper is borderline, with a slight leaning towards acceptance due to its innovative approach and potential impact. However, the unresolved issues, particularly regarding experimental robustness and practical scalability, suggest that further revisions and clarifications are necessary for a more confident endorsement.
Decision: Accept (poster)