Title: Adaptive Preference Scaling for Reinforcement Learning with Human Feedback
Venue: NeurIPS 2024
Accepted: yes
Keywords: Reinforcement Learning from Human Feedback

Large Language Models

Alignment
Abstract: Reinforcement learning from human feedback (RLHF) is a prevalent approach to align AI systems with human values by learning rewards from human preference data. Due to various reasons, however, such data typically takes the form of rankings over pairs of trajectory segments, which fails to capture the varying strengths of preferences across different pairs. In this paper, we propose a novel adaptive preference loss, underpinned by distributionally robust optimization (DRO), designed to address this uncertainty in preference strength. By incorporating an adaptive scaling parameter into the loss for each pair, our method increases the flexibility of the reward function. Specifically, it assigns small scaling parameters to pairs with ambiguous preferences, leading to more comparable rewards, and large scaling parameters to those with clear preferences for more distinct rewards. Computationally, our proposed loss function is strictly convex and univariate with respect to each scaling parameter, enabling its efficient optimization through a simple second-order algorithm. Our method is versatile and can be readily adapted to various preference optimization frameworks, including direct preference optimization (DPO). Our experiments with robotic control and natural language generation with large language models (LLMs) show that our method not only improves policy performance but also aligns reward function selection more closely with policy optimization, simplifying the hyperparameter tuning process.

Adaptive Preference Scaling for Reinforcement Learning with Human Feedback

Abstract

Reinforcement learning from human feedback (RLHF) is a prevalent approach to align AI systems with human values by learning rewards from human preference data. Due to various reasons, however, such data typically takes the form of rankings over pairs of trajectory segments, which fails to capture the varying strengths of preferences across different pairs. In this paper, we propose a novel adaptive preference loss, underpinned by distributionally robust optimization (DRO), designed to address this uncertainty in preference strength. By incorporating an adaptive scaling parameter into the loss for each pair, our method increases the flexibility of the reward function. Specifically, it assigns small scaling parameters to pairs with ambiguous preferences, leading to more comparable rewards, and large scaling parameters to those with clear preferences for more distinct rewards. Computationally, our proposed loss function is strictly convex and univariate with respect to each scaling parameter, enabling its efficient optimization through a simple second-order algorithm. Our method is versatile and can be readily adapted to various preference optimization frameworks, including direct preference optimization (DPO). Our experiments with robotic control and natural language generation with large language models (LLMs) show that our method not only improves policy performance but also aligns reward function selection more closely with policy optimization, simplifying the hyperparameter tuning process.

1 Introduction

In the field of artificial intelligence, aligning AI systems with human preferences has become increasingly crucial, particularly for applications involving complex data and models like large language models (LLMs) in natural language processing [38, 28]. Reinforcement learning from human feedback (RLHF) has gained popularity for customizing AI systems [12, 3, 47]. RLHF involves learning a reward function from human preference data, then using a reinforcement learning algorithm to train a policy to optimize the learned reward model.

A key challenge in RLHF lies in the complexity of reward modeling, which primarily stems from the reliance on preference labels. Since preference labels only provide comparative rankings of trajectory segments without quantifying the scale of underlying preference strengths, previous methods have employed the Bradley-Terry (BT) model [8] in conjunction with cross-entropy loss to learn the reward function from preference data [12, 38]. This approach assumes that the logit of the preference distribution scales linearly with the reward difference across all sample pairs. However, such linear scaling is often insufficient to account for the variations in preference strength among different pairs, restricting the reward function's ability to capture a broader range of reward differences. This restrictive approach to reward modeling limits the flexibility of the learned reward function, hindering its capacity to produce the versatile rewards essential for the downstream policy optimization.

To overcome this shortcoming, we introduce a novel adaptive preference loss function inspired by distributionally robust optimization (DRO) [16]. Our approach incorporates an instance-specific scaling factor to change the scaling between the preference distribution and the reward difference to be non-linear. These factors are learned during training and enable the model to accommodate varying uncertainties of preference strength, thereby enhancing the flexibility of the reward. For pairs showing strong preference (i.e., low preference uncertainty), our method learns a large scaling factor, which enables the model to learn a larger reward difference. In contrast, for pairs showing ambiguous preferences (i.e., high preference uncertainty), our method assigns a smaller scaling factor, enabling the model to learn a smaller reward difference. The additional computational overhead of involving this scaling factor into training is negligible, as the proposed loss function is strictly convex and univariate with respect to each scaling parameter. Therefore, it can be easily optimized by a simple second-order algorithm within a few iterations.

Our experiments on robotic control tasks [39] demonstrate that our method can learn a more flexible reward function, resulting in an improved policy. Surprisingly, we also discover that our method better aligns the learned reward function with downstream policy optimization. Specifically, when tuning hyperparameters for reward modeling, the simplest approach is to select the reward model according to preference prediction accuracy. However, the selected reward function (with the highest accuracy) often yields a downstream policy with poor performance. To address this misalignment, we usually have to jointly tune the parameters across both stages according to downstream policy performance, resulting in significant computational burden and tuning effort. Our proposed method can mitigate this misalignment: When using our adaptive loss, we can select the reward model based on preference prediction accuracy alone and yield a reasonably well-performing policy. This allows separate tuning of the two stages, easing tuning overhead. To our knowledge, the challenge of this misalignment issue is almost untouched in the RLHF literature, and we are the first to propose a principal approach to mitigate this issue.

Moreover, our method is generalizable and can be applied to other preference optimization algorithms. For instance, we implement it with direct preference optimization (DPO) [31] and evaluate its effectiveness on natural language generation tasks using Llama-2 7B [40]. Our results demonstrate that integrating adaptive preference scaling into DPO boosts policy performance, while preserving the benefits of alignment. Alignment is especially critical in this setting, where we employ proprietary models like Claude 3 [1] as judges for policy selection, which demands substantial costs for using the APIs. In the case without access to LLM assessment, we must select policy based solely on preference accuracy, under which our approach substantially outperforms other baselines.

2 Related works

Loss functions for reward learning. Prior work on this topic is very limited. For example, Song et al. [37] propose using different loss functions for strong and ambiguous preference data in natural language generation tasks. They apply heavy-tailed loss functions for open-ended questions, where preference ambiguity is desirable, and light-tailed loss functions for close-ended questions requiring clear-cut rewards. However, their approach requires knowing the question type a priori, necessitating extra labeling effort, and may fail for complex questions containing both open and closed aspects. Zhao et al. [47] propose using a hinge loss, which results in zero gradient when the learned reward difference exceeds a margin of 1. This limits the ability to learn very large differences in rewards. Azar et al. [2] develop Ψ Preference Optimization with Identity Mapping (IPO), which modifies DPO with a loss function matching the scaling of KL-divergence between the learned policy and the initial policy to avoid overfitting due to weak regularization. In contrast to prior work, our method is

more broadly applicable to complex preference learning tasks without needing additional labeling or sacrificing the ability to learn arbitrarily large reward differences.

Adaptive temperature scaling (ATS). Temperature scaling (TS) aims to adjust the entropy of probabilistic models by rescaling their logit outputs before the softmax function is applied. This simple method not only enables confidence calibration [19], but also plays a vital role in various machine learning methods, including knowledge distillation [20], reinforcement learning [25], and contrastive learning [43]. Building on TS, adaptive temperature scaling (ATS) enhances flexibility by using instance-specific scalars. Most ATS method trains an additional network for predicting the temperature parameter, which is further integrated into the softmax operator to calibrate the prediction probabilities [44, 14, 4, 21]. In contrast to the aforementioned ATS methods, the proposed adaptive preference scaling (APS) is not designed for classical confidence calibration, but is crafted specifically to enhance the training process of reward function in RLHF. Consequently, the interpretations of scaling factors in ATS and APS are opposite . In ATS, a larger scaling parameter is applied to data with higher uncertainty (e.g., data that the classifier is likely to misclassify), which reduces the magnitude of the corresponding logit. Conversely, in APS, a larger scaling factor is assigned to data with clearer preferences, resulting in a larger logit. This distinction clarifies why the scaling parameter in our approach does not correspond to the concept of 'temperature" from statistical physics. Additionally, we propose a principled framework for learning scaling parameter based on DRO, which avoids the complexities of designing specific temperature networks and does not rely on heuristically designed loss functions.

Distributionally robust optimization (DRO). DRO is a technique that trains machine learning models to be robust against uncertainty in the data distribution. Specifically, DRO finds a solution that performs well under the worst-case distribution within a specified uncertainty set around the empirical data distribution [5, 7, 23, 35, 16]. DRO has been applied in various AI/ML domains to improve generalization when the test distribution differs from the training distribution [27, 18, 26, 10, 45, 30]. Our framework is motivated by Qi et al. [30], which tackles KL-constrained DRO problem. However, our approach differs in two significant ways. First, instead of using a single KL constraint for the entire training dataset, we apply a separate KL constraint to each individual training data. Second, since each training data involves just two distributional variables, we can use a deterministic method to optimize these efficiently. Note that while our proposed method is inspired by DRO, it serves a distinct purpose: improving reward learning in RLHF, which is orthogonal to distributional robustness.

3 Method

In this section, we first outline the problem setup, derive the loss function with adaptive preference scaling, and then provide theoretical motivation for our proposed loss. At last, we present an optimization algorithm, extend the approach to direct preference optimization, and introduce a variant of our proposed loss that incorporates quadratic regularization.

3.1 Problem setup

We consider a reward-free Markov decision process M = ( S , A , p, γ ) with state s ∈ S , action a ∈ A , state transition function p , and discount factor γ . The ground truth reward function r : S × A → R is assumed to be unknown, but only human preferences over pairs of trajectory segments are observed. A trajectory segment is a sequence of consecutive state and action pairs z = { ( s m , a m ) , ( s m +1 , a m +1 ) , . . . , ( s k -1 , a k -1 ) } ∈ ( S × A ) k -m . We denote z 1 ≻ z 2 to indicate that the human preferred trajectory segment z 1 over the trajectory segment z 2 and denote the preferred one with a subscript w and the dispreferred one with a subscript l (i.e., z w and z l ). Here, we are given a human preference dataset of trajectory segments D pref = ( z w,i , z l,i ) N i =1 . Our goal is to find a reward ˆ r ( s, a ) , which is well-aligned with human preferences. Once we learn the reward, we then find a policy π ∈ ∆ S A such that it maximizes the expected sum of discounted rewards,

where ˆ r ( z ) = ∑ ( s t ,a t ) ∈ z γ t ˆ r ( s t , a t ) and D π denotes the stationary distribution of the state-action pair induced by π .

3.2 Reward learning with adaptive preference scaling

We now focus on the reward learning phase in RLHF, a crucial stage for capturing human preferences across various trajectory segments. The standard reward learning procedure assumes that the reward function determines a preference distribution, also known as the Bradley-Terry (BT) model [8],

where σ denotes the sigmoid function. The reward function is then learned by minimizing the expectation of negative log-likelihood of r over the preference data [12]:

As can be seen from (1), the BT model essentially assumes that the logit of the preference distribution σ -1 ( p r ( z w ≻ z l )) scales linearly with the reward difference, regardless of the specific pair of samples. Such linear scaling, however, may not align well with downstream policy learning. Human preferences are often influenced by numerous factors that interact in non-linear ways, making the BT model suboptimal as a reward model. For example, when the reward difference is small, even slight changes in certain features might lead to significant shifts in preference. The BT model may struggle to capture such rapid shifts due to its slower transition.

To address this challenge, we propose an adaptive preference loss based on KL-constrained distributionally robust optimization formulation [30], which can implicitly change the scaling between the logit and the reward difference to be non-linear. Specifically, given a pair of trajectory segments ( z 1 , z 2 ) , we denote d r ( z 1 , z 2 ) = 1 ( z 1 ≻ z 2 ) · ( r ( z 2 ) -r ( z 1 )) and p = ( p 1 , p 2 ) . We define the following instance-level loss:

where ∆ 2 = { p ∈ R 2 : p 1 + p 2 = 1 , 0 ≤ p 1 , p 2 ≤ 1 } , 1 / 2 is denoted for the uniform distribution, and ρ 0 , τ 0 > 0 are shared prespecified parameters across all instances. KL( · , · ) denotes the KL divergence. Note that without the KL-constraint, (3) is reduced to the cross-entropy loss with τ 0 = 1 . Unlike general KL-constrained DRO formulation, which considers a distribution p over all training samples, the distributional variable p in (3) is associated specifically with binary preference comparisons for each pair.

We then convert (3) into an equivalent minimax formulation based on the Lagrangian duality,

where λ is the Lagrange multiplier. By defining τ as τ = λ + τ 0 and applying the optimality condition for p , we have

where

We refer to Appendix A.1 for the full derivation. Note that the preference scaling factor τ in (4) and (5) serves as the Lagrange multiplier of (3). This scaling parameter τ is used specifically for training the reward function r , rather than calibrating the preference distribution p r,τ ( z w ≻ z l ) . The scaler τ is used exclusively during the reward learning phase and is no longer needed in subsequent policy optimization, where the reward function r alone is used.

Moreover, the scaling parameter τ is defined to be an instance-specific parameter corresponding to the pair of trajectory segments ( z w , z l ) . Therefore, when applying our adaptive loss to reward learning, for each pair ( z w,i , z l,i ) , we need to define a corresponding scaling parameter denoted by τ i . The overall loss function over the training set D pref is as follows:

where T = ( τ 1 , . . . , τ N ) , Ω = { τ : τ 0 ≤ τ ≤ τ max } with τ max as another prespecified parameter, and ρ = ρ 0 -log 2 > -log 2 . Here, we also involve an upper bound τ max > 0 in (6), and we will explain why it is needed in the next subsection.

3.3 Theoretical insights

We next provide some theoretical insights on why the scaling parameter τ can help gain adaptivity by a proposition. For simplicity, we only consider a pair of trajectories.

Proposition 3.1. Assume we have a pair of trajectories z 1 , z 2 , and the preference distribution p ( z 1 ≻ z 2 ) = p ∗ ∈ (0 , 1) , i.e., the probability, that z 1 is preferred over z 2 , is p ∗ . Consider the problem of minimizing the expectation of our adaptive loss function over the preference distribution:

Then the minimizer τ ∗ and r ∗ of the expected loss satisfy

Here, σ -1 is the inverse of sigmoid function.

Note that the expected loss (7) is only for easing theoretical analysis, as p ∗ is not accessible in practice. From Proposition 3.1, we can see that when p ∗ is close enough to 0 . 5 , (i.e., the uncertainty of preference is large), the corresponding optimal τ ∗ is at the lower bound τ 0 . The resulting optimal reward difference is τ 0 σ -1 ( p ∗ ) , which is smaller than the counterpart obtained by the cross-entropy loss when τ 0 < 1 . Conversely, when p ∗ is close to 0 or 1 , (i.e., the uncertainty of preference is small), the resulting optimal τ ∗ is at the upper bound τ max . Here, we introduce the upper bound τ max to ensure that the optimal τ ∗ is bounded. The resulting reward difference in this case is τ max σ -1 ( p ∗ ) , which is larger than the counterpart obtained by the cross-entropy loss when τ max > 1 . Our theoretical analysis suggests that the adaptive scaling factor essentially changes the correspondence between the logit of preference distribution and the reward difference for each pair of trajectory segments, which could lead to a more flexible reward model.

We further visualize our adaptive preference loss in Figure 1, setting τ 0 and τ max to 0 . 1 and 5 . 0 , respectively. As depicted, our adaptive preference loss behaves distinctly compared to the cross-entropy loss. With large learned reward differences, the cross-entropy tends to be very flat, while our loss maintains a non-trivial gradient, allowing us to continually decrease the loss function. In contrast, for small positive learned reward differences, our loss yields a smaller gradient, thereby

ρ

=-0.2

ρ

=-0.5

ρ

=-0.6

CE

Loss

-

2

0

2

4

6

Reward Difference

-

5

0

5

ρ

=-0.2

ρ

=-0.5

ρ

=-0.6

CE

Gradient

-

1.0

-

0.5

0

Reward Difference

-

10

-

5

0

5

10

less encouraging the reward model to further distinguish pairs of ambiguous trajectory segments. This is consistent with our theoretical analysis.

3.4 Algorithm

We present an efficient algorithm for solving (6). Suppose we parameterize r as a neural network with parameter ϕ . At the m -th iteration, we have the iterate ϕ ( m ) , and we sample a pair of trajectory segments z w,i and z l,i . We initialize τ (0) i = 1 and then optimize τ i by a projected Newton method subject to a simple interval constraint Ω [6]. Specifically, for k = 0 , ..., K -1 , we take

where ∆ ( k ) i denotes the descent direction

Once we get τ ( K ) i , we update ϕ by a stochastic gradient descent step

where η ϕ is the learning rate. We summarized our proposed algorithm in Algorithm 1, which is presented in a per-data manner for clarity but can be directly adapted for mini-batch learning.

Algorithm 1 Algorithm for reward learning with adaptive preference scaling

1: Input : τ 0 , τ max , ρ , η ϕ ; 2: for m = 0 , 1 , 2 , . . . , M -1 do 3: Sample a pair of trajectory segements from D pref ; 4: Set τ 0 i = 1 ; 5: for k = 0 , 1 , 2 , . . . , K -1 do 6: Compute ∆ ( k ) i using (9) and update τ ( k ) i using (8); 7: end for 8: Update ϕ ( m ) using (10) or Adam-style step; 9: end for

Remark 3.1. Note that since ℓ i ( ϕ, τ i ) is strictly convex and univariate with respect to τ i , in each iteration m , τ ( K ) i is guaranteed to be near-optimal (i.e., τ ( K ) i ≈ τ ⋆ i ). Therefore, the convergence of Algorithm 1 can be guaranteed by the convergence of stochastic gradient descent on the reward model parameter ϕ .

Remark 3.2. Computationally, Algorithm 1 incurs negligible additional cost. The inner minimization problem (Lines 5-7 in Algorithm 1) can be solved to near-optimality efficiently within a few iterations (e.g. K = 5 ) given its convex and univariate nature. The additional overhead of each update is minor compared to the overall RLHF pipeline.

3.5 Extension to direct preference optimization (DPO)

Our adaptive preference scaling approach is generic and can be extended to DPO [31], which is another popular method for policy learning from human preferences. DPO directly learns the policy in supervised manner using the preference data of state-action pairs D pref = ( s i , a w,i , a l,i ) N i =1 . This approach forgoes the need to learn the reward function explicitly by the reparameterization of reward function r with respect to its optimal policy π r ,

where Z ( s ) = ∑ a π ref ( a | s ) exp ( r ( s, a ) /β ) and π ref denotes the reference policy. By plugging in (11) back into (2), we have the policy optimization problem

where r π ( a | s ) = log( π ( a | s ) /π ref ( a | s )) denotes the log-probability ratio.

Similarly. we can integrate adaptive preference scaling into DPO by plugging in (11) into (6). By merging β with the τ i and ρ , we can further obtain the adaptive DPO (Ada-DPO) formulation as

Remark 3.3. Note that the proposed adaptive preference loss can be further combined with other RLHF approaches, such as PEBBLE [24], SURF [29], and PARL [11], which still optimize the standard cross-entropy loss (see [24, Eq. (4)], [29, Eq. (3)], and [11, Eq. (5)]).

3.6 Extension to quadratic regularization

We now introduce a variant of our adaptive preference loss that uses quadratic regularization for τ . This modification removes the need for the hyperparameter τ max in Ω , easing the tuning effort. We define the following instance-level adaptive preference loss with quadratic regularization:

Compared to (4), which includes a linear regularization term of ( ρ 0 -log 2) τ , (12) modifies the regularization term with coefficient ρ 0 to be quadratic while keeping the term log 2 τ linear. Additionally, in (12), the constraint on τ only specifies a lower bound τ 0 and no longer includes an upper bound τ max . The following proposition provides theoretical insights for this modification.

Proposition 3.2. Assume we have a pair of trajectories z 1 , z 2 , and the preference distribution p ( z 1 ≻ z 2 ) = p ∗ ∈ (0 , 1) . Consider the problem of minimizing the expectation of our adaptive loss function with quadratic regularization over the preference distribution:

0 Then the minimizer τ ∗ and r ∗ of the expected loss satisfy ⋆ ∗ ∗ ∗ ∗

Here, σ -1 is the inverse of sigmoid function.

Note that unlike the adaptive preference loss with linear regularization described in Proposition 3.1, the optimal value τ ⋆ for quadratic regularization does not involve the upper bound τ max .

4 Experiments

In this section, we examine the effectiveness of our adaptive preference loss based on robotic control and natural language generation tasks. Due to space limit, we defer the experiments with quadratic regularization, ablation studies, and discussions on hyperparameter selection to Appendix C.

4.1 Robotic control

Experiment setup. We apply our proposed reward learning method on 3 robotic control tasks from the PyBullet [13] environments: HalfCheetah , Ant , and Hopper . These environments are similar to those available in OpenAI Gym [9] but they are known to be much harder to solve [34]. Similarly to Gao et al. [17], our setting is synthetic, where we use the ground truth rewards to provide preference labels on each pair of samples due to high expense of collecting human preferences. For the reward function, we use two-hidden-layer MLPs, each containing 64 hidden units. This configuration is aligned with the designs of both the policy and value networks. Following Christiano et al. [12], we repeat the following three steps for each stage: (i) We sample a set of trajectories by the policy π , and update the policy with proximal policy optimization (PPO, Schulman et al. [36]) alongside a reward function ˆ r . (ii) We split the segments (the sequence of state-action pairs) into a training set and a testing set. Then, we randomly sample pairs of segments from the training set, and generate D pref with preference labels. We do the same to the testing set, and generate D ′ pref . (iii) We train the reward function ˆ r on D pref , and use D ′ pref for evaluating the preference prediction of ˆ r .

For notational simplicity, we name our proposed adaptive preference scaling method for reward learning as 'Ada-Pref'. We compare Ada-Pref with the baseline method 'Pref', which uses the standard cross-entropy loss for reward learning. For every 10000 timesteps the policy π runs, we evaluate the learned policy based on 20 test episodes. We also compute the average preference prediction accuracy of the learned reward function across stages. We set the budget to 3 million timesteps and perform training over 10 different seeds. For hyperparameter tuning in both reward learning and policy optimization, we apply two different criteria: 1) We identify the best policy based on its performance (the one with the highest return) and subsequently select the corresponding reward function. 2) We choose the best reward function based on its performance (the one with the highest average preference prediction accuracy) and then select the corresponding policy. Details of the implementations and hyperparameter tuning procedures are in Appendix B.1.

Results. We summarize the results on three PyBullet tasks as follows:

Table 1 and Figure 2 illustrate the results for Pref and Ada-Pref on the PyBullet tasks, based on the first hyperparameter tuning criterion. In Table 1, we report the highest return of the best policy and the average preference accuracy of the corresponding reward function. We can see that Ada-Pref consistently outperforms Pref in terms of return on all three tasks and achieves comparable preference accuracy. The upper panel of Figure 2 shows the learning curve plots. We can see that Ada-Pref surpasses Pref at nearly every timestep and reaches a higher plateau across all tasks. The lower panel of Figure 2 presents percentile plots from different seeds to demonstrate individual run behaviors. As shown, we confirm that Ada-Pref consistently outperforms Pref at every percentile across all tasks.

Table 2 presents the results for Pref and Ada-Pref based on the second hyperparameter tuning criterion. From Table 2, we can see that both methods show a decrease in performance compared to Table 1,

(a) HalfCheetah

(b) Ant

(c) Hopper

3500

3000

3000

2000

2000

2500

1500

Return

Return

1000

1500

1000

0

1000

Pref

500

Pref

500

Pref

-1000

Ada-Pref

Ada-Pref

Ada-Pref

0

0

0.0

0.5

1.0

1.5

2.02.5

3.0

0.0

0.5

1.0

1.5

2.0

2.5

3.0

0.0

0.5

1.0

1.5

2.0

2.5

3.0

Timesteps(1e6)

Timesteps(1e6)

Timesteps (1e6)

3600

2500

3200

3500

2000

3400

3000

3300

1500

Retur

Retur

3200

R

P1000

3100

2600

Pref

3000

Pref

500

Pref

Ada-Pref

2900

Ada-Pref

Ada-Pref

2400

0

20

40

60

80

100

2800

20

40

60

80

100

0

0

20

40

60

80

100

Percentile

Percentile

Percentile

while Ada-Pref still outperforms Pref in terms of both preference accuracy and return on all three tasks. Furthermore, Ada-Pref demonstrates greater resistance to performance degradation than Pref, indicating its superior ability to align the learned reward function with policy optimization. This alignment allows for effective policy selection based on preference accuracy without the need to evaluate the policy using ground truth rewards.

4.2 Natural language generation

Experiment setup. We apply DPO with our proposed adaptive loss (Ada-DPO) method to two open-ended text generation tasks: summarization and single-turn dialogue . We adopt the Llama-2 7B model [40] as the backbone and conduct instruction tuning on each task to obtain the initial reference models. For summarization, the policy generates summaries given posts collected from Reddit. We use the filtered TL;DR summarization dataset [41] for instruction tuning, which contains more than 117K Reddit posts, each with a human-written summary. We apply the human preferences collected by Stiennon et al. [38] for preference optimization, where each transcript contains a pair of responses along with a preference label. For single-turn dialogue, the policy responds to various human queries ranging from simple questions to complex demands. We utilize the Anthropic Helpful and Harmless dialogue preferences dataset [3] for both instruction tuning and preference optimization. This dataset contains 170k human-AI dialogues, with each dialogue containing two AI responses and a human preference label. We use the preferred responses for instruction tuning and the full set of preferences for optimization. For instruction tuning stage, we fine-tune the entire Llama-2 model. For the alignment stage using Ada-DPO and different baselines, we apply LoRA fine-tuning for computational efficiency concerns, as we need to simultaneously tune multiple hyperparameters.

The rank of the LoRA adaptor is 64 . We consider three baseline methods: DPO [31], Ψ Preference Optimization with Identity Mapping (IPO) [2] and Sequence Likelihood Calibration with Human Feedback (SLiC-HF) [47].

As human evaluation is prohibitively expensive, we use Claude 3 [1], a proprietary large language model, to automatically evaluate responses based on summary quality and helpfulness/harmlessness for the summarization and dialogue tasks, respectively. Prior work has shown that Claude 3 and GPT-4 can effectively measure a quantitative improvement over the instruction-tuned model [15]. We split a small subset from each instruction tuning dataset for testing and calculate the win rate against the instruction-tuned reference model as the evaluation metric. The percentage of instances where the response generated by policy A is preferred over policy B is referred to as the win rate of A against B. We also split a subset from each preference optimization dataset to validate the preference prediction accuracy. Details of the implementations and hyperparameter selections are in Appendix B.2.

Results. We summarize the results on the two natural language generation tasks as follows:

In Figure 3, we select the model with the highest win rate and present the win rate and its preference accuracy for all baselines. We observe that Ada-DPO outperforms the other baselines on both tasks in terms of win rate and achieves comparable preference accuracy. In Figure 4, we display the performance of the model selected with the highest accuracy (not win rate). As shown, Ada-DPO achieves a significant improvement beyond the DPO baseline in terms of win rate and obtains a comparable preference accuracy. This again indicates that Ada-DPO yields better alignment between the learned reward function and policy optimization, allowing good policy selection based on preference accuracy without a proprietary LLM judge.

Summarization

Dialogue

SLiC-HF

IPO

DPO

Ada-DPO

45

50

55

60

65

70

Summarization

Dialogue

SLiC-HF

IPO

DPO

Ada-DPO

50

60

70

80

4.3 Detailed analysis

We present detailed analyses of Ada-Pref and Ada-DPO for both the Ant and summarization tasks. Figure 5(a) presents a histogram of the learned scaling factors τ for the Ant task. We can see that around 60% of these scaling factors reach the upper bound, while about 10% converge to the lower bound, and the rest are distributed across the region. In Figure 5(b), we explore the relationship between preference strength and the learned scaling factors τ , and in Figure 5(c), we investigate the relationship between preference strength and the learned reward difference for Pref and Ada-Pref. We measure preference strength using the true reward difference, categorize it into five percentile bins, and then bin the scaling factors and the learned reward differences accordingly to compute the average. As can be seen, the learned scaling factor increases monotonically with preference strength, demonstrating that the our method successfully adapts the loss scaling to the varying degrees of preference in the data. Furthermore, Ada-Pref learns smaller reward differences for pairs with ambiguous preferences and learns larger reward differences for those with strong preferences, compared to Pref. This indicates that our method leads to a more flexible reward function.

In Figure 6(a), we plot a histogram of the learned scaling factors τ for the summarization task. We can see that around 40% of the scaling factors converge to the upper bound, with the rest distributed across the region. We also display the relationship between the confidence scores and the scaling factors in Figure 6(b). The confidence score is an integer from 1 to 4 included in the dataset, and a higher score denotes a stronger preference. We bin the scaling factors based on confidence scores and compute the average. As shown, the scaling factors positively correlate with confidence scores, justifying that we learn larger τ for strong preferences and smaller τ for ambiguous ones.

We further present two pairs of preference samples where Ada-DPO assigns large or small scaling factors in Figure 7. We observe that the sample pair with a large scaling factor shows a strong

Summarization

Dialogue

DPO

Ada-DPO

0

10

20

30

40

50

60

Summarization

Dialogue

DPO

Ada-DPO

0

20

40

60

80

(a) Histogram of

τ

(b) Preference strength and

τ

(c)

Preference

strength and

60

0.95

Difference

0.025

Pref

(%)

Percentage

50

40

Average

0.85

0.90

0.020

0.015

Ada-Pref

0.80

Reward

0.010

20

0.75

0.005

10

0.70

0

0.2

0.4

T

0.6

0.8

1.0

1

PreferenceStrength

2

3

4

5

0.000

1

PreferenceStrength

2

3

4

learned reward difference

Percentage (%)

0

20

40

τ

0

2

4

(a) Histogram of

τ

Average

τ

3.0

3.2

3.4

3.6

Confidence Score

1

2

3

4

(b) Confidence score and

τ

Sample pair with large 𝝉 ( 𝝉 = 4.0)

Chosen: 8 spoiled brat girls got stuck at the airport due to their immaturity and obliviousness, and I'd like to know how I can help them.

Rejected: Teenage girls, who we had no idea were teenage girls, were being obnoxiously obnoxiously obnoxiously obnoxiously obnoxiously obnoxiously obnoxiously obnoxiously

POST: This morning, I was walking between terminals in

the Minneapolis airport. A bunch of us got stuck behind

a group of 8 teenage girls slowly walking

…

Prompt

Sample pair with small 𝝉 ( 𝝉 = 0.1)

Chosen: I have bipolar 2, and recently my depression and moodiness is getting to my girlfriend. She is afraid to come over because she thinks im losing interest in her. How do I help her?

Rejected: I have bipolar 2, and recently my depression and moodiness is getting to my girlfriend. She is afraid to come over because she is afraid I will lose interest in her.

POST: On our second date I told her I had Bipolar 2 and

she was fine with it. She borrowed a book I had on the

disorder that helped her understand.

Everything

…

Prompt

DPO

Ada-DPO

1.13

30.4

Learned Reward Dif.

0

10

20

30

DPO

Ada-DPO

0.385

0.031

Learned Reward Dif.

0

0.1

0.2

0.3

0.4

preference, as the rejected response is nonsensical while the chosen one is clear. Ada-DPO learns a larger reward difference for such data, while it is much smaller with DPO. Conversely, for the sample pair with a small scaling factor, the two responses are very similar, indicating its ambiguity. Ada-DPO learns a small reward difference on this pair, while DPO gets a large reward difference.

5 Conclusion

RLHF is an emerging challenge in machine learning. Prior to the popularity of models like ChatGPT, research on designing proper loss functions for reward learning was limited. To bridge this gap, we explore uncertainties in underlying preference strengths and propose an adaptive preference loss function. This loss function incorporates instance-specific scaling factors to modulate the correspondence between reward differences and preference distributions. Taking the result in this paper as an initial start, we expect more sophisticated and stronger follow-up work that applies to RLHF with similar structures. All of these efforts may ultimately assist in developing more principled RLHF methods to better control risks associated with advanced AI systems.

References

Anthropic. Claude, 2024. URL https://www.anthropic.com .

Mohammad Gheshlaghi Azar, Zhaohan Daniel Guo, Bilal Piot, Remi Munos, Mark Rowland, Michal Valko, and Daniele Calandriello. A general theoretical paradigm to understand learning from human preferences. In International Conference on Artificial Intelligence and Statistics , pages 4447-4455. PMLR, 2024.

Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, Nicholas Joseph, Saurav Kadavath, Jackson Kernion, Tom Conerly, Sheer El Showk, Nelson Elhage, Zac Hatfield-Dodds, Danny Hernandez, Tristan Hume, Scott Johnston, Shauna Kravec, Liane Lovitt, Neel Nanda, Catherine Olsson, Dario Amodei, Tom B. Brown, Jack Clark, Sam McCandlish, Chris Olah, Benjamin Mann, and Jared Kaplan. Training a helpful and harmless assistant with reinforcement learningvfrom human feedback. CoRR , abs/2204.05862, 2022.

Sergio A Balanya, Juan Maroñas, and Daniel Ramos. Adaptive temperature scaling for robust calibration of deep neural networks. Neural Computing and Applications , pages 1-23, 2024.

Aharon Ben-Tal, Dick Den Hertog, Anja De Waegenaere, Bertrand Melenberg, and Gijs Rennen. Robust solutions of optimization problems affected by uncertain probabilities. Management Science , 59(2):341-357, 2013.

Dimitri P Bertsekas. Projected newton methods for optimization problems with simple constraints. SIAM Journal on control and Optimization , 20(2):221-246, 1982.

Dimitris Bertsimas, Vishal Gupta, and Nathan Kallus. Data-driven robust optimization. Mathematical Programming , 167:235-292, 2018.

Ralph Allan Bradley and Milton E Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika , 39(3/4):324-345, 1952.

Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym. arXiv preprint arXiv:1606.01540 , 2016.

Samuel Broscheit, Quynh Do, and Judith Gaspers. Distributionally robust finetuning bert for covariate drift in spoken language understanding. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics , pages 1970-1985, 2022.

Souradip Chakraborty, Amrit Bedi, Alec Koppel, Huazheng Wang, Dinesh Manocha, Mengdi Wang, and Furong Huang. Parl: A unified framework for policy alignment in reinforcement learning. In The Twelfth International Conference on Learning Representations (ICLR) , 2024.

Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. Advances in neural information processing systems , 30, 2017.

Erwin Coumans and Yunfei Bai. Pybullet, a python module for physics simulation for games, robotics and machine learning, 2016-2019. URL https://pybullet.org/ .

Zhipeng Ding, Xu Han, Peirong Liu, and Marc Niethammer. Local temperature scaling for probability calibration. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 6889-6899, 2021.

Yann Dubois, Chen Xuechen Li, Rohan Taori, Tianyi Zhang, Ishaan Gulrajani, Jimmy Ba, Carlos Guestrin, Percy S Liang, and Tatsunori B Hashimoto. Alpacafarm: A simulation framework for methods that learn from human feedback. Advances in Neural Information Processing Systems , 36, 2024.

John C Duchi, Peter W Glynn, and Hongseok Namkoong. Statistics of robust optimization: A generalized empirical likelihood approach. Mathematics of Operations Research , 46(3): 946-969, 2021.

Leo Gao, John Schulman, and Jacob Hilton. Scaling laws for reward model overoptimization. In International Conference on Machine Learning , pages 10835-10866. PMLR, 2023.

Tejas Gokhale, Abhishek Chaudhary, Pratyay Banerjee, Chitta Baral, and Yezhou Yang. Semantically distributed robust optimization for vision-and-language inference. In 60th Annual Meeting of the Association for Computational Linguistics , pages 1493-1513, 2022.

Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. On calibration of modern neural networks. In International conference on machine learning , pages 1321-1330. PMLR, 2017.

Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531 , 2015.

Tom Joy, Francesco Pinto, Ser-Nam Lim, Philip HS Torr, and Puneet K Dokania. Sampledependent adaptive temperature scaling for improved calibration. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 37, pages 14919-14926, 2023.

Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. International Conference on Learning Representations , 2015.

Daniel Kuhn, Peyman Mohajerin Esfahani, Viet Anh Nguyen, and Soroosh ShafieezadehAbadeh. Wasserstein distributionally robust optimization: Theory and applications in machine learning. In Operations research & management science in the age of analytics , pages 130-166. Informs, 2019.

Kimin Lee, Laura Smith, and Pieter Abbeel. Pebble: Feedback-efficient interactive reinforcement learning via relabeling experience and unsupervised pre-training. In International Conference on Machine Learning , 2021.

Xuezhe Ma, Pengcheng Yin, Jingzhou Liu, Graham Neubig, and Eduard Hovy. Softmax qdistribution estimation for structured prediction: A theoretical interpretation for raml. arXiv preprint arXiv:1705.07136 , 2017.

Paul Michel, Tatsunori Hashimoto, and Graham Neubig. Modeling the second player in distributionally robust optimization. International Conference on Learning Representations , 2021.

Yonatan Oren, Shiori Sagawa, Tatsunori B Hashimoto, and Percy Liang. Distributionally robust language modeling. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) , pages 4227-4237, 2019.

Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems , 35:27730-27744, 2022.

Jongjin Park, Younggyo Seo, Jinwoo Shin, Honglak Lee, Pieter Abbeel, and Kimin Lee. SURF: Semi-supervised reward learning with data augmentation for feedback-efficient preference-based reinforcement learning. In International Conference on Learning Representations , 2022.

Qi Qi, Jiameng Lyu, Kung-Sik Chan, Er-Wei Bai, and Tianbao Yang. Stochastic constrained dro with a complexity independent of sample size. Transactions on Machine Learning Research , 2023.

Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems , 36, 2024.

Antonin Raffin. Rl baselines3 zoo, 2020. URL https://github.com/DLR-RM/ rl-baselines3-zoo .

Antonin Raffin, Ashley Hill, Adam Gleave, Anssi Kanervisto, Maximilian Ernestus, and Noah Dormann. Stable-baselines3: Reliable reinforcement learning implementations. Journal of Machine Learning Research , 22(268):1-8, 2021.

Antonin Raffin, Jens Kober, and Freek Stulp. Smooth exploration for robotic reinforcement learning. In Conference on Robot Learning , pages 1634-1644. PMLR, 2022.

Shiori Sagawa, Pang Wei Koh, Tatsunori B Hashimoto, and Percy Liang. Distributionally robust neural networks for group shifts: On the importance of regularization for worst-case generalization. arXiv preprint arXiv:1911.08731 , 2019.

John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347 , 2017.

Ziang Song, Tianle Cai, Jason D. Lee, and Weijie J. Su. Reward collapse in aligning large language models. CoRR , abs/2305.17608, 2023.

Nisan Stiennon, Long Ouyang, Jeff Wu, Daniel M. Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F. Christiano. Learning to summarize from human feedback. Advances in Neural Information Processing Systems , 2020.

Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ international conference on intelligent robots and systems , pages 5026-5033. IEEE, 2012.

Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288 , 2023.

Michael Völske, Martin Potthast, Shahbaz Syed, and Benno Stein. Tl; dr: Mining reddit to learn automatic summarization. In Proceedings of the Workshop on New Frontiers in Summarization , pages 59-63, 2017.

Leandro von Werra, Younes Belkada, Lewis Tunstall, Edward Beeching, Tristan Thrush, Nathan Lambert, and Shengyi Huang. Trl: Transformer reinforcement learning, 2020. URL https: //github.com/huggingface/trl .

Feng Wang and Huaping Liu. Understanding the behaviour of contrastive loss. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 2495-2504, 2021.

Pei-Hsin Wang, Sheng-Iou Hsieh, Shih-Chieh Chang, Yu-Ting Chen, Jia-Yu Pan, Wei Wei, and Da-Chang Juan. Contextual temperature for language modeling. arXiv preprint arXiv:2012.13575 , 2020.

Hongyi Wen, Xinyang Yi, Tiansheng Yao, Jiaxi Tang, Lichan Hong, and Ed H Chi. Distributionally-robust recommendations for improving worst-case user experience. In Proceedings of the ACM Web Conference 2022 , pages 3606-3610, 2022.

Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al. Transformers: Stateof-the-art natural language processing. In Proceedings of the 2020 conference on empirical methods in natural language processing: system demonstrations , 2020.

Yao Zhao, Rishabh Joshi, Tianqi Liu, Misha Khalman, Mohammad Saleh, and Peter J. Liu. Slic-hf: Sequence likelihood calibration with human feedback. CoRR , abs/2305.10425, 2023.

A Derivation and proofs of Section 3

A.1 Derivation of Equation (4)

In this subsection, we present the full derivation of Equation (4). Recall the following loss:

Now, we consider the optimality conditions for the inner constrained maximization problem by defining the following Lagrangian function:

where µ is the Lagrange multiplier. The optimal solutions p r,τ to the inner maximization problem satisfy the following KKT conditions:

Then we have

Plugging in p and p back into the inner maximization problem, we have

Without loss of generality, we let z 1 = z w and z 2 = z l , and obtain

where σ is the logistic function. This completes the derivation.

A.2 Proof of Proposition 3.1

We first derive the expectation of the adaptive loss. By taking expectation of (4) with ρ = ρ 0 -log 2 , we have:

where σ -1 is the inverse of sigmoid function. Plugging (13) into the objective in (7), we obtain min [ -p ∗ log( p ∗ ) -(1 -p ∗ ) log(1 -p ∗ ) + ρ ] τ, whose objective is essentially linear in τ . Hence, when -p ∗ log( p ∗ ) -(1 -p ∗ ) log(1 -p ∗ )+ ρ > 0 , the corresponding optimal τ ∗ is at the lower bound τ 0 and the optimal reward difference r ∗ ( z 1 ) -r ∗ ( z 2 ) = τ 0 σ -1 ( p ∗ ) given the optimality condition. Conversely, when -p ∗ log( p ∗ ) -(1 -p ∗ ) log(1 -p ∗ )+ ρ < 0 , we have τ ∗ = τ max and r ∗ ( z 1 ) -r ∗ ( z 2 ) = τ max σ -1 ( p ∗ ) . This completes the proof.

A.3 Proof of Proposition 3.2

Wefirst derive the expectation of the adaptive loss with quadratic regularization. By taking expectation of (12), we have:

By the optimality condition of r ( z 1 ) -r ( z 2 ) and τ , we have

Note that (16) is always bounded without the need of an upper bound of τ . Specifically, with any p ⋆ ∈ (0 , 1) , we have -p ∗ log( p ∗ ) -(1 -p ∗ ) log(1 -p ∗ ) -log 2 ≤ 0 and τ ⋆ = max { τ 0 , ( p ∗ log( p ∗ )+ (1 -p ∗ ) log(1 -p ∗ ) + log 2) / (2 ρ 0 ) } . This completes the proof.

B Implementation details

B.1 Robotic control

Our implementations of robotic control tasks are based on Stable-Baselines3 [33] and RL Zoo training framework [32]. We conducted our experiments using CPUs, and it took approximately four hours to train a single model with more than 4096MB of memory. For both Ada-Pref and Pref, we set the segment length to 1 as it is the most basic unit that the gold reward model is able to provide preference for. Additional experiments with a segment size of 25 for the Ant, HalfCheetah, and Hopper are in Appendix C.2. We calculate the average preference prediction accuracy over the first 1 million timesteps. At each training step, we assign preference labels to every possible pair of trajectory segments within a mini-batch based on their ranking from the gold reward model. We set the batch size to 64 for the HalfCheetah and Ant tasks and 4 for the Hopper task. We tune the number of epochs in { 1 , 3 , 5 } . We use Adam optimizer [22] and tune the learning rate in { 5 e -3 , 1 e -3 , 5 e -4 , 1 e -4 } for the Ant and HalfCheetah, and set the learning rate to 1 e -2 for the Hopper. For Ada-Pref, we tune the τ max in { 1 . 0 , 3 . 0 } and the ρ 0 in { 0 . 1 , 0 . 3 , 0 . 5 } . We fix τ 0 = 0 . 1 and the number of Newton iterations to 3 for all experiments. Details of the chosen hyperparameters for reward learning for all three tasks are summarized in Tables 3 and 4. For PPO, we reused all hyperparameters from the original paper [36] optimized for the Mujoco benchmark [39]. Details of the hyperparameters for PPO are in Table 5.

B.2 Natural language generation

Our implementations of natural language generation tasks are based on transformers [46] and trl training framework [42]. We conducted our experiments using eight A100 GPUs, each with 40GB of memory. Training a single model took approximately two hours. We provide more details on each task as follows:

B.2.1 Summarization

For the instruction tuning stage, we randomly select 800 data from the filtered TL;DR summarization dataset [41] for testing the policy and leave the rest for supervised tuning. In the preference optimization stage, we split the preference dataset [38] into a training and testing set to evaluate the preference accuracy. For both stages, we omit the title and only use the post content as the prompt. The prompt format follows: "POST: post content.\n\nTL;DR:".

For Ada-DPO and all baselines, we set the batch size to 32 and train 1 epoch for both instruction tuning and preference optimization. We set the α parameters of LoRA fine-tuning to 16, and tune the other parameters by grid search. The learning rate is tuned in { 5 e -6 , 5 e -5 , 1 e -4 , 5 e -4 } . SLiC-HF, IPO and DPO include parameter β , which is tuned in a range of { 0 . 01 , 0 . 1 , 0 . 3 , 0 . 5 } . For Ada-DPO, we tune the ρ 0 in { 0 . 05 , 0 . 1 , 0 . 3 , 0 . 5 } and the τ max in { 1 . 0 , 4 . 0 , 5 . 0 , 10 . 0 } . We fix τ 0 = 0 . 1 and the number of Newton iterations to 5 for all experiments. The best Ada-DPO is achieved with lr = 5 e -5 , ρ 0 = 0 . 1 , and τ max = 4 . 0 .

B.2.2 Single-turn dialogue

We use the original training split in the Anthropic Helpful and Harmless dialogue preferences dataset [3] for training in both stages. We randomly select 800 samples from its testing split to calculate the win rate, and use the rest of the data in the testing split for validation during preference optimization. We use the original data format.

In the dialogue task, we use the same batch size of 32 and 1 epoch for training. The learning rate is tuned in { 5 e -6 , 5 e -5 , 1 e -4 } . The parameter β for baselines is tuned in a range of { 0 . 01 , 0 . 1 , 0 . 3 } . For Ada-DPO, we tune the ρ 0 in { 0 . 05 , 0 . 1 , 0 . 3 , 0 . 5 } , τ max in { 1 , 5 , 10 } and fix τ 0 = 0 . 1 . The best Ada-DPO is achieved with lr = 5 e -5 , ρ 0 = 0 . 05 , and τ max = 5 . 0 .

B.2.3 Evaluation prompt.

We apply Claude 3 for win rate judgments. Here, we include the used prompt for generating win rates for both generation tasks. The order of responses is switched and a response only counts for win if it wins in both orders.

Summarization win rate prompt.

Dialogue win rate prompt.

C Additional experiments and discussions

C.1 Experiments with quadratic regularization

(a) Learning curve

(b) Percentile plot

(c) Histogram of

τ

(d)

Preference

strength and

τ

(e)

Preference

strength and learned

reward difference

3000

3600

Return

2000

1500

2500

1000

Return

3200

3400

(%）

Percentage

17.5

15.0

12.5

10.0

7.5

Average

0.40

0.45

0.35

0.50

Difference

Reward

0.020

0.015

0.010

Pref

Ada-Pref-Quad

500

Pref

3000

Pref

5.0

0.30

0.005

Ada-Pref-Quad

Ada-Pref-Quad

2.5

0.25

0.0

0.5

Timesteps (1e6)

1.0

1.5

2.0

2.5

3.0

2800

0

20

Percentile

40

60

80

100

0.0

0.2

0.4

1

0.6

0.8

1.0

1

2

3

4

0.000

1

2

3

4

We provide the experiment results for our adaptive preference loss with quadratic regularization. Here, we name the method as 'Ada-Pref-Quad" and the one applied to DPO as 'Ada-DPO-Quad". Table 6 and Figure 8 show the results for Pref and Ada-Pref-Quad on the Ant task, and DPO and Ada-DPO-Quad on the single-turn dialogue. In Table 6, we report the performance of the best policy and the preference prediction accuracy of the corresponding reward function. From Table 6, we can see that Ada-Pref-Quad outperforms Pref on the Ant task, and Ada-DPO-Quad surpasses DPO on the single-turn dialogue in terms of return and win rate, respectively. Figures 8(a) and 8(b) present the learning curve and the percentile plot for the Ant task. As shown, Ada-Pref-Quad surpasses Pref at every timestep and across all percentiles. Figure 8(c) shows a histogram of the learned scaling

factors τ . Compared to Figure 5(a), we can see much smoother distribution of τ due to the quadratic regularization. Figures 8(d) and 8(e) illustrate the relationship between the learned scaling factors τ and preference strength, and the relationship between the learned reward difference and preference strength. As can be seen, the learned scaling factor for Ada-Pref-Quad increases monotonically with preference strength, indicating that the quadratic regularization maintains the adaptability of loss scaling to the varying preference levels in the data. Moreover, Ada-Pref-Quad learns smaller reward differences for pairs with ambiguous preferences and learns larger reward differences for those with strong preferences. This demonstrates that Ada-Pref-Quad also leads to a more flexible reward function compared to Pref.

C.2 Ablation studies

In this subsection, we present the results for three PyBullet tasks, using a segment size of 25. Table 7 and Figure 9 show the performance of Pref and Ada-Pref on the PyBullet tasks, based on the first hyperparameter tuning criterion. Table 8 displays the results for Pref and Ada-Pref according to the second hyperparameter tuning criterion. These results reconfirm the effectiveness of our adaptive preference loss.

(a) HalfCheetah

(b) Ant

(c) Hopper

3000

3000

1750

2500

1500

2000

1250

2000

Return

1000

Return

1500

750

0

1000

500

Pref

500

Pref

250

Pref

-1000

Ada-Pref

Ada-Pref

Ada-Pref

0

0

0.0

Timesteps (1e6)

0.5

1.0

1.5

2.0

0.0

Timesteps (1e6)

0.5

1.0

1.5

2.0

0.0

Timesteps (1e6)

0.5

1.0

1.5

2.0

3300

3000

2000

3200

2800

3100

1500

Return

2600

Retur

3000

2400

2900

2200

Pref

2800

Pref

500

Pref

Ada-Pref

2700

Ada-Pref

Ada-Pref

2000

0

0

20

40

60

80

100

0

20

40

60

80

100

0

20

40

60

80

100

Percentile

Percentile

Percentile

C.3 Discussions on hyperparameter tuning

Compared to the cross-entropy loss, our method needs three additional hyperparameters: the bounds on the scaling factors τ 0 and τ max , and the regularization parameter ρ . In our experiments, we fixed τ 0 at 0.1 without tuning it, as this value worked well for all five tasks. We did tune τ max to adjust the scale of τ , but this can be avoided by using the quadratic regularization formulation described in Section 3.6. The parameter ρ turns out to be more important, because it controls the distribution of the scaling factors. We performed a careful grid search to tune ρ in our experiments. Figure 10 shows the hyperparameter sensitivity of ρ on the Ant and summarization tasks. Overall, we found that smaller values of ρ often lead to better performance.

Rewards

Return

3000

3100

3200

Preference Acc.

86

88

90

92

94

ρ

-

0.6

-

0.4

-

0.2

Win rate

Win rate

20

30

40

50

Preference Acc.

64

66

68

70

ρ

-

0.6

-

0.4

-

0.2

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: Contributions and scope are explained in the methodology section 3 and well-supported by results presented in section 4.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [NA]

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: The full derivation of the proposed loss and the proofs for the propositions are provided in Appendix A.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: The proposed algorithm is summarized in Appendix B and the experiment details are included in Section 4 and Appendix C.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [No]

Justification: We will release the code after the submission deadline.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: The basic experiment settings are included in Section 4 and more implementation details are in Appendix C.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: Error bars are included in the robotic control experiments to indicate variability. Additionally, percentile plots are provided to support the validity of the experimental results. See Figures 2, 7, and 8 for reference. Only one seed is used for natural language tasks due to computational concerns.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: See implementation details in Appendix C.

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: This research adheres to the NeurIPS Code of Ethics.

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences of our work, none of which we feel must be specifically highlighted.

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: This paper does not release data or models.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: The used models and data are properly credited in experimental settings.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: This paper does not release new assets.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: This paper does not involve crowdsourcing nor research with human subjects.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: This paper does not involve crowdsourcing nor research with human subjects.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nTo learn a versatile rewards essential for the downstream policy optimization, this paper introduces a novel adaptive preference loss function inspired by distributionally robust optimization (DRO).\nThe proposed approach incorporates an learnable instance-specific scaling factor to accommodate varying uncertainties of preference strength and change the scaling between the preference distribution and reward difference to be non-linear.\nThe proposed loss function for learning the scaling parameters is strictly convex, and thus they only induce negligible additional computational overhead.\nExperiment results on robotic control tasks and LLMs verify the proposed method.\n\nSTRENGTHS:\n1. Adapting the optimization strength by factoring in the characteristics of each preference pair is an interesting direction.\n2. The proposed method is well (theoretically) motivated.\n3. The proposed method work well in practice.\n\nWEAKNESSES:\n1. The adaptive scaling is defined on a per-instance basis, which necessitates significant compute costs and hinders real-world mini-batch learning. This is evident from Appendix, where the algorithm box shows that the training process operates in a per-datapoint manner\n2. Some of the derivation may be simplified, both to make the paper more succinct and leave room for algorithm box as well as other technical details.\n3. L221: could you theoretically justify the incorporation of adaptive preference scaling into DPO. e.g., in the framework of KL-control as in the original DPO paper?\n\nQUESTIONS:\n1. L71 \"Prior work on this topic is very limited.\": There are actually a vast pool of literature concerning loss functions for (RLHF's) reward modeling, a quick pointer is [1] for language tasks and [2]  for image generation, as well as the literature review in them. I encourage the authors to add proper discussions and citations to make the claims more precise.\n2. L140-141: Could you elaborate this sentence more: \"For some applications, it may lead to a reward function that is not flexible enough to differentiate a pair of segments\"? In particular, why is the resulted reward function not flexible enough? And how is this related to the linear scaling?\n3. In Eq. (3), why do we need to have $KL(p, 1/2)$ in both the objective function and constraint?\n\n\n\n***\n[1] Yang, Shentao, et al. \"Preference-grounded token-level guidance for language model fine-tuning.\" Advances in Neural Information Processing Systems 36 (2023).\n\n[2] Yang, Shentao, Tianqi Chen, and Mingyuan Zhou. \"A Dense Reward View on Aligning Text-to-Image Diffusion with Preference.\" Forty-first International Conference on Machine Learning. 2024.\n\nLIMITATIONS:\nA discussion on limitations and/or potential negative societal impact seems missing.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-06-20"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the problem of learning from preference data and introduces a learnable scaling parameter for each preference sample. The authors propose an adaptive preference loss function that assign small scaling parameters to ambiguous preferences pairs and large scaling parameters to clear preferences. Experiments demonstrate the improvement of policy optimization performance and efficient of the hyperparameter tuning.\n\nSTRENGTHS:\nConsidering different samples have different preference strength, using an adaptive preference scaling makes sense for preference learning.\n\nExperiments are conducted on both robotic control and natural language generation tasks.\n\nWEAKNESSES:\nThe increase in flexibility of the proposed reward function is not verified. The authors claim that one of the limitations of the BT model is the logit of the preference distribution scales linearly with the reward difference. However, according to Proposition 3.1, the limitation still exists. There is no empirical result that supports the claim.\t\n\nMissing the ablation study of the regularization term in the proposed loss function.\n\nMissing the RLHF baseline in the natural language generation task. \n\n“Ada-Pref demonstrates greater resistance to performance degradation than Pref, indicating its superior ability to align the learned reward function with policy optimization.” It seems that it is not true in the Hopper task?\n\nMinors: \nLine 120: The notation of state transition function is conflicted with preference distribution.\nPlease clarify the meaning of M and K in Algorithm 1.\n\nQUESTIONS:\nI do not believe the proposed loss function is convex, can you provide the proof of Remark 3.1?\nWhy the reward models trained with the proposed method can better guide policy model selection considering the reward model performance is not improved?\n\nHow do the authors obtain the true reward difference in Section 4.3?\n\nLIMITATIONS:\nThe computation overhead introduced by learning the scaling parameter is not discussed.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}

comments: SUMMARY:
The paper identifies a limitation in RLHF methods, noting that ranking over pairs of trajectory segments often fails to capture the varying strengths of preferences across different pairs. To address this, the paper proposes a new adaptive preference loss (Ada-DPO), underpinned by distributionally robust optimization (DRO). The proposed method improves policy performance and is supported by theoretical proofs on convexity and univariate analysis.

STRENGTHS:
- The paper identifies a key limitation in previous RLHF methods and provides a clear theoretical analysis of its methods and claims. 
- It presents robust experiments and quantitative analysis on both robotic controls and natural language generation tasks.

WEAKNESSES:
- It seems that only a single run was conducted for the experiments in the NLP tasks. More repeated runs would be more convincing, especially since the differences in performance are so close. Additionally, p-value testing for that domain would help determine if the differences are truly significant.

QUESTIONS:
- I would like to see more examples like Figure 7, comparing the learned reward differences with different scaling factors.

LIMITATIONS:
- More discussion on the limitations of the proposed method and future directions can be added.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-13

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper focuses on redesigning the loss function with adaptive scaling parameters to deal with the uncertainty in the preferences and thus improving the reward modeling flexibility.  In the context of both robotics and NLP, the algorithm with the new loss shows improved performance.\n\nSTRENGTHS:\n1. The proposed loss is flexible and can be incorporated in majority of the current RLHF frameworks\n2. The objective has strong connections to DRO which is critical since preferences will be noisy and sub-optimal in practical scenarios.\n\nWEAKNESSES:\n1. The experimental ablation is weak and doesn't provide concrete indications of the strength of the proposed algorithm. For ex in Figure 3, the win-rate gap is marginal and not significant. Why is the performance in summarization better in Figure 4 is not extremely clear?\n\n2. The experimental evaluations lack important benchmarks and comparisons. For the Robotics environment, it's crucial to compare with Pebble, SURF, and PARL to understand the true efficacy of the proposed approach.\n\nQUESTIONS:\n1. Can the authors provide more concrete mathematical justifications regarding the issue with linear depedence with the logit of the preference distribution? Specifically, connecting with the relation to noisy preferences?\n2. \"For some applications, it may lead to a reward function that is not flexible enough to differentiate a\n141 pair of segments, which are supposed to have significantly different rewards.\" Can the authors please highlight such examples intuitively or mathematically, why such a collapse might happen?\n3. The authors mention \"Our proposed method is inspired by DRO, it serves a distinct purpose: improving reward learning in RLHF, which is orthogonal to distributional robustness\". Can the authors pls justify this statement and discuss why it's orthogonal? DRO in reward learning will also improve the robustness and flexibility of the reward model to noisy preferences, in what sense it does better than DRO?\n4. Can the authors provide a more concrete definition of the best win rate and best prediction accuracy?\n5. It will be helpful if the authors can provide more details on how the trajectories are constructed for the robotics experiments.\n\nLIMITATIONS:\nCheck above", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-15"}