Adaptive Preference Scaling for Reinforcement Learning with Human Feedback

Authors: ilgee  hong
zichong  li
alexander  bukharin
yixiao  li
haoming  jiang
tianbao  yang
tuo  zhao
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-13 10:41:16

Abstract: Reinforcement learning from human feedback (RLHF) is a prevalent approach to align AI systems with human values by learning rewards from human preference data. Due to various reasons, however, such data typically takes the form of rankings over pairs of trajectory segments, which fails to capture the varying strengths of preferences across different pairs. In this paper, we propose a novel adaptive preference loss, underpinned by distributionally robust optimization (DRO), designed to address this uncertainty in preference strength. By incorporating an adaptive scaling parameter into the loss for each pair, our method increases the flexibility of the reward function. Specifically, it assigns small scaling parameters to pairs with ambiguous preferences, leading to more comparable rewards, and large scaling parameters to those with clear preferences for more distinct rewards. Computationally, our proposed loss function is strictly convex and univariate with respect to each scaling parameter, enabling its efficient optimization through a simple second-order algorithm. Our method is versatile and can be readily adapted to various preference optimization frameworks, including direct preference optimization (DPO). Our experiments with robotic control and natural language generation with large language models (LLMs) show that our method not only improves policy performance but also aligns reward function selection more closely with policy optimization, simplifying the hyperparameter tuning process.

Main paper: 
 introduction: 
 In the field of artificial intelligence, aligning AI systems with human preferences has become increasingly crucial, particularly for applications involving complex data and models like large language models (LLMs) in natural language processing [38, 28] . Reinforcement learning from human feedback (RLHF) has gained popularity for customizing AI systems [12, 3, 47] . RLHF involves learning a reward function from human preference data, then using a reinforcement learning algorithm to train a policy to optimize the learned reward model.A key challenge in RLHF lies in the complexity of reward modeling, which primarily stems from the reliance on preference labels. Since preference labels only provide comparative rankings of trajectory segments without quantifying the scale of underlying preference strengths, previous methods have employed the Bradley-Terry (BT) model [8] in conjunction with cross-entropy loss to learn the reward function from preference data [12, 38] . This approach assumes that the logit of the preference distribution scales linearly with the reward difference across all sample pairs. However, such linear scaling is often insufficient to account for the variations in preference strength among different pairs, restricting the reward function's ability to capture a broader range of reward differences. This restrictive approach to reward modeling limits the flexibility of the learned reward function, hindering its capacity to produce the versatile rewards essential for the downstream policy optimization.To overcome this shortcoming, we introduce a novel adaptive preference loss function inspired by distributionally robust optimization (DRO) [16] . Our approach incorporates an instance-specific scaling factor to change the scaling between the preference distribution and the reward difference to be non-linear. These factors are learned during training and enable the model to accommodate varying uncertainties of preference strength, thereby enhancing the flexibility of the reward. For pairs showing strong preference (i.e., low preference uncertainty), our method learns a large scaling factor, which enables the model to learn a larger reward difference. In contrast, for pairs showing ambiguous preferences (i.e., high preference uncertainty), our method assigns a smaller scaling factor, enabling the model to learn a smaller reward difference. The additional computational overhead of involving this scaling factor into training is negligible, as the proposed loss function is strictly convex and univariate with respect to each scaling parameter. Therefore, it can be easily optimized by a simple second-order algorithm within a few iterations.Our experiments on robotic control tasks [39] demonstrate that our method can learn a more flexible reward function, resulting in an improved policy. Surprisingly, we also discover that our method better aligns the learned reward function with downstream policy optimization. Specifically, when tuning hyperparameters for reward modeling, the simplest approach is to select the reward model according to preference prediction accuracy. However, the selected reward function (with the highest accuracy) often yields a downstream policy with poor performance. To address this misalignment, we usually have to jointly tune the parameters across both stages according to downstream policy performance, resulting in significant computational burden and tuning effort. Our proposed method can mitigate this misalignment: When using our adaptive loss, we can select the reward model based on preference prediction accuracy alone and yield a reasonably well-performing policy. This allows separate tuning of the two stages, easing tuning overhead. To our knowledge, the challenge of this misalignment issue is almost untouched in the RLHF literature, and we are the first to propose a principal approach to mitigate this issue. Moreover, our method is generalizable and can be applied to other preference optimization algorithms. For instance, we implement it with direct preference optimization (DPO) [31] and evaluate its effectiveness on natural language generation tasks using Llama-2 7B [40] . Our results demonstrate that integrating adaptive preference scaling into DPO boosts policy performance, while preserving the benefits of alignment. Alignment is especially critical in this setting, where we employ proprietary models like Claude 3 [1] as judges for policy selection, which demands substantial costs for using the APIs. In the case without access to LLM assessment, we must select policy based solely on preference accuracy, under which our approach substantially outperforms other baselines. 
related works: 
 Loss functions for reward learning. Prior work on this topic is very limited. For example, Song et al. [37] propose using different loss functions for strong and ambiguous preference data in natural language generation tasks. They apply heavy-tailed loss functions for open-ended questions, where preference ambiguity is desirable, and light-tailed loss functions for close-ended questions requiring clear-cut rewards. However, their approach requires knowing the question type a priori, necessitating extra labeling effort, and may fail for complex questions containing both open and closed aspects. Zhao et al. [47] propose using a hinge loss, which results in zero gradient when the learned reward difference exceeds a margin of 1. This limits the ability to learn very large differences in rewards. Azar et al. [2] develop Ψ Preference Optimization with Identity Mapping (IPO), which modifies DPO with a loss function matching the scaling of KL-divergence between the learned policy and the initial policy to avoid overfitting due to weak regularization. In contrast to prior work, our method is more broadly applicable to complex preference learning tasks without needing additional labeling or sacrificing the ability to learn arbitrarily large reward differences.Adaptive temperature scaling (ATS). Temperature scaling (TS) aims to adjust the entropy of probabilistic models by rescaling their logit outputs before the softmax function is applied. This simple method not only enables confidence calibration [19] , but also plays a vital role in various machine learning methods, including knowledge distillation [20] , reinforcement learning [25] , and contrastive learning [43] . Building on TS, adaptive temperature scaling (ATS) enhances flexibility by using instance-specific scalars. Most ATS method trains an additional network for predicting the temperature parameter, which is further integrated into the softmax operator to calibrate the prediction probabilities [44, 14, 4, 21] . In contrast to the aforementioned ATS methods, the proposed adaptive preference scaling (APS) is not designed for classical confidence calibration, but is crafted specifically to enhance the training process of reward function in RLHF. Consequently, the interpretations of scaling factors in ATS and APS are opposite. In ATS, a larger scaling parameter is applied to data with higher uncertainty (e.g., data that the classifier is likely to misclassify), which reduces the magnitude of the corresponding logit. Conversely, in APS, a larger scaling factor is assigned to data with clearer preferences, resulting in a larger logit. This distinction clarifies why the scaling parameter in our approach does not correspond to the concept of "temperature" from statistical physics. Additionally, we propose a principled framework for learning scaling parameter based on DRO, which avoids the complexities of designing specific temperature networks and does not rely on heuristically designed loss functions.. DRO is a technique that trains machine learning models to be robust against uncertainty in the data distribution. Specifically, DRO finds a solution that performs well under the worst-case distribution within a specified uncertainty set around the empirical data distribution [5, 7, 23, 35, 16] . DRO has been applied in various AI/ML domains to improve generalization when the test distribution differs from the training distribution [27, 18, 26, 10, 45, 30] . Our framework is motivated by Qi et al. [30] , which tackles KL-constrained DRO problem. However, our approach differs in two significant ways. First, instead of using a single KL constraint for the entire training dataset, we apply a separate KL constraint to each individual training data. Second, since each training data involves just two distributional variables, we can use a deterministic method to optimize these efficiently. Note that while our proposed method is inspired by DRO, it serves a distinct purpose: improving reward learning in RLHF, which is orthogonal to distributional robustness. 
method: 
 In this section, we first outline the problem setup, derive the loss function with adaptive preference scaling, and then provide theoretical motivation for our proposed loss. At last, we present an optimization algorithm, extend the approach to direct preference optimization, and introduce a variant of our proposed loss that incorporates quadratic regularization.We consider a reward-free Markov decision process M = (S, A, p, γ) with state s ∈ S, action a ∈ A, state transition function p, and discount factor γ. The ground truth reward function r : S × A → R is assumed to be unknown, but only human preferences over pairs of trajectory segments are observed. A trajectory segment is a sequence of consecutive state and action pairs z = {(s m , a m ), (s m+1 , a m+1 ), . . . , (s k-1 , a k-1 )} ∈ (S × A) k-m . We denote z 1 ≻ z 2 to indicate that the human preferred trajectory segment z 1 over the trajectory segment z 2 and denote the preferred one with a subscript w and the dispreferred one with a subscript l (i.e., z w and z l ). Here, we are given a human preference dataset of trajectory segments D pref = (z w,i , z l,i ) N i=1 . Our goal is to find a reward r(s, a), which is well-aligned with human preferences. Once we learn the reward, we then find a policy π ∈ ∆ S A such that it maximizes the expected sum of discounted rewards,max π E z∼Dπ r(z) ,where r(z) = (st,at)∈z γ t r(s t , a t ) and D π denotes the stationary distribution of the state-action pair induced by π.We now focus on the reward learning phase in RLHF, a crucial stage for capturing human preferences across various trajectory segments. The standard reward learning procedure assumes that the reward function determines a preference distribution, also known as the Bradley-Terry (BT) model [8] ,EQUATIONwhere σ denotes the sigmoid function. The reward function is then learned by minimizing the expectation of negative log-likelihood of r over the preference data [12] :EQUATIONAs can be seen from ( 1), the BT model essentially assumes that the logit of the preference distribution σ -1 (p r (z w ≻ z l )) scales linearly with the reward difference, regardless of the specific pair of samples. Such linear scaling, however, may not align well with downstream policy learning. Human preferences are often influenced by numerous factors that interact in non-linear ways, making the BT model suboptimal as a reward model. For example, when the reward difference is small, even slight changes in certain features might lead to significant shifts in preference. The BT model may struggle to capture such rapid shifts due to its slower transition.To address this challenge, we propose an adaptive preference loss based on KL-constrained distributionally robust optimization formulation [30] , which can implicitly change the scaling between the logit and the reward difference to be non-linear. Specifically, given a pair of trajectory segments(z 1 , z 2 ), we denote d r (z 1 , z 2 ) = 1(z 1 ≻ z 2 ) • (r(z 2 ) -r(z 1 )) and p = (p 1 , p 2 ). We define the following instance-level loss:EQUATIONwhere∆ 2 = {p ∈ R 2 : p 1 + p 2 = 1, 0 ≤ p 1 , p 2 ≤ 1}, 1/2is denoted for the uniform distribution, and ρ 0 , τ 0 > 0 are shared prespecified parameters across all instances. KL(•, •) denotes the KL divergence. Note that without the KL-constraint, ( 3) is reduced to the cross-entropy loss with τ 0 = 1. Unlike general KL-constrained DRO formulation, which considers a distribution p over all training samples, the distributional variable p in (3) is associated specifically with binary preference comparisons for each pair.We then convert (3) into an equivalent minimax formulation based on the Lagrangian duality,min λ≥0 max p∈∆2 p 1 d r (z 1 , z 2 ) + p 2 d r (z 2 , z 1 ) -(λ + τ 0 )(KL(p, 1/2) -ρ 0 ),where λ is the Lagrange multiplier. By defining τ as τ = λ + τ 0 and applying the optimality condition for p, we haveEQUATIONwhereEQUATIONWe refer to Appendix A.1 for the full derivation. Note that the preference scaling factor τ in ( 4) and ( 5) serves as the Lagrange multiplier of (3). This scaling parameter τ is used specifically for training the reward function r, rather than calibrating the preference distribution p r,τ (z w ≻ z l ). The scaler τ is used exclusively during the reward learning phase and is no longer needed in subsequent policy optimization, where the reward function r alone is used.Moreover, the scaling parameter τ is defined to be an instance-specific parameter corresponding to the pair of trajectory segments (z w , z l ). Therefore, when applying our adaptive loss to reward learning, for each pair (z w,i , z l,i ), we need to define a corresponding scaling parameter denoted by τ i . The overall loss function over the training set D pref is as follows:EQUATIONwhere T = (τ 1 , . . . , τ N ), Ω = {τ : τ 0 ≤ τ ≤ τ max } with τ max as another prespecified parameter, and ρ = ρ 0 -log 2 > -log 2. Here, we also involve an upper bound τ max > 0 in ( 6), and we will explain why it is needed in the next subsection.We next provide some theoretical insights on why the scaling parameter τ can help gain adaptivity by a proposition. For simplicity, we only consider a pair of trajectories. Proposition 3.1. Assume we have a pair of trajectories z 1 , z 2 , and the preference distribution p(z 1 ≻ z 2 ) = p * ∈ (0, 1), i.e., the probability, that z 1 is preferred over z 2 , is p * . Consider the problem of minimizing the expectation of our adaptive loss function over the preference distribution: minEQUATION)Then the minimizer τ * and r * of the expected loss satisfyτ * = τ 0 if -p * log(p * ) -(1 -p * ) log(1 -p * ) + ρ > 0, τ max if -p * log(p * ) -(1 -p * ) log(1 -p * ) + ρ < 0, r * (z 1 ) -r * (z 2 ) = τ * σ -1 (p * ).Here, σ -1 is the inverse of sigmoid function. Note that the expected loss ( 7) is only for easing theoretical analysis, as p * is not accessible in practice. From Proposition 3.1, we can see that when p * is close enough to 0.5, (i.e., the uncertainty of preference is large), the corresponding optimal τ * is at the lower bound τ 0 . The resulting optimal reward difference is τ 0 σ -1 (p * ), which is smaller than the counterpart obtained by the cross-entropy loss when τ 0 < 1. Conversely, when p * is close to 0 or 1, (i.e., the uncertainty of preference is small), the resulting optimal τ * is at the upper bound τ max . Here, we introduce the upper bound τ max to ensure that the optimal τ * is bounded. The resulting reward difference in this case is τ max σ -1 (p * ), which is larger than the counterpart obtained by the cross-entropy loss when τ max > 1. Our theoretical analysis suggests that the adaptive scaling factor essentially changes the correspondence between the logit of preference distribution and the reward difference for each pair of trajectory segments, which could lead to a more flexible reward model. We further visualize our adaptive preference loss in Figure 1 , setting τ 0 and τ max to 0.1 and 5.0, respectively. As depicted, our adaptive preference loss behaves distinctly compared to the cross-entropy loss. With large learned reward differences, the cross-entropy tends to be very flat, while our loss maintains a non-trivial gradient, allowing us to continually decrease the loss function. In contrast, for small positive learned reward differences, our loss yields a smaller gradient, thereby less encouraging the reward model to further distinguish pairs of ambiguous trajectory segments. This is consistent with our theoretical analysis.We present an efficient algorithm for solving (6) . Suppose we parameterize r as a neural network with parameter ϕ. At the m-th iteration, we have the iterate ϕ (m) , and we sample a pair of trajectory segments z w,i and z l,i . We initialize τ (0) i = 1 and then optimize τ i by a projected Newton method subject to a simple interval constraint Ω [6] . Specifically, for k = 0, ..., K -1, we takeEQUATIONwhereEQUATION.Once we get τ(K) i, we update ϕ by a stochastic gradient descent stepEQUATIONwhere η ϕ is the learning rate. We summarized our proposed algorithm in Algorithm 1, which is presented in a per-data manner for clarity but can be directly adapted for mini-batch learning.Algorithm 1 Algorithm for reward learning with adaptive preference scaling 1: Input: τ 0 , τ max , ρ, η ϕ ; 2: for m = 0, 1, 2, . . . , M -1 do for k = 0, 1, 2, . . . , K -1 do Update ϕ (m) using (10) or Adam-style step; 9: end for Remark 3.1. Note that since ℓ i (ϕ, τ i ) is strictly convex and univariate with respect to τ i , in each iteration m, τ (K) i is guaranteed to be near-optimal (i.e., τ(K) i ≈ τ ⋆ i ).Therefore, the convergence of Algorithm 1 can be guaranteed by the convergence of stochastic gradient descent on the reward model parameter ϕ. Remark 3.2. Computationally, Algorithm 1 incurs negligible additional cost. The inner minimization problem (Lines 5-7 in Algorithm 1) can be solved to near-optimality efficiently within a few iterations (e.g. K = 5) given its convex and univariate nature. The additional overhead of each update is minor compared to the overall RLHF pipeline.Our adaptive preference scaling approach is generic and can be extended to DPO [31] , which is another popular method for policy learning from human preferences. DPO directly learns the policy in supervised manner using the preference data of state-action pairs D pref = (s i , a w,i , a l,i ) N i=1 . This approach forgoes the need to learn the reward function explicitly by the reparameterization of reward function r with respect to its optimal policy π r , r(s, a) = β log(π r (a|s)/π ref (a|s)) + β log Z(s), Similarly. we can integrate adaptive preference scaling into DPO by plugging in (11) into (6) . By merging β with the τ i and ρ, we can further obtain the adaptive DPO (Ada-DPO) formulation asmin π,τ1,...,τ N ∈Ω L Ada-DPO (π, τ 1 , ..., τ N ) := 1 N N i=1 -τ i log σ r π (a w,i |s i ) -r π (a l,i |s i ) τ i + ρτ i .Remark 3.3. Note that the proposed adaptive preference loss can be further combined with other RLHF approaches, such as PEBBLE [24] , SURF [29] , and PARL [11] , which still optimize the standard cross-entropy loss (see [24, Eq. ( 4)], [29, Eq. ( 3)], and [11, Eq. ( 5)]).We now introduce a variant of our adaptive preference loss that uses quadratic regularization for τ . This modification removes the need for the hyperparameter τ max in Ω, easing the tuning effort. We define the following instance-level adaptive preference loss with quadratic regularization:EQUATIONCompared to (4), which includes a linear regularization term of (ρ 0 -log 2)τ , ( 12) modifies the regularization term with coefficient ρ 0 to be quadratic while keeping the term log 2τ linear. Additionally, in (12) , the constraint on τ only specifies a lower bound τ 0 and no longer includes an upper bound τ max . The following proposition provides theoretical insights for this modification.Proposition 3.2. Assume we have a pair of trajectories z 1 , z 2 , and the preference distribution p(z 1 ≻ z 2 ) = p * ∈ (0, 1). Consider the problem of minimizing the expectation of our adaptive loss function with quadratic regularization over the preference distribution: minr,τ ≥τ0 -τ p * log σ (r(z 1 ) -r(z 2 ))/τ -τ (1 -p * ) log σ (r(z 2 ) -r(z 1 ))/τ + ρ 0 τ 2 -log 2τ.Then the minimizer τ * and r * of the expected loss satisfyτ ⋆ = max{τ 0 , (p * log(p * ) + (1 -p * ) log(1 -p * ) + log 2)/(2ρ 0 )}, r * (z 1 ) -r * (z 2 ) = τ * σ -1 (p * ).Here, σ -1 is the inverse of sigmoid function.Note that unlike the adaptive preference loss with linear regularization described in Proposition 3.1, the optimal value τ ⋆ for quadratic regularization does not involve the upper bound τ max . 
experiments: 
 In this section, we examine the effectiveness of our adaptive preference loss based on robotic control and natural language generation tasks. Due to space limit, we defer the experiments with quadratic regularization, ablation studies, and discussions on hyperparameter selection to Appendix C.Experiment setup. We apply our proposed reward learning method on 3 robotic control tasks from the PyBullet [13] environments: HalfCheetah, Ant, and Hopper. These environments are similar to those available in OpenAI Gym [9] but they are known to be much harder to solve [34] . Similarly to Gao et al. [17] , our setting is synthetic, where we use the ground truth rewards to provide preference labels on each pair of samples due to high expense of collecting human preferences. For the reward function, we use two-hidden-layer MLPs, each containing 64 hidden units. This configuration is aligned with the designs of both the policy and value networks. Following Christiano et al. [12] , we repeat the following three steps for each stage: (i) We sample a set of trajectories by the policy π, and update the policy with proximal policy optimization (PPO, Schulman et al. [36] ) alongside a reward function r. (ii) We split the segments (the sequence of state-action pairs) into a training set and a testing set. Then, we randomly sample pairs of segments from the training set, and generate D pref with preference labels. We do the same to the testing set, and generate D ′ pref . (iii) We train the reward function r on D pref , and use D ′ pref for evaluating the preference prediction of r. For notational simplicity, we name our proposed adaptive preference scaling method for reward learning as "Ada-Pref". We compare Ada-Pref with the baseline method "Pref", which uses the standard cross-entropy loss for reward learning. For every 10000 timesteps the policy π runs, we evaluate the learned policy based on 20 test episodes. We also compute the average preference prediction accuracy of the learned reward function across stages. We set the budget to 3 million timesteps and perform training over 10 different seeds. For hyperparameter tuning in both reward learning and policy optimization, we apply two different criteria: 1) We identify the best policy based on its performance (the one with the highest return) and subsequently select the corresponding reward function. 2) We choose the best reward function based on its performance (the one with the highest average preference prediction accuracy) and then select the corresponding policy. Details of the implementations and hyperparameter tuning procedures are in Appendix B.1.Results. We summarize the results on three PyBullet tasks as follows: Table 1 and Figure 2 illustrate the results for Pref and Ada-Pref on the PyBullet tasks, based on the first hyperparameter tuning criterion. In Table 1 , we report the highest return of the best policy and the average preference accuracy of the corresponding reward function. We can see that Ada-Pref consistently outperforms Pref in terms of return on all three tasks and achieves comparable preference accuracy. The upper panel of Figure 2 shows the learning curve plots. We can see that Ada-Pref surpasses Pref at nearly every timestep and reaches a higher plateau across all tasks. The lower panel of Figure 2 presents percentile plots from different seeds to demonstrate individual run behaviors. As shown, we confirm that Ada-Pref consistently outperforms Pref at every percentile across all tasks. while Ada-Pref still outperforms Pref in terms of both preference accuracy and return on all three tasks. Furthermore, Ada-Pref demonstrates greater resistance to performance degradation than Pref, indicating its superior ability to align the learned reward function with policy optimization. This alignment allows for effective policy selection based on preference accuracy without the need to evaluate the policy using ground truth rewards. Experiment setup. We apply DPO with our proposed adaptive loss (Ada-DPO) method to two open-ended text generation tasks: summarization and single-turn dialogue. We adopt the Llama-2 7B model [40] as the backbone and conduct instruction tuning on each task to obtain the initial reference models. For summarization, the policy generates summaries given posts collected from Reddit. We use the filtered TL;DR summarization dataset [41] for instruction tuning, which contains more than 117K Reddit posts, each with a human-written summary. We apply the human preferences collected by Stiennon et al. [38] for preference optimization, where each transcript contains a pair of responses along with a preference label. For single-turn dialogue, the policy responds to various human queries ranging from simple questions to complex demands. We utilize the Anthropic Helpful and Harmless dialogue preferences dataset [3] for both instruction tuning and preference optimization. This dataset contains 170k human-AI dialogues, with each dialogue containing two AI responses and a human preference label. We use the preferred responses for instruction tuning and the full set of preferences for optimization. For instruction tuning stage, we fine-tune the entire Llama-2 model. For the alignment stage using Ada-DPO and different baselines, we apply LoRA fine-tuning for computational efficiency concerns, as we need to simultaneously tune multiple hyperparameters.The rank of the LoRA adaptor is 64. We consider three baseline methods: DPO [31] , Ψ Preference Optimization with Identity Mapping (IPO) [2] and Sequence Likelihood Calibration with Human Feedback (SLiC-HF) [47] .As human evaluation is prohibitively expensive, we use Claude 3 [1] , a proprietary large language model, to automatically evaluate responses based on summary quality and helpfulness/harmlessness for the summarization and dialogue tasks, respectively. Prior work has shown that Claude 3 and GPT-4 can effectively measure a quantitative improvement over the instruction-tuned model [15] . We split a small subset from each instruction tuning dataset for testing and calculate the win rate against the instruction-tuned reference model as the evaluation metric. The percentage of instances where the response generated by policy A is preferred over policy B is referred to as the win rate of A against B. We also split a subset from each preference optimization dataset to validate the preference prediction accuracy. Details of the implementations and hyperparameter selections are in Appendix B.2.Results. We summarize the results on the two natural language generation tasks as follows:In Figure 3 , we select the model with the highest win rate and present the win rate and its preference accuracy for all baselines. We observe that Ada-DPO outperforms the other baselines on both tasks in terms of win rate and achieves comparable preference accuracy. In Figure 4 , we display the performance of the model selected with the highest accuracy (not win rate). As shown, Ada-DPO achieves a significant improvement beyond the DPO baseline in terms of win rate and obtains a comparable preference accuracy. This again indicates that Ada-DPO yields better alignment between the learned reward function and policy optimization, allowing good policy selection based on preference accuracy without a proprietary LLM judge. We present detailed analyses of Ada-Pref and Ada-DPO for both the Ant and summarization tasks. Figure 5 (a) presents a histogram of the learned scaling factors τ for the Ant task. We can see that around 60% of these scaling factors reach the upper bound, while about 10% converge to the lower bound, and the rest are distributed across the region. In Figure 5 (b), we explore the relationship between preference strength and the learned scaling factors τ , and in Figure 5 (c), we investigate the relationship between preference strength and the learned reward difference for Pref and Ada-Pref. We measure preference strength using the true reward difference, categorize it into five percentile bins, and then bin the scaling factors and the learned reward differences accordingly to compute the average. As can be seen, the learned scaling factor increases monotonically with preference strength, demonstrating that the our method successfully adapts the loss scaling to the varying degrees of preference in the data. Furthermore, Ada-Pref learns smaller reward differences for pairs with ambiguous preferences and learns larger reward differences for those with strong preferences, compared to Pref. This indicates that our method leads to a more flexible reward function.In Figure 6 (a), we plot a histogram of the learned scaling factors τ for the summarization task. We can see that around 40% of the scaling factors converge to the upper bound, with the rest distributed across the region. We also display the relationship between the confidence scores and the scaling factors in Figure 6 (b). The confidence score is an integer from 1 to 4 included in the dataset, and a higher score denotes a stronger preference. We bin the scaling factors based on confidence scores and compute the average. As shown, the scaling factors positively correlate with confidence scores, justifying that we learn larger τ for strong preferences and smaller τ for ambiguous ones.We further present two pairs of preference samples where Ada-DPO assigns large or small scaling factors in Figure 7 . We observe that the sample pair with a large scaling factor shows a strong Rejected: preference, as the rejected response is nonsensical while the chosen one is clear. Ada-DPO learns a larger reward difference for such data, while it is much smaller with DPO. Conversely, for the sample pair with a small scaling factor, the two responses are very similar, indicating its ambiguity. Ada-DPO learns a small reward difference on this pair, while DPO gets a large reward difference. 
conclusion: 
 RLHF is an emerging challenge in machine learning. Prior to the popularity of models like ChatGPT, research on designing proper loss functions for reward learning was limited. To bridge this gap, we explore uncertainties in underlying preference strengths and propose an adaptive preference loss function. This loss function incorporates instance-specific scaling factors to modulate the correspondence between reward differences and preference distributions. Taking the result in this paper as an initial start, we expect more sophisticated and stronger follow-up work that applies to RLHF with similar structures. All of these efforts may ultimately assist in developing more principled RLHF methods to better control risks associated with advanced AI systems.A Derivation and proofs of Section 3A.1 Derivation of Equation (4)In this subsection, we present the full derivation of Equation ( 4). Recall the following loss:ℓ r (z 1 , z 2 ) = max p∈∆2 p 1 d r (z 1 , z 2 ) + p 2 d r (z 2 , z 1 ) -τ 0 KL(p, 1/2) s.t. KL(p, 1/2) ≤ ρ 0 .Using the Lagrangian duality, we have maxp∈∆2 min λ≥0 p 1 d r (z 1 , z 2 ) + p 2 d r (z 2 , z 1 ) -τ 0 KL(p, 1/2) -λ(KL(p, 1/2) -ρ 0 ).By strong duality theorem, we have minλ≥0 max p∈∆2 p 1 d r (z 1 , z 2 ) + p 2 d r (z 2 , z 1 ) -τ 0 KL(p, 1/2) -λ(KL(p, 1/2) -ρ 0 ), which is equivalent to min λ≥0 max p∈∆2 p 1 d r (z 1 , z 2 ) + p 2 d r (z 2 , z 1 ) -(λ + τ 0 )(KL(p, 1/2) -ρ 0 ) -τ 0 ρ 0 .We let τ = λ + τ 0 and obtain minτ ≥τ0 max p∈∆2 p 1 d r (z 1 , z 2 ) + p 2 d r (z 2 , z 1 ) -τ (KL(p, 1/2) -ρ 0 ) -τ 0 ρ 0 .Now, we consider the optimality conditions for the inner constrained maximization problem by defining the following Lagrangian function:L r,τ (p, µ) = p 1 d r (z 1 , z 2 ) + p 2 d r (z 2 , z 1 ) -τ (KL(p, 1/2) -ρ 0 ) -µ 2 k=1 p k -1 ,where µ is the Lagrange multiplier. The optimal solutions p r,τ to the inner maximization problem satisfy the following KKT conditions: Without loss of generality, we let z 1 = z w and z 2 = z l , and obtaind r (z 1 , z 2 ) -τ (log(p r,τ ) + 1) -µ = 0, d r (z 2 , z 1 ) -τ (log(p r,τ ) + 1) -µ = 0,min τ ≥τ0 -τ log σ r(z w ) -r(z l ) τ + (ρ 0 -log 2)τ,where σ is the logistic function. This completes the derivation.A.2 Proof of Proposition 3.1We first derive the expectation of the adaptive loss. By taking expectation of (4) with ρ = ρ 0 -log 2, we have:E z1,z2 1(z 1 ≻ z 2 ) -τ log σ (r(z 1 ) -r(z 2 ))/τ + ρτ + 1(z 2 ≻ z 1 ) -τ log σ (r(z 2 ) -r(z 1 ))/τ + ρτ = p * -τ log σ (r(z 1 ) -r(z 2 ))/τ + ρτ + (1 -p * ) -τ log σ (r(z 2 ) -r(z 1 ))/τ + ρτ = -τ p * log σ (r(z 1 ) -r(z 2 ))/τ -τ (1 -p * ) log σ (r(z 2 ) -r(z 1 ))/τ + ρτ.By the optimality condition of r(z 1 ) -r(z 2 ) and τ , we have r(z 1 ) -r(z 2 ) = τ σ -1 (p * ), (13) where σ -1 is the inverse of sigmoid function. Plugging (13) into the objective in (7) , we obtain minτ ∈Ω -p * log(p * ) -(1 -p * ) log(1 -p * ) + ρ τ,whose objective is essentially linear in τ . Hence, when -p * log(p * )-(1-p * ) log(1-p * )+ρ > 0, the corresponding optimal τ * is at the lower bound τ 0 and the optimal reward difference r * (z 1 )-r * (z 2 ) = τ 0 σ -1 (p * ) given the optimality condition. Conversely, when -p * log(p * )-(1-p * ) log(1-p * )+ρ < 0, we have τ * = τ max and r * (z 1 ) -r * (z 2 ) = τ max σ -1 (p * ). This completes the proof. 
 
 Appendix: We first derive the expectation of the adaptive loss with quadratic regularization. By taking expectation of ( 12), we have:EQUATION) By the optimality condition of r(z 1 ) -r(z 2 ) and τ , we haveEQUATIONwhere σ -1 is the inverse of sigmoid function. Plugging (15) into the objective in (14) , we obtain minEQUATION)Note that ( 16) is always bounded without the need of an upper bound of τ . Specifically, with any p ⋆ ∈ (0, 1), we have -p * log(p * )-(1-p * ) log(1-p * )-log 2 ≤ 0 and τ ⋆ = max{τ 0 , (p * log(p * )+ (1 -p * ) log(1 -p * ) + log 2)/(2ρ 0 )}. This completes the proof.Our implementations of robotic control tasks are based on Stable-Baselines3 [33] and RL Zoo training framework [32] . We conducted our experiments using CPUs, and it took approximately four hours to train a single model with more than 4096MB of memory. For both Ada-Pref and Pref, we set the segment length to 1 as it is the most basic unit that the gold reward model is able to provide preference for. Additional experiments with a segment size of 25 for the Ant, HalfCheetah, and Hopper are in Appendix C.2. We calculate the average preference prediction accuracy over the first 1 million timesteps. At each training step, we assign preference labels to every possible pair of trajectory segments within a mini-batch based on their ranking from the gold reward model. We set the batch size to 64 for the HalfCheetah and Ant tasks and 4 for the Hopper task. We tune the number of epochs in {1, 3, 5}. We use Adam optimizer [22] and tune the learning rate in {5e -3, 1e -3, 5e -4, 1e -4} for the Ant and HalfCheetah, and set the learning rate to 1e -2 for the Hopper. For Ada-Pref, we tune the τ max in {1.0, 3.0} and the ρ 0 in {0.1, 0.3, 0.5}. We fix τ 0 = 0.1 and the number of Newton iterations to 3 for all experiments. Details of the chosen hyperparameters for reward learning for all three tasks are summarized in Tables 3 and 4 . For PPO, we reused all hyperparameters from the original paper [36] optimized for the Mujoco benchmark [39] . Details of the hyperparameters for PPO are in Table 5 . Our implementations of natural language generation tasks are based on transformers [46] and trl training framework [42] . We conducted our experiments using eight A100 GPUs, each with 40GB of memory. Training a single model took approximately two hours. We provide more details on each task as follows: For the instruction tuning stage, we randomly select 800 data from the filtered TL;DR summarization dataset [41] for testing the policy and leave the rest for supervised tuning. In the preference optimization stage, we split the preference dataset [38] into a training and testing set to evaluate the preference accuracy. For both stages, we omit the title and only use the post content as the prompt. The prompt format follows: "POST: post content.\n\nTL;DR:".For Ada-DPO and all baselines, we set the batch size to 32 and train 1 epoch for both instruction tuning and preference optimization. We set the α parameters of LoRA fine-tuning to 16, and tune the other parameters by grid search. The learning rate is tuned in {5e -6, 5e -5, 1e -4, 5e -4}. SLiC-HF, IPO and DPO include parameter β, which is tuned in a range of {0.01, 0.1, 0.3, 0.5}. For Ada-DPO, we tune the ρ 0 in {0.05, 0.1, 0.3, 0.5} and the τ max in {1.0, 4.0, 5.0, 10.0}. We fix τ 0 = 0.1 and the number of Newton iterations to 5 for all experiments. The best Ada-DPO is achieved with lr = 5e -5, ρ 0 = 0.1, and τ max = 4.0.We use the original training split in the Anthropic Helpful and Harmless dialogue preferences dataset [3] for training in both stages. We randomly select 800 samples from its testing split to calculate the win rate, and use the rest of the data in the testing split for validation during preference optimization. We use the original data format.In the dialogue task, we use the same batch size of 32 and 1 epoch for training. The learning rate is tuned in {5e -6, 5e -5, 1e -4}. The parameter β for baselines is tuned in a range of {0.01, 0.1, 0.3}. For Ada-DPO, we tune the ρ 0 in {0.05, 0.1, 0.3, 0.5}, τ max in {1, 5, 10} and fix τ 0 = 0.1. The best Ada-DPO is achieved with lr = 5e -5, ρ 0 = 0.05, and τ max = 5.0.We apply Claude 3 for win rate judgments. Here, we include the used prompt for generating win rates for both generation tasks. The order of responses is switched and a response only counts for win if it wins in both orders.Summarization win rate prompt.factors τ . Compared to Figure 5 (a), we can see much smoother distribution of τ due to the quadratic regularization. Figures 8(d ) and 8(e) illustrate the relationship between the learned scaling factors τ and preference strength, and the relationship between the learned reward difference and preference strength. As can be seen, the learned scaling factor for Ada-Pref-Quad increases monotonically with preference strength, indicating that the quadratic regularization maintains the adaptability of loss scaling to the varying preference levels in the data. Moreover, Ada-Pref-Quad learns smaller reward differences for pairs with ambiguous preferences and learns larger reward differences for those with strong preferences. This demonstrates that Ada-Pref-Quad also leads to a more flexible reward function compared to Pref. In this subsection, we present the results for three PyBullet tasks, using a segment size of 25. Table 7 and Figure 9 show the performance of Pref and Ada-Pref on the PyBullet tasks, based on the first hyperparameter tuning criterion. Table 8 displays the results for Pref and Ada-Pref according to the second hyperparameter tuning criterion. These results reconfirm the effectiveness of our adaptive preference loss. Compared to the cross-entropy loss, our method needs three additional hyperparameters: the bounds on the scaling factors τ 0 and τ max , and the regularization parameter ρ. In our experiments, we fixed τ 0 at 0.1 without tuning it, as this value worked well for all five tasks. We did tune τ max to adjust the scale of τ , but this can be avoided by using the quadratic regularization formulation described in Section 3.6. The parameter ρ turns out to be more important, because it controls the distribution of the scaling factors. We performed a careful grid search to tune ρ in our experiments. Figure 10 shows the hyperparameter sensitivity of ρ on the Ant and summarization tasks. Overall, we found that smaller values of ρ often lead to better performance. 


Decision: The paper presents a novel adaptive preference loss for RLHF, integrating a distributionally robust optimization approach to handle varying strengths in human preferences. The reviewers generally appreciate the motivation behind the method and acknowledge its potential impact, especially in aligning rewards with policy optimization across robotic control and NLP tasks.

**Strengths:**
- The proposed method is theoretically sound, with robust experimentation across different domains.
- The flexibility introduced by the adaptive scaling parameters addresses a significant limitation in previous RLHF methods.

**Weaknesses:**
- Reviewers noted concerns about the weak experimental ablation, particularly in NLP tasks, where only a single run was initially conducted. The authors provided additional results with repeated runs, which addressed most concerns, but some reviewers remain unconvinced about the conclusiveness of these results.
- The lack of comparison with key benchmarks like PEBBLE, SURF, and PARL in the robotics domain was pointed out. The authors argue that their method is orthogonal to these benchmarks and could complement rather than directly compete with them.
- The computational overhead of the proposed method was discussed, with the authors asserting that the additional costs are negligible. However, the practical implications of this overhead in real-world settings remain a point of contention among reviewers.

**Unresolved Points:**
- Some reviewers were not entirely satisfied with the mathematical justifications provided regarding the linear dependence issue in preference modeling, indicating that the explanation could be clearer.
- The necessity of the proposed method versus simpler alternatives (e.g., reward margin methods) is still debated, with some reviewers questioning if the complexity of the method is justified by the performance gains.
- Concerns about the practical implementation and scalability of the method in real-world mini-batch learning scenarios were raised and only partially addressed by the authors.

**Recommendation:** 
The paper is borderline, with a slight leaning towards acceptance due to its innovative approach and potential impact. However, the unresolved issues, particularly regarding experimental robustness and practical scalability, suggest that further revisions and clarifications are necessary for a more confident endorsement.

Reviews: review: paper_summary: paper_summary: The paper focuses on redesigning the loss function with adaptive scaling parameters to deal with the uncertainty in the preferences and thus improving the reward modeling flexibility.  In the context of both robotics and NLP, the algorithm with the new loss shows improved performance.
strength_weakness: strength_weakness: 1. The experimental ablation is weak and doesn't provide concrete indications of the strength of the proposed algorithm. For ex in Figure 3, the win-rate gap is marginal and not significant. Why is the performance in summarization better in Figure 4 is not extremely clear?

2. The experimental evaluations lack important benchmarks and comparisons. For the Robotics environment, it's crucial to compare with Pebble, SURF, and PARL to understand the true efficacy of the proposed approach.
questions: questions: 1. Can the authors provide more concrete mathematical justifications regarding the issue with linear depedence with the logit of the preference distribution? Specifically, connecting with the relation to noisy preferences?
2. "For some applications, it may lead to a reward function that is not flexible enough to differentiate a
141 pair of segments, which are supposed to have significantly different rewards." Can the authors please highlight such examples intuitively or mathematically, why such a collapse might happen?
3. The authors mention "Our proposed method is inspired by DRO, it serves a distinct purpose: improving reward learning in RLHF, which is orthogonal to distributional robustness". Can the authors pls justify this statement and discuss why it's orthogonal? DRO in reward learning will also improve the robustness and flexibility of the reward model to noisy preferences, in what sense it does better than DRO?
4. Can the authors provide a more concrete definition of the best win rate and best prediction accuracy?
5. It will be helpful if the authors can provide more details on how the trajectories are constructed for the robotics experiments.
limitations: limitations: Check above
paper_summary: paper_summary: The paper identifies a limitation in RLHF methods, noting that ranking over pairs of trajectory segments often fails to capture the varying strengths of preferences across different pairs. To address this, the paper proposes a new adaptive preference loss (Ada-DPO), underpinned by distributionally robust optimization (DRO). The proposed method improves policy performance and is supported by theoretical proofs on convexity and univariate analysis.
strength_weakness: strength_weakness: - It seems that only a single run was conducted for the experiments in the NLP tasks. More repeated runs would be more convincing, especially since the differences in performance are so close. Additionally, p-value testing for that domain would help determine if the differences are truly significant.
questions: questions: - I would like to see more examples like Figure 7, comparing the learned reward differences with different scaling factors.
limitations: limitations: - More discussion on the limitations of the proposed method and future directions can be added.
paper_summary: paper_summary: This paper studies the problem of learning from preference data and introduces a learnable scaling parameter for each preference sample. The authors propose an adaptive preference loss function that assign small scaling parameters to ambiguous preferences pairs and large scaling parameters to clear preferences. Experiments demonstrate the improvement of policy optimization performance and efficient of the hyperparameter tuning.
strength_weakness: strength_weakness: The increase in flexibility of the proposed reward function is not verified. The authors claim that one of the limitations of the BT model is the logit of the preference distribution scales linearly with the reward difference. However, according to Proposition 3.1, the limitation still exists. There is no empirical result that supports the claim.	

Missing the ablation study of the regularization term in the proposed loss function.

Missing the RLHF baseline in the natural language generation task. 

“Ada-Pref demonstrates greater resistance to performance degradation than Pref, indicating its superior ability to align the learned reward function with policy optimization.” It seems that it is not true in the Hopper task?

Minors: 
Line 120: The notation of state transition function is conflicted with preference distribution.
Please clarify the meaning of M and K in Algorithm 1.
questions: questions: I do not believe the proposed loss function is convex, can you provide the proof of Remark 3.1?
Why the reward models trained with the proposed method can better guide policy model selection considering the reward model performance is not improved?

How do the authors obtain the true reward difference in Section 4.3?
limitations: limitations: The computation overhead introduced by learning the scaling parameter is not discussed.
paper_summary: paper_summary: To learn a versatile rewards essential for the downstream policy optimization, this paper introduces a novel adaptive preference loss function inspired by distributionally robust optimization (DRO).
The proposed approach incorporates an learnable instance-specific scaling factor to accommodate varying uncertainties of preference strength and change the scaling between the preference distribution and reward difference to be non-linear.
The proposed loss function for learning the scaling parameters is strictly convex, and thus they only induce negligible additional computational overhead.
Experiment results on robotic control tasks and LLMs verify the proposed method.
strength_weakness: strength_weakness: 1. The adaptive scaling is defined on a per-instance basis, which necessitates significant compute costs and hinders real-world mini-batch learning. This is evident from Appendix, where the algorithm box shows that the training process operates in a per-datapoint manner
2. Some of the derivation may be simplified, both to make the paper more succinct and leave room for algorithm box as well as other technical details.
3. L221: could you theoretically justify the incorporation of adaptive preference scaling into DPO. e.g., in the framework of KL-control as in the original DPO paper?
questions: questions: 1. L71 "Prior work on this topic is very limited.": There are actually a vast pool of literature concerning loss functions for (RLHF's) reward modeling, a quick pointer is [1] for language tasks and [2]  for image generation, as well as the literature review in them. I encourage the authors to add proper discussions and citations to make the claims more precise.
2. L140-141: Could you elaborate this sentence more: "For some applications, it may lead to a reward function that is not flexible enough to differentiate a pair of segments"? In particular, why is the resulted reward function not flexible enough? And how is this related to the linear scaling?
3. In Eq. (3), why do we need to have $KL(p, 1/2)$ in both the objective function and constraint?



***
[1] Yang, Shentao, et al. "Preference-grounded token-level guidance for language model fine-tuning." Advances in Neural Information Processing Systems 36 (2023).

[2] Yang, Shentao, Tianqi Chen, and Mingyuan Zhou. "A Dense Reward View on Aligning Text-to-Image Diffusion with Preference." Forty-first International Conference on Machine Learning. 2024.
limitations: limitations: A discussion on limitations and/or potential negative societal impact seems missing.

Comments: title: Thank you for the discussion
Thank you for the discussion
Response to Rebuttal by Authors
Response to the authors
Thank you for the thorough review
Response to the authors
comment: Thank you for your insightful comment. We would like to clarify that the BT model does not completely fail but is indeed insufficient in some cases. For example, when the relationship between the ground truth reward difference and the logit is non-linear, the BT model cannot fully capture and learn this correspondence. In our synthetic case, we only know the ground truth reward, not the logit of the preference distribution, so we can't provide direct evidence even in a toy example. However, the superior performance of our method across various tasks suggests that non-linear relationships do exist in real data. While the overall performance of the BT model is ok, our method offers greater flexibility and consistently delivers better results.
I appreciate the authors' detailed reply. They have addressed most of my concerns. 

However, there is no evidence (even without a toy example) demonstrating that the BT model would fail due to its linear relationship. This makes it unclear for researchers to determine when the proposed method, which seems hard to re-implement,  should be used.

Therefore, I would keep my previous evaluation.
Thank you for your thoughtful comments!

Due to NeurIPS regulations, we are unable to upload figures or provide links. However, we would like to clarify the following points:

The BT model assumes a linear relationship between the reward difference and the logit of the preference distribution. This means it cannot accurately represent situations where the relationship is non-linear. In contrast, our method models this relationship as non-linear. Specifically, with linear regularization, the relationship becomes piecewise-linear (see Proposition 3.1), and with quadratic regularization, the relationship is more complexly non-linear (see Proposition 3.2).

In both cases, it is important to note that when the logit value is small, our method learns a smaller reward difference. Conversely, when the logit value is large, our method learns a larger reward difference compared to the BT model. This adaptive, non-linear relationship makes our method more flexible and better suited for capturing complex preference dynamics that the BT model cannot handle.

Further examples of the benefits of this non-linear relationship are provided in our response to Q1 of reviewer E9fU. In Example 1 (low preference strength), Example 2 (moderate preference strength), and Example 3 (large preference strength), the reward differences for DPO are 0.64, 1.07, and 1.47, respectively. Meanwhile, the reward differences for Ada-DPO are 0.31, 0.88, and 2.83, respectively, demonstrating that Ada-DPO scales more appropriately across varying levels of preference strength.

Regarding the learned scaling parameter shown in Figure 5(b), we want to clarify that the near-linearity of the scaling factor does not imply a linear relationship between the learned logits and preference strength.

Lastly, in the additional experiment, we implemented the reward margin method from [1] on three robotic control tasks. We did not include other reward margin methods, such as those from [2] and [3], because they require additional data called "score", which is expensive to obtain, making a fair comparison difficult. As shown in the table below, the reward margin method from [1] not only fails to match our method’s performance but also performs significantly worse than standard RLHF.

| Task         | Method   | Return  |
|--------------|----------|---------|
| HalfCheetah  | Pref     | 2724.42 |
|                       | Margin | 577.98 |
|              | Ada-Pref | **2875.45** |
| Ant          | Pref     | 2917.81 |
|              | Margin | 866.5 |
|              | Ada-Pref | **3177.11** |
| Hopper       | Pref     | 1324.91 |
|              | Margin | 40.13 |
|              | Ada-Pref | **1692.1**  |

[1] Qin, Bowen, Duanyu Feng, and Xi Yang. "Towards Understanding the Influence of Reward Margin on Preference Model Performance." arXiv preprint arXiv:2404.04932 (2024). 

[2] Touvron, Hugo, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov et al. "Llama 2: Open foundation and fine-tuned chat models." arXiv preprint arXiv:2307.09288 (2023).

[3] Amini, Afra, Tim Vieira, and Ryan Cotterell. "Direct preference optimization with an offset." arXiv preprint arXiv:2402.10571 (2024).
Thank you for your valuable comments. We will include the mentioned references and the discussion in the paper. Additionally, we would like to address your remaining concern regarding the mathematical justification related to the issue of linear dependence.

The BT model assumes a linear relationship between the reward difference ($\Delta r$) and the logit of the preference distribution ($\ell$), expressed as $\Delta r = \ell$. This assumption inherently limits the model’s ability to accurately capture scenarios where this relationship is non-linear.

In contrast, our method models this relationship as non-linear. Specifically, in the case of linear regularization, the relationship is piecewise-linear and can be expressed as: $\Delta r = \tau_0\ell , \text{if } -t < \ell \leq t; \Delta r=\tau_\max\ell, \text{otherwise},$ where $t$ is a threshold related to $\rho$, and $\tau_0 < 1 < \tau_\max$ (see Proposition 3.1). For quadratic regularization, the relationship becomes even more complexly non-linear (see Proposition 3.2). Importantly, in both cases, when $\ell$ is small, $\Delta r$ is smaller, and when $\ell$ is large, $\Delta r$ is larger compared to the BT model. This adaptive, non-linear relationship enhances our method’s flexibility and allows it to better capture complex preference dynamics that the BT model cannot.

Although we intended to include visualizations of these relationships to make them more intuitive, NeurIPS regulations prevent us from doing so in the rebuttal. However, we will add these plots in the paper to provide better clarification.

Additionally, please refer to the examples on the summarization task in our response to Q1 for reviewer E9fU, which further illustrate how this non-linearity adapts to the data.

Lastly, we noticed that, despite your mention of potentially increasing the score, it has remained unchanged. We would greatly appreciate it if you could reconsider the score in light of the explanations we provided.
Thanks for providing concrete answers to my queries and issues. Most of my concerns have been resolved, hence I am increasing my score. Please add the references mentioned in detail and explain the reason for not comparing them with the baselines.

However,  mathematical justifications regarding the issue with linear dependence with the logit of the preference distribution is not very clear and would request authors to provide a clear explaination mathematically.
Thank you for the informative response!

Can you further visualize situations in which the BT model cannot express, maybe in your synthetic setting where the true preference strengths can be captured? The learned scaling parameter also seems linear regarding preference strength, as shown in Figure 5(b).  

If this motivation is not well-established, adding Adaptive Preference Scaling that has complicated implementation is not worth the cost. I also wonder if using simpler approaches, such as adding a reward margin to ranking loss or adding another layer of (non-) linear function outside the sigmoid function, can achieve similar or better performance.
Dear Reviewer,

Thank you for your thoughtful feedback and for increasing the scores toward acceptance. We will incorporate your suggestions in our next version.

Best regards,

The Authors
Dear authors, 

Thank you so much for the helpful responses. I hope these discussions and revisions can be incorporated into the next version of the manuscript. I've increased my rating to 6.