Title: KL Penalty Control via Perturbation for Direct Preference Optimization
Venue: NeurIPS 2025
Accepted: yes
Keywords: Direct Alignment Algorithm

Reinforcement Learning from Human Feedback

Human Preference

LLM

Alignment
Abstract: Direct Preference Optimization (DPO) demonstrates the advantage of aligning a large language model with human preference using only an offline dataset. However, DPO has the limitation that the KL penalty, which prevents excessive deviation from the reference model, is static throughout the training process. Several methods claim to change this static KL penalty of DPO into a dynamic one, but no approach can adaptively assign different KL penalties for each preference pair. In this paper, we propose $\varepsilon$-Direct Preference Optimization ($\varepsilon$-DPO), which allows adaptive control of the KL penalty strength $\beta$ for each preference pair. Specifically, $\varepsilon$-DPO adaptively controls $\beta$ for each preference pair based on the monotonicity of logits as a preference model under the perturbation of $\beta$ during training. This is equivalent to adjusting the KL penalty by checking whether the change in training-time temperature can lead to better preference confidence as preference models by simply reusing the logit of the current policy and the reference policy. Experimental results show that the simple criterion of $\varepsilon$-DPO for KL penalty relaxation significantly improves DPO compared to most existing direct alignment algorithms on general chatbot benchmarks and reveal that this KL penalty control criterion can reflect confusion as a preference model and provide an efficient KL trade-off, highlighting the significance of instance-level adaptive KL penalty control in DPO.

KL Penalty Control via Perturbation for Direct Preference Optimization

Abstract

Direct Preference Optimization (DPO) demonstrates the advantage of aligning a large language model with human preference using only an offline dataset. However, DPO has the limitation that the KL penalty, which prevents excessive deviation from the reference model, is static throughout the training process. Several methods claim to change this static KL penalty of DPO into a dynamic one, but no approach can adaptively assign different KL penalties for each preference pair. In this paper, we propose ε -Direct Preference Optimization ( ε -DPO), which allows adaptive control of the KL penalty strength β for each preference pair. Specifically, ε -DPO adaptively controls β for each preference pair based on the monotonicity of logits as a preference model under the perturbation of β during training. This is equivalent to adjusting the KL penalty by checking whether the change in training-time temperature can lead to better preference confidence as preference models by simply reusing the logit of the current policy and the reference policy. Experimental results show that the simple criterion of ε -DPO for KL penalty relaxation significantly improves DPO compared to most existing direct alignment algorithms on general chatbot benchmarks and reveal that this KL penalty control criterion can reflect confusion as a preference model and provide an efficient KL trade-off, highlighting the significance of instance-level adaptive KL penalty control in DPO. 1

1 Introduction

Aligning large language models with human preferences for helpfulness and harmless principles [2, 4, 9] is a crucial requirement for general chatbot agents. Reinforcement Learning from Human Feedback (RLHF) [46] is the pioneering approach that regards the alignment of large language models as a reward maximization problem and solves it by reinforcement learning [37]. However, the complicated training pipeline of RLHF increases the training complexity and computation cost of the rollout for online reinforcement learning, in addition to the difficulty of collecting human preference datasets. Moreover, introducing a trained reward model as a proxy reward function to replace the intractable ground-truth human preference reward function makes large language models suffer from the side effect of reward over-optimization [13] inherited from the reward models.

Direct Preference Optimization (DPO) [32] proposes an alternative approach to reform the limitation of RLHF by converting the policy optimization problem into a preference modeling problem and performing alignment using only the offline preference dataset. It shows comparable performance

1) Monotonically Decreasing

2) Monotonically Increasing

while skipping the reward modeling process required by RLHF and has become an effective alternative approach for alignment. In particular, subsequent studies with various modifications to the DPO objective function open a new research domain called direct alignment algorithms [33], which perform alignment directly from offline preference datasets without training separate reward models.

However, DPO assumes that β and the reference policy, which define a KL penalty that prevents excessive deviations from the reference model in RLHF, are fixed for exploiting the existence of a closed-form solution derived from the objective function of the RLHF. However, this assumption can lead to suboptimal results, since the KL penalty can be regarded as a Lagrangian relaxation of the constraint optimization defined by the trust region [37]. In this regard, β -DPO [41] argues that β should be adaptively chosen according to the quality of the preference pair but fails to control β at the instance-level and proposes a batch-level control method. On the other hand, TR-DPO [14] claims to periodically update the reference policy to reduce over-optimization [33], but it may induce unnecessary KL divergence for improvement since the update is not adaptive.

In this paper, we present ε -Direct Preference Optimization ( ε -DPO) , a simple instance-level adaptive KL penalty control for DPO that neither TR-DPO nor β -DPO achieves. Specifically, we check the advantage of adjusting β for each preference pair by observing the monotonicity of the log-likelihood ratio between the chosen response and the rejected response if the β used during training was perturbed, as described in Figure 1. Here, the criterion for controlling β does not require batch-level statistics, and the policy under the perturbed β can be estimated by reusing the logits from the policy and reference policy. This criterion results in independence from the choice of micro-batch size and no additional computation requirements for model updates, unlike β -DPO and TR-DPO.

Experimental results demonstrate that the instance-level adaptive criterion of ε -DPO remarkably improves DPO, better than β -DPO and TR-DPO, to outperform most direct alignment algorithms that modify the DPO objective function [43, 45, 3, 42, 12, 18, 31, 29]. This reveals that the static KL penalty of DPO is the major bottleneck to final model performance and highlights the importance of instance-level adaptive KL penalty control. Furthermore, we confirm that the variation of β determined by the adaptive criterion in ε -DPO reflects the confusion as a preference model, which is not addressed in the adaptive β control criterion proposed by β -DPO. We also find that the adaptive KL penalty control of ε -DPO is crucial for an efficient KL trade-off compared to TR-DPO, which is not an adaptive KL penalty control because of the periodic update of the reference policy.

Our contribution to the alignment of large language models can be summarized as threefold: (1) We present ε -DPO that changes the static KL penalty of DPO and adaptively controls the KL penalty at the instance-level with a simple criterion, which can outperform most direct alignment algorithms proposed as alternatives to DPO in general chatbot benchmarks. (2) We show that DPO's perspective, which reparameterizes the policy to a preference model, can be converted to an approach for controlling the KL penalty at the instance-level by estimating the preference confidence changes according to the perturbation of β used during training. (3) We demonstrate that this instance-level adaptive KL penalty control distinguishes confusing preference pairs and achieves an efficient KL trade-off, and neither is addressed in existing research that performs KL penalty relaxation.

2 Preliminaries

Reinforcement Learning from Human Feedback To obtain a language model that aligns with human preference, RLHF [46] introduces reinforcement learning. It is equivalent to approaching preference alignment as a reward maximization problem, where we find a policy π that maximizes a ground-truth reward function r ∗ representing a human preference score for a response y obtained

from a corresponding policy for a given prompt x . However, since the ground-truth reward function cannot be accessed, a reward model trained from the preference dataset is introduced as a proxy reward function. On the other hand, to prevent the policy update from deviating too much from the current policy from the initial policy, the KL divergence from the reference policy π ref serves as a penalty and regards the initial policy as a reference policy. At this time, the coefficient β controls the strength of the penalty. The optimal policy that satisfies the maximization of the modified objective function under β has a closed-form solution π ∗ β with an intractable normalizing constant Z ∗ β ,

Direct Preference Optimization RLHF has a limitation in efficiency due to the additional training step of the reward model. In this respect, DPO [32] proposes an approach that can perform preference alignment without training the reward model. DPO focuses on the fact that the ground-truth reward function can be implicitly reparameterized by the closed-form solution π ∗ β and reference policy π ref . If we assume the Bradley-Terry model [6] for the ground-truth human preference function, then the human preference can be modeled by the margin between the reward of the chosen response y w and the rejected response y l with the sigmoid function σ , which can cancel out the intractable term Z ∗ β . From this observation, DPO performs preference alignment through preference model optimization using an offline dataset in the sense that obtaining an optimal policy through policy optimization in RLHF can be obtained by training a preference model given by the implicit reward r θ,β ,

3 ε -Direct Preference Optimization

In this section, we describe our proposed method, ε -Direct Preference Optimization ( ε -DPO) , that adaptively controls the KL penalty coefficient β at the instance-level based on the logit monotonicity as a preference model according to the perturbation of β . Figure 2 illustrates the difference between ε -DPO and existing KL penalty relaxation methods for DPO, β -DPO [41] and TR-DPO [14].

3.1 Relaxation of the KL Penalty in DPO

The KL penalty introduced by RLHF can be regarded as an approach to solve the constrained optimization problem in the trust region [36] defined near the reference policy π ref as an unconstrained optimization by treating β as a Lagrange multiplier [37]. From this perspective, even though DPO reformulates the problem of finding an optimal policy under fixed π ref and β as a preference modeling problem, using a single β and a fixed trust region for all instances may lead to suboptimal results. This hypothesis regarding relaxation of KL penalty can be supported by the experimental results of β -DPO [41] that adaptively control β based on the statistics of implicit reward margin during the training process and TR-DPO [14] that updates π ref during the training process for preventing over-optimization [33] from the vanishing curvature of the loss landscape.

However, β -DPO fails to perform instance-level β control despite claiming that the quality of each preference pair should determine β . Instead, it performs batch-level β control using momentumbased estimation of batch-level margin disparities, which is strongly affected by the micro-batch size. In addition, TR-DPO updates the reference model without adaptive criteria, which can lead to inefficient KL divergence trade-off between performance and incur computational costs for updating the reference model. Therefore, instance-level adaptive KL penalty control without requiring additional computational cost that achieves an efficient KL trade-off is still undiscovered for DPO.

-DPO

TR-DPO

-DPO

Update the

reference policy

Check

batch-level statistics by momentum

Compare

Perturbations of

for each pair

βbatch

Tθ

Tref

log

π(β)(3|)

T(β-)(31x)

log

π0(3)(y|x)

π()(y|x)

log

π(3)(3|a)

π(3)(3lx)

ch = Ebatch[ro,β(x,y") -r0,β(x,y')]

Tref ← απθ + (1 -α)πref

ifmonotonicallydecrease

h ←[1 + α(Mbatch － Mhistory)]B

h

or

β←

if monotonically increase

Tref ← πo at each T step

otherwise

Mbatch = Ebatch[ro,β(x, y") - ro,3(c,y)]

πref < απe + (1 -α)πref

βbatch ← [1 + α(Mbatch － Mhistory)]β

or

Mhistory

←mMhistory

y+ (1-m)Mbatch

Tref ←πo at each T step

3.2 Logit Monotonicity Under the KL Penalty Perturbation

Establishing a criterion for adaptively changing the KL penalty for each instance in the preference dataset is not a trivial problem. As a proxy criterion, we can revisit the DPO's assumption on the ground-truth preference model, the Bradley-Terry model [6], which assumes that the reward difference between two candidates imposes a total ordering. Formally, the policy obtained via DPO can function as a preference model P θ,β which can be expressed as a binary classifier,

by regarding the log-likelihood ratio between the chosen response and the rejected response from a preference triplet ( x, y w , y l ) ∈ D as a logit z θ ( x, y w , y l ) and adaptive margin γ ( x, y w , y l ) ,

This reveals that the KL penalty coefficient β also serves as an inverse temperature of a binary classifier. For a given β , we define β -ε and β + ε with a positive constant ε > 0 . That is, β -ε and β + ε refer to values that have been perturbed to be slightly larger or slightly smaller than the β ,

Let us denote the parameters obtained via DPO as a function of β , θ ( β ) : R + → Θ . Consider the case we observe the strict monotonicity of logits happens according to the perturbation of β on θ ( β ) ,

Suppose we assume that the observation corresponds to the hard label in the ground-truth preference model (i.e. P ( y w ≻ y l | x ) = 1 ). In this case, the preference model with a larger logit is considered more accurate under the Bradley-Terry model. Alternatively, it is equivalent to checking monotonic changes in preference confidence under the perturbation of training-time inverse temperature β by whether the better separation of y w and y l can be obtainable at the same test-time temperature scaling [15] in the neighborhood of 1 β . From this criterion, we can estimate the direction of adjusting β for each instance within the neighborhood defined by ε to increase preference confidence.

3.3 Estimating Policies Under the KL Penalty Perturbation

Note that θ ( β ) is an intractable function since it is equivalent to having access to models trained on each β in the definition. However, Liu et al. [28] shows that optimal policy under β λ can be expressed through π ∗ β by re-weighting with importance ratio using π ref. If we assume the autoregressive prior of optimal policy, then the optimal policy under β λ can be estimated by the optimal policy under β and the reference policy, as we respecify the observation of Liu et al. [28] as Proposition 1,

Proposition 1 (Liu et al. [28]) Under the assumption of optimal autoregressive policy π ∗ where the prompt x ∈ X , response vocabulary y i ∈ V , and logit f : X × V i -1 → R |V| , the optimal policy π ∗ β λ can be approximated by the arithmetic mean of logits between π ∗ β and reference policy π ref ,

Proof. See Appendix A.

Using Proposition 1, we can approximate π θ ( β -ε ) and π θ ( β + ε ) by trained policy and reference policy without accessing θ ( β ) since they are the approximated policies for π ∗ β -ε and π ∗ β + ε . To adaptively control β for each preference triplet ( x, y w , y l ) during the training process, we regard the policy π θ obtained in the current training step as the our best approximation of the optimal policy defined under current β and estimate π θ ( β -ε ) and π θ ( β + ε ) for approximating intractable z θ ( β -ε ) and z θ ( β + ε ) ,

Recall that we need not only the logit of the current policy f θ but also the logit of the reference policy f ref to compute the estimated log-likelihood ratio. However, in order to compute the loss function of DPO, L DPO, the log-likelihood from the reference policy must be computed for each training instance, which allows us to simply reuse f ref for estimation without any additional computation cost of model forward passes. Therefore, we determine the ˜ β , which is used for the KL penalty coefficient in the current training step for each training preference triple instance ( x, y w , y l ) ,

After the model update, the β , which corresponds to the optimal policy that the current policy targets, should be changed depending on ˜ β used in L DPO for each instance. Therefore, we need to modify the baseline β for the next training step, and we simply update the β with the mean statistics of ˜ β determined across the batch used in the update. Note that ˜ β is determined independently of the batch-level statistic, so the adaptive control of β in ε -DPO can be performed independently of the choice of micro-batch size. Algorithm 1 summarizes the entire training process of ε -DPO.

Algorithm 1 ε -Direct Preference Optimization

Require: policy π θ , reference policy π ref, initial KL penalty coefficient β , and perturbation size ε

1: while not converged do

2: Sample training batch of preference triplet ( x, y w , y l ) ∼ D .

3: Estimate the policies under the perturbation π ˆ θ ( β -ε ) and π ˆ θ ( β + ε ) according to 3 and 4.

4: Determine instance-level KL penalty coefficients ˜ β ( x, y w , y l ; θ ) according to 5.

5: Update π θ by L DPO with ˜ β ( x, y w , y l ; θ ) and then β ← E x,y w ,y l [ ˜ β ( x, y w , y l ; θ )] .

6: end while

7: return aligned policy π θ .

4 Experiments

In this section, we conduct experiments to validate the ε -DPO. We mainly check the feasibility of ε -DPO for general chatbot alignment using UltraFeedback [9], compared to the direct alignment algorithms [32, 43, 45, 3, 42, 12, 18, 31, 29]. We also use Anthropic-HH [4] for analyzing the proposed adaptive KL penalty control and comparing with existing methods for KL penalty relaxation of DPO [41, 14]. The implementation details for each experimental setting are in Appendix B.

4.1 Datasets and Evaluations

UltraFeedback UltraFeedback [9] is an AI feedback dataset where GPT-4 [1] rates responses obtained from four different language models. We strictly follow the experimental setting proposed by SimPO [29], which conducts broad range of hyperparameter search then comparing best performance of various direct alignment algorithms including DPO [32], RRHF [43], SLiC-HF [45], IPO [3], CPO [42], KTO [12], ORPO [18], and R-DPO [31] for robust comparison due to the hyperparamter sensitivity of direct alignment algorithms. Specifically, we use the Instruct setting starting from Mistral-7B-Instruct-v0.2 [20] and Meta-Llama-3-8B-Instruct [10]. We evaluate resulting models by general chatbot benchmarks, AlpacaEval 2 [11], Arena-Hard [25], and MT-Bench [21].

Anthropic-HH Anthropic-HH [4] is a human preference dialogue dataset containing two subsets based on the helpfulness and harmlessness principle. Here, we use helpful-base and harmless-base splits to validate the criterion using logit monotonicity for instance-level β control used in ε -DPO and the efficiency in terms of trade-off between performance and KL divergence [33]. We choose gemma-2-2B [39] to obtain the reference policy through Supervised Fine-tuning with chosen responses. Following DPO [32], we evaluate the models trained with each method under various β in the single-turn dialogue setting. We regard PairRM [21] as an external evaluator for checking performance by win rate, comparing their responses and chosen responses in the test splits.

4.2 Experimental Results on UltraFeedback

Overall Performance of ε -DPO In Table 1, we observe that ε -DPO shows notable performances compared to DPO across AlpacaEval 2 [11], Arena-Hard [25], and MT-Bench [21]. In particular, we find that the performance of ε -DPO outperforms most direct alignment algorithms, which generally modify the loss function, highlighting that the major assumption of fixed KL penalty in DPO is overlooked. Simultaneously, we observe that ε -DPO performs better than other KL penalty relaxation approaches [41, 14] from Table 2. We further consider an experimental setting

10

20

30

(

||

)

38

40

42

44

46

Length-controlled Win Rate (%)

0

200

400

Training Step

0.0

0.1

0.2

0.3

0.4

0.5

0.6

0

200

400

Training Step

0.0

0.1

0.2

0.3

0.4

0.5

0.6

+

0.005

0.01

0.02

KL

ref

that uses Qwen2.5-7B-Instruct [40] as the base model, extending the original SimPO's experimental setting. Specifically, we train the base model with the best hyperparameters obtained in the Llama-3-Instruct setting, without performing a hyperparameter search, as shown in Table 3. Interestingly, ε -DPO outperforms DPO even with hyperparameters obtained in the Llama-3-Instruct setting, whereas SimPO is inferior to DPO despite SimPO showing comparable performance in the Llama-3-Instruct . We speculate that introducing a fixed margin as a hyperparameter to mitigate the KL penalty, as in SimPO, can be an effective approach only if adequate hyperparameter search precedes it; adherence to the reference policy remains crucial otherwise. In addition to general chatbot benchmarks for preference alignment, we also check the Huggingface Open LLM Leaderboard [5] to see the impact ε -DPO on specific downstream tasks in Appendix C, but we find ε -DPO also follows the general trend of direct alignment algorithms [29]. Thus, we can confirm that instance-level KL penalty control significantly impacts the performance of general chatbot agents.

Influence of ε to the Training Dynamics In ε -DPO, the perturbation scale ε is used for checking logit monotonicity as a preference model in the neighborhood of the current β , for estimating policies under perturbation of the KL penalty. Therefore, it can be chosen within a reasonable range to estimate the approximated policies corresponding to β -ε and β + ε . However, ε can influence training dynamics since ε determines the scale of the instance-level KL penalty coefficient ˜ β . Wefurther analyze the intra-epoch training dynamics on Llama-3-Instruct settings according to ε . We compare the forward KL divergence D KL ( π ref || π θ ) [33] and performance on AlpacaEval 2 using checkpoints obtained at 0.2 intervals during the training, along with the in-batch occurrence ratio of β -ε and β + ε , as shown in Figure 3. We find that adaptive control occurs more frequently for both β -ε and β + ε as ε increases, leading to an acceleration of the increase of the KL divergence and performance. We also observe that the performance at the beginning of training tends to be lower when ε is higher. We speculate that the trained policy at the beginning of training is insufficient to estimate the optimal policy, making the approximation unstable at the high ε level.

Analysis of Computation Cost Although the instance-level KL penalty control of ε -DPO improves performance over DPO and incurs no additional model forward passes cost, it incurs additional computation costs for estimating policies with the training-time β perturbation, which warrants further analysis. Formally, the estimated forward passes cost C f and backward passes cost C b per token in FLOPs, following C f ≈ 2 N and C b ≈ 2 C f for a given model parameter size N , excluding the embedding

layer [23]. In the case of DPO, since forward and backward passes for the policy model and forward pass for the reference model occur, the FLOPs per token can be approximated as 8 N . When we approximate the policy model under perturbation of β , (2 v + v +5 v ) FLOPs are added per token for a given vocabulary size v , which corresponds to two scalar-vector multiplications, vector addition, and log-softmax operation, respectively. This implies that the relative ratio of additional computation cost in FLOPs per token compared to the computation cost of DPO can be roughly approximated as 2 v N ; therefore, because v ≪ N in general, the additional computation cost required by ε -DPO is negligible. To verify whether ε -DPO follows such a small computational cost empirically, we compared the wall-time increment ∆ t during training of ε -DPO compared to DPO under Mistral-Instruct and Llama-3-Instruct settings as Table 4, which confirms our computation analysis.

4.3 Experimental Results on Anthropic-HH

Variants of KL Penalty Control Strategy By default, ε -DPO shares the same ε that defines the neighborhood to check logit monotonicity and determine the relaxation strength performed on β in each instance. However, to understand how ε affects the training dynamics, it is beneficial to compare the default strategy with alternative strategies that use different values of ε for each case. We compare the default strategy with strategies that use different values of ε c and ε s ,

which are used to check logit monotonicity and to define the step size of the KL penalty control, respectively, when β is fixed at 0.05. Simultaneously, we also check the average occurrence of β -ε and β + ε through the corresponding average step direction for each instance sgn ( ε i ) in the batch to compare how the scale of ε affects the KL penalty relaxation. Table 5 shows that using different ε c and ε s produces suboptimal result compared to the default strategy. Furthermore, we can see that the adaptive KL penalty control of ε -DPO is strongly influenced by ε c since a similar level of sgn ( ε i ) is observed for the same ε c , and only imposing a higher ε c can lead to worse performance than DPO compared to ε s . In the sense that adopting higher ε for the estimating perturbed policy can be understood as the stronger extrapolation for approximation of distribution, we can see that it is necessary to set an appropriately small size of ε , as large ε risks increasing the probability of making a wrong decision for KL penalty relaxation in terms of weaker approximation to the optimal policy.

Behavior of Logit Monotonicity Criterion β -DPO [41] chooses a higher β for preference pairs with larger implicit reward margins to update the current policy conservatively from the reference policy. This is motivated by the claim that large implicit reward margins reflect higher quality gaps of response pairs corresponding to meaningless training signals. In this respect, we analyze the implicit reward margin of preference pairs where logit monotonicity according to the perturbation of β happened in policies trained by DPO using Antropic-HH, as shown in Figure 4a. We find that ε -DPO performs opposite decisions compared to β -DPO, assigning a higher β for preference pairs, revealing high confusion based on the observation that preference pairs with monotonically increasing logits show low confidence as a preference model. Also, this implies that ε -DPO reflects confusion on the preference label to the training signals by scaling the gradient of DPO loss through controlling β [32]. Furthermore, we confirm that implicit reward margins do not always represent the quality of preference pairs through qualitative analysis in Appendix D. Therefore, we suspect that β -DPO fails

r

r

r

r

r

r

r

r

r

r

r

r

r

r

r

r

Logit Monotonicity

40

30

20

10

0

10

20

30

r

,

(

x

,

y w

)

r

,

(

x

,

y l

)

z

(

)>

z

( )>

z

(

+ )

z

(

)<

z

( )<

z

(

+ )

0.01

0.05

0.1

0.5

(a)

20

30

40

KL

(

ref

||

)

45

50

55

60

65

70

75

80

DPO

-DPO

TR-DPO

TR-DPO

(b)

on the instance-level KL penalty control because it strongly relies on the implicit reward margins that do not always represent the quality of preference pairs, so that it fails to detect confusing examples.

KL Trade-off Efficiency of Adaptive Control As TR-DPO [14] claims, increasing the KL divergence would be desirable as a trade-off when deviating from the reference policy improves the performance. However, the over-optimization of direct alignment algorithms [33] emphasizes that it is necessary to check the Pareto frontier to determine whether performance improvements can be achieved without indiscriminately expanding the KL divergence because of degenerating behavior as the KL divergence grows. Figure 4b depicts the Pareto frontier between forward KL divergence and win rate compared with chosen responses in the test split, measured using Antropic-HH. Each model is trained through DPO, ε -DPO and two variants of TR-DPO, TR-DPO τ , which hard-updates the reference policy by the fixed interval, and TR-DPO α , which soft-updates the reference policy through weight merging, sharing the same β range, [0.01, 0.05, 0.1, 0.5]. We can see that ε -DPO shows better performance than DPO, simultaneously achieving better KL trade-off efficiency than TR-DPO. Also, we can observe that regardless of the two variants, TR-DPO induces more KL divergence than DPO and ε -DPO and cannot achieve similar performance under the same KL budget as ε -DPO. This highlights the efficiency of ε -DPO in the KL trade-off and implies that controlling the KL penalty in a non-adaptive manner can induce excessive relaxation for performance improvements.

Sensitivity of ε on the Approximation The logit monotonicity criterion assumes two conditions: (1) The current policy can sufficiently approximate the optimal policy for a given β . (2) When observing logit monotonicity, the logit function z θ ( β ) maintains monotonic order in the entire neighborhood ( β -ε , β + ε ) . However, these conditions can be significantly affected by the current policy and the choice of ε during training. To verify how much these conditions can be satisfied, we additionally check the upper bound of ε that defines a neighborhood of all values in the neighborhood consistently satisfies the logit monotonicity criterion for triplets ( x, y w , y l ) , through checkpoints obtained in 0.1 epoch intervals on DPO when β is fixed as 0.05.

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1.0

Epoch

0.0070

0.0075

0.0080

0.0085

0.0090

Consistent Upper Bound of

That is, assuming that the approximation of the optimal policy for the current β improves with the increase in training steps, we verify the smoothness of logit monotonicity with respect to ε by observing that the upper bound of ε yields consistent decisions compared to smaller values. We test 100 uniform sample points of ε over the range (0.005, 0.02). We observe that ε ↓ and ε ↑ , which correspond to the

expected upper bound of ε for monotonic decreasing and increasing logits, respectively, converged almost at 0.008 after 0.2 epochs, similar to the best result of previous experiments, as shown in Figure 5. Furthermore, the low value at 0.1 epoch is consistent with the phenomenon observed in the early stages of training in the Llama-3-Instruct . Therefore, we can confirm that relatively stable estimations of policy under perturbation of β , except for the early stage of training.

5 Related Works

Direct Alignment Algorithms Many variants of direct alignment algorithms perform alignment on offline preference datasets without an external reward model. DPO [32] performs alignment through preference modeling with the implicit reward derived from the optimal policy of reward maximization under the KL divergence regularization. RRHF [43] performs alignment by training to maintain the likelihood margin between preference ranks. KTO [12] changes the assumptions of the Bradley-Terry model [6] used by DPO and introduces Prospect Theory [22], and IPO [3] converts to the root-finding problem for strengthening the KL constraint. SLiC-HF [45], CPO [42], ORPO [18], and SimPO [29] train without reference models by compensating the KL penalty through behavior cloning, margin loss, contrastive loss, odds ratio loss, and fixed margin by replacing the implicit rewards.

Reward Over-optimization and KL Penalty Since RLHF [46] utilizes a trained reward model, it amplifies the limitations of the reward model as it is optimized toward an imperfect reward, according to Goodhart's Law [19], and this is called reward over-optimization [13]. However, Rafailov et al. [33] finds that direct alignment algorithms also experience similar reward over-optimization, regardless of the variant. Direct alignment algorithms commonly show humped curves of performance according to the increase of the KL divergence from the reference model during training. TR-DPO [14] argues that this is due to the Hessian of the loss landscape converging to zero as the implicit reward margin grows, so they update the reference model for mitigating this phenomenon. On the other hand, β -DPO [41], which also performs relaxation of the KL penalty, claims that adaptively changing β through the statistics of the implicit reward margin is required to reflect the quality of the preference pair.

Combining Sampling Distribution Combining sampling distributions can be utilized to estimate a new sampling distribution with specific characteristics. Contrastive Decoding [26] shows that the log-likelihood margins of the expert and amateur language models can enhance response diversity by penalizing incorrect response patterns favored by the amateur language model. Sanchez et al. [35] shows that classifier-free guidance [17] can enhance prompt relativity in language modeling by treating prompts as conditions. Mitchell et al. [30] estimates the importance ratio of the optimal distribution in RLHF by combining the change during instruction-tuning in a small language model with the large language model to approximate fine-tuning. Inspired by the theoretical motivation of Mitchell et al. [30], Liu et al. [28] shows that the sampling distribution of the policy trained by DPO with different β can be approximated by importance sampling using the reference policy.

6 Conclusion

In this paper, we present ε -Direct Preference Optimization ( ε -DPO), an instance-level adaptive KL penalty control method for DPO, adjusting the KL penalty coefficient β by observing the monotonicity of the log-likelihood ratio between the chosen response and the rejected response when the β used during training is perturbed. This simple criterion only requires estimating the policy under the perturbed β , which can be efficiently estimated by reusing the policy and reference policy logits without relying on batch-level statistics and requiring computation of reference policy updates. ε -DPO shows significantly better performance than DPO and also surpasses most existing direct alignment algorithms in general chatbot benchmarks. In particular, the criterion of ε -DPO shows a more efficient KL trade-off than the non-adaptive KL penalty relaxation while reflecting the confusion on preference pairs, emphasizing the importance of an appropriate instance-level KL penalty relaxation.

Limitations ε -DPO requires the reference policy because it has a KL penalty from the reference policy, like DPO in default. It leads to requirements of additional memory consumption and computation for the reference policy compared to other direct alignment algorithms that do not perform regularization through the reference policy [45, 42, 18, 29]. Still, ε -DPO can reduce additional resources by pre-computing the logits of the responses from the reference policy, similar to DPO.

References

J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt, S. Altman, S. Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 , 2023.

A. Askell, Y. Bai, A. Chen, D. Drain, D. Ganguli, T. Henighan, A. Jones, N. Joseph, B. Mann, N. DasSarma, et al. A general language assistant as a laboratory for alignment. arXiv preprint arXiv:2112.00861 , 2021.

M. G. Azar, Z. D. Guo, B. Piot, R. Munos, M. Rowland, M. Valko, and D. Calandriello. A general theoretical paradigm to understand learning from human preferences. In International Conference on Artificial Intelligence and Statistics , pages 4447-4455. PMLR, 2024.

Y. Bai, A. Jones, K. Ndousse, A. Askell, A. Chen, N. DasSarma, D. Drain, S. Fort, D. Ganguli, T. Henighan, et al. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862 , 2022.

E. Beeching, C. Fourrier, N. Habib, S. Han, N. Lambert, N. Rajani, O. Sanseviero, L. Tunstall, and T. Wolf. Open llm leaderboard. https://huggingface.co/spaces/ open-llm-leaderboard-old/open_llm_leaderboard , 2023.

R. A. Bradley and M. E. Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika , 39(3/4):324-345, 1952.

P. Clark, I. Cowhey, O. Etzioni, T. Khot, A. Sabharwal, C. Schoenick, and O. Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.05457 , 2018.

K. Cobbe, V. Kosaraju, M. Bavarian, M. Chen, H. Jun, L. Kaiser, M. Plappert, J. Tworek, J. Hilton, R. Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168 , 2021.

G. Cui, L. Yuan, N. Ding, G. Yao, B. He, W. Zhu, Y. Ni, G. Xie, R. Xie, Y. Lin, et al. Ultrafeedback: Boosting language models with scaled ai feedback. In International Conference on Machine Learning , pages 9722-9744. PMLR, 2024.

A. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. Al-Dahle, A. Letman, A. Mathur, A. Schelten, A. Yang, A. Fan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783 , 2024.

Y. Dubois, B. Galambosi, P. Liang, and T. B. Hashimoto. Length-controlled alpacaeval: A simple way to debias automatic evaluators. arXiv preprint arXiv:2404.04475 , 2024.

K. Ethayarajh, W. Xu, N. Muennighoff, D. Jurafsky, and D. Kiela. Kto: Model alignment as prospect theoretic optimization. arXiv preprint arXiv:2402.01306 , 2024.

L. Gao, J. Schulman, and J. Hilton. Scaling laws for reward model overoptimization. In International Conference on Machine Learning , pages 10835-10866. PMLR, 2023.

A. Gorbatovski, B. Shaposhnikov, A. Malakhov, N. Surnachev, Y. Aksenov, I. Maksimov, N. Balagansky, and D. Gavrilov. Learn your reference model for real good alignment. arXiv preprint arXiv:2404.09656 , 2024.

C. Guo, G. Pleiss, Y. Sun, and K. Q. Weinberger. On calibration of modern neural networks. In International conference on machine learning , pages 1321-1330. PMLR, 2017.

D. Hendrycks, C. Burns, S. Basart, A. Zou, M. Mazeika, D. Song, and J. Steinhardt. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300 , 2020.

J. Ho and T. Salimans. Classifier-free diffusion guidance. arXiv preprint arXiv:2207.12598 , 2022.

J. Hong, N. Lee, and J. Thorne. Orpo: Monolithic preference optimization without reference model. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing , pages 11170-11189, 2024.

K. Hoskin. The 'awful idea of accountability': inscribing people into the measurement of objects. Accountability: Power, ethos and the technologies of managing , 265, 1996.

A. Q. Jiang, A. Sablayrolles, A. Mensch, C. Bamford, D. S. Chaplot, D. d. l. Casas, F. Bressand, G. Lengyel, G. Lample, L. Saulnier, et al. Mistral 7b. arXiv preprint arXiv:2310.06825 , 2023.

D. Jiang, X. Ren, and B. Y. Lin. Llm-blender: Ensembling large language models with pairwise ranking and generative fusion. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages 14165-14178, 2023.

D. Kahneman and A. Tversky. Prospect theory: An analysis of decision under risk. In Handbook of the fundamentals of financial decision making: Part I , pages 99-127. World Scientific, 2013.

J. Kaplan, S. McCandlish, T. Henighan, T. B. Brown, B. Chess, R. Child, S. Gray, A. Radford, J. Wu, and D. Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361 , 2020.

S. Lee, S. Kim, A. Yousefpour, M. Seo, K. M. Yoo, and Y. Yu. Aligning large language models by on-policy self-judgment. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages 11442-11459, 2024.

T. Li, W.-L. Chiang, E. Frick, L. Dunlap, T. Wu, B. Zhu, J. E. Gonzalez, and I. Stoica. From crowdsourced data to high-quality benchmarks: Arena-hard and benchbuilder pipeline. arXiv preprint arXiv:2406.11939 , 2024.

X. L. Li, A. Holtzman, D. Fried, P. Liang, J. Eisner, T. B. Hashimoto, L. Zettlemoyer, and M. Lewis. Contrastive decoding: Open-ended text generation as optimization. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages 12286-12312, 2023.

S. Lin, J. Hilton, and O. Evans. Truthfulqa: Measuring how models mimic human falsehoods. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages 3214-3252, 2022.

T. Liu, S. Guo, L. Bianco, D. Calandriello, Q. Berthet, F. Llinares, J. Hoffmann, L. Dixon, M. Valko, and M. Blondel. Decoding-time realignment of language models. In Proceedings of the 41st International Conference on Machine Learning , pages 31015-31031, 2024.

Y. Meng, M. Xia, and D. Chen. Simpo: Simple preference optimization with a reference-free reward. Advances in Neural Information Processing Systems , 37:124198-124235, 2024.

E. Mitchell, R. Rafailov, A. Sharma, C. Finn, and C. D. Manning. An emulator for fine-tuning large language models using small language models. arXiv preprint arXiv:2310.12962 , 2023.

R. Park, R. Rafailov, S. Ermon, and C. Finn. Disentangling length from quality in direct preference optimization. In Findings of the Association for Computational Linguistics ACL 2024 , pages 4998-5017, 2024.

R. Rafailov, A. Sharma, E. Mitchell, C. D. Manning, S. Ermon, and C. Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in neural information processing systems , 36:53728-53741, 2023.

R. Rafailov, Y. Chittepu, R. Park, H. S. Sikchi, J. Hejna, B. Knox, C. Finn, and S. Niekum. Scaling laws for reward model overoptimization in direct alignment algorithms. Advances in Neural Information Processing Systems , 37:126207-126242, 2024.

K. Sakaguchi, R. L. Bras, C. Bhagavatula, and Y. Choi. Winogrande: An adversarial winograd schema challenge at scale. Communications of the ACM , 64(9):99-106, 2021.

G. V. Sanchez, A. Spangher, H. Fan, E. Levi, and S. Biderman. Stay on topic with classifier-free guidance. In Proceedings of the 41st International Conference on Machine Learning , pages 43197-43234, 2024.

J. Schulman, S. Levine, P. Abbeel, M. Jordan, and P. Moritz. Trust region policy optimization. In International conference on machine learning , pages 1889-1897. PMLR, 2015.

J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347 , 2017.

F. Tajwar, A. Singh, A. Sharma, R. Rafailov, J. Schneider, T. Xie, S. Ermon, C. Finn, and A. Kumar. Preference fine-tuning of llms should leverage suboptimal, on-policy data. In International Conference on Machine Learning , pages 47441-47474. PMLR, 2024.

G. Team, M. Riviere, S. Pathak, P. G. Sessa, C. Hardin, S. Bhupatiraju, L. Hussenot, T. Mesnard, B. Shahriari, A. Ramé, et al. Gemma 2: Improving open language models at a practical size. arXiv preprint arXiv:2408.00118 , 2024.

Q. Team et al. Qwen2 technical report. arXiv preprint arXiv:2407.10671 , 2(8), 2024.

J. Wu, Y. Xie, Z. Yang, J. Wu, J. Gao, B. Ding, X. Wang, and X. He. β -dpo: Direct preference optimization with dynamic β . Advances in Neural Information Processing Systems , 37:129944129966, 2024.

H. Xu, A. Sharaf, Y. Chen, W. Tan, L. Shen, B. Van Durme, K. Murray, and Y. J. Kim. Contrastive preference optimization: Pushing the boundaries of llm performance in machine translation. In International Conference on Machine Learning , pages 55204-55224. PMLR, 2024.

Z. Yuan, H. Yuan, C. Tan, W. Wang, S. Huang, and F. Huang. Rrhf: Rank responses to align language models with human feedback without tears. arXiv preprint arXiv:2304.05302 , 2023.

R. Zellers, A. Holtzman, Y. Bisk, A. Farhadi, and Y. Choi. Hellaswag: Can a machine really finish your sentence? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics , pages 4791-4800, 2019.

Y. Zhao, R. Joshi, T. Liu, M. Khalman, M. Saleh, and P. J. Liu. Slic-hf: Sequence likelihood calibration with human feedback. arXiv preprint arXiv:2305.10425 , 2023.

D. M. Ziegler, N. Stiennon, J. Wu, T. B. Brown, A. Radford, D. Amodei, P. Christiano, and G. Irving. Fine-tuning language models from human preferences, 2020.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: See Abstract and Section 1.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: See Section 6.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory assumptions and proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental result reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: See Abstract and supplemental materials.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental setting/details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment statistical significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: This paper includes experimental results such as Figure 4a and Figure 5.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments compute resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code of ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: We conducted the research following the NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: This paper does not deal with topics that can have societal impacts.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to

generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: This paper does not introduce new assets having a high risk for misuse.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: This paper does not release new assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and research with human subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: This paper does not involve crowdsourcing nor research with human subjects. Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional review board (IRB) approvals or equivalent for research with human subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: This paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

16. Declaration of LLM usage

Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.

Answer: [NA]

Justification: This paper only involves LLM usage for editting manuscripts.

Guidelines:

The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.

Please refer to our LLM policy ( https://neurips.cc/Conferences/2025/LLM ) for what should or should not be described.

A Proof of Proposition 1

Proposition 1 (Liu et al. [28]) Under the assumption of optimal autoregressive policy π ∗ where the prompt x ∈ X , response vocabulary y i ∈ V , and logit f : X × V i -1 → R |V| , the optimal policy π ∗ β λ can be approximated by the arithmetic mean of logits between π ∗ β and reference policy π ref ,

Proof of Proposition 1 . Recall that the optimal policy π ∗ β has a closed-form solution, and ground-truth reward function r ∗ can be reparameterized using the normalizing constant Z ∗ β ,

Here, we plug the reparameterization of r ∗ to the close-form solution of π ∗ β λ and simple algebra yield,

where Z denotes the normalizing constant of reparameterized π ∗ β λ . Now, we use the assumption of the autoregressive policy π ∗ β . This assumption allows us to evade the intractable normalizing constant Z ,

with eliminating ( ∑ v ∈V exp ( f ∗ β ( x, y 1: i -1 ) ) v ) λ ( ∑ v ∈V exp ( f ref ( x, y 1: i -1 ) ) v ) 1 -λ from nominator and denominator. Note that the geometric mean acts as the arithmetic mean mean on log scales,

Therefore, π ∗ β λ can be approximated by the arithmetic mean of logit between π ∗ β and π ref ,

B Implementation Details

The implementation of ε -DPO and experiments are all based on the TRL 2 library. Here, we explain the experimental settings, including hyperparameters, for UltraFeedback [9] and Antropic-HH [4].

B.1 UltraFeedback

For a fair comparison with direct alignment algorithms and existing approaches for KL penalty relaxation, we follow the Instruct setting suggested by SimPO [29]. The Instruct setting starts with Mistral-7B-Instruct-v0.2 3 [20] and Meta-Llama-3-8B-Instruct 4 [10] as reference policies, each named as Mistral-Instruct and Llama-3-Instruct . First, rollouts using prompts from UltraFeedback [9] are performed, then PairRM [21] serves as an external evaluator to build preference datasets to approximate on-policy learning [38, 24]. We use corresponding datasets publicly released by SimPO, each denoted as mistral-instruct-ultrafeedback 5 and llama3-ultrafeedback 6 . Additionally, we also include experiments using Qwen2.5-7B-Instruct 7 as a reference model for further analysis, still following the same dataset construction process of SimPO. We perform a hyperparameter search for ε -DPO, keeping it similar to the hyperparameter grid used by SimPO for other direct alignment algorithms to ensure fairness; the learning rate within the range of [3e-7, 5e-7, 7e-7, 1e-6] and ε within the [0.005, 0.01, 0.02] range while β is fixed to 0.01, following the best hyperparameter of DPO reported from SimPO. Other common hyperparameters are fixed in the same way as SimPO. Every experiment is conducted using 16 NVIDIA A100-SXM4-40GB GPUs within 2 hours. We evaluate resulting models through AlpacaEval 2 [11], Arena-Hard [25], and MT-Bench [21] following the same sampling configuration settings reported by SimPO. Since this experimental setting reports the best-performing model due to the hyperparameter sensitivity of the direct alignment algorithms, we report the results of a single model in a hyperparameter grid with the best rank, prioritizing AlpacaEval 2 (LC), Arena-Hard, and MT-Bench in that order. Table 6 summarizes the training configurations for Mistral-Instruct and Llama-3-Instruct .

5

huggingface.co/datasets/princeton-nlp/mistral-instruct-ultrafeedback, MIT License

□

B.2 Anthropic-HH

We use helpful-base and harmless-base splits for experiments using Anthropic-HH 8 [4]. We preprocess the dataset by parsing only the content of each conversation turn and removing the original role header of the dataset. We use gemma-2-2b 9 [39] as a base model for obtaining the reference policy through Supervised Fine-tuning (SFT) with chosen responses by applying the chat template of gemma-2-2b-it 10 [39]. We fix all hyperparameters except β for a fair comparison between methods. We use ε = 0 . 01 in ε -DPO and τ = 128 , α = 0 . 6 in TR-DPO [14] as the method-specific hyperparameter and β within the [0.01, 0.05, 0.1, 0.5] range. Following DPO [32], we evaluate resulting models in the single-turn dialogue setting by comparing with chosen responses from the test split through PairRM 11 [21] as an external evaluator to check the win rate. We set the temperature to 1.0 and the max token length to 1024 when sampling responses from each model for evaluation. Every experiment is conducted using 4 NVIDIA A100-SXM4-40GB GPUs within 7 hours. Table 7 shows the common training configurations for each experiment.

C Evaluation on Specific Downstream Tasks

Beyond the main evaluation through general chatbot benchmarks [11, 25, 21], SimPO [29] used the Huggingface Open LLM Leaderboard [5] to see the impact of direct alignment algorithms on specific downstream tasks. This includes MMLU [16], AI2 Reasoning Challenge (ARC) [7], HellaSwag [44], TruthfulQA [27], Winograd [34], and GSM8K [8] as target evaluation tasks. SimPO only analyzes the general tendencies of direct alignment algorithms since the impact of different direct alignment algorithms on downstream tasks can be strongly dependent on pretrained models and preference datasets. We similarly observe that the impact of instance-level adaptive KL penalty control in

ε -DPO still follows the general tendency of direct alignment algorithms; improvements in knowledge (MMLU), reading comprehension (ARC), commonsense reasoning (HellaSwag, Winograd), and truthfulness (TruthfulQA), but a score drop happens in math skills (GSM8K).

D Qualitative Analysis of Logit Monotonicity and Implicit Reward Margin

We compare preference pairs whose implicit reward margin is maximized among the preference pairs showing monotonically increasing or decreasing logits in the helpful-base split of AntropicHH [4]. Similarly, we compare preference pairs whose implicit reward margins are minimized among those with monotonically increasing or decreasing logits. We obtain these preference pairs by training the policy with DPO under β = 0 . 1 . If we follow the claim of β -DPO, the higher β should be selected for both preference pairs that sufficiently maximize the implicit reward margin, regardless of logit monotonicity. However, Table 9 shows the case close to the label flipping compared to the case of Table 10 in which the adaptive control decision of ε -DPO and β -DPO matches in the high implicit reward margin. On the other hand, Table 11 shows the case of the rejected response with a significantly lower quality than the chosen response, compared to the case of Table 12 in which the adaptive control decision of ε -DPO and β -DPO matches in the low implicit reward margin. However, β -DPO will assign a low β to the corresponding example, contrary to the original claim, since it shows a low implicit reward margin. These qualitative examples demonstrate that the implicit reward margin cannot fully reflect the quality of preference data, as claimed by the β -DPO.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a variant of DPO, termed $\\varepsilon$-DPO, which introduces an adaptive mechanism to control the KL penalty parameter $\\beta$. By perturbing the KL term at the instance level, $\\varepsilon$-DPO dynamically balances the trade-off between staying close to the reference model and aligning with preference data. Experimental results demonstrate that $\\varepsilon$-DPO outperforms existing baselines, achieving improved alignment and generalization in preference optimization tasks.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\nThe paper introduces a principled and effective way to adaptively control the KL penalty parameter $\\beta$ in DPO through instance-level perturbations. This addresses a key limitation of standard DPO, which relies on a fixed penalty term, and allows the model to better accommodate diverse preference samples.\n\nWeaknesses:\n1. While the introduction of an adaptive KL penalty in DPO is a useful refinement, the core idea, which modulates the regularization strength based on instance-level signals, may be seen as a relatively incremental extension of existing approaches. The paper would benefit from a stronger emphasis on the broader implications or theoretical insights enabled by this adaptation.\n2. The theoretical development builds heavily on existing results and offers limited additional insight into the underlying mechanisms that make the proposed method effective. A more principled analysis of why and when the approach works would strengthen the contribution. (See details in the question part)\n\nQUESTIONS:\n1. Do equations (1), (2), and (5) imply that the choice of $\\beta_ \\varepsilon^+$ and $\\beta_ \\varepsilon^-$ are determined by the sign of  $z_\\theta(x,y^w,y^l)$? \n2. For Equation (4), I understand that $\\pi_{\\theta(\\beta_ \\varepsilon^+)}$ can be interpreted as the softmax over a convex combination of the logits $f_\\theta$ and $f_{\\textnormal{ref}}$. However, for Equation (3), the interpretation is less clear. Could the authors provide an intuitive explanation of this expression?\n\nLIMITATIONS:\nThe authors have discussed the limitations in conclusion and I raise my score for acceptance.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-23"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces ϵ-Direct Preference Optimization (ϵ-DPO), a new method for aligning large language models with human preferences. The key innovation of ϵ-DPO is its ability to adaptively control the strength of the KL penalty for each individual data point (instance-level) during training. This contrasts with standard Direct Preference Optimization (DPO), which uses a fixed KL penalty, and other methods that adjust it at a batch-level or through periodic, non-adaptive updates.\n\nThe core mechanism of ϵ-DPO involves checking the \"monotonicity of logits.\" Essentially, for each preference pair (a chosen and a rejected response), the method observes how the model's confidence (represented by the log-likelihood ratio) changes when the KL penalty coefficient, β, is slightly perturbed. This check allows the algorithm to decide whether to increase or decrease the KL penalty for that specific training example, without requiring extra model updates or being dependent on the batch size. The policy under this perturbed penalty is efficiently estimated by reusing the logits from the current and reference policies.\n\nSTRENGTHS AND WEAKNESSES:\n### Strength\n- The experimental results show that ϵ-DPO doesn't only improve upon DPO but also outperforms most existing direct alignment algorithms on several key chatbot benchmarks. The paper demonstrates that ϵ-DPO achieves a more efficient Pareto frontier between performance and KL divergence compared to DPO and TR-DPO. This shows that the adaptive control is not just indiscriminately increasing KL divergence but is doing so efficiently for performance gains.\n- The paper provides strong evidence that the static KL penalty in DPO is a \"major bottleneck to final model performance\". The success of ϵ-DPO underscores the importance of instance-level adaptive control for unlocking better performance.\n- While the idea of a dynamic KL penalty is not new , this paper is the first to propose a method that is both instance-level and adaptive. \n\n### Weakness\n - The method's effectiveness hinges on the quality of the policy approximation in Proposition 1. The authors acknowledge that a large perturbation size $\\epsilon$ can lead to a \"weaker approximation\" and potentially \"a wrong decision for KL penalty relaxation\". The paper could be strengthened by a more in-depth analysis of the error bounds or the conditions under which this approximation is most reliable.\n- As the authors correctly state in their limitations section, ϵ-DPO, being an extension of DPO, requires a reference policy. This entails additional memory consumption and computation compared to reference-free methods like ORPO and SimPO. \n- The evaluation on specific downstream tasks in Appendix C shows that ϵ-DPO has a performance drop compared with DPO on math reasoning (GSM8K). The average scores reported in Table 6 do not show a consistent, decisive advantage for ϵ-DPO, suggesting its benefits are most pronounced in conversational ability rather than across all specialized tasks.\n- $\\beta$ for the next training step is updated by taking the mean of the instance-level beta values used in the current batch. I think this could be replaced by a potentially more robust update scheme (e.g., an exponential moving average with a learnable parameter).\n\nQUESTIONS:\n1. In Line 232, what is step size $\\mathbb{E}(\\epsilon_i)$? Where is $\\epsilon_i$ defined? How is this related to $\\epsilon_c$ and $\\epsilon_s$?\n2. Can you analyze why on the same dataset, e.g. Arena-Hard, Mistral and Llama will have very different results? (in Table 1), could you provide experiments on more models? such as Qwen?\n3. Does computing (3) (4) requires more computational resources compared with the DPO objective? In other words, how much more FlOPs are needed (in percentile) in addition to the DPO objective? \n4. In \"Variants of KL Penalty Control Strategy\", the authors claim that we should set $\\epsilon_c=\\epsilon_\\beta$. In the Table 3, it sets $\\epsilon_c=\\epsilon_\\beta=0.01$. Can you set the fixed $\\epsilon$ to other values? such as when $\\epsilon_c$ or $\\epsilon_\\beta$ equals $0.005, 0.01,0.02$, and varying another one. Or in general, you should enumerate all choices of $\\epsilon_c$, $\\epsilon_\\beta$, in $0.005,0.01, 0.02$ (or more) and say $\\epsilon_c=\\epsilon_\\beta=0.01$ is the best choice.\n\nI am willing to increase my score if the weaknesses and questions are properly resolved.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nAs an RLHF algorithm, DPO has its limitations, in that it uses a constant KL penalty coefficient β, which may lead to suboptimal performance.\n\nTo address this, ε-DPO proposes an instance-level adaptive KL penalty control mechanism. It follows the direction of β that brings better separability of winner samples and loser samples ($z_\\theta\\left(x, y^w, y^l\\right):=\\log \\frac{\\pi_\\theta\\left(y^w \\mid x\\right)}{\\pi_\\theta\\left(y^l \\mid x\\right)}$). However, to estimate the separability, one has to be able to estimate the solution of DPO $\\theta(\\beta)$ for each $\\beta$, which is not possible.\n\nTo address this, ε-DPO utilizes the estimation from Liu et al., to model the policy obtained by $\\theta(\\beta/\\gamma)$ by an interpolation of  the policy obtained by $\\theta(beta)$ and the reference policy. Therefore, ε-DPO updates $\\beta$ at each iteration. \n\nExperiments on benchmarks such as AlpacaEval 2, Arena-Hard, and MT-Bench demonstrate that ε-DPO outperforms DPO and several other direct alignment methods in LC win rate.\n\nSTRENGTHS AND WEAKNESSES:\nStrength:\n1. The paper is clearly written.\n\n2. The paper utilizes an effective way of estimating the expected change in separability under a small change in KL penalty coefficient β.\n\n3. The paper clearly discussed its relationship with prior related work $\\beta$-DPO and TR-DPO, with performance comparison.\n\n4. The performance on  Llama-3-Instruct (8B) Arena-Hard is good, showing high potential.\n\nWeakness:\n\n1. The performance in Table 1 is not pronounced. I see that the performance based on Mistral has lower WR on AlpacaEval, ArenaHard, and the performance based on Llama3-Instrcut has a lower MT-Bench score.\n\n2. While I understand that the performance may not outperform on every benchmark and every dataset, I suspect that this might be due to the sensitivity of $\\epsilon$, when changing datasets.\n\n3. The algorithm requires additional computation overhead, and the training time is not compared.\n\n4. Intuitively, the result of one step of optimizing $\\theta$ might be far from the estimation in this paper, leading to an inaccurate estimation of the direction, the accuracy of this estimation, and the accuracy of the direction of $\\beta$ has not been discussed.\n\n5. The performance comparison with $\\beta$-DPO and TR-DPO on base model Mistral has not been shown.\n\nQUESTIONS:\n1. How many epochs do you train, for each baselines, and your algorithms? Do you use the same epoch number?\n\n2. How do you set the initial $\\beta$ value, do you have to tune $\\beta$ and $\\epsilon$ at the same time?\n\nLIMITATIONS:\nThe authors discussed the limitations in the last paragraph.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 4, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces ε-DPO, an instance-level adaptive extension to Direct Preference Optimization (DPO) for aligning large language models with human preferences in offline setting. Instead of static KL penalty in DPO, this paper proposes a novel mechanism that is adaptive to each preference pair. This gives more informative KL divergence info, because of the monotonicity of the log-likelihood ratio between preferred and rejected responses.\n\nSTRENGTHS AND WEAKNESSES:\nStrength: \n1. Instance-level Adaptive KL Control:\nProposes ε-DPO, which replaces DPO’s static KL penalty with a simple yet effective adaptive mechanism at the instance level, achieving superior performance on general chatbot benchmarks.\n\n2. Preference Confidence–Driven Adjustment:\nReinterprets DPO’s preference modeling framework as a basis for dynamic KL control, by estimating how preference confidence changes under small perturbations of the KL coefficient β.\n\n3. Experimental results are comprehensive and promising: \nExperiments on UltraFeedback and Anthropic-HH datasets demonstrated the good performance of the proposed method in the majority of the trials.\n\nWeakness:\n1. Although the proposal is theoretically justified from the monotonicity of the log-likelihood ratio, the assumption is dependent on the transitivity of the reward function, which might not hold in a real dataset given human annotations.\n- The literature below studied representative preference datasets in the real world, where the 'transitive' relationship between preference annotations may not always hold. \n- https://arxiv.org/abs/2409.19325 (Duan et al, 2017) \n\n2. Previous works have inefficiencies in computational cost, as described in Lines 114-116; however, this paper does not discuss this aspect with evidence in its proposed algorithms and experiments.\n\nQUESTIONS:\nQ1. The monotonicity of the log-likelihood ratio holds assuming scalar evaluation. Does the intransitivity observed in real datasets at the instance and annotator levels have any potential impact or connection to the proposed method? How does this relate to the discussed 'confusion on preference pairs'?\n\nQ2. How to elaborate the additional computational cost in the proposed adaptive instance-level operations, when compared with batch operations?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 4, "ORIGINALITY": 2, "DATE": "2025-07-06"}