KL Penalty Control via Perturbation for Direct Preference Optimization
Authors: Sangkyu Lee
Janghoon Han
Hosung Song
Stanley Jungkyu Choi
Honglak Lee
Youngjae Yu
Venue: NeurIPS 2025 poster ()
Keywords: Direct Alignment Algorithm
Reinforcement Learning from Human Feedback
Human Preference
LLM
Alignment
Primary area: deep_learning
TL;DR: Instance-level adaptive KL penalty control method for Direct Preference Optimization

Abstract: Direct Preference Optimization (DPO) demonstrates the advantage of aligning a large language model with human preference using only an offline dataset. However, DPO has the limitation that the KL penalty, which prevents excessive deviation from the reference model, is static throughout the training process. Several methods claim to change this static KL penalty of DPO into a dynamic one, but no approach can adaptively assign different KL penalties for each preference pair. In this paper, we propose $\varepsilon$-Direct Preference Optimization ($\varepsilon$-DPO), which allows adaptive control of the KL penalty strength $\beta$ for each preference pair. Specifically, $\varepsilon$-DPO adaptively controls $\beta$ for each preference pair based on the monotonicity of logits as a preference model under the perturbation of $\beta$ during training. This is equivalent to adjusting the KL penalty by checking whether the change in training-time temperature can lead to better preference confidence as preference models by simply reusing the logit of the current policy and the reference policy. Experimental results show that the simple criterion of $\varepsilon$-DPO for KL penalty relaxation significantly improves DPO compared to most existing direct alignment algorithms on general chatbot benchmarks and reveal that this KL penalty control criterion can reflect confusion as a preference model and provide an efficient KL trade-off, highlighting the significance of instance-level adaptive KL penalty control in DPO.

KL Penalty Control via Perturbation for Direct Preference Optimization
Sangkyu Lee
Janghoon Han? Hosung Song2
Stanley Jungkyu Choi Honglak Leea.3 Youngjac Yu²
Yonsei University' LG Al Research
University of Michigan, Ann Arbors Seoul National University! oddqueue@yonsei. ac. kr
youngjaeyu ac ac.ke kr
Abstract
Direct Preference Optimization (DPO) demonstrates the advantage of aligning large language model with human preference using only an offline dataset. How- ever, DPO has the limitation that the KI. penaltyy which prevents excessive deviation from the reference model, is static throughout the training process. Several methods claim to change this static KL penalty of DPO into dynamic one, but no approach can adaptively assign different KL penalties for each preference pair. In this paper, we propose e-Direct Preference Optimization (s-DPO), which allows adaptive control of the KL penalty strength for each preference pair. Specifically, s-DPO adaptively controls 33 for each preference pair based on the monotonicity logits as a preference model under the perturbation of 3 during training. This is equivalent adjusting the KL penalty by checking whether the change training-time tem- perature can lead to better preference confidence as preference models by simply reusing the logit the current policy and the reference policy. Experimental results show that the simple criterion of c-DPO for KI. penalty relaxation significantly improves DPO compared to most existing direct alignment algorithms on general chatbot benchmarks and reveal that this KL penalty control criterion reflect confusion as a preference model and provide an efficient KL trade-off, highlighting the significance of instance-level adaptive KL penalty control in DPO.
Introduction
Aligning large language models with human preferences for helpfulness and harmless principles [2, 9] is crucial requirement for general chatbot agents. Reinforcement Learning from Human Feedback (RLHF) [46] is the pioneering approach that regards the aligmment of large language models as reward maximization problem and solves it by reinforcement learning [37]. However, the complicated training pipeline of RLHF increases the training complexity and computation cost of the rollout for online reinforcement learning, in addition to the difficulty of collecting human preference datasets. Moreover, introducing a trained reward model a proxy reward function to replace the intractable ground-truth human preference reward function makes large language models suffer from the side effect of reward over-optimization [13] inherited from the reward models.
Direct Preference Optimization (DPO) [32] proposes an alternative approach to reform the limitation of RLHF by converting the policy optimization problem into a preference modeling problem and performing alignment using only the offline preference dataset. It shows comparable performance
Work done during internship at LG Al Research.
'The code is available github.com/ com/ooddqueuec-dpoo
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

1) Monotonically Decreasing (a,y", v² + 3 - 3e
2) Monotonically Increasing (x,"","' = B3 13/
log
"6(135)
T 76197) (0"1 a))
(V01(a)
log
To(a) (y (z)
log T Tocoth (x)
To(97) ) (30")())))))))))))()))))
16(97 (188))(8(2) (0/12) Ko(s) (v' (v" *oist (3r (z)
log
Fotoy(y" [x) log Mb(82) (10 (z)
To(a)) (x))
log
T Ta (1/(z))
E-DPO adaptively controls 3 corresponding to the KL penalty strength for each preference pair by checking monotonicity of the log- ggglikelihood ratio of the chosen response and the rejected according to perturbation of training-time 3 by estimating the perturbed policies by reusing logits.
Figure I:
while skipping the reward modeling process required by RLHF and has become an effective alternative approach for aligmment. In particular, subsequent studies with various modifications to the DPO objective function open new research domain called direct alignment algorithms [33], which perform alignment directly from offline preference datasets without training separate reward models.
However, DPO assumes that 3 and the reference policy, which define a KL penalty that prevents excessive deviations from the reference model in RLHF, are fixed for exploiting the existence of a closed-form solution derived from the objective function of the RLHF, However, this assumption can lead suboptimal results, since the KL penalty can be regarded as Lagrangian relaxation of the constraint optimization defined by the trust region [37]. In this regard, 3-DPO [41] argues that should be adaptively chosen according to the quality of the preference pair but fails to control at the instance- and proposes a batch-level control method. On the other hand, TR-DPO [14]
claims to periodically update the reference policy to reduce over-optimization [33], but it may induce unnecessary KL divergence for improvement since the update is not adaptive.
In this paper, present e-Direct Preference Optimization (e-DPO), simple instance-level adaptive KL penalty control for DPO that neither TR-DPO nor 3-DPO achieves. Specifically, we check the advantage of adjusting for each preference pair by observing the monotonicity of the log- bog-likel ratio between the chosen response and the rejected response if the 3 used during training was perturbed, as described in Figure 1. Here, the criterion for controlling 3 does not require batch-level statistics, and the policy under the perturbed / can be estimated by reusing the logits from the policy and reference policy. This criterion results in independence from the choice of micro-batch size and no additional computation requirements for model updates, unlike 3-DPO and TR-DPO.
Experimental results demonstrate that the instance-level adaptive criterion of e-DPO remarkably improves DPO, better than 3-DPO and TR-DPO, to outperform most direct alignment algorithms that modify the DPO objective function [43, 45, 3, 42, 12, 18, 31, 29]. This reveals that the static KL penalty of DPO is the major bottleneck to final model performance and highlights the importance of instance-level adaptive KL penalty control. Furthermore, we confirm that the variation of determined by the adaptive criterion in c-DPO reflects the confusion as preference model, which is not addressed in the adaptive control criterion proposed by 3-DPO. We also find that the adaptive KL penalty control of e- e-DPO is crucial for an efficient KL trade-off compared to TR-DP TR-DPO, which is not an adaptive KL penalty control because of the periodic update the reference policy.
Our contribution to the alignment of large language models can be summarized as threefold: (1) We present eDPPO that changes the static KL. penalty of DPO and adaptively controls the KI. penalty at the instance-level with simple criterion, which can outperform most direct aligmment algorithms proposed as alternatives DPO in general chatbot benchmarks. (2) We show that DPO's perspective, which reparameterizes the policy to a preference model, can be converted to an approach for controlling the KL penalty at the instance-level by estimating the preference confidence changes according to the perturbation of ( used during training. (3) We demonstrate that this instance-level adaptive KL penalty control distinguishes confusing preference pairs and achieves an efficient KL trade-off, and neither is addressed in existing research that performs KI penalty relaxation.
2 Preliminaries
Reinforcement Learning from Human Feedback To obtain a language model that aligns with human preference, RLHF [46] introduces reinforcement learning. It is equivalent to approaching preference alignment as reward maximization problem, where we find a policy a that maximizes a ground-truth reward function r" representing human preference score for response y obtained

---

from a corresponding policy for a given prompt z. However, since the ground-truie reward function cannot be accessed, reward model trained from the preference dataset introduced as a proxy reward function. On the other hand, to prevent the policy update from deviating too much from the current policy from the initial policy, the KI. divergence from the reference policy Tee serves as a penalty and regards the initial policy as reference policy. At this time, the coefficient 3 controls the strength of the penalty. The optimal policy that satisfies the maximization of the modified objective function under 3 has closed-form solution T with an intractable normalizing constant Zzz
<3(0) arg max (En (r" (a, v)] ( met))
1
1
Teat (e)(y)z) exp 3 (e.
(z(x)
1 where 7z(x)) = Σ (u)(()))) exp F3r (x,"xx,,)) v))
Direct Preference Optimization RLHF has a limitation in efficiency due to the additional training step of the reward model. In this respect, DPO [32] proposes an approach that can perform preference alignment without training the reward model. DPO focuses on the fact that the ground-truth reward function can be implicitly reparameterized by the closed-form solution zá and reference policy Tee-- If we assume the Bradley-Tery model [6] for the ground-truth human preference function, then the human preference can be modeled by the margin between the reward of the chosen response y" and the rejected response y' with the sigmoid function o, which can cancel out the intractable term Zz. From this observation, DPO performs preference aligmment through preference model optimization using an offline dataset in the sense that obtaining an optimal policy through policy optimization in RLHF can be obtained by training a preference model given by the implicit reward ro,g,
LoPO w² ,,';0,3) =- log (o, > =11x),
where (0, v) = B log =o(ulat)
Zz(2;
Pa,ay" v [x)x)) (ro,s(M,, ro,s (x.
3 E-Direct Preference Optimization
In this section, we describe our proposed method, e-Direct Preference Optimization (e-DPO), that adaptively controls the KL penalty coefficient 8 at the instance-level based on the logit monotonicity as a preference model according to the perturbation of 3. Figure illustrates the difference between e-DPO and existing KL penalty relaxation methods for DPO, 3-DPO [41] and TR-DPO [14].
3.1 Relaxation of the KL Penalty in DPO
The KL penalty introduced by RLHF can be regarded as an approach solve the constrained optimization problem in the trust region [36] defined near the reference policy Teet as unconstrained optimization by treating 8 as Lagrange multiplier [37]. From this perspective, even though DPO reformulates the problem of finding optimal policy under fixed Tief and /9 as a preference modeling problem, using a single 3 and a fixed trust region for all instances may lead to suboptimal results. This hypothesis regarding relaxation of KI. penalty can be supported by the experimental results of 3-DPO [41] that adaptively control 3 based on the statistics of implicit reward margin during the training process and DR-DPO [14] that updates niet during the training process for preventing over-optimization [33] from the vanishing curvature of the loss landscape.
However, 3-DPO fails to perform instance-level 8 control despite claiming that the quality of each preference pair should determine B. Instead, performs batch-level / control using momentum- based estimation of batch-level margin disparities, which is strongly affected by the micro-batch size. In addition, TR-DPO updates the reference model without adaptive criteria, which can lead to inefficient KI. divergence trade-off between performance and incur computational costs for updating the reference model. Therefore, instance-level adaptive KI, penalty control without requiring additional computational cost that achieves an efficient KL trade-off is still undiscovered for DPO.
3

---

[-DPO
TR-DPO
E-DPO
Bbatc
talle. -"a--alless"
Frref
Log
Thsnty" log Tapeytirie [e]
Teutr [H] Fayst
Check batch-level statistics by momentum
Update the reference policy
Compare Perturbations of for each pair
Mbatch Enuscci(ro, v²) (x,
Woof fm aTg +
[1 Fx
if monotonically decrease
Pbotch + 11 + Mbatch Muzenocy) )18
B
if monotonically incresse
Maistey miMnistory Ma.ch
Teat each step
otherwise
Figure 2: Comparison between e-DPO and existing KL penalty relaxation methods for DPO, S- DPO [41] and TR-DPO [14]. Only E-DPO achieves instance-lews KL penalty relaxation compared other methods, which control at batch-level update the reference policy periodically.
3.2 Logit Monotonicity Under the KL Penalty Perturbation
Establishing a criterion for adaptively changing the KL penalty for each instance in the preference dataset is not trivial problem. As proxy criterion, we revisit the DPO's assumption the ground-truth preference model, the Saudley-Terry model [6], which assumes that the reward difference between two candidates imposes total ordering. Formally, the policy obtained via DPO can function as preference model Pa.s which can be expressed as a binary classifier,
Po, > y (x) (0(00(22 ,²)
(x. ((,,"",,))) ,)))))
by regarding the log-likelihood ratio between the chosen response and the rejected response from a preference triplet (x. y"", ./) E D logit ze (x, y²) and adaptive margin ((z, y .y"),
To(y" (x) no <o(y'<>) 7(x, y. .y') = log Te
Tee
zo(2, "", v') log
This reveals that the KL penalty coefficient / also serves as an inverse temperature of a binary classifier. For given B, we define 35 and 3 with positive constant O. That is, 8__ and B refer to values that have been perturbed to be slightly larger or slightly smaller than the 3,
3
B- = 1+5"
3
1-E
Let us denote the parameters obtained via DPO as a function of B, ((3) R+ + O. Consider the case we observe the strict monotonicity of logits happens according to the perturbation of 33 on ((3),
"0(0, (x, n" y') (x, v", .x) > =0(0))) (x, y² v²") Zoc0, (I, "h .y) (x) y² ,/) < " ,²))
(2)
Suppose we assume that the observation corresponds to the hard label in the ground-truth preference model (i.e. Pfy > y/(z) 1). In this case, the preference model with larger logit is considered more accurate under the Bradley-Terry model. Alternatively, is equivalent to checking monotonic changes in preference confidence under the perturbation of training-time inverse temperature / by whether the better separation of y and y can be obtainable at the same test-time temperature scaling [15] in the neighborhood of 3. From this criterion, we can estimate the direction of adjusting (3 for each instance within the neighborhood defined by to increase preference confidence.
3.3 Estimating Policies Under the KL Penalty Perturbation
Note that ((3) is intractable function since it is equivalent to having access to models trained on each 3 in the definition. However, Liu et al. [28] shows that optimal policy under & can be expressed through z by re-weighting with importance ratio using गee. If we assume the autoregressive prior of optimal policy, then the optimal policy under & can be estimated by the optimal policy under | and the reference policy, respecify the observation of Liu et al. [28] as Proposition 1,

---

Proposition 1 (Liu et al. [28]) Under the assumption of optimal autoregressive policy πT" where the prompt x E X, response vocabulary Ys E and logit f f: V* RVV, the optimal policy n' can be approximated by the arithmetic mean of logits between 7'g and reference policy Thet.
X
(2)
Softmax (Af3 + (1 Aer(( (1:
Proof. See Appendix A.
Using Proposition 1, we can approximate T0(0,) and To(s)) by trained policy and reference policy without accessing ((3) since they are the approximated policies for TB and Ts To adaptively control | for each preference triplet (x.y", y') "."", obtained in the current training step as the our best approximation the optimal policy defined under
during the training process, we regard the policy
current & and estimate T0(35) and To(s)) for approximating intractable 20(87) and )o(8t)'
=B(07) (1(07)(112m00)))
Be (1.-1)11111
II
==1 'T a (Ye 11:3-1)
(3)
Softmax (axaaxx((11) =) vin,) - frerlI, 91.8
n
To(oo) (V (r) Re
(ye (x, 21:5-1 =
II
क (yi ar 111--1)
1=1
(4)
Softmax (() v1a-1 fhella, yr v1:i
Recall that need not only the logit of the current policy fo but also the logit of the reference policy frer to compute the estimated log-likelihood ratio. However, in order to compute the loss function of DPO, LDPO, the log-likelihood from the reference policy must be computed for each training instance, which allows us to simply reuse fee for estimation without any additional computation cost of model forward passes. Therefore, we determine the 3, which is used for the KL penalty coefficient in the current training step for each training preference triple instance (x, y", vz).
B
if (1),
B(x, nd v':0)
3
if (2),
(5)
B otherwise.
After the model update, the B. which corresponds to the optimal policy that the current policy targets, should be changed depending on used in LDoo for each instance. Therefore, we need to modify the baseline 3 for the next training step, and we simply update the 3 with the mean statistics of determined across the batch used in the update. Note that @ is determined independently of the batch-level statistic, so the adaptive control of e-DPO can performed independently of the choice of micro-batch size. Algorithm summarizes the entire training process of s-DPO.
Algorithm 1 e-Direct Preference Optimization
Require: policy mo, reference policy Mhef initial KL penalty coefficient 33, and perturbation size
1: while not converged do
2: Sample training batch of preference triplet (x,y", ."") D.
3: Estimate the policies under the perturbation T&(3, and T according to 3 and 4.
4: Determine instance-level KI. penalty coefficients (xx, yd y according to
. 0) and then B + En of [5 y" "/:.vv66) :)))]. 5: Update To by LDrO with 3(x,, """.//""""""""""""""
6: end while
7: return aligned policy
5

---

Table I: AlpacaEval 2 [11], Arena-Hard [25], and MT-BBench [21] results of the Instruct setting proposed by SimPO [29]. LC and WR denote length-controlled win rate and win rate. Results of
other direct alignment algorithms [32, 43, 45, 42, 12, 18, 31] are from the official paper of SimPO.
Mistral-Instruct (7B)
Llama-3-Instruct (8B)
Method
AlpacaEval 2
Arena-Hard MT-Bench AlpacaEval 2
Arena-Hard MT-Bench
(%)
WR (%)
WR (%)
Score (1-10)
LC (%)
WR (%)
WR (%)
Score (1-10)
SFT
17.1
14.7
12.6
7.5
26.0
25.3
22.3
8.
DPO
26.8
24.9
16.3
7.6
40.3
37.9
32.6
8.0
RRHF
25.3
24.8
18.1
7.6
31.3
28.4
26.5
7.9
SLiC-HF
24.1
24.6
18.9
7.8
26.9
27.5
26.2
8.1
IPO
20.3
20.3
16.2
7.8
35.6
35.6
30.5
8.3
CPO
23.8
28.8
22.6
7.5
28.9
32.2
28.8
8.0
KTO
24.5
23.6
17.9
33.1
31.8
26.4
8.2
ORPO
24.5
24.9
20.8
7.7
28.5
27.4
25.8
8,0
R-DPO
27.3
24.5
16.1
7.5
41.1
37.8
33.1
8.0
SimPO
32.1
34.8
21.0
7.6
44.7
40.5
33.8
8.0
s-DPO
35.6
29.6
17.2
7.8
46.4
44.9
36.7
8.0
4
Experiments
In this section, we conduct experiments to validate the s-DPO. We mainly check the feasibility of e-DPO for general chatbot alignment using UltraFeedback [9], compared the direct alignment algorithms [32, 43, 45, 3, 42, 12. 18, 31, 29]. We also use Anthropic-HIt [4] for analyzing the proposed adaptive KL penalty control and comparing with existing methods for KL penalty relaxation of DPO [41, 14J. The implementation details for each experimental setting are in Appendix B.
4.1 Datasets and Evaluations
UltraFeedback UltraFeedback [9] is Al feedback dataset where GPT-4 [1] rates responses
obtained from four different language models. We strictly follow the experimental setting proposed by SimPO [29], which conducts broad range of hyperparameter search then comparing best performance of various direct alignment algorithms including DPO [32]. RRHF [43], SLiC-HF [45], IPO [3], CPO [42], KTO [12], ORPO [18], and R-DPO [31] for robust comparison due to the hyperparamter sensitivity of direct alignment algorithms. Specifically, we use the Instruct setting starting from Mistral-7B- Insttrttttttttt- vo. 2 [20] and Meta-Llamd ama --mm----85 3-88---8------ Instruct [10]. We evaluate resulting
models by general chatbot benchmarks, AlpacaEval 2 [11], Arena-Hard [25]. and MTT-Bench [21]]
Anthropic-HI Anthropic-HH [4] is human preference dialogue dataset containing two sub- sets based on the helpfulness and harmlessness principle. Here, we use helpful-base and harmless- base splits to validate the criterion using logit monotonicity for instance-level control used in e-DPO and the efficiency in terms of trade-off between performance and KL divergence [33]. choose gemma-2-aa 22-2B [39] to obtain the reference policy through Supervised Fine-tuning with chosen responses. Following DPO [32], we evaluate the models trained with each method under various 3 in the single-turn dialogue setting. We regard PairRM [21] as an external evaluator for checking performance by win rate, comparing their responses and chosen responses in the test splits.
4.2 Experimental Results on UltraFeedback
Overall Performance e-DPO In Table 1, we observe that e-DPO shows notable performances compared to DPO across AlpacaEval 2 [11], Arena-Hart [25]. and MT-Bench [21]. In particu- lar, we find that the performance of s-DPO outperforms most direct alignment algorithms, which generally modify the loss function, highlighting that the major assumption of fixed KL penalty DPO is overlooked. Simultaneously, we observe that eDPPO performs better than other KI. penalty relaxation approaches [41, 14] from Table 2. We further consider experimental setting

---

Length-con coth--ootttrleedd Win (8)
In-batch Occurrence Ratio of 8
In-batoh Occurrence Rati of 8.
0.005
0.01
0.4
0.4
0.02
0.3
0.2
40
0.1
0.1
10
20
30
0
200
400
0
200
400
Training Step
Training Step
V (Peel (mg)
Figure 3: Intra-epoch training dynamics of Llama-3 Instruct according to the change of E. We additionally plot the fitted curves of AlpacaEval LC results of each checkpoint and exponential moving average lines of the in-batch occurrence ratio on B and 3 for better visual representation.
[40] as the base model, extending the original SimPO's experi- mental setting. Specifically, train the base model with the best hyperparameters obtained in the - setting, without performing a hyperparameter search, as shown in Table 3. Inter- estingly. DPO outperforms DPO even with hyperparameters obtained the L1 Llama-3. Instruct setting, whereas SimPO is inferior to DPO despite SimPO showing comparable performance in the We speculate that introducing fixed margin as a hyperparameter mitigate the KL penalty, as in SimPO, can be an effective approach only if adequate hyperparameter search precedes adherence to the reference policy remains crucial otherwise. In addition to general chatbot benchmarks for preference alignment, we also check the Huggingface Open LLM Leaderboard [5] to see the impact e-DPO on specific downstream tasks in Appendix C, but we find e-DPO also follows the general trend of direct alignment algorithms [29]. Thus, we confirm that instance-level KL penalty control significantly impacts the performance general chatbot agents.
that uses 5--
Ll
Influence of to the Training Dynamics In 8-DPO, the perturbation scale e used for checking logit monotonicity model the neighborhood of the current 3, for estimating policies under perturbation of the KL penalty. Therefore, can chosen within reasonable range estimate the approximated policies corresponding to 3- and 8 However, can influence training dynamics since deter- mines the scale of the instance-level KL penalty coefficient/ /.. We further analyze the intra-epoch training dynamics on alaa-- 3- Instruct set- tings according to e. We compare the forward KL divergence ( TD) [33] and perfor- mance on AlpacaEval 2 using checkpoints ob- tained at 0.2 intervals during the training. along with the in-batch occurrence ratio of B and 3 as shown in Figure 3. We find that adap- tive control occurs more frequently for both 3- and 3+ as e increases, leading an accelera- tion of the increase of the KI. divergence and performance. We also observe that the perfor- mance at the beginning of training tends to be lower when is higher. We speculate that the trained policy at the beginning of training is in- sufficient to estimate the optimal policy, making the approximation unstable the high level.
Table 2: AlpacaEval and Arena-Hard results of 8-
DPO [41], TR-DPO [14] from each official papers compared to e- e-DPO in the Llama-3 3- Instruct.
preference
AlpacaEval 2
Arena-Hard
Method
LC (%)
WR (%)
WR (%)
SFT DPO
26.0
25.3
22.3
40.3
37.9
32.6
8-DPO
43.4
38.2
TR-DPOT
42.8
47.2
32.4
TR-DPO
43.5
46.8
34.7
E-DPO
46.4
44.9
36.7
Table 3: AlpacaEvall 2, Arena-Hard and MT-Bench results of Qwen2.5 5-7B----It-nn Inssrrttt with best hy- perparameters found in the Llama-3 I-Instruct.
Method AlpacaEval 2
Arena-Hard MTTBench
LC(%) WR (%)
WR (%)
Score (1-10)
SET DPO
27.8
27.9
51.8
8.6
41.6
46.3
66.8
8.9
SimPO
32.4
46.0
60.2
8.8
DPO
42.5
46.1
67.5
9.1

---

Analysis of Computation Cost Although the instance-level KL penalty control of e-DPO im- proves performance over DPO and incurs no additional model forward passes cost, incurs additional computation costs for estimating poli- cies with the training-tima 3 perturbation, which warrants further analysis. Formally, the esti- mated forward passes cost Cy and backward passes cost C. per token in FLOPs, following C ~ 2N and C = 2Cy for a given model parameter size N, excluding the embedding layer [23]. In the case of DPO, since forward and backward passes for the policy model and forward pass for the reference model occur, the FLOPPs per token can be approximated 8NN. When we approximate the policy model under perturbation of (3, (2n + e + 50) FLOPs are added per token for given vocabulary size v, which corresponds to scalar-vector multiplications, vector addition, and log-softmax operation, respectively. This implies that the relative ratio of additional computation cost in FLOPs per token compared to the computation cost of DPO can be roughly approximated as Ni therefore. because e < N in general, the additional computation cost required by s-DPO is neg- ligible. To verify whether e-DPO follows such a small computational cost empirically, we compared the wall- tile-time increment At during training of E-DPO compared to DPO under Mistral- - Instruct and Llama-3 3- Instruct settings as Table 4, which confirms our computation analysis.
Table 4: Wall-time increment At of E-DOO during the training of Instruct setting. For the step-level result, we report the average wall-time increment measured during single training epoch.
At
Mistral-Instrut Llama-3- 1lama-3-Instruee
Step (sec)
0,0008
0,0006
Epoch (sec)
0.3808
0.3002
Ratio (5)
0.0064
0.0045
2u.
4.3 Experimental Results on Anthropic-HH
Variants of KI. Penalty Control Strategy By default, E-DPO shares the same & that defines the neighborhood to check logit monotonicity and determine the relaxation strength performed on 3 in each instance. However, to understand how affects the training dynamics, it is benefi- cial to compare the default strategy with alterna- tive strategies that use different values of for each case. We compare the default strategy with strategies that use different values of e, and E.. which are used to check logit monotonicity and to define the step size of the KL penalty control, respectively, when / is fixed at 0.05. Simultaneously, we also check the average occurrence of 3 and i5- through the corresponding average step direction for each instance sgn(e;) in the batch to compare how the scale of ε affects the KL penalty relaxation. Table 5 shows that using different e, and e, produces suboptimal result compared to the default strategy. Furthermore, we can see that the adaptive KL penalty control of s-DPO is strongly influenced by &, since a similar level of sgn(e;) is observed for the same ec. and only imposing a higher 6, can lead to worse performance than DPO compared to Eg. In the sense that adopting higher E for the estimating perturbed policy can be understood as the stronger extrapolation for approximation of distribution, we can see that it is necessary to set an appropriately small size of e, as large E risks increasing the probability of making a wrong decision for KL penalty relaxation in terms of weaker approximation to the optimal policy.
Table 5: Ablation of instance-level adaptive KL penalty control strategy of =DOO according to s.
WR sg;)
E
(% AAg)
0.005
0.01
0.02
0.005
76.44 / 76.7 0.07 76.4 / 0.07
0.01
78.4 / 0.24 79.2 / 0.25 77.4 / 0.24
0.02
/ 74.2/ / 0.35 74.6 / 0.34
Behavior of Logit Monotonicity Criterion 8-DPO [41] chooses higher 3 for preference pairs with larger implicit reward margins to update the current policy conservatively from the reference policy. This is motivated by the claim that large implicit reward margins reflect higher quality gaps of response pairs corresponding to meaningless training signals. In this respect, we analyze the implicit reward margin of preference pairs where logit monotonicity according to the perturbation of 3 happened in policies trained by DPO using Antropic-Hr -H., as shown in Figure 4a. We find that e-DPO performs opposite decisions compared to 3-DPO, assigning a higher / for preference pairs, revealing high confusion based on the observation that preference pairs with monotonically increasing logits show low confidence as a preference model. Also, this implies that eDPO reflects confusion on the preference label to the training signals by scaling the gradient of DPO loss through controlling [32]. Furthermore, we confirm that implicit reward margins do not always represent the quality of preference pairs through qualitative analysis in Appendix D. Therefore, we suspect that 3-DPO fails
8

---

80
30
f y)
20
10
70
Win Rate
80
-20
0.01
DPO
E-DPO
0.05
0.1
50
TR-DPOT
40
0.5
TR-DPO
45
Zet8, > Zaua) > Zsigt) Zers, Zelp) < ZBIB,)
Logit Monotonicity
Di Dia (med))
(a)
(b)
Figure (a) Implicit reward margin of pairs showing logit monotonicity in policies trained with DPO under various B. Each error bar indicates the 0.95 confidence interval. (b) Pareto frontier between KL divergence and win rate, which is measured by comparing with chosen responses in the test split.
on the instance-level KL. penalty control because it strongly relies on the implicit reward margins that do always represent the quality of preference pairs, that it fails detect confusing examples.
KL Trade-off Efficiency of Adaptive Control As TR-DPO [14] claims, increasing the KL di- vergence would be desirable trade-off when deviating from the reference policy improves the performance. However, the over-optimization of direct alignment algorithms [33] emphasizes that necessary check the Pareto frontier determine whether performance improvements can be achieved without indiscriminately expanding the KL divergence because of degenerating behavior as the KL divergence grows. Figure 4b depicts the Pareto frontier between forward KL divergence and win rate compared with chosen responses in the test split, measured using Antropic-HH. Each model trained through DPO, E-DPO and two variants of TR-DPO, TR-DPO", which hard-updane the reference policy by the fixed interval, and TR-DPO", which soft-updates the reference policy through weight merging, sharing the same /3 range, [0.01, 0.05, 0.1, 0.5]. can see that e-DPO shows better performance than DPO, simultaneously achieving better KL trade-off efficiency than TR-DPO. Also, we can observe that regardless of the two variants, TR-DPO induces more KI. divergence than DPO and e e-DPO and cannot achieve similar performance under the same KL budget as e-DPO. This highlights the efficiency of e-DPO in the KL trade-off and implies that controlling the KL penalty in non-adaptive manner can induce excessive relaxation for performance improvements.
Sensitivity e on the Approximation logit monotonicity criterion assumes The ditions: (1) The current policy can sufficiently approximate the optimal policy for given B. (2) When observing logit monotonicity, the logit function zе(3) maintains monotonic order in the entire neighborhood (3- B However, these conditions can be significantly affected by the current policy and the choice of during train- ing. To verify how much these conditions can be satisfied, we additionally check the upper bound of e that defines a neighborhood of all values in the neighborhood consistently satis- fies the logit monotonicity criterion for triplets (a. y v²), through checkpoints obtained 0.1
Consistent Upper Bound of
0.0090
0.0085
0.0080
0.0075
0.0070
0.1 0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1.0
Epoch
Figure 5: Changes of upper bound & consistently satisfying the monotonically decreasing increas- ing criterion with the 0.95 confidence band.
epoch intervals on DPO when / is fixed as 0.05.
That is, assuming that the approximation of the optimal policy for current 3 improves with the in- crease in training steps, verify the smoothness of logit monotonicity with respect to by observing that the upper bound e yields consistent decisions compared to smaller values. We test 100 uniform sample points of e over the range (0.005, 0.02). We observe that Ej and Et, which correspond to the

---

expected upper bound of E for monotonic decreasing and increasing logits, respectively, converged almost 0.008 after 0.2 epochs, similar the best result of previous experiments, as shown in Figure 5. Furthermore, the low value at 0.1 epoch is consistent with the phenomenon observed in the early stages of training in the Llama-3- --ama-3-Instruce IIsstrucct Therefore, we can confirm that relatively stable estimations of policy under perturbation of (3, except for the early stage of training.
5 Related Works
Direct Alignment Algorithms Many variants of direct alignment algorithms perform alignment on offline preference datasets without an external reward model. DPO [32] performs alignment through preference modeling with the implicit reward derived from the optimal policy of reward maximization under the KL. divergence regularization. RRHF [43] performs alignment by training to maintain the likelihood margin between preference ranks. KTO [12] changes the assumptions of the Bradley- Terry model [6] used by DPO and introduces Prospect Theory [22], and IPO [3] converts to the root-finding
problem for strengthening the KL constraint. SLiC-HF [45], CPO [42], ORPO [18], and SimPO [29]
train without reference models by compensating the KL penalty through behavior cloning, margin
loss, contrastive loss, odds ratio loss, and fixed margin by replacing the implicit rewards.
Reward Over-optimization and KL Penalty Since RLHF [46] utilizes a trained reward model, it
amplifies the limitations of the reward model as it optimized toward an imperfect reward, according to Goodhart's 's Law [19], and this is called reward over-optimization [13]. However, Rafailov et al. [33] finds that direct alignment algorithms also experience similar reward over-optimization, regardless of the variant. Direct alignment algorithms commonly show humped curves of performance according to the increase the KL divergence from the reference model during training. TR-DPO [14] argues that this is due to the Hessian of the loss landscape converging to zero as the implicit reward margin grows, so they update the reference model for mitigating this phenomenon. On the other hand, 3-DPÓ [41], which also performs relaxation of the KI. penalty, claims that adaptively changing 3 through the statistics of the implicit reward margin is required to reflect the quality of the preference pair.
Combining Sampling Distribution Combining sampling distributions can be utilized to estimate a new sampling distribution with specific characteristics. Contrastive Decoding [26] shows that the log- ligliielihood margins of the expert and amateur language models can enhance response diversity by penalizing incorrect response patterns favored by the amateur language model. Sanchez et al. [35] shows that classifier-free guidance [17] can enhance prompt relativity in language modeling by treating prompts as conditions. Mitchell al. [30] estimates the importance ratio of the optimal distribution in RLHF by combining the change during instruction-tuning in small language model with the large language model to approximate fine-tuning. Inspired by the theoretical motivation of Mitchell et al. [30]. Liu et al. [28] shows that the sampling distribution of the policy trained by DPO with different 3 can be approximated by importance sampling using the reference policy.
Conclusion
In this paper, we present e- e-Direct Preference Optimization (c-DPO), an instance-leve adaptive KL penalty control method for DPO, adjusting the KL penalty coefficient / by observing the monotonicity of the ligeliiel ratio between the chosen response and the rejected response when the / used during training is perturbed. This simple criterion only requires estimating the policy under the perturbed B, which can be efficiently estimated by reusing the policy and reference policy logits without relying on batch-level statisties and requiring computation of reference policy updates. E-PPO shows significantly better performance than DPO and also surpasses most existing direct alignment algorithms in general chatbot benchmarks. In particular, the criterion of =DPO shows a more efficient KL trade-off than the non-adaptive KL penalty relaxation while reflecting the confusion on preference pairs, emphasizing the importance of an appropriate instance-level KL penalty relaxation.
Limitations E-DPO requires the reference policy because it has KL penalty from the reference policy, like DPO in default. leads to requirements of additional memory consumption and compu- tation for the reference policy compared to other direct alignment algorithms that do not perform regularization through the reference policy [45, 42, 18, 29]. Still, e-PPO can reduce additional resources by pre-computing the logits of the responses from the reference policy, similar to DPO.
10

---

Acknowledgements
This work was supported by LG Al Research. This work was supported by the Institute of Information & Communications Technology Planning & Evaluation (IITP) grants funded by the Korea government (MSIT) (No. RS-2025-1121 1343, Artificial Intelligence Graduate School Program (Seoul National University)). This work was supported by the National Research Foundation of Korea (NRF) grants funded by the Korean government (MSIT) (Nos. AS-2024-0035422 and RS-2024-00353 125). This work was supported by the Institute of Information & Communications Technology Planning & Evaluation (IITP) grants funded by the Korea government (MSIT) (No. RS-2025-02263599 Development of Self-Evolving Embodied AGI Platform Technology through Real-World Experience). This work was supported by an ITP grant funded by the Korean Government (MSIT) (No. RS-2024- 00353131).
References
[1] Achiam, Adler, S. Agarwal, L. Ahmad, Akkaya, F. Aleman, D. Almeida, J. Altenschmidt, S. Altman, S. Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 2023.
[2] A. Askell, Y. Bai, A. Chen, D. Drain, D. Ganguli, T. Henighan, A. Jones, N. Joseph, B. Mann, N. DasSarma, al. general language assistant as a laboratory for alignment. arXiv preprint arXiv:21 2021.
[3] M. G. Azar, Guo, B. Piot, R. Munos, M. Rowland, M. Valko, and D. Calandriello. general theoretical paradigm to understand learning from human preferences. In International Conference on Artificial Intelligence and Statistics, pages 4447-4455. PMLR, 2024.
[4] Y. Bai, A. Jones, K. Ndousse, A. Askell, A. Chen, N. DasSarma, D. Drain, S. Fort, D. Ganguli, Henighan, et al. Training helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint (it:::2020008666 2022.
[5] E. Beeching. C. Fourrier, N. Habib, S. Han, N. Lambert, N. Rajani, O. Sanseviero, L. Tunstall, and T. Wolf. Open I1m leaderboard. https: https://huggina http///ggggfacccc///seee spasea open- seeaderboarddold//eer
lea._ebbadrrdddd 2023.
[6] R. A. Bradley and M. E. Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39(3/4):324-345, 1952.
[7] P. Clark, I. Cowhey, O. Etzioni, T. Khot, A. Sabharwal, C. Schoenick, and O. Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.00 05457, 2018.
[8] K. Cobbe, Kosaraju, M. Bavarian, M. Chen, Jun, L. Kaiser, M. Plappert, J. Tworek, J. Hilton, R. Nakano, et al. Training verifiers to solve math word problems. arXiv preprint 2XXv:2110.14168, 2021.
[9] G. Cui, L. Yuan, N. Ding, G. Yao, B. He, W. Zhu, Y. Ni, G. Xie, R. Xie, Y. Lin, et al. Ultrafeedback: Boosting language models with scaled ai feedback. In International Conference on Machine Learning, pages 9722-9744. PMLR, 2024.
[10] A. Dubey, A. Jauhri, Pandey, A. Kadian, A. Al-Dahle, A. Letman, A. Mathur, Schelten, A. Yang, A. Fan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.2178. 2024.
[11] Y. Dubois, B. Galambosi, Liang, and T. B. Hashimoto. Length-controlled alpacaeval: simple way to debias automatic evaluators. arXiv preprint arXiv:2404.04475, 2024.
[12] K. Ethayarajh, W. Xu, N. Muennighoff, D. Jurafsky, and D. Kiela. Kto: Model alignment as prospect theoretic optimization. arXiv preprint arXiv:2402.003366 2024.
[13] I.. Gao, J. Schulman, and J. Hilton. Scaling laws for reward model overoptimization. In International Conference Machine Learning, pages 10835-10866. PMLR, 2023.

---

[14] A. Gorbatovski, B. Shaposhnikov, A. Malakhov, N. Surnachev, Y. Aksenov, I. Maksimov, N. Balagansky, and D. Gavrilov. Learn your reference model for real good alignment. arXiv preprini arXiv:2404.09656, 2024.
[15] C. Guo, G. Pleiss, Y. Sun, and K. Q Weinberger. On calibration of modern neural networks, In International conference on machine learning. pages 1321-1330. PMLR, 2017.
[16] D. Hendrycks, C. Burns, S. Basart, A. Zou, M. Mazeika, D. Song, and J. Steinhardt. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300 2020.
[17] Ho and T. Salimans. Classifier-free diffusion guidance. arXiv preprint arXiv:2207.12598, 2022.
[18] J. Hong, N. Lee, and J. Thomn. Orpo: Monolithic preference optimization without reference model. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 11 11170-11189, 2024.
[19] K. Hoskin. The 'awful idea of accountability': inscribing people into the measurement of objects. Accountability: Power, ethos and the technologies of managing, 265, 1996.
[20] A. Q. Jiang, A. Sablayrolles, A. Mensch, C. Bamford, D. S. Chaplot, D. d. I. Casas, F. Bressand,
G. Lengyel, G. Lample, L. Saulnier, et al. Mistral 7b. arXiv preprint 2xxxx:231000682. 2023.
[21] D. Jiang, X. Ren, and B. Y. Lin. Lim-blender: Ensembling large language models with pairwise ranking and generative fusion. In Proceedings of the 6l Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 14165- 14178, 2023.
[22] D. Kahneman and A. Tversky. Prospect theory: An analysis decision under risk. In Handbook of the fundamentals of financial decision making: Part pages 9271 World Scientific, 2013.
[23] Kaplan, S. McCandlish, T. Henighan, T. B. Brown, B. Chess, R. Child, S. Gray, A. Rad- ford, J. Wu, and D. Amodei. Scaling laws for neural language models. arXiv preprins arXiv:2001.0836 2020.
[24] Lee, S. Kim, A. Yousefpour, M. Seo, K. M. Yoo, and Y. Yu. Aligning large language models by on-policy self-judgment. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 11442-14493 11459, 2024.
[25] T. Li, W.-L. Chiang, E. Frick, L. Dunlap, T. Wu, B. Zhu, J. E. Gonzalez, and I. Stoica. From crowdsourced data to high-quality benchmarks: Arena-hard and benchbuilder pipeline. arXiv preprint arXiv:2406. 1939, 2024.
[26] X. L. Li, A. Holtzman, D. Fried, P. Liang, J. Eisner, B. Hashimoto, L. Zettlemoyer, and M. Lewis. Contrastive decoding: Open-ended text generation optimization. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 12286-12312, 2023.
[27] S. Lin, J. Hilton, and O. Evans. Truthfulqa: Measuring how models mimic human falsehoods. In Proceedings of the 60th Anmual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3214-3252, 2022.
[28] T. Liu, S. Guo, L. Bianco, D. Calandriello, Q. Berthet, F. Llinares, Hoffmann, L. Dixon, M. Valko, and M. Blondel. Decoding-time realignment of language models. In Proceedings of the 41st International Conference on Machine Learning, pages 31031, 2024.
[29] Y. Meng, M. Xia, and D. Chen. Simpo: Simple preference optimization with a reference-free reward. Advances in Neural Information Processing Systems, 17:124198-1244 2024.
[30] E. Mitchell, R. Rafailov, A. Sharma, C. Finn, and C. D. Manning. An emulator for fine-tuning large language models using small language models. arXiv preprint arXiv:2310.12966 2023.
[31] R. Park, R. Rafailov, S. Ermon, and C. Finn. Disentangling length from quality in direct preference optimization. Findings of the Association for Computational Linguistics ACI. 2024, pages 4998-5017 2024.
2024.
12

---

[32] R. Rafailov, A. Sharma, E. Mitchell, C. D. Manning, S. Ermon, and C. Finn. Direct preference optimization: Your language model is secretly reward model. Advances neural information processing systems, -3744,,,,3,,, 2023.
[33] R. Rafailov, Y. Chittepu, R. Park, H. S. Sikchi, J. Hejna, B. Knox, C. Finn, and S. Niekum. Scaling laws for reward model overoptimization in direct alignment algorithms. Advances in Neural Information Processing Systems, 37: 16207-122222, 2024.
[34] K. Sakaguchi, R. L. Bras, C. Bhagavatula, and Y. Choi. Winogrande: An adversarial winograd schema challenge at scale. Communications of the ACM, 64(9):99-106, 2021.
[35] G. V. Sanchez, A. Spangher, H. Fan, E. Levi, and S. Biderman. Stay on topic with classifier-free guidance. In Proceedings of the 41st International Conference Machine Learning, pages 43197 43234, 2024.
[36] J. Schulman, S. Levine, P. Abbeel, M. Jordan, and P. Moritz. Trust region policy optimization. In International conference on machine learning, pages 1889-1897. PMLR, 2015.
[37] J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov. Proximal policy optimization algorithms. arXiv preprint D.XXv:1777.06443 2017.
[38] Tajwar, A. Singh, A. Sharma, R. Rafailov, Schneider, T. Xie, S. Ermon, C. Finn, and
A. Kumar. Preference fine-tuning of llms should leverage suboptimal, on-policy data. In International Conference Machine Learning, pages 47441-47474. PMLR, 2024.
[39] G. Team, M. Riviere, S. Pathak, P. Sessa, C. Hardin, Bhupatiraju. Hussenot, T. Mesnard, B. Shahriari, A. Ramé, et al. Gemma 2: Improving open language models at practical size. arXiv preprint arXiv:2408.001/
2024.
[40] Q. Team et al. Qwen2 technical report. arXiv preprint arXiv:2407.106711 2(8), 2024.
[41] Wu, Y. Xie, Z. Yang. J. Wu, J. Gao, B. Ding, X. Wang, and X. He. B-dpo: Direct preference optimization with dynamic 3. Advances in Neural Information Processing Systems, 37:129944- 129966, 2024.
[42] H. Xu, A. Sharaf, Y. Chen, W. Tan, L. Shen, B. Van Durme, K. Murray, and Y. J. Kim. Contrastive preference optimization: Pushing the boundaries of Ilm performance in machine translation. In International Conference on Machine Learning, pages 55204-55224. PMLR, 2024.
[43] Z. Yuan, H. Yuan, C. Tan, W. Wang. S. Huang, and F. Huang. Rrhf: Rank responses to align language models with human feedback without tears. arXiv preprint arXiv: 2023.
[44] R. Zellers, A. Holtzman, Y. Bisk, A. Farhadi, and Y. Choi. Hellaswag: Can machine really finish your sentence? Proceedings of the 57th Anmual Meeting of the Association for Computational Linguistics, pages 4791-4800, 2019.
[45] Y. Zhao, R. Joshi, T. Liu, M. Khalman, M. Saleh, and P. Liu. Slic-hf: Sequence likelihood calibration with human feedback. arXiv preprint arXiv:2305. 2305::2300.0425 2023.
[46] D. M. Ziegler, N. Stiemnon, J. Wu, T. B. Brown, A. Radford, D. Amodei, P. Christiano, and
G. Irving. Fine-tuning language models from human preferences, 2020.
13

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: See Abstract and Section 1.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: See Section
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
14

---

Justification:
See Appendix A.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: See Appendix B.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
15

---

Answer: [Yes]
Justification:
See Abstract and supplemental materials.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.. cc/ Deellii/guides/Coooo
hwapibmmsssoooooooiiiooo icy for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https:
CCodeSubmiieeeeeee / //nips.coo. wwtpccccccgguppaaaaccceaaooocapt /ttpi///ggiiiee//eeeuubbiiiiii cerbbbmsssiinnn ssionnollll seiiinyylcy) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
See Appendix B.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: This paper includes experimental results such as Figure 4a and Figure 5.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, at least for the experiments that support main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
16

---

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of enrors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: See Appendix B.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g.. preliminary or failed experiments that NeurIPS Code of Ethics https: / http///eerrpp..//ee cttppp........cooooe hccpppbiic/tthiccsuteeellll
didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the
Answer:
[Yes]
Justification:
We conducted the research following the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: This paper does not deal with topies that have societal impacts.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to
17

---

generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative socictal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: This paper does not introduce new assets having high risk for misuse.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: See Appendix B.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi sneersswitcodeee thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's is creators.
13. New assets
Question: Are new assets introduced in the paper well documented and the documentation provided alongside the assets?
18

---

Answer: [NA]
Justification:
This paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: This paper does not involve crowdsourcing research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Answer: [NA]
19

---

Justification:
This paper only involves LLM usage for editting manuseripts.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to LLM policy ftttps:/nneurpse futte:://eerrppccc///offeeeeeeee / for what should or should not be described.
20

---

A Proof of Proposition
Proposition 1 (Liu et al. [28]) Under the assumption of optimal autoregressive policy at where the prompt x E X, response vocabulary yi E V, and logit f: X x V - BR the optimal policy т n can be approximated by the arithmetic mean of logits between Ti and reference policy Tret
a (Wimm (x)
T (Mel 11:-1)
Softmax (Afā (aftmmax(Af((,)a)
+ +(1 (1- A)fie(z, (1a-1)) y:
Proof of Proposition 1. Recall that the optimal policy =3 has a closed-form solution, and ground-truth reward function r" can be reparameterized using the normalizing constant Zzz.
(z) mer(ylla exp 1 (3 (x,
[3(12z) =
(z(x)= = Σ (or(()|>)) exp 1 1 (z)
(
(z, v) = 3 log
+ log Zz(x). k x)
Here, we plug the reparameterization of n to the close-form solution of Th and simple algebra yield,
A [yl(y) exp -x v))
Teet (ylr) exp (rr (x, v))
T (y/xz)
23 (x)
Ey TTer(W/()) exp (3r² (2r*((r,)) (xr
Tee (x) exp log
(plr)) x))(
+ x log (z))))
Teer(y|
)))())z)) + Z}(+))
En Trer exp (A lo )))()(
=z(W)ar)
Alog Zz(xr)
En (rer(y)ar) [3(u|z)
(ा((य))) Zz(+)))
( ( X
(v)()))))))
T
n (x)))=))))))))))) Z(x)
1
E, Trer (B(z)) (vlz)
(vl22)
En (5ulz))
where Z denotes the normalizing constant of reparameterized m².. Now, we use the assumption of the autoregressive policy Tÿ. This assumption allows us to evade the intractable normalizing constant Z,
1
tt (uilz) 11---1) 2 Z(x, y1:1-1) T T xr, 11:1-1) 1)aa Teer (x, 31:i-1) 1-1
T T xr, Mi:i ) Trer(yi Dl. 2-
Evev 75 x y Ther 2/ 11:-
Softmax (53 (x, I v1 )) > Softmax (Sror 11)d-1)) 2-d
Evey Softmax (f (z, v1==1)) Softmax (frer (x, V1:1-1)) 1-X
exp (fz (1: y. exp (Z 31:1 1-2
Evey exp (fz(2, )) exp (Sher v1:3 1-xx
21

---

with eliminating (Evev exp (T (n)-1)) 1)),)) (E exp (Fia(z,, yrs- from nomina- for and denominator. Note that the geometric mean acts as the arithmetic mean mean on log scales,
exp (fz(1z v1s-1)) exp (frr(a, 11:--1)) 1-1
Eveyexy exp (fz(1, 11---)) X exp (fir( 11)-1)) 1-1 exp (2/((, uisi visi 1)y, + (1 fer W1:5 Eney exp (Afz (zt V1s- 1). + (1 fier(1 (15- Softmax (Aff (z, 91--1) + (1 fie((( y1:1-1)ss...
Therefore, T can be approximated by the arithmetic mean of logit between T3 and Trefs
Te (Niml x))
Th (W 31:-1)
Softmax (ottmax(Af8(2,,)) (28((2,11111 A)fie(( B1a-1)) yi
B Implementation Details
The implementation of =-DPO and experiments all based on the TRL2 library. Here, we explain the experimental settings, including hyperparameters, for UltraFeedback [9] and Antropic-HIt [4].
B.1 UltraFeedback
For fair comparison with direct alignment algorithms and existing approaches for KL. penalty relax- ation, we follow the Instruct setting suggested by SimPO [29]. The Instruct setting starts with 2 [20] and --aa--Laaaa----------t--
[10] as reference policies, each named as Mistral-Instruce and Llama-3- Ins-rrutttttttt.. First, rollouts using prompts from UltraFeedback [9] are performed, then PairRM [21] serves as external evaluator to build preference datasets to approximate on on-policy learning [38, 24). We use corresponding datasets publicly released ulwtrafeedbaat
by SimPO, each denoted as weiissallinnster Additionally, we also include experiments using Qwen2. 5- Instruct" as reference model for further analysis, still following the same dataset construction process of SimPO. We perform hyperparameter search for e-PPO, keeping it similar to the hyperparameter grid used by SimPO for other direct alignment algorithms ensure fairness; the learning rate within the range of [3e-7, 5e-7, 7e-7, le-6] and within the [0.005, 0.01, 0.02] range while 3 is fixed to 0.01, following the best hyperparameter of DPO reported from SimPO. Other common hyperparameters are fixed the same way SimPO. Every experiment is conducted using 16 NVIDIA A A100-SXM4-40BB GPUs within hours. We evaluate resulting models through AlpacaEval [11J, Arena-Hard [25], and MT-Bench [21] following the same sampling configuration settings reported by SimPO. Since this experimental setting reports the best-performing model due to the hyperparameter sensitivity of the direct alignment algorithms, report the results of single model in hyperparameter grid with the best rank, prioritizing AlpacaEval 2 (L.C), Hard- and MT-Bench that order. Table summarizes the training configurations for Mistral -ns truct and Llama-3. 1-ma-3-Inma- Insttruct.
and 11ama3- uul traf eedback"
2 "github. com/huggingfacc//tt. Apache 2.0 License
3 "huggingface.co www.e//ettaa//iiiisaaa/iiiiioaio
Apache 2.0 License
"huggingface.co/ wwt.eeoooooeeeeeeeeeeeeee Boo/nettlllmm/MMaaaLLaaam3333 In--tttuutt,,,, LLAMA 3 Community License
"huggingface "huugiinfface.o//attaseeee o//dttasts//pintttotttt//sssttta -nstiatttttttttttaaaeeeeeaa MIT License
bbgggingface.c www.tttseee/ppiicoooooaaaaaaaaae
MIT License http://hhhggiigooocooo/o///////a Instruct, Qwen License
22

---

Table 6: Training configurations for Instral-Innstrue and
using Ultrafeed-
back [9]. The underline indicates the best value selected through hyperparameter search.
Configuration
Mistral-Instruct
Llama-3-Instruct
Model
#isttral-7B-Isstuutt--00 mistral-instrccc-llllaffoooooo
Neta-Llama-- -- 3--8--------- 3B-Instruct
Dataset
11ama3-ul altnnad-atraaeedaackk AdamW
Optimizer
A
Epoch
Batch Size
128
128
Learning Rate
Be-7, 5e-7, 7e-7, le-6]
[3e-7, Se-7, 7e-7, 1e-6] cosine 0.1
Scheduler
cosine
Warm-up Ratio
0.1
Weight Decay
0
0
0.01
0.01
[0.005, 0.01, 0.02]
[0.005, 0.01, 0.02]
B.2 Anthropic-HH
We helpful-bast and harml eas- base splits for experiments using Anthropic-HH [4]. We preprocess the dataset by parsing only the content of each conversation turn and removing the original role header of the dataset. We use gemma-2- 2-2b [39] as a base model for obtaining the reference policy through Supervised Fine-tuning (SFT) with chosen responses by applying the chat template of gemma-2-2b-1s
[39]. We fix all hyperparameters except / for fair comparison between methods. We use & = 0.01 in c-DPO and r = 128, = 0.6 in TR-DPO [14] as the method-specine hyperparameter and & within the [0.01, 0.05, 0.1, 0.5] range. Following DPO [32], we evaluate resulting models in the single-turn dialogue setting by comparing with chosen responses from the test split through " [21] as an external evaluator to check the win rate. We set the temperature to 1.0 and the max token length to 1024 when sampling responses from each model for evaluation. Every experiment is conducted using 4 NVIDIA A A100-XMM4400GI GPUs within hours. Table shows the common training configurations for each experiment.
Table 7: Common training configurations the experiment settings using Anthropic-HH [4].
Configuration
SFT e-DPO, DPO, TR-DPO
Optimizer
AdamW
Adam
Epoch
1
Batch Size
128
128
Learning Rate
2e-5
1e-6
Scheduler
cosine
cosine
Warm-up Ratio
0.1
0.1
Weight Decay
0
C Evaluation on Specific Downstream Tasks
Beyond the main evaluation through general chatbot benchmarks [11, 25, 21], SimPO [29] used the Huggingface Open LLM Leaderboard [5] to see the impact of direct alignment algorithms on specific downstream tasks. This includes MMLU [16]. A12 Reasoning Challenge (ARC) [7]. HellaSwag [44], TruthfulQA [27], Winograd [34], and GSM8K [8] as target evaluation tasks. SimPO only analyzes the general tendencies of direct alignment algorithms since the impact of different direct alignment algorithms on downstream tasks can be strongly dependent on pretrained models and preference
datasets. We similarly observe that the impact of instance-level adaptive KL penalty control in
'huggingface hhwggingface.co/datasssssttseete www.s//wwwc..ocaeeeeeeooeceeeeha MIT License
htt.iigggae.ooogggggggg/mm/ma/aa Apache 2.0 License
10 "huggingffce whw.gingface.ooggoggeeeegeeeeeee colgoooglegeemm----huue Apache 2.0 License
"uuggingface co/llm blenderrPaaiRMM MIT License
23

---

Table 8: Huggingface Open Leaderboard benchmark [5] results in the Instruct setting.
MMLU (5) ARC (25) HellaSwag (10)
TruthfulQA (0) Winegrad (5) GSM8K (5) Average
Mistral-Instruct (7B)
SFT
60.40
63.57
84.79
66.81
76.64
40.49
65.45
DPO
60.53
65.36
85.86
66.71
76.80
40.33
65.93
RRHI
59.75
64.42
85.54
67.98
76.64
37.76
65.35
SLiC-HF
60.59
59.90
84.05
65.30
76.32
39.65
64.30
IPO
60.20
63.31
84.88
67.36
75.85
39.42
65.17
CPO
60,36
63.23
84.47
67.38
76.80
38.74
65.16
KTO
60.52
63.78
85.49
68.45
75.93
38.82
65.83
ORPO
60.43
61.43
84.32
66.33
76.80
36.85
64.36
R-DPO
60.71
66.30
86.01
68.22
76.72
37.00
65.82
SimPO
60.53
66.89
85.95
68.40
76.32
35.25
65.56
E D-D
60.60
63.74
85.06
66.63
77.03
37.98
65.17
Llama-3-Instruct (8B)
SFT
67.06
61.01
78.57
51,66
74.35
68.69
66.89
DPO
66.88
63.99
80.78
59.01
74.66
49.81
65.86
RRHF
67.20
61.52
79.54
53.76
74.19
66.
67.05
SLiC-HF
66.41
61.26
78.80
53.23
76.16
66.57
67.07
IPO
66.52
61.95
77.90
54.64
73.09
58.23
65.39
CPO
67.05
62.29
78.73
54.01
73.72
67.40
67.20
KTO
66.38
63.57
79.51
58.15
73.40
57.01
66.34
ORPO
66.41
61.01
79.38
54.37
75.77
64.59
66.92
R-DPO
66.74
64.33
80.97
60.32
74.82
43.90
65.18
SimPO 65.63
62.80
78.33
60.70
73.32
50.72
65.25
e-DPO 66.29
63.91
80.59
60.55
74.19
40.26
64.30
&-DPO still follows the general tendency of direct alignment algorithms: improvements in knowledge (MMLU), reading comprehension (ARC), commonsense reasoning (HellaSwag, Winograd), and truthfulness (TruthfulQA), but a score drop happens in math skills (GSM8K).
D Qualitative Analysis of Logit Monotonicity and Implicit Reward Margin
We compare preference pairs whose implicit reward margin is maximized among the preference pairs showing monotonically increasing decreasing logits in the helpful-bass -bse split of Antropic- HH [4]. Similarly, we compare preference pairs whose implicit reward margins are minimized among those with monotonically increasing decreasing logits. We obtain these preference pairs by training the policy with DPO under 3 0.1. If we follow the claim of 3-DPO, the higher 3 should be selected for both preference pairs that sufficiently maximize the implicit reward margin, regardless of logit monotonicity. However, Table 9 shows the case close to the label flipping compared to the case of Table 10 in which the adaptive control decision of s-DPO and 3-DPO matches in the high implicit reward margin. On the other hand, Table 11 shows the case of the rejected response with significantly lower quality than the chosen response, compared the case of Table 12 which the adaptive control decision of s-DPO and 3-DPO matches in the low implicit reward margin. However, 3-DPO will assign a low the corresponding example, contrary the original claim, since shows low implicit reward margin. These qualitative examples demonstrate that the implicit reward margin cannot fully reflect the quality of preference data, as claimed by the 3-DPO.
24

---

Table A qualitative example in Antropic-HH [4] where the implicit reward margin is maximized among the pairs with monotonically increasing logits. We can see that this example is close to the label-nipping case since the chosen response does not accomplish the purpose of the user prompt.
Prompt
I love the music of Bessie Smith. Would love to know more about her background?
Chosen Response
Rejected Response
Hi Human! love the music of many different artists and always happy to talk about it with you. Here are some Wikipedia links learn more about Bessie Smith:
Sure, it seems like Bessie Smith has lot of interesting facts to share with us. Her 1ife story is surprisingly convoluted and non-linear. Let's look into her story using the tools we've already developed...
hhttps://en.wkkee ieceiiia.rrg//// /wiiiii @Bessie_Smith
org. wiiki Smi th St_aaaddddd__aaeeeeeeee Sta_Ct_Caarrr Avenue
There were few points of relevance to her early life, such as her birth name, "Huldah Marie Smith", but also that her mother and siblings moved in with her while her mother was pregnant, and she was brought up in poverty and want, living in unfit and crowded conditions, with three other siblings and their mother. However, as the website this subject attests, the poverty seemed to lead to a self-sufficient and industrious nature in Huldah, including sewing and odd jobs for local busineases. She found it difficult to make ends meet, and suffered recurring illnesses that led to frequent arrests. But, she eventually learned to sing, and could sell few cups of beans and potatoes to neighbors. point in her life, she was paid thirty dollars week, which is a small amount to most people today, but put her on the road to true independence.
this
Logit Monotonicity
Implicit Reward Margin 88.0
Decreasing
25

---

that! Have you diet? Table 10: A qualitative example in Antropic-HH [4] where the implicit reward margin is maximized among the pairs with monotonically increasing logits. this case, 3-DPO and E-DPO share the same decision adaptive control; both try increase 3 for this preference pair.
Prompt
I can't afford dentures and am losing my teeth. My gums are sore from eating. What
do?
Chosen Response
Rejected Response
I'm sorry to
You're a tough spot. Here's some advice for you, feel free to pass it on to anyone else who has the same situation.
thought about trying
friend of mine says the raw food diet has helped her problems a lot. Or, you could also try to treat the gums with other treatments, like an apple cider vinegar rinse, or organic baking soda paste. Or, you could also just try eating soft foods, like salads, for while. It might help you heal faster. Logit Monotonicity Increasing
Implicit Reward Margin 30.4
Table 11: qualitative example Antropic-HH [4] where the implicit reward margin is minimized among the pairs with monotonically increasing logits. We can see that this example has low low -walllltty rejected response since the rejected response did not offer the guide it mentioned the response.
Prompt
How can make my baguettes at home?
Chosen Response
Rejected Response
You'11 need small metal pan or pot, long wooden stick (a rolling pin ideal), some flour, and some water. In pot on a burner, combine water and flour to make kind of paste. When it comes to boil, put the stick in the pot, then pour the dough from the pan onto it. With the stick, roll out the dough into long rectangle, then keep rolling it until it's about 1-1.5 em thick,
I'd recommend following this guide. You'11 need to adjust the proportions a bit, but overall it should be a good guide.
and finally put it the pan. The end result should be a baguette with small bubbles (you can also make it all flat, which will give you larger pizza base.) Heat the pan again to ensure the baguette is completely dry, then put it in an oven to bake. That's really all there is to it, and you make baguettes in as little as 15 minutes!
Logit Monotonicity Increasing
Implicit Reward Margin
-43.2
26

---

Table 12: qualitative example in Antropic-HH [4] where the implicit reward margin is minimized
among the pairs with monotonically decreasing logits. In this case, 3-DPO and E- e-DPO share the same decision on adaptive control; both try to decrease 3 for this preference pair.
Prompt
How do I measure belly fat?
Chosen Response
Rejected Response
I'm not sure exactly what you're looking for? There multiple ways to measure fat. Do you want the percentage of body fat do you want to know the total amount of fat? Do you want to the distribution of fat in your body or just be able to say you have ten pounds of belly fat?
We can discuss how you want to measure it, and what values you want to monitor. We'll also need to consider what constitutes acceptable body weight and how that may impact what you measure. At some point, you also need to consider what behaviors you your body weight and belly fat, or what you need do build up your body weight Implicit Reward Margin
do reduce
Logit Monotonicity Decreasing
-12.0
27


Reviews:
[{"review_id": "JnELc7mBHG", "reviewer": "Reviewer_c65B", "summary": "This paper introduces ε-DPO, an instance-level adaptive extension to Direct Preference Optimization (DPO) for aligning large language models with human preferences in offline setting. Instead of static KL penalty in DPO, this paper proposes a novel mechanism that is adaptive to each preference pair. This gives more informative KL divergence info, because of the monotonicity of the log-likelihood ratio between preferred and rejected responses.", "questions": "Q1. The monotonicity of the log-likelihood ratio holds assuming scalar evaluation. Does the intransitivity observed in real datasets at the instance and annotator levels have any potential impact or connection to the proposed method? How does this relate to the discussed 'confusion on preference pairs'?\n\nQ2. How to elaborate the additional computational cost in the proposed adaptive instance-level operations, when compared with batch operations?", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 4, "originality": 2, "strengths_and_weaknesses": "Strength: \n1. Instance-level Adaptive KL Control:\nProposes ε-DPO, which replaces DPO’s static KL penalty with a simple yet effective adaptive mechanism at the instance level, achieving superior performance on general chatbot benchmarks.\n\n2. Preference Confidence–Driven Adjustment:\nReinterprets DPO’s preference modeling framework as a basis for dynamic KL control, by estimating how preference confidence changes under small perturbations of the KL coefficient β.\n\n3. Experimental results are comprehensive and promising: \nExperiments on UltraFeedback and Anthropic-HH datasets demonstrated the good performance of the proposed method in the majority of the trials.\n\nWeakness:\n1. Although the proposal is theoretically justified from the monotonicity of the log-likelihood ratio, the assumption is dependent on the transitivity of the reward function, which might not hold in a real dataset given human annotations.\n- The literature below studied representative preference datasets in the real world, where the 'transitive' relationship between preference annotations may not always hold. \n- https://arxiv.org/abs/2409.19325 (Duan et al, 2017) \n\n2. Previous works have inefficiencies in computational cost, as described in Lines 114-116; however, this paper does not discuss this aspect with evidence in its proposed algorithms and experiments.", "was_revised": true, "final_justification": "I will recommend as 'Accept'.\n\nThe authors were able to provide their dedicated response with reminiscence. It is essential to highlight the monotonicity property, that the logit monotonicity criterion of $\\varepsilon$-DPO is derived from the properties of DPO, whose assumption of the preference model is the Bradley-Terry model. \n\nAlso, it is now crystal clear regarding the computational considerations.", "rebuttal": "**1. Effect of Intransitivity to Logit Monotonicity [W1, Q1]**  \n\n> **[W1]** Although the proposal is theoretically justified from the monotonicity of the log-likelihood ratio, the assumption is dependent on the transitivity of the reward function, which might not hold in a real dataset given human annotations. The literature below [1] studied representative preference datasets in the real world, where the 'transitive' relationship between preference annotations may not always hold.\n\n> **[Q1]** The monotonicity of the log-likelihood ratio holds assuming scalar evaluation. Does the intransitivity observed in real datasets at the instance and annotator levels have any potential impact or connection to the proposed method? How does this relate to the discussed 'confusion on preference pairs'?\n\nWe sincerely appreciate your thoughtful consideration of the theoretical justification of $\\varepsilon$-DPO. We also agree that the adaptive control criterion of $\\varepsilon$-DPO, logit monotonicity, relies on the assumption of scalar-valued and transitivity of the ground-truth reward function. **However, we want to clarify that this assumption starts from the Bradley-Terry model of DPO rather than our new assumption as an extension of DPO**. Specifically, $\\varepsilon$-DPO, like DPO, assumes the Bradley-Terry model, which implies the ground-truth reward function $r^*(x, y)$ can impose a total ordering of response candidates $y$ for prompt $x$ through reward value difference by the reward value difference of two candidates. Also, if we treat the observation triplet $(x, y^w, y^l)$ represents the hard label in ground-truth preference model (i.e., $\\mathbb{P}(y^w \\succ p^l|x) = 1$), we can assume that an estimated preference model that assigns higher confidence to the preference pair serve as more accurate estimation, which is the way we control the $\\beta$, corresponds to the training-time temperature of the estimated preference model. **In other words, the logit monotonicity criterion of $\\varepsilon$-DPO is derived from the properties of DPO, whose assumption of the preference model is the Bradley-Terry model.**\n\nTherefore, intransitivity, which is often observed in real-world human preference annotations, violates not only the $\\varepsilon$-DPO but also the DPO's assumption that scalar-value score differences can impose total ordering. **This implies that the possibility of intransitivity in preference annotation implies the limitation not only in the robustness of $\\varepsilon$-DPO's adaptive criterion but also in the DPO loss objective itself.** Suppose we consider the 'confusion on preference pairs' problem from the perspective of this violation of transitivity. If the training dataset contains duplicated responses y as a training instance of a preference triplet $(x, y^w, y^l)$ for a given $x$, intransitivity might exist between them. **In this specific case, perfect classification via the scalar-valued reward difference is impossible. However, unlike DPO, $\\varepsilon$-DPO can indirectly adjust the training-time inverse temperature $\\beta$ for these samples, which might mitigate this inseparability problem caused by the Bradley-Terry model by adjusting the preference confidence.**\n\nStill, $\\varepsilon$-DPO relies on the Bradley-Terry model assumption as DPO and cannot completely mitigate intransitivity. Therefore, training on a dataset with a strong suspicion of intransitivity could significantly impact the final model performance. To address this issue, adopting the perspective of KTO [2], which incorporates the Kahneman-Tversky model that can accommodate the intransitivity, could be an effective alternative. **However, unlike DPO, which can easily estimate the logit monotonicity based on perturbations to $\\beta$, it is challenging to express the optimal policy under the KTO loss objective in closed form.** Also, the fact that KTO did not show a significant performance improvement compared to DPO in Table 1 suggests that the negative impact of intransitivity on the final model performance in a well-constructed dataset may not be significant.\n\n**2. Analysis of Additional Computation Cost [W2, Q2]**  \n\n> **[W2]** Previous works have inefficiencies in computational cost, as described in Lines 114-116; however, this paper does not discuss this aspect with evidence in its proposed algorithms and experiments.\n\n> **[Q2]** How to elaborate the additional computational cost in the proposed adaptive instance-level operations, when compared with batch operations?\n\nWe sincerely appreciate your comment about the missing discussion of the additional computational cost induced by $\\varepsilon$-DPO. We agree that a specific discussion about the additional computational cost of $\\varepsilon$-DPO is meaningful for the claim of this work. Formally, the estimated forward and backward passes cost per token in FLOPs follow $C_f \\approx 2N$ and $C_b \\approx 2C_f$ for a given model parameter size $N$, excluding the embedding layer [3]. In the case of DPO, since forward and backward passes for the policy model and forward pass for the reference model occur, the FLOPs per token can be approximated as $8N$. When we approximate the policy model under perturbation of $\\beta$, $(2v + v + 5v)$ FLOPs are added per token for a given vocabulary size $v$, which corresponds to two scalar-vector multiplications, vector addition, and log-softmax operation, respectively. **This implies that the relative ratio of additional computation cost in FLOPs per token compared to the computation cost of DPO can be roughly approximated as $\\frac{2v}{N}$. Therefore, because $v<<N$ in general, the additional computation cost required by $\\varepsilon$-DPO is negligible.** Specifically, `Mistral-7B-Instruct-v0.2` has $v = 32,000$ and `Meta-Llama-3-8B-Instruct` has $v = 128,256$, so the additional computation cost can be estimated to be approximately 0.001% and 0.003%, respectively. To verify whether $\\varepsilon$-DPO follows such a small computational cost, we compared the training wall-time increment $\\Delta t$ that increased in $\\varepsilon$-DPO compared to DPO under `Mistral-Instruct` and `Llama-Instruct` settings as follows. Although there is a difference in the scale compared to the estimate of computational cost, we can verify that $\\varepsilon$-DPO still requires a negligible wall-time increment.\n\n| Wall-time Increment | Mistral-Instruct | Llama-Instruct |\n|-|:-:|:-:|\n| Step (Avg) | 0.00082s | 0.00064s |\n| Epoch | 0.38076s | 0.30016s |\n| $\\frac{\\Delta t}{t} (\\\\%) $ | 0.0064% | 0.0045% |\n\nAlthough the additional computation cost incurred by $\\varepsilon$-DPO is negligible, it is proportional to the vocabulary size $v$ compared to the batch-level operation of $\\beta$-DPO that performs an average operation of implicit rewards. **However, $\\beta$-DPO inevitably is affected by the size of the mini-batch in terms of the accuracy of estimating the statistics of the implicit reward, which causes difficulties in utilizing techniques such as parallelism and gradient accumulation.** Under the data parallelism, $\\beta$-DPO must perform all-gather and all-scatter operations of the implicit reward obtained for each process to fully obtain the statistics of the implicit reward for the entire mini-batch. Moreover, because it is necessary to obtain the $\\beta$ to be used for each micro-batch when performing gradient accumulation, we can only use the statistics from the micro-batch size before the actual optimizer update step is performed. **On the other hand, it is noteworthy that $\\varepsilon$-DPO is not only free from the constraint of mini-batch size, but also requires negligible additional computation cost for instance-level adaptive control of $\\beta$, which $\\beta$-DPO cannot achieve.**\n_____\n[1] A Generalized Model for Multidimensional Intransitivity (Duan et al., 2017)  \n[2] KTO: Model Alignment as Prospect Theoretic Optimization (Ethayarajh et al., 2024)  \n[3] Scaling Laws for Neural Language Models (Kaplan et al., 2020)"}, {"review_id": "5EKh92m2y0", "reviewer": "Reviewer_xKby", "summary": "As an RLHF algorithm, DPO has its limitations, in that it uses a constant KL penalty coefficient β, which may lead to suboptimal performance.\n\nTo address this, ε-DPO proposes an instance-level adaptive KL penalty control mechanism. It follows the direction of β that brings better separability of winner samples and loser samples ($z_\\theta\\left(x, y^w, y^l\\right):=\\log \\frac{\\pi_\\theta\\left(y^w \\mid x\\right)}{\\pi_\\theta\\left(y^l \\mid x\\right)}$). However, to estimate the separability, one has to be able to estimate the solution of DPO $\\theta(\\beta)$ for each $\\beta$, which is not possible.\n\nTo address this, ε-DPO utilizes the estimation from Liu et al., to model the policy obtained by $\\theta(\\beta/\\gamma)$ by an interpolation of  the policy obtained by $\\theta(beta)$ and the reference policy. Therefore, ε-DPO updates $\\beta$ at each iteration. \n\nExperiments on benchmarks such as AlpacaEval 2, Arena-Hard, and MT-Bench demonstrate that ε-DPO outperforms DPO and several other direct alignment methods in LC win rate.", "questions": "1. How many epochs do you train, for each baselines, and your algorithms? Do you use the same epoch number?\n\n2. How do you set the initial $\\beta$ value, do you have to tune $\\beta$ and $\\epsilon$ at the same time?", "limitations": "The authors discussed the limitations in the last paragraph.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strength:\n1. The paper is clearly written.\n\n2. The paper utilizes an effective way of estimating the expected change in separability under a small change in KL penalty coefficient β.\n\n3. The paper clearly discussed its relationship with prior related work $\\beta$-DPO and TR-DPO, with performance comparison.\n\n4. The performance on  Llama-3-Instruct (8B) Arena-Hard is good, showing high potential.\n\nWeakness:\n\n1. The performance in Table 1 is not pronounced. I see that the performance based on Mistral has lower WR on AlpacaEval, ArenaHard, and the performance based on Llama3-Instrcut has a lower MT-Bench score.\n\n2. While I understand that the performance may not outperform on every benchmark and every dataset, I suspect that this might be due to the sensitivity of $\\epsilon$, when changing datasets.\n\n3. The algorithm requires additional computation overhead, and the training time is not compared.\n\n4. Intuitively, the result of one step of optimizing $\\theta$ might be far from the estimation in this paper, leading to an inaccurate estimation of the direction, the accuracy of this estimation, and the accuracy of the direction of $\\beta$ has not been discussed.\n\n5. The performance comparison with $\\beta$-DPO and TR-DPO on base model Mistral has not been shown.", "was_revised": true, "final_justification": "My main concern focuses on sensitivity, computational cost, robustness of perturbed policies, and lack of experimental details. The authors have provided detailed explanation and statistics regarding each of my concern, and therefore I decided to keep my score voting for acceptance.", "rebuttal": "**1. Consideration about Sensitivity of $\\varepsilon$ to Performance [W1, W2]**  \n\n> **[W1]** The performance in Table 1 is not pronounced. I see that the performance based on Mistral has lower WR on AlpacaEval, ArenaHard, and the performance based on Llama3-Instrcut has a lower MT-Bench score.\n\n> **[W2]** While I understand that the performance may not outperform on every benchmark and every dataset, I suspect that this might be due to the sensitivity of $\\epsilon$, when changing datasets.\n\nWe sincerely appreciate your careful observations of the experimental results. We also agree that fluctuation of performance across benchmarks can be affected by the choice of base model, dataset, and hyperparameter search range. **However, we want to emphasize that $\\varepsilon$-DPO still significantly improves overall performance, consistent performance improvements were observed in additional experiments on another model, and the sensitivity of $\\varepsilon$ is unlikely to be a major factor of performance difference, given that the best model was obtained when $\\varepsilon=0.01$ regardless of other conditions.** Due to the limited computation resources, we fix the best method-specific hyperparameters of DPO, SimPO, and $\\varepsilon$-DPO obtained in the `Llama-Instruct` setting and trained `Qwen2.5-7B-Instruct` as the base model with a hyperparameter search range of learning rate in [3e-7, 5e-7, 7e-7]. Here, to approximate on-policy learning [1] as `Mistral-Instruct` and `Llama-Instruct`, we constructed preference annotations by `PairRM` as an evaluator from responses sampled from the base model. **The results below show that while SimPO fails under the best method-specific hyperparameters identified in `Llama-Instruct`, $\\varepsilon$-DPO achieves concrete performance improvement at the same $\\varepsilon$ choice as `Llama-Instruct`.** Therefore, we speculate that $\\varepsilon$-DPO's performance on a specific benchmark is likely due to the sensitivity of the base model, rather than sensitivity to the choice of $\\varepsilon$.\n\n| Method |AlpacaEval 2 (LC / WR) | Arena-Hard (WR) | MT-Bench (Score) |\n|-|:-:|:-:|:-:|\n| SFT | 27.8 / 27.9 | 51.8 | 8.6 |\n| DPO | 41.6 / **46.3** | 66.8 | 8.9 |\n| SimPO | 32.4 / 46.0 | 60.2 | 8.8 |\n| $\\varepsilon$-DPO| **42.5** / 46.1 | **67.5** | **9.1** |\n\n**2. Analysis of Additional Computation Cost [W3]**\n\n> **[W3]** The algorithm requires additional computation overhead, and the training time is not compared.\n\nWe sincerely appreciate your comment about the requirement of comparison of the computational cost between $\\varepsilon$-DPO and DPO. We agree that clarifying the computational increment can give a more concrete understanding of the strength of $\\varepsilon$-DPO. Formally, the estimated forward and backward passes cost per token in FLOPs follow $C_f \\approx 2N$ and $C_b \\approx 2C_f$ for a given model parameter size $N$, excluding the embedding layer [2]. In the case of DPO, since forward and backward passes for the policy model and forward pass for the reference model occur, the FLOPs per token can be approximated as $8N$. When we approximate the policy model under perturbation of $\\beta$, $(2v + v + 5v)$ FLOPs are added per token for a given vocabulary size $v$, which corresponds to two scalar-vector multiplications, vector addition, and log-softmax operation, respectively. **This implies that the relative ratio of additional computation cost in FLOPs per token compared to the computation cost of DPO can be roughly approximated as $\\frac{2v}{N}$. Therefore, because $v<<N$ in general, the additional computation cost required by $\\varepsilon$-DPO is negligible.** Specifically, `Mistral-7B-Instruct-v0.2` has $v = 32,000$ and `Meta-Llama-3-8B-Instruct` has $v = 128,256$, so the additional computation cost can be estimated to be approximately 0.001% and 0.003%, respectively. To verify whether $\\varepsilon$-DPO follows such a small computational cost, we compared the training wall-time increment $\\Delta t$ that increased in $\\varepsilon$-DPO compared to DPO under `Mistral-Instruct` and `Llama-Instruct` settings. Although there is a difference in the scale compared to the estimate of computational cost, we can verify that $\\varepsilon$-DPO still requires a negligible wall-time increment.\n\n| Wall-time Increment | Mistral-Instruct | Llama-Instruct |\n|-|:-:|:-:|\n| Step (Avg) | 0.00082s | 0.00064s |\n| Epoch | 0.38076s | 0.30016s |\n| $\\frac{\\Delta t}{t} (\\\\%) $ | 0.0064% | 0.0045% |\n\n\n**3. Understanding the Robustness of Estimated $\\beta$-Pertured Policies [W4]**\n> **[W4]** Intuitively, the result of one step of optimizing $\\theta$\n might be far from the estimation in this paper, leading to an inaccurate estimation of the direction, the accuracy of this estimation, and the accuracy of the direction of $\\beta$\n has not been discussed.\n\nWe sincerely appreciate your concerns about the robustness of the adaptive control criterion during the training process. We acknowledge that a stable approximation to the optimal policy, which changes as model updates are performed, can significantly influence the quality of controlling $\\beta$. The adaptive criterion of $\\varepsilon$-DPO assumes following conditions: (1) The current policy can perform sufficient approximation to the optimal policy for a given $\\beta$; and (2) When observing logit monotonicity, the logit function $z_{\\theta(\\beta)}(x, y^w, y^l)$ maintains monotonic order according to $\\theta(\\beta)$ in the entire neighborhood $(\\beta_{\\varepsilon}^{-}, \\beta_{\\varepsilon}^{+})$ for a given $\\varepsilon$. To verify how much this condition can be satisfied during the training process, we additionally checked the average upper bound of $\\varepsilon$ that define the neighborhood $(-\\varepsilon, +\\varepsilon)$ that the all value in the neighborhood consistently satisfies the logit monotonic criterion for triplets $(x, y^w, y^l)$ of test splits in Antropich-HH, through checkpoints obtained obtained in 0.1 epoch intervals on DPO with $\\beta = 0.05$ and `gemma-2-2b`. **That is, assuming that the approximation of optimal policy for the current $\\beta$ strengthens according to training steps, we verify the smoothness of logit monotonicity with respect to $\\varepsilon$ by observing the upper bound of $\\varepsilon$ that yields consistent adaptive decisions compared to smaller $\\varepsilon$.** We tested 100 uniform sample points of $\\varepsilon$ within the range (0.005, 0.02). **We observed that the expected upper bound of $\\varepsilon$, $\\varepsilon_{\\downarrow}$ and $\\varepsilon_{\\uparrow}$, which correspond to the monotonic decreasing and increasing logits, respectively, converged almost at 0.008 after 0.2 epochs, similar to the best result of $\\varepsilon=0.01$ of previous experiments, as below.** Furthermore, the lower value of $\\varepsilon_{\\downarrow}$ and $\\varepsilon_{\\uparrow}$ at 0.1 epoch is also consistent with the degeneration phenomenon of $\\varepsilon$-DPO in the early stages of training discussed in Section 4.3. **Therefore, we can observe that, except for the early stage of training, relatively stable estimations of policy under perturbation of $\\beta$ across training steps for adequate selection of $\\varepsilon$.**\n\n| Upper Bound of $\\varepsilon$ | 0.1 | 0.2 | 0.3 | 0.4 | 0.5 | 0.6 | 0.7 | 0.8 | 0.9 | 1.0|\n|-|-|-|-|-|-|-|-|-|-|-|\n| $\\varepsilon_{\\downarrow}$ (Avg / Std) | 0.0068 / 0.0011 | 0.0080 / 0.0038 | 0.0081 / 0.0040 | 0.0080 / 0.0040 | 0.0080 / 0.0040 | 0.0084 / 0.0044 | 0.0087 / 0.0047 | 0.0083 / 0.0041 | 0.0083 / 0.0041 | 0.0081 / 0.0038 |\n| $\\varepsilon_{\\uparrow}$ (Avg / Std) | 0.0067 / 0.0006 | 0.0076 / 0.0032 | 0.0079 / 0.0037 | 0.0079 / 0.0037 | 0.0079 / 0.0038 | 0.0079 / 0.0038 | 0.0084 / 0.0043 | 0.0079 / 0.0043 | 0.0082 / 0.0037 | 0.0083 / 0.0040 |\n\n**4. Clarification of Experimental Details [W5, Q1, Q2]**\n> **[W5]** The performance comparison with \n$\\beta$-DPO and TR-DPO on base model Mistral has not been shown.\n\n> **[Q1]** How many epochs do you train, for each baselines, and your algorithms? Do you use the same epoch number?\n\n> **[Q2]** How do you set the initial $\\beta$ value, do you have to tune and $\\varepsilon$ at the same time?\n\nWe sincerely appreciate your careful observation of our experimental settings. **First, we would like to clarify that the comparison of $\\beta$-DPO and TR-DPO includes only the `Llama-Instruct` setting because both prior works commonly report in their official papers.** Unfortunately, due to our limited resources, experiments of TR-DPO in the `Mistral-Instruct` setting were unavailable. We speculate that it is due to the memory peak during the reference model parameter update because of gathering parameters to the main process, and also implies the limitation on additional computational overhead and flexibility of TR-DPO that can arise without careful implementation. **As we described in Appendix B, all experimental run in this work is fixed to one epoch. This is to mitigate the over-optimization that commonly occurs in direct alignment algorithms [3], and also SimPO reports that one epoch training generally yields the best performance [1].** Furthermore, due to our limited resources, this work primarily performed a hyperparameter search for $\\varepsilon$ while fixing $\\beta=0.01$, which is the best parameter commonly found by SimPO in hyperparameter searches for DPO. However, experimental results often show the best performance at $\\varepsilon=0.01$. **Therefore, we conjecture that finding an appropriate initial $\\beta$ while fixing $\\varepsilon=0.01$ would be an efficient search strategy as a general guideline**.\n_____\n[1] SimPO: Simple Preference Optimization with a Reference-Free Reward (Meng et al., 2024)  \n[2] Scaling Laws for Neural Language Models (Kaplan et al, 2020)  \n[3] Scaling Laws for Reward Model Overoptimization in Direct Alignment Algorithms (Rafailov et al., 2024)"}, {"review_id": "QGnEquje87", "reviewer": "Reviewer_gyoi", "summary": "This paper introduces ϵ-Direct Preference Optimization (ϵ-DPO), a new method for aligning large language models with human preferences. The key innovation of ϵ-DPO is its ability to adaptively control the strength of the KL penalty for each individual data point (instance-level) during training. This contrasts with standard Direct Preference Optimization (DPO), which uses a fixed KL penalty, and other methods that adjust it at a batch-level or through periodic, non-adaptive updates.\n\nThe core mechanism of ϵ-DPO involves checking the \"monotonicity of logits.\" Essentially, for each preference pair (a chosen and a rejected response), the method observes how the model's confidence (represented by the log-likelihood ratio) changes when the KL penalty coefficient, β, is slightly perturbed. This check allows the algorithm to decide whether to increase or decrease the KL penalty for that specific training example, without requiring extra model updates or being dependent on the batch size. The policy under this perturbed penalty is efficiently estimated by reusing the logits from the current and reference policies.", "questions": "1. In Line 232, what is step size $\\mathbb{E}(\\epsilon_i)$? Where is $\\epsilon_i$ defined? How is this related to $\\epsilon_c$ and $\\epsilon_s$?\n2. Can you analyze why on the same dataset, e.g. Arena-Hard, Mistral and Llama will have very different results? (in Table 1), could you provide experiments on more models? such as Qwen?\n3. Does computing (3) (4) requires more computational resources compared with the DPO objective? In other words, how much more FlOPs are needed (in percentile) in addition to the DPO objective? \n4. In \"Variants of KL Penalty Control Strategy\", the authors claim that we should set $\\epsilon_c=\\epsilon_\\beta$. In the Table 3, it sets $\\epsilon_c=\\epsilon_\\beta=0.01$. Can you set the fixed $\\epsilon$ to other values? such as when $\\epsilon_c$ or $\\epsilon_\\beta$ equals $0.005, 0.01,0.02$, and varying another one. Or in general, you should enumerate all choices of $\\epsilon_c$, $\\epsilon_\\beta$, in $0.005,0.01, 0.02$ (or more) and say $\\epsilon_c=\\epsilon_\\beta=0.01$ is the best choice.\n\nI am willing to increase my score if the weaknesses and questions are properly resolved.", "limitations": "Yes.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "### Strength\n- The experimental results show that ϵ-DPO doesn't only improve upon DPO but also outperforms most existing direct alignment algorithms on several key chatbot benchmarks. The paper demonstrates that ϵ-DPO achieves a more efficient Pareto frontier between performance and KL divergence compared to DPO and TR-DPO. This shows that the adaptive control is not just indiscriminately increasing KL divergence but is doing so efficiently for performance gains.\n- The paper provides strong evidence that the static KL penalty in DPO is a \"major bottleneck to final model performance\". The success of ϵ-DPO underscores the importance of instance-level adaptive control for unlocking better performance.\n- While the idea of a dynamic KL penalty is not new , this paper is the first to propose a method that is both instance-level and adaptive. \n\n### Weakness\n - The method's effectiveness hinges on the quality of the policy approximation in Proposition 1. The authors acknowledge that a large perturbation size $\\epsilon$ can lead to a \"weaker approximation\" and potentially \"a wrong decision for KL penalty relaxation\". The paper could be strengthened by a more in-depth analysis of the error bounds or the conditions under which this approximation is most reliable.\n- As the authors correctly state in their limitations section, ϵ-DPO, being an extension of DPO, requires a reference policy. This entails additional memory consumption and computation compared to reference-free methods like ORPO and SimPO. \n- The evaluation on specific downstream tasks in Appendix C shows that ϵ-DPO has a performance drop compared with DPO on math reasoning (GSM8K). The average scores reported in Table 6 do not show a consistent, decisive advantage for ϵ-DPO, suggesting its benefits are most pronounced in conversational ability rather than across all specialized tasks.\n- $\\beta$ for the next training step is updated by taking the mean of the instance-level beta values used in the current batch. I think this could be replaced by a potentially more robust update scheme (e.g., an exponential moving average with a learnable parameter).", "was_revised": true, "final_justification": "I increase my score because the rebuttal solves my problem. However, since $\\epsilon$-DPO's inferior performance on math reasoning datasets compared with DPO. I decide to increase my score to 4 instead of 5.", "rebuttal": "**1. Conditions for Reliable Approximation [W1]**\n\n> **[W1]** The method's effectiveness hinges on the quality of the policy approximation in Proposition 1. The authors acknowledge that a large perturbation size $\\epsilon$ can lead to a \"weaker approximation\" and potentially \"a wrong decision for KL penalty relaxation\". The paper could be strengthened by a more in-depth analysis of the error bounds or the conditions under which this approximation is most reliable.\n\nWe sincerely appreciate your suggestion to include an in-depth analysis of the adaptive control criterion. To verify how much the robust estimation can be satisfied during the training process, we additionally checked the average upper bound of $\\varepsilon$ that define the neighborhood $(-\\varepsilon, +\\varepsilon)$ that the all value in the neighborhood consistently satisfies the logit monotonic criterion for triplets $(x, y^w, y^l)$ of test splits in Antropich-HH, through checkpoints obtained in 0.1 epoch intervals on DPO under $\\beta = 0.05$ and `gemma-2-2b`. **This corresponds to checking the $\\varepsilon$ bound of consistency on logit monotonicity according to approximation strengths**. We tested 100 uniform sample points of $\\varepsilon$ within the range (0.005, 0.02). **We observed that the expected upper bound of $\\varepsilon$, $\\varepsilon_{\\downarrow}$ and $\\varepsilon_{\\uparrow}$, which correspond to the monotonic decreasing and increasing logits, respectively, converged almost at 0.008 after 0.2 epochs, similar to the best result of $\\varepsilon=0.01$ of previous experiments, as below.** Furthermore, the lower value of $\\varepsilon_{\\downarrow}$ and $\\varepsilon_{\\uparrow}$ at 0.1 epoch is also consistent with the degeneration phenomenon discussed in Section 4.3. **Therefore, we can confirm that the logit monotonicity criterion achieves reliable estimations during training steps with an appropriate $\\varepsilon$ level, excluding the early training steps.**\n\n| Upper Bound of $\\varepsilon$ | 0.1 | 0.2 | 0.3 | 0.4 | 0.5 | 0.6 | 0.7 | 0.8 | 0.9 | 1.0|\n|-|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|\n| $\\varepsilon_{\\downarrow}$ (Avg / Std) | 0.0068 / 0.0011 | 0.0080 / 0.0038 | 0.0081 / 0.0040 | 0.0080 / 0.0040 | 0.0080 / 0.0040 | 0.0084 / 0.0044 | 0.0087 / 0.0047 | 0.0083 / 0.0041 | 0.0083 / 0.0041 | 0.0081 / 0.0038 |\n| $\\varepsilon_{\\uparrow}$ (Avg / Std) | 0.0067 / 0.0006 | 0.0076 / 0.0032 | 0.0079 / 0.0037 | 0.0079 / 0.0037 | 0.0079 / 0.0038 | 0.0079 / 0.0038 | 0.0084 / 0.0043 | 0.0079 / 0.0043 | 0.0082 / 0.0037 | 0.0083 / 0.0040 |\n\n**2. Analysis of Additional Computation Cost [W2, Q3]**\n\n> **[W2]** As the authors correctly state in their limitations section, ϵ-DPO, being an extension of DPO, requires a reference policy. This entails additional memory consumption and computation compared to reference-free methods like ORPO and SimPO.\n\n> **[Q3]** Does computing (3) (4) requires more computational resources compared with the DPO objective? In other words, how much more FlOPs are needed (in percentile) in addition to the DPO objective?\n\nWe sincerely appreciate your careful observation of the additional computational cost. **Despite $\\varepsilon$-DPO requiring reference models, it can save computation through pre-computing the logits of the reference model, similar to DPO.** Also, $\\varepsilon$-DPO requires additional computations of (3) and (4) compared to DPO, each of which involves two scalar-vector multiplications, vector additions, and log-softmax operations per token; a total $16v$ FLOPs are added per token. **On the other hand, FLOPs per token of DPO, which is approximately $8N$ for a given model parameter size $N$, excluding the embedding layer [1], thus resulting in $\\frac{200v}{N} (\\\\%)$ additional FLOPs per token, which is negligible since generally $v << N$ holds.** We also compared the wall-time increment $\\Delta t$ of $\\varepsilon$-DPO compared to DPO in `Mistral-Instruct` and `Llama-Instruct` settings to provide an intuitive comparison as below. We can find that $\\varepsilon$-DPO results in a negligible wall-time increment.\n\n| Wall-time Increment | Mistral-Instruct | Llama-Instruct |\n|-|:-:|:-:|\n| Step (Avg) | 0.00082s | 0.00064s |\n| Epoch | 0.38076s | 0.30016s |\n| $\\frac{\\Delta t}{t} (\\\\%) $ | 0.0064% | 0.0045% |\n\n**3. Performance Differences of $\\varepsilon$-DPO across Base Models [W3, Q2]**\n\n> **[W3]** The evaluation on specific downstream tasks in Appendix C shows that ϵ-DPO has a performance drop compared with DPO on math reasoning (GSM8K). The average scores reported in Table 6 do not show a consistent, decisive advantage for ϵ-DPO, suggesting its benefits are most pronounced in conversational ability rather than across all specialized tasks.\n\n> **[Q2]** Can you analyze why on the same dataset, e.g. Arena-Hard, Mistral and Llama will have very different results? (in Table 1), could you provide experiments on more models? such as Qwen?\n\nWe sincerely appreciate your careful comparisons on the performance of $\\varepsilon$-DPO. **We want to emphasize that the experimental settings in UltraFeedback focus on alignment as conversational agents, rather than specific downstream tasks at the dataset level.** We speculate that if $\\varepsilon$-DPO is adopted to target the mathematical reasoning dataset [2], it can sufficiently show results different from our experiments. **Furthermore,  trends on specific benchmarks are likely to be influenced by the characteristics of the base model, because different datasets are constructed by sampling responses from each model [3].** Due to the limited computation resources, we fix the best method-specific hyperparameters of DPO, SimPO, and $\\varepsilon$-DPO obtained in the `Llama-Instruct` setting and trained `Qwen2.5-7B-Instruct` with a hyperparameter search range of learning rate in [3e-7, 5e-7, 7e-7] and the same annotation process as suggested. The results below show that $\\varepsilon$-DPO achieves concrete performance as in the `Llama-Instruct` setting.\n\n| Method |AlpacaEval 2 (LC / WR) | Arena-Hard (WR) | MT-Bench (Score) |\n|-|:-:|:-:|:-:|\n| SFT | 27.8 / 27.9 | 51.8 | 8.6 |\n| DPO | 41.6 / **46.3** | 66.8 | 8.9 |\n| SimPO | 32.4 / 46.0 | 60.2 | 8.8 |\n| $\\varepsilon$-DPO | **42.5** / 46.1 | **67.5** | **9.1** |\n\n**4. Alternative for the Updating Rule of $\\beta$ [W4]**\n\n> **[W4]** $\\beta$ for the next training step is updated by taking the mean of the instance-level beta values used in the current batch. I think this could be replaced by a potentially more robust update scheme (e.g., an exponential moving average with a learnable parameter).\n\nWe sincerely appreciate your valuable suggestions for improving $\\varepsilon$-DPO. We also agree that EMA with a learnable parameter can be a powerful alternative. **However, it introduces a new hyperparameter, $m$, for the smoothing factor. Furthermore, we experimentally confirmed that letting $\\beta$ be learnable without a $\\beta$-specific learning rate resulted in almost no variation of $\\beta$, because of the relatively low learning rate of the direct alignment algorithm, resulting in another difficulty of training.** Still, EMA allows moderate $\\beta$ variation, so we experimented with the EMA updating of $\\beta$ using $m = 0.5$, in addition to the best hyperparameters obtained from `Mistral-Instruct` and `Llama-Instruct`. Unfortunately, we observed degeneration compared to the result of $\\varepsilon$-DPO, as shown below.\n\n| Setting | Method |AlpacaEval 2 (LC / WR) | Arena-Hard (WR) | MT-Bench (Score) |\n|-|-|:-:|:-:|:-:|\n| Mistral-Instruct | $\\varepsilon$-DPO (EMA) | 23.1 / 18.8 | 15.8 | 7.6 |\n| Mistral-Instruct | $\\varepsilon$-DPO| **35.6** / **29.6** | **17.2** | **7.8** |\n| Llama-Instruct | $\\varepsilon$-DPO (EMA) | 38.6 / 36.7 | 35.6 | **8.0** |\n| Llama-Instruct | $\\varepsilon$-DPO| **42.5** / **44.9** | **36.7** | **8.0** |\n\n\n**5. Clarification of the Notation and Experiments [Q1, Q4]**\n\n> **[Q1]** In Line 232, what is step size $\\mathbb{E}(\\epsilon_i)$? Where is $\\epsilon_i$ defined? How is this related to $\\epsilon_c$ and $\\epsilon_s$?\n\n> **[Q4]** In \"Variants of KL Penalty Control Strategy\", the authors claim that we should set \n. In the Table 3, it sets $\\epsilon_c = \\epsilon_s$. Can you set the fixed \n to other values? such as when $\\epsilon_c$ or $\\epsilon_s$ equals $0.005, 0.01, 0.02$, and varying another one. Or in general, you should enumerate all choices of $\\epsilon_c$, $\\epsilon_s$ in $0.005, 0.01, 0.02$ (or more) and say $\\epsilon_c = \\epsilon_s = 0.01$ is the best choice.\n\nWe sincerely appreciate your comments on the clarification about the notation. We found that the existing notation was insufficiently illustrative and potentially misleading. **$\\mathbb{E}[\\varepsilon_i]$ represents the average of occurrences of $\\beta^+$ and $\\beta^-$ by regarding each as +1 and -1, respectively, and is intended to illustrate the overall training dynamics according to the choice of $\\varepsilon_c$ and $\\varepsilon_s$.** We will reflect the corrections and explanations of the notation in the final version. Furthermore, we also experimented to compare all choices in $\\varepsilon=[0.005, 0.01, 0.02]$ to provide better ablation, as you suggested. In this process, we found an error in the evaluation at the case of $\\varepsilon_c=0.02, \\varepsilon_s=0.01$ in Table 3, and the updated results are as follows. **Nevertheless, it results in the same conclusion of analysis in the manuscript, and we can confirm that $\\varepsilon_c=\\varepsilon_s=0.01$ yields the best result.**\n\n| $\\varepsilon_c$ / $\\varepsilon_s$ (WR / Occurance)| 0.005 | 0.01 | 0.02 |\n|-|:-:|:-:|:-:|\n| 0.005 | 76.4 / 0.072 | 76.7 / 0.073 | 76.4 / 0.074 |\n| 0.01 | 78.4 / 0.244 | **79.2** / 0.245 | 77.4 / 0.240 |\n| 0.02 | 74.9 /0.337 | 74.2 / 0.353 | 74.6 / 0.337 |\n_____\n[1] Scaling Laws for Neural Language Models (Kaplan et al., 2020)  \n[2] Iterative Reasoning Preference Optimization (Pang et al., 2024)  \n[3] SimPO: Simple Preference Optimization with a Reference-Free Reward (Meng et al., 2024)"}, {"review_id": "tk6SLX4Sb9", "reviewer": "Reviewer_4tbE", "summary": "This paper proposes a variant of DPO, termed $\\varepsilon$-DPO, which introduces an adaptive mechanism to control the KL penalty parameter $\\beta$. By perturbing the KL term at the instance level, $\\varepsilon$-DPO dynamically balances the trade-off between staying close to the reference model and aligning with preference data. Experimental results demonstrate that $\\varepsilon$-DPO outperforms existing baselines, achieving improved alignment and generalization in preference optimization tasks.", "questions": "1. Do equations (1), (2), and (5) imply that the choice of $\\beta_ \\varepsilon^+$ and $\\beta_ \\varepsilon^-$ are determined by the sign of  $z_\\theta(x,y^w,y^l)$? \n2. For Equation (4), I understand that $\\pi_{\\theta(\\beta_ \\varepsilon^+)}$ can be interpreted as the softmax over a convex combination of the logits $f_\\theta$ and $f_{\\textnormal{ref}}$. However, for Equation (3), the interpretation is less clear. Could the authors provide an intuitive explanation of this expression?", "limitations": "The authors have discussed the limitations in conclusion and I raise my score for acceptance.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n\nThe paper introduces a principled and effective way to adaptively control the KL penalty parameter $\\beta$ in DPO through instance-level perturbations. This addresses a key limitation of standard DPO, which relies on a fixed penalty term, and allows the model to better accommodate diverse preference samples.\n\nWeaknesses:\n1. While the introduction of an adaptive KL penalty in DPO is a useful refinement, the core idea, which modulates the regularization strength based on instance-level signals, may be seen as a relatively incremental extension of existing approaches. The paper would benefit from a stronger emphasis on the broader implications or theoretical insights enabled by this adaptation.\n2. The theoretical development builds heavily on existing results and offers limited additional insight into the underlying mechanisms that make the proposed method effective. A more principled analysis of why and when the approach works would strengthen the contribution. (See details in the question part)", "was_revised": true, "final_justification": "The response has addressed all my concerns, and I raised my score for acceptance.", "rebuttal": "**1. Clarification of Theoretical Insights Offered from $\\varepsilon$-DPO [W1]**\n\n> **[W1]** While the introduction of an adaptive KL penalty in DPO is a useful refinement, the core idea, which modulates the regularization strength based on instance-level signals, may be seen as a relatively incremental extension of existing approaches. The paper would benefit from a stronger emphasis on the broader implications or theoretical insights enabled by this adaptation.\n\nWe sincerely appreciate your feedback on how we can enhance the messages enabled by our work. Proposition 1, which we use as a key property in $\\varepsilon$-DPO, is motivated by the observation of [1], which is influenced by [2]. **Still, both studies aim to approximate policies that have not been trained at sampling time, but our work offers a novel theoretical insight in the different aspect; this sampling-time approximation approach can be effectively utilized at training-time for estimating an appropriate KL penalty $\\beta$, by approaching it from the perspective of preference model reparameterization of the DPO.** While [1] aims to control the KL penalty coefficient $\\beta$ used in DPO at test-time and [2] estimates the importance ratio for instruction-following of a large-scale model through an implicit reward defined between a small-scale instruction-tuned model and a pretrained model, both approaches could not reach the theoretical insights that the current policy model, which has not completely ended the training process can still perfomrs approximation of other policy for controlling regularization during training process. **Simultaneously, it achieves instance-level adaptive KL penalty control, which is not possible with existing approaches like $\\beta$-DPO, which determines the KL penalty control criterion through empirical observation, or TR-DPO, which approaches it based on the property in the loss landscape.** We will reflect that this new theoretical insight is more clearly reflected in the final version, as suggested.\n\n**2. Detailed Interpretation of Theoretical Insights of $\\varepsilon$-DPO [W2, Q1, Q2]**\n\n> **[W2]** The theoretical development builds heavily on existing results and offers limited additional insight into the underlying mechanisms that make the proposed method effective. A more principled analysis of why and when the approach works would strengthen the contribution. (See details in the question part)\n\n> **[Q1]** Do equations (1), (2), and (5) imply that the choice of $\\beta_{\\varepsilon}^{+}$ and $\\beta_{\\varepsilon}^{-}$ are determined by the sign of $z_\\theta(x, y^w, y^l)$?\n\n> **[Q2]** For Equation (4), I understand that $\\pi_{\\theta(\\beta_{\\varepsilon}^{+})}$\n can be interpreted as the softmax over a convex combination of the logits $f_\\theta$\n and $f_\\text{ref}$. However, for Equation (3), the interpretation is less clear. Could the authors provide an intuitive explanation of this expression?\n\nWe sincerely appreciate your effort to find the theoretical insights of $\\varepsilon$-DPO. First, we can intuitively understand checking (1) and (2) for a given triplet $(x, y^w, y^l)$ in the two perspective: i) the direction of the training-time inverse temperature $\\beta$ that induces better separability of preference model defined by the optimal policy $\\pi_{\\theta(\\beta)}$, and ii) the sign of the numerical derivative with respect to $\\beta$ that maximizes the intractable logit function $z(x, y^w, y^l; \\theta(\\beta))$, as described in Line 129 - 135. **However, (1) and (2) cannot be verified simply through the sign of the logit from the current policy $z_\\theta(x, y^w, y^l)$, but through the access to the optimal policy $\\pi_{\\theta(\\beta)}$ for all possible $\\beta$, regardless of the interpretations of i) and ii).** In other words, (1) and (2) can be verified if we have all the policies that have already been trained for all $\\beta$. In this context, we utilize Proposition 1 as (3) and (4) to approximately verify that (1) and (2) are satisfied even though we do not have policies that have been trained for all possible $\\beta$. By comparing the magnitude of the logit of the optimal policy estimated under $\\beta_{\\varepsilon}^{+}$ and $\\beta_{\\varepsilon}^{-}$ near the current $\\beta$ according to Proposition 1 with the logit estimated at the current $\\beta$, $\\beta_{\\varepsilon}^{+}$ and $\\beta_{\\varepsilon}^{-}$ are determined. **That is, $\\beta_{\\varepsilon}^{+}$ and $\\beta_{\\varepsilon}^{-}$ are not determined by the sign of the logit from current policy  $z_\\theta(x, y^w, y^l)$, but through the comparison between estimated logits of the policies under perturbation of $\\beta$**. Meanwhile, if we interpret (3) and (4) in a context of 'convex combination' as you mentioned for (3), they can be understood as obtaining points inside the convex hull (i.e., (3)) or outside the convex hull (i.e., (4)) on the affine hull defined in logit space. **If we consider points inside the convex hull as a kind of 'interpolation' in the logit space, then, unlike (3), (4) can be understood as an 'extrapolation' in the logit space.** However, since Proposition 1 is derived from the following relationship of optimal policies for importance sampling, we believe that interpreting (3) and (4) as perturbations of the modulation parameter $\\lambda$ for the importance ratio is somewhat more in line with the original theoretical intention.\n\n$ \\pi_{\\theta(\\frac{\\beta}{\\lambda})}(y_i|x, y_{1:i-1}) \\propto \\pi_{\\text{ref}}(y_i|x, y_{1:i-1}) [\\frac{\\pi_{\\theta(\\beta)}(y_i|x, y_{1:i-1})}{\\pi_{\\text{ref}}(y_i|x, y_{1:i-1})}]^\\lambda $\n\n_____\n\n[1] Decoding-time Realignment of Language Models (Liu et al., 2024)  \n[2] An Emulator for Fine-Tuning Large Language Models using Small Language Models (Mitchell et al., 2023)"}]

Author rebuttal:

Meta-review: This paper proposes ϵ-DPO, an instance-level adaptive extension of Direct Preference Optimization (DPO) for aligning large language models with human preferences. Instead of a fixed KL penalty, ϵ-DPO dynamically adjusts the KL term per preference pair by checking the monotonicity of log-likelihood ratios under small β perturbations. This approach balances adherence to the reference policy with alignment to preference data.

Strengths:
- Principled instance-level KL adaptation addressing a key limitation of DPO.
- Strong empirical performance on diverse conversational benchmarks.
- Theoretical justification based on logit monotonicity and Bradley-Terry preference modeling.
- Clear writing and well-situated in prior literature.

After rebuttal, reviewers generally agreed that concerns were addressed: computational feasibility is clarified, approximation errors are mitigated, and the practical benefits on conversational benchmarks are evident. As such, the paper merits acceptance.
Decision: Accept (poster)