$$\texttt{C2-DPO}$$: Constrained Controlled Direct Preference Optimization
Authors: Kavosh Asadi
Xingzi Xu
Julien Han
Ege Beyazit
Idan Pipano
Dominique Perrault-Joncas
Shoham Sabach
Mohammad Ghavamzadeh
Karim Bouyarmane
Venue: Accepted by TMLR ()

Abstract: Direct preference optimization (\texttt{DPO}) has emerged as a promising approach for solving the alignment problem in AI. In this paper, we make two counter-intuitive observations about \texttt{DPO}. First, we show that the \texttt{DPO} loss could be derived by starting from an alternative optimization problem that only defines the KL guardrail on in-sample responses, unlike the original RLHF problem where guardrails are defined on the entire distribution. Second, we prove a surprising property of this alternative optimization problem, where both the preferred and rejected responses tend to decrease in probability under its optimal policy, a phenomenon typically displayed by DPO in practice. To control this behavior, we propose a set of constraints designed to limit the displacement of probability mass between the preferred and rejected responses in the reference and target policies. The resulting algorithm, which we call Constrained Controlled DPO (\texttt{C2-DPO}), has a meaningful RLHF interpretation. By hedging against the displacement, \texttt{C2-DPO} provides practical improvements over vanilla \texttt{DPO} when aligning several language models using standard preference datasets.

Published in Transactions on Machine Learning Research (01/2026)
C2-DPO: Constrained Controlled Direct Preference Optimiza- tion
Kavosh Asadi
kavosadi@omazon.com
Amozon
Xingzi Xu
zingzinu@omazon.coo
Amazon
Julien Han
hameng@amazzno
Amazon
Ege Beyazit
beyazit@amazon, com
Amazon
Idan Pipano
idan. pipanoo haappaooooaamppsssseeiiioooooo
Technioo
Dominique Perrault-Joncas
joncas@amazon.comm
Amazon
Shoham Sabach
ssabach com
Amazon
Mohammad Ghavamzadeh
ghavemac@amazon.com com
Amazon
Karim Bouyarmane
bongkari@@mazzon.com
Amazon
Reviewed on OpenReview:
https: //
// openreview. net/ forum? id= 755.o9
Abstract
Direct preference optimization (DPO) has emerged a promising approach for solving the alignment problem in AI. In this paper, make two counter-intuitive observations about DPO. First, we show that the DPO loss could be derived by starting from alternative optimization problem that only defines the KL guardrail on in-sample responses, unlike the original RLHF problem where guardrails are defined on the entire distribution. Second, we prove a surprising property of this alternative optimization problem, where both the preferred and rejected responses tend decrease in probability under its optimal policy, phenomenon typically displayed by DPO in practice. To control this behavior, we propose set of constraints designed to limit the displacement of probability mass between the preferred and rejected responses in the reference and target policies. The resulting algorithm, which we call Constrained Controlled DPO (C2-DPO), has meaningful RLHF interpretation. By hedging against the displacement, C2-DPO provides practical improvements over vanilla. DPO when aligning several language models using standard preference datasets.
1 Introduction
Ensuring AI systems act in accordance with human preferences, also known as the alignment problem, has become critical focus in machine learning. Reinforcement Learning from Human Feedback (RLHF) has emerged as one promising approach [Christiano et al.) 2017). RLHF proceeds by first learning a reward model (RM), and then employing standard RL algorithms to maximize the RM while keeping the model close a reference model. Recent years have witnessed the emergence of algorithms that solve the two RL.F sub-problems in single step, chief among them being the Direct Preference Optimization (DPO) algorithm RRfaillvv al. 2023 DPO proceeds by leveraging the closed-form solution of the RLHF objective
* *Corresponding author.

---

Published in Transactions on Machine Learning Research (01/2026)
and using the preference dataset to align the model, thus bypassing the explicit reward-learning and the need to sample new responses during training. Since then, numerous extensions and successors have been proposed, e.g., IPO |Azrr et al. 2024) and CDPO (Mitchell| 2024), underscoring the need for a deeper investigation into this emerging class of algorithms to connect the underlying principles.
In this paper; we start with the counter-intuitive observation that the DPO loss can be derived from an alternative optimization problem that imposes the KL penalty only on in-sample responses those present in the preference dataset rather than the full output distribution, done in traditional RLHF. We show that this subtle shift has significant implication: the alternative optimization problem incentivizes in-sample probability reduction in DPO. We formally prove that under the optimal solution to this new problem, both the preferred and rejected responses tend to decrease in probability. This phenomenon, while counter- inttter-ittuiiive mirrors recent findings about DPO behavior (e.g.. Adler et al. 2024 Xu et al. 2024 Pal
ct all.2. 2024 Fisch et al. (024) Xiao et al. 2024 Shen et al. 2024 Wu et al. 2024 D'Oosterlicck et al. 2024 Pang
et al. 2024). and is referred as likelihood displacement by Razin et
2025). We then show that the above
in-sample probability reduction phenomenon is shared among DPO exlensions/succesee
by developing
simple classification framework that unifies the family of DPO-style algorithms.
Leaning on these insights, we propose family of constraints that provably control likelihood displacement in DPO-style algorithms. The constraints are designed to limit the movement of winmer-loser probability mass between the reference and target policies. Our proposed algorithm, Constrained Controlled DPO (C2-DPO), optimizes the DPO objective under these constraints, has a meaningful RLHF interpretation, and requires no extra computation. We evaluate the effectiveness of our constraints in enhancing preference alignment across two datasets and three models with up to 13B parameters, and show that C2-DPO outperforms vanilla DPO and several other baselines, delivering higher-quality final models when assessed holistically the standard -rT-Bench dataset (Zheng et al. 2023
2 Preliminaries
We present the key ingredients of preference optimization on which we will build in the subsequent sections. In this setting, we are given a dataset D of triplets (I, yw: Iw: M1), where x is a prompt, while yo and y reflect our preference choosing response yo over y conditioned on æ. We are also given a reference policy Fye (often the SFT checkpoint TSFT) which serves as a guardrail.
In RLHF, first employ DD to train parameterized RM, ré; and then use it to solve the following:
max
Eg E [ro(z, v)] 3KLL (T0(()=)]]lmm(((
(1)
where 3 > 0 is a low deviation from Tref. The RM is learned by minimizing the cross-entropy (CE) loss:
denoting the relative importance of reward maximization against ensuring
min Σ - log a (ro(s, - (a((,,)))
(2)
(DD
assuming that preferences follow the Bradley-Terry (BT) model: p(u > y1 I x) o(((z, vw) r(z, vi) where (I) = 1/(1 (1+xx((-x))) is the sigmoid function and r is the latent reward of the annotator. Fine-tuning in the RLHF approach splits into two stages: reward-learning using the BT model, followed by a policy optimization using equation I More recently, an emerging family of algo solve the above two problems in a single stage: Direct Preference Optimization (DPO)-style algorithms. The loss function of DPO is derived from the RLHF problem equation | using the recipe from (Rafailov et al.) 2023). The key insight here is that problem equation I admits the following closed-form solution: n² (1/2z) Teet exp (r(z,)//3) (z)))))))((( with Zzz) the partition function. We can rewrite this as
((x) [rer(y(x))
r(x, y) = 3 log
(3)
Substituting r(z. y) from equation 3 into equation 2 the partition function (xx) cancels out, leading to the optimization problem solved by DPO:
2

---

Published in Transactions on Machine Learning Research (01/2026)
Toe (y xx)
B log 7Te (311 Teee( (x)
min 0
Σ - logo (3log
(4)
Te yvo (x)
3 KL in DPO: Implicit Guardrailing with a Counter-intuitive Side Effect
Consider the optimization problem formed by replacing the KI-penalty in equation I with one that only operates on in-sample responses S = yu: Vi (z, you y1) D:
max Es Eywno [ro(z, v]] B Σ To To log To(y|2)) Teet (y/la)))))))
(5)
yES,
Comparing to equation recall that (L( [+))(+1))) == Ey To (o( log (.(()))) sums over all y whereas equation 5 sums only over Sg. At first glance, the two optimization problems look quite similar,
ne(ulz)
but finding the closed-form solution of equation requires delicate analysis since the penalty only sums over Sz, leading to more involved KKT optimality conditions. Via this KKT analysis (details in Appendix problem equation [ admits closed-form solution:
Remark 3.1 (Explicit form of the solution). The optimal policy for equation lakes one of two forms depending on whether out-of-sample responses have higher rewards than in-sample ones. Defining Pa. maxy Es, To)
Pglm.y) Case I: If > Boog Eves, Tredyla)e 8-1 then
<)(1 a (o (y(()) =0
y E Sz:
y $ Sz. ( 9) < P
(6)
E E Th (1((x) =1 Eves, Trr(W//)eerotee0---1111-
y $ Sz r T (I,z) f..
Case 11: Otherwise,
Nnc(ulz)e 7ols
(b((a) Σ Ex'e, [nn(V(1a)""oteurm E Sz; y $ S..
(7)
TZ (5(l)) = 0
Despite this complex form, both cases satisfy the key property that for any v.v' E Sa.
ne (v(x)) (y)r)
To To(v(xx)
() (=, 3/) 3 log
8 log
(8)
Fer xx)
We further prove that substituting the explicit solutions into the Bradley- -eery model recovers the standard DPO loss equation 4
Lemma 3.1. Problem equation yields the same DPO loss as problem equation
At first glance, problems equation and equation 5 appear similar, as both maximize expected reward with a KL-based penalty. However, Lemma 3.1 reveals that DPO can equivalently be derived from the latter, where regularization applies only to the small subset Sg rather than the full response space. This has important implications for understanding DPO's behavior. We note that the introduced penalty equation proper divergence, in the sense that it could be negative. Thus, the Lemma 3.1 could be viewed evidence that when move from RLHF to DPO, no longer explicitly enforce the KL penalty. However, as using larger values of results in smaller KL deviations during DPO training (sce Figure [eft), we can still empirically use KL regularization as an effective guardrail. This arguably makes sense, because in DPO there is little incentive for the model to shift probability mass for responses that are quite different from in-sample responses, and explicit out-of-sample guardrailing may not be necessary.
not a
While this minimal guardrailing effective, leads to counter-intuitive behavior. The penalty term in equa- tion 5 be negative, unlike the standard KT. divergence, which is always non-negative. More importantly,
3

---

Published in Transactions on Machine Learning Research (01/2026)
10a
Vr.ye.VilED Bedlaaki +og modekk
-10
-10
10a
[ (rrr))
10a
log Radyalkl
-10
B=1
B=1
10a
B 10
-10
B
log
8 8= 8=10-
8 =
10-1
8 10
-107
==
10
10
Epoch
Epoch
Figure 1: Left: Increasing 3 in DPO leads to effective out-of-sample guardrailing. We ran DPO with different values of B starting from the Zephyr-7B initial checkpoint on the UltraFeedback dataset. We then estimated the KL divergence between Te and Tee by autoregressively sampling N = 32 responses from To for each prompt in the test set, followed by computing * > log Ka(s.la) and averaging over prompts. Right: Reduction of in-sample probabilities in DPO training.
the overall objective in equation 5 can increase by simultaneously decreasing the probabilities of both winner and loser responses (making log (y x)/Trel(y æ) < 0 for all Sz). In contrast, this behavior is not incentivized in the standard RLHF problem equation !] There, the KL penalty is non-negative by defini- tion, so reducing in-sample probabilitics would require increasing out-of-sample probabilities correspondingly, providing no net benefit. We now formalize this observation:
Lemma 3.2. Consider pairs (I, y) with E Sx- If rolo, v) s max ro(x, then any oplimal sobution to equation satisfies To(y|æ)) S e m))(()z)))
The proof (Appendix A follows from the subset-constrained penalty structure in equation 5 This result does not hold for standard RLHF equation II where the full KL penalty prevents this counter-intuitiv sys- tematic probability reduction behavior. The lemma shows that in-sample responses whether preferred or rejected rejected-experienee experience probability roduction whenever their rewards are dominated by out-of-sample rewards. Since [Sa] is typically very small relative to the full response space (in the extreme case, single preferred and rejected response), this condition is likely to hold frequently, explaining the widespread likelihood dis- placement observed in practice.
Recent work has observed that during DPO training, all in-sample probabilities including those of preferred
responses tend to decrease (c.g., Adler et al.
2024 Xu et al. 2024 Pal et al. 2024 Fisch et al. 2024
Xiao
lt
2024
Liu et al. 2024 Yin et al. 2024
Guo
al. 2024
Yuzi et al. 2025 Razin et al. 2025
Deng
et
2025
Xiliang al. 2025 Huang al.
2025 Yunan al. 2025 Pang al. 2024). Figurel Riihht)
confirms this trend: the sum of log-ratios log Toly
[..r(y.]))) + log [ea((g(]]] decreases substantially during training.
Lemma 3.2 hints at. the underlying reason for this counter-intuitive behavior, which has been referred to
as liketihood displacement of in-sample probabilities (Razin et al. 2025). To the best of our knowledge, while this has boen reported in previous empirical studies, Lemma 3.2 is among very few theoretical results explaining this counter-intuilive phenomenon.
We conclude this section by noting that we proved in-sample probability reduction for DPO alone, and it would be natural to ask if some of its main successors, e.g., IPO (Azar et al., 2024 and CDPO (Mitchell, 2024), share the same property. In the next section, answer this question affirmatively by developing classification framework that unifies the family of DPO-style algorithms.
4 A Classification View of DPO-style Algorithms
We now show that DPO-style algorithms can be interpreted as classification mothods, where the objective is defined solely in-sample responses. As a result, similar to DPO, of these algorithms applies any
4

---

Published in Transactions on Machine Learning Research (01/2026)
direct guardrailing out-of-sample responses. Recall that the standard classification setting has three main ingredients.
First, we construct a hypothesis space by defining probabilities assigned to each class. In DPO-style algo- rithms, these probabilities are implicitly defined as:
po yw 9t) == softmax (ro(z, yw), ro(2, (1),
(9)
where re is the reward defined in equation 3
having substituted x² with Tp- Under po, the probability assigned to the winner (preferred) response Yws denoted by py does not depend on the partition function (ar) and can be written as:
Tte [T lar) 8
10 Pe (I, yw: 11)
(10)
sta (Mw[2z) (ecr((brlz))
/
+
s 2) [x]])
The probability assigned to the loser (rejected) response Mi, denoted by phe can be defined similarly. Note that the distribution pe in equation 9 can be thought of as a generalization of the conditional probability of a response given that E 3t).
Second, the standard classification setting, the dataset gives us access to labels, which we use to extract target probabilities associated with each class. To obtain these target probabilities, we simply use any distribution p (p",p') from the simplex A2, which is defined as the set of all vectors p 122 satisfying p" ,p' 2 0 and pd + p' = 1. In the most basic case, we just use the one-hot vector (p"",p") (+1,0) akin to using hard labels. More generally, we can use soft labels, meaning that we put some non-zero weight behind each class (Mülerr et al. 2019).
Third, we define a classification loss L between two distributions pe and p, leading us to the optimization problem: ming En C(pe. p). good example the CF loss.
We can now show that a large number of DPO-style algorithms can be viewed as specific instances of this classification framework. The generality arises from the ability to use (a) hard or soft labels for the target distribution p and (b) different classification losses L.
Remark 4.1 (DPO). Suppose that we use the CE loss and hard labels == (10 = (+1, 0) in the above
classification framework. Then, using equation 10, we can write
To To(yult)) B log Tee To /xx) Trer(Un
E(po, p) = (2"" (1" lo + logr') =-
=- log Pa log a Blog
Tren(
which is exactly the DPO loss equation if it is summed over D.
Another popular DPO-style algorithm is IPO (Azar et al. 2024). While the derivation of IPO in the original paper looks completely different than DPO, we now show that IPO can also be viewed as a specific instance of our classification framework (see Appendix B.1 for the detailed derivation).
Remark 4.2 (IPO). We can recover IPO (Fq. 17 in Azar al. 2024) using the loss L(po: p) = ( (log(rö /rb) log("" /p'))" and soft labels p (p" (0(1/
Table 1 shows that several DPO-style algorithms can be formulated using this framework. Given this framework, we can formulate the set of optimal solutions (those that achieve 0 loss) for any DPO-style algorithm. An optimal parameter 0 is one that achioves 0 loss for all samples in D, i.c., po( Bw W)) = p. V(x, Bw m1) E D. Setting P% (ar, Uw w) equation 10 equal to p = - we obtain
Trer(ywlar)
Tg- (ymoax)) 7 na- (21(x), with 1 :== (11 e)/ē. Tref (3)
(11)
Note that the derivation using ii yiclds the same result. Thus, we have two probabilities, ne. (awalar) and Te- (Mllz), that we aim to learn, but minimizing the loss only gives us one constraint specified in equation I1 This means that the original classification problem (loss- mosimiiiiiiaio in DPO-style algorithms) is under- specified. In Figure [ we provide an illustration of this phenomenon. We also provide a concrete example to further highlight that winner-loser probabilities can move in arbitrary directions, and most notably, can both go to zero.
5

---

Published in Transactions on Machine Learning Research (01/2026)
Data
Algorithm
Labels
Loss
Table 1: Unifying DPO-style algorithms. By (PL,, we mean DPO with Plackett-Luce model. See RPO (Adler et al.) 2024) and
(BT)
Hard
CE
Pairs
CDPO
Soft
CE
proofs for
IPO
Soft
Remark 4.2
List (Appendix.)
B
(PO
Hard
CE
Distilled DPO (Fisch et al. 2024 in Appendix B
Auxiliary Info
RPO
Soft
Appendix
(Appendix.
B
Distilled DPO
Soft
Appendix
Remark 4.3 (A Concrete Example). Suppose we have E = 1/11 and for simplicity we set B = (re((Wllz)) 0.02, and [z) 0.01, which result in 1) 20. (See Eqr 11 for the definition of n) We identify two pairs of probabilities that satisfy equation 17
1. (To- (Uull:)) no- no- (wilzz)) = (0.4, 0.02)
2. (To- (Uwlz), ne- (uul=)) = (0.001, 0.0002)
In the first case both probabilities (including the loser) increase under πT- relative to Tref- In sharp controst, both probabilities (including the winner) decrease under Tg- relative to Tref- Note that the increase is bounded as the two probabilities can go up until they hit Te- (ywlx)) + TT. (ın(z) = 1))= (yilx) = 1. Perhaps more concerning is the observation that the two probabilities con decrease arbitrarily and even collapse to O while stall maintaining Ppp =.
List of Preferences Our classification framework can can extend to work with lists, rather than pairs, of preferences. In particular, assume that we have N responses for each prompt x, giving us dataset of the form D = ((x, yı, 12, ..,...... (yN)) In this case, we can define list version of the probability vector pa(z, .1..... yN) similarly to equation together with target distribution p. With this simple extension now incorporate cxisting DPO-style algorithms that work with lists. For instance, DPO with the Plackett-Luce model for preferences Rafailov et al. (2023 can be captured in our classification framework, again using hard labels and the CE loss. See Appendix B.3 for a proof.
show that
Auxiliary Information A second important extension pertains the definition of soft labels in our classification setting. So far we have only worked with soft labels that are fixed across the entire dataset, for instance, p := (p", p') about any extra information we may have access to some auxiliary scores, Sw, St, (e.g., ratings) associated with each response, which can then be used to enrich our soft labels. More formally, suppose now that our dataset comprised of D = (x, yor Sw: Mt st). To obtain the soft labels we can employ, for instance, p softmax(s s1). Combining this with the IPO loss, we recover Distilled DPO (Eq. in Fisch al. 2024). Using the same soft labels, but with the CE loss recovers RPO (see Sec. 3.3.2 in Adler el. al. 2024). We provide more details both algorithms in Appendix [.4 These natural extensions further demonstrate that our classification framework is fairly general as well sufficiently flexible to capture large number of existing DPO-style algorithms.
o((1/),,o-11/2)) for all (z, yus 31) in IPO. These fixed labels are agnostic may have about our data triplets (z, Bw. Bw) However, in some applications
5 Constrained Controlled DPO (C2-DPO)
We now present general family of constraints to control the likelihood displacement of in-sample probabili- ties in DPO-style algorithms described in Scction 3 These constraints can also help with the under-specified nature of these algorithms discussed in Section 4 We define the constraints on the probability mass of the winner-loser pair and use them to control how much this mass changes from the reference policy Tyel to the target policy Tg. We then show how these constraints can be incorporated into any DPO-style loss function and propose our algorithm, which we refer the Constrained Controlled Direct Preference Optimization (C2-DPO).

---

Published in Transactions on Machine Learning Research (01/2026)
Σ yi Dur X))
m x)
An illustration of the set of solutions that achieve loss in DPO-style algorithmss The shaded gray is the set of fcasible solutions. The red line pass- ing through the feasible indicates the set of optimal solutions. Note that the case where the probability belonging to in-sample responses displace entirely to out-of-sample responses, i.c,, pz( Vw V1) = (0+,0+) also lies on this line.
Figure 2:
1g xx)
F x)
mdhi x) 7 x
The constraint, with respect to an arbitrary function & PR + R, takes the general form
((m)(()))) + (TT(((l)) = (v (z)) + up (TTrar(()(()))
(12)
Note that the RHS fixed during training, so the two terms on the LHS cannot move in the same direction. We now generalize this intuition by showing that when the constraint function yp is monotonic and added the solution characterization of DPO-style algorithms given by
Tg (yno (r)
To- (uilx) x))
(13)
Tref (e x))
then we can control the direction of the movement of probability mass for all winner-loser pairs.
Proposition 5.1. Let & IR IR be a monotonic function and assume that equation 12 holds and that the target labels satisfy (p" p') (1 E,E) for some < 1/2. Then, ag (ywlz) Tyef(y and Tg. (yt (r) Tref(yn (z:).
Proposition 5.1 whose proof is reported in Appendix C.1 shows that any monotonic constraint function y guarantees that the learned policy, Te,, assigns a higher (lower) probability to the winner (loser) response than the one assigned to it by the reference policy Tref- It is natural to ask what &p should be used in the context of this constraint, and we present two interesting candidates below.
5.1 Logarithmic Constraint <p(z) := log x
Our first choice is to employ the logarithmic constraint:
log (TT(H=|=)) + log (T)())[=) log (Tre((Uwll)) + log (mver(Hr(z))
(14)
which is nice to work with empirically in light of the fact that all terms are in the log-space. Morcover, these log probabilities are already computed in DPO, which makes the implementation the corresponding C2-DPD algorithm more efficient.
Rather than using hard constraints, add the deviation from the constraint soft penalty to the original DPO-style loss with regularization parameter A. We consider both & and &a penalties; for conciseness, we present the & version here. Equipping the DPO loss equation 4 with the logarithmic constraint equation 14] we obtain the following loss for C2-DPO:
min Σ - logo (glog (Bw B log (b((2) Trer(() (xr) + A llo
To (v (z) Tree (x:) Tret (3)
+ log To (a))
(15)
Teet(Ver (x)
The E version simply replaces the squared term with an absolute value. We evaluate both variants in Section 0 In contrast to the hard constraint, in this case we do not necessarily force the winner (loser) probability to go up (down). Rather, we impose a penalty when the learner violates the constraint. Notice also that we added the penalty term to the original loss of DPO for simplicity, but in principle, the penalty term can be added to any DPO-style loss covered in our classification framework.

---

Published in Transactions on Machine Learning Research (01/2026)
Further, can show that employing the logarithmic constraint equation 14 has meaningful RLHF in- terpretation. Recall that Rafailov et al. (2023) defined Po(ar, v) := Blg (y) (y)l/))))) Vy E y as an implicit reward learned during DPO training. Using this notation, we can rewrite objective equation 15 simply as
- llog ofog Bw) fa(z, u1)) + ga (fo(I, + fe(x,
Under ((z) = loga;, x, we solve the original RLHF problem akin to DPO, but we also incentivize the sum of the implicit rewards for the winner and loser to remain around zero. It follows that the constraint regularizes the implicit rewards so as to avoid rewards that are (a) very large and (b) have the same sign. These two properties cannot co-exist when employing ()x) = log æ, since doing so would yield a large magnitude inside the square and ultimately large magnitude in the second term of the loss. Intuitively, this can hedge against the likelihood displacement, because in the case of displacement both implicit rewards are large in magnitude and both have negative sign, which the constraint will greatly penalize.
5.2 Identity Constraint (0z)= x
A second interesting choice would be to simply the identity constraint:
TT(wll))) + no <o(u]<>) = Tre((yula)) + mes((ul
(16)
While equation 16 is also plausible constraint, at first glance it is unclear how to implement it since the constraint is no longer in the log-space and is specified in terms of raw probabilities. Working with raw probabilities is prone to numerical underflow issues; thus, we would like to derive constraint which is equivalent to equation and operates in the log-space. To do so, we make use of the following lemma whose proof is reported in Appendix C.2
Lemma 5.1. For any lwo numbers a and b, we have log(a + b) = log logo log b).
Applying log both sides of equation 16
we obtain log( (v) mo(u]>>) log (Te + )))((Ml
which can be rewritten using Lemma 5.1
To TOw lx))
log (mTc((/ll>)) - log (log Fret(yarl (z:)
log (v)) log
log
(17)
To( x))
Tred(
Moving from equation I6 to equation T7 we have rewritten the constraint entirely the log-space, thus
avoiding numerical issues, and similar to the logarithmic constraint in Section 5.1 allowing a straightforward
implementation of the corresponding C2-DPO algorithm. Equipping the DPO loss equation 4 with the identity constraint equation
17.
we obtain the following loss for C2-DPO using &g penalty:
To (Vmllt)) 33 log
To(yil (r)
min
Σ D
log
Blog Tre((Hwlx))
Tree (z)
+ A log (o(Wu(z)) + log log
Trer (Yw (2)
Te (yh (x) To [z)
log log
(18)
Trat Trar(y1(2)
An l, variant can be obtained by replacing the squared term with absolute value. We evaluate both penalties in our experiments in Section 0 Note that unlike p(x) logær, deriving an RLHF interpretation under ((xx) x is subtle. That said, interesting property under ((x) = is that the winner probability
increase only by an amount equal to (r)r))) This means that we will not put the entire probability mass on Ye-
6 Experiments
In this section, we present results across multiple evaluation settings: head-to-head win rates comparing model outputs (Figure 3), win rates against the preferred responses in the datasets (Figure 4 and holistic evaluation via. MT-Bench (Figure 7 to demonstrate consistent improvements across diverse metrics (Zheng et al. (023). Importantly, C2-DPO achieves these improvements with negligible additional computa- tional cost compared to DPO, constraints only operate in-sample responses.
8

---

Published in Transactions on Machine Learning Research (01/2026)
C20PO 1og -h Vs. DPO
C2DPO -Log-ty
C20P0-1-6 DPO
G2DPO-1-4 1-00-11-6 Vs
033
330
Figure 3: Head-to- head win-rate of C2-DPO against DPO, For each individual plot, we take the prompts from the held-out test set Ultrafeedback Binarized and generate responses from the C-3DPO and DPO model. Then we present the prompt and two responses to Anthropic Claude Sonnet and ask which of the two answers is more helpful and honest. We compute the average win rates across 10 different random seeds.
DPO
0.00
DPO
6
0.687
C2DPO-log-d:
C20PD
0.556
0.546
0.577
566
0.534
0.554
0.446
0540
Win Rates
0.571
0377
win-rates against the preferred response in the test set of the UltraFeedback dataset. Right: win rates on the TL;DR dataset.
Figure 4: Left:
0.399
0,422
COPOO -1 B
C2PO 1
0.25
0.50
0.75
1.00
Temperature
Reasening
C2-DPO-LogHLI L2
Reascring
C2-0P-LLggLi
L20
DPG
CaloPo
Mart
Malh
Rolepkay
SPPO
Coding
Willing
Coding
Willing
Extrccion
Humanites
3TEM
STEM
Figure 5: A comparison between C2-DPO and baselines. (Left) The two variants of C2-DPO-Log better align Zephyr-7b-SFT relative to vanilla DPO as well as the other baselines (Right).
6.1 Ultrafeedback Binarized
This datasct is comprised of 64k prompts, and winner (preferred) and loser (rejected) continuation per prompt (ye and 31). We used the standard DPO implementation published with the paper (Mitchel] 2023 Following Rasul et al. (2024), we train Zephyr-7B-SFT.
We present the head-to-head win rate of C2-DPO against DPO in Figure 3 by using 200 prompts from the held-out test set. We compute the win-rate by asking Anthropic's Claude Connet-3.5-v2 which response is more helpful. The exact prompt used for Claude in Appendix

---

Published in Transactions on Machine Learning Research (01/2026)
DV
DVG
DIO
DVD
C20PO -kg
C2DPO
C20RO 1-b
-10F0-1-4
CADPO
0.557
0.554
0554
0.546
0.546
6.s4
0.546
0,540
0.8
0.58
0.5
0.5
0.5
0.54
0.4
0.4
3,4
0.4
0.52
300
420
100
100
103
100
Stops
St.55
Stops
Stops
Figure 6: Win rates comparison of Zephyr-7B-SFT aligned on Ultrafeedback Binarized using DPO and C-2DPO The first 4 plots show win rates C-2DPO at individual checkpoints across different training trajectories. The last plot shows mean and standard error of win rates across all checkpoints and all inference runs.
Recall from Section that we proposed two candidates for the constraint function ş,, namely the logarithmic and identity functions. Moreover, notice from (18) that we measure the deviation from the constraint using an &g penalty. Alternatively, we can measure this deviation using an & penalty. Altogethor, we have four specific implementatiinss of C2-DPO: C2-DPO-Log-(1, C2-DPO-Log-(2, C2-DPO-T-(1, and C2-DPO-I- In Appendix
D we include pseudo-code for each of these 4 variations. In Figure 3 we show the head to head win rate of each of these 4 implementations against DPO, and in Figure (Left), show win-rate against the preferred response from the dataset. For all 4 implementations, we used the cyper-parameter À = 2 x 10 and did not tune for each the 4 implementations separately. From this result, it is clear that all variations improve upon DPO, with C2-DPO-Log being the highest performer. Figure shows win rates comparison between Zephyr-7B-SFT aligned with DPD and C-2DPD. We align Zephyr-7B-SFT following Rasul et al. (024)) using DPO, C-2DPO-Log-f1 C-2DPO-Log-(2, C-2DPO-I-L,, and C-2DPO-I-(2 for one epoch, hyper-paramcter X 2 x 10-4 With each checkpoint, we generate responses using test split of Ultrafeedback Binarized using typerparameters max tokens=1000, temperature-11.0, top p-0.9, Lop_k=0.. Different from the head to head setting, we ask Claude to compare the generated response directly with the preferred response in the dataset. The win rates and standard errors are calculated based on 10 different inference runs.
all C-2DPO
Further, holistically evaluate the final model, we used MT-Bench, multi-turn benchmark that uses GPT-4 to judge models' performance in different categories: Writing, Roleplay, Reasoning, Math, Coding, Extraction, STEM, and Humanities Rasul et al. (024). In Figure 5 we show the MT-Bench evaluation for C2-DPO against vanilla DPO, well as related DPO-styl algorithms that discuss the collapse of probabilities in DPO and aim to mitigate it, such as Cal-DPO (Xiao et al. 2024), SPPO (Wu et al. 2024). and DPOP (Pal et al. 2024). C2-DPO is the most competitive algorithm.
We then used C2-DPO-Log on a larger initial checkpoint, namely the Olmo-13B-SFT model from Al- lenAI OLMo al. (2024). We compared C2-DPO-Log against DPO. In Figure 7 we see that C2-DPO-Log outperforms DPO, indicating that C2-DPO improvement may scale to larger models.
Across all evaluation settings the UltraFeedback dataset, the four C2-DPO variants consistently outperform vanilla DPO. The logarithmic constraint variants (C2 (C2-DPO-Lo--4 and C2-DPO-Log-6>) achieve the strongest performances, with head-to-head win rates around 55% against DPO and improvements across all MT- Bench categories. Notably, all variants use the same hyperparameter without individual tuning, suggesting the method is robust to the choice of constraint function and penalty. The improvements on the larger Olmo-13B model indicate that the benefits of controlling likelihood displacement extend beyond the 7B scale.
10

---

Published in Transactions on Machine Learning Research (01/2026)
proferred responses
rojected responses
Wah
DPO
SFT
-80
C2DRO-ugee
log procabiity
-70
-100
UPO
-90
Codiig
Warr
-140
-160
Epoch
Epoch
STEM
Figure 7: Left: A comparison between C2-DPO and DPO in terms of the probability of winner and loser responses when training the GPT-J model with the Reddit TL;DR dataset. Right: Comparison between C2-DPD and DPO for aligning the 13B-STT model using the Ultrafeedbackk-Bnaaaa dataset.
6.2 Reddit. TL;DR
We then evaluate proposed method summarization task with human-assigned scores for pairs of summaries. For this purpose, employ the Reddit TL;DR dataset from Stiennon et al. (2022). We follow Amini et al. (2024) in creating the dataset and include pairs of summaries where one received a higher quality score than the other. During training and testing, we select the highest-scoring summary as Vho and a randomly selected alternative as yt-
Temperature-O.1.
Temperature-tt
Temperature-o0
Temperature -1.0
DFD C2DPO-log.i
0.8
0.587
ame
0.72 4577
Win Rates
Win Rates
0.023 A59
0.5
0.4
0.75 Epoch
0.25
1.00
Epoch
Epoch
Epoch
Temperature
Figure 8: Win rates against Ve comparison of GPT-J aligned on TL:DR using DPO and C-2DPO. The first 4 plots show win rates of C-2DPO at individual checkpoints across different training trajectories. The last plot shows mean and standard error of win rates across all checkpoints and all inference runs.
We align GPT-J Wang & Komatsuzaki (2021) with vanilla DPO well C2-DPO algorithms. Specifically, we first SFT epoch with the Reddit TL;DR dataset GPT-J, and then perform subsequent preference alignment. We evaluate the final checkpoints by computing win rates against Vo following Rafailov et al.) (2023)g Amini et al. (2024). We use Claude Sonnet 3.5 v2 as a judge, and provide prompt used for Claude in the Appendix F As shown in Figure 8 C2-DPO variants consistently oulperform vanilla DPO on this summarization task. The improvements demonstrate that controlling likelihood displacement beneficial beyond conversational tasks, extending specializod domains like summarization. Figure 7 (Left) further shows that C2-DPO successfully mitigates the probability reduction of both preferred and rejected responses during training, consistent with our theoretical predictions from Section
11

---

Published in Transactions on Machine Learning Research (01/2026)
7 Related Work
A few recent papers have proposed a unifying perspective on DPO-style algorithms. Notably, Tang et al. (2024) presented a generalization of DPO where different supervised learning losses are applied to the difference of implicit rewards (f(z. yw) - f(zz /3. In contrast, we make a clear connection between DPO-style algorithms and classífication, and also extend our results to lists and auxiliary information, as do similar efforts by (Su et al.
2025 Zhao et al. 2025 Im & Li, 2024 Yao et al.) 2025). A second notable example was to generalize from KL to any f-divergence when measuring the discrepancy between the target and reference model (Han et al. 2024). We also note that the first ingredient of our classification framework T was
used by Sharifnassab et al. [2024) to propose soft version of DPO.
Earlier work studied the decrease of likelihood during training (Feng et al. 2024 Xie et al. 2024), explained this phenomenon from different angles, and proposed different losses to address it Here we provide a brief overview of a number of these results, especially those that we experimentally compare against. Pal et al. (2024) proposed DPOP which addresses the phenomenon by adding the penalty term max(0, -f 3w within the log-sigmoid of the DPO loss equation DPOP can also be viewed as DPO with modified BT model.
In this sense, it has similarities with the DPD Shen et al. 2024 (see also (Choi et al. 2025 Shao et al. 2025
Xiao et al. (024) attributed the undesirable behavior to the contrastive loss of DPO not being scale-culibrated, i.e., ignoring the absolute values of implicit rewards fw) and f(z, y1). They address this by constraining the implicit rewards scale that matches the ground-truth reward r. Thus, in their proposed loss, Cal-DPO, they add the square-loss (f(r, v) - r(a,, v))², y E (y ) to the DPO loss (without 3). Of course, since they do not have access to the ground-truth reward, they replace it with 1/2 and 112 for the and yt, respectively. A loss similar to Cal-DPO was proposed by Wu et al. (024)) and they named it SPPO. It is simply Cal-DPO without the DPO loss. Finally, D'Oosterlinck et al. (2024) proposed APO, which offer fine-graine control over the implicit rewards.
While our work focuses on the RLHF setting, some underlying concepts, including pairwise Bradley-Teryy modeling and con-identifiabiiity of scores from pairwise comparisons, are well-established in the learning-
to-rank and preference learning literature (Newman 2023; Hamilton et al., 2025 Casper et al., 2023 Zhu ct al. 2024 Khorasani et al. 2025). A key distinction is that in RLHF, scores are not free paramcters but implicit rewards lied lo policy through r(z,y) = B (w)())) manipulating probability ratios directly affects the learned policy, unlike standard ranking formulations. Our contributions, including proving that DPO derives from a subset-constrained KL objective (Section 3 establishing the resulting likelihood displacement with an explicit e e-1 bound (Lemma 3.2), and proposing constraints that exploit the (To, Nyet) structure to control this displacement (Section 5) are specific to policy optimization with preference data in RLHE.
This coupling means that
8 Conclusion & Future Work
In this work, we revisited the derivation of DPO and revealed two counter-intuiivv findings. We first proved that the standard DPO loss can arise from a formulation that regularizes only in-sample responses. This, in turn, uncovered another surprising result: both the preferred and rejected responses are likely to experience a decrease in likelihood. This insight shed light on the thelihoodddisplleemmnn phenomenon observed in prior empirical studics. We then generalized this result through a unifying classification framework, highlighting broader absence of out-of-sample KL regularization. Building on these findings, we introduced the Con- strained Controlled DPO (C2-DPO), principled extension of DPO that controls probability displacement by constraining the redistribution of likelihood between preferred and rejected responses. C2-DPO retains the computational efficiency of standard DPO while still offering a clear RLHF interpretation.
We know that DPO displaces probabilities to unseen samples, but we do not have a clear picture in terms of the kinds of responses for which the probability increascs. While Fisch et al. (2024) and Razin et al. (2025) present some findings for special cases, it would be interesting to do a more systematic study of the kinds of unseen responses whose likelihood increases during training.
12

---

Published in Transactions on Machine Learning Research (01/2026)
References
Bo Adler, Niket Agarwal, Ashwath Aithal, Dong H Anh, Pallab Bhattacharya, Annika Brundyn, Jared Casper, Bryan Catanzaro, Sharon Clay, Jonathan Cohen, et al. Nemotron-4 340B technical report. arXiv preprint arXic:2406. 1 1774, 2024.
Afra Amini, Tim Vieira, and Ryan Cotterell. Direct preference oplimization with an offset, 2024. URL
Mohammad Gheshlaghi Azar, Zhaohan Daniel Guo, Bilal Piot, Remi Munos, Mark Rowland, Michal Valko, and Daniele Calandriello. A general theoretical paradigm to understand learning from human preferences. In International Conference on Artificial Intelligence and Statistics, pp.
PMLR, 2024.
Stephen Casper, Xander Davies, Claudia Shi, Thomas Krendl Gilbert, Jérémy Scheurer, Javier Rando, Rachel Freedman, Tomasz Korbak, David Lindner, Pedro Freire, Tony Wang, Samuel Marks, Charbel- Raphaël Segerie, Micah Carroll, Andi Peng, Phillip Christoffersen, Mehul Damani, Stewart Slocum, Usman Anwar, Anand Siththaranjan, Max Nadeau, Eric J. Michaud, Jacob Pfan, Dmitrii Krasheninnikov, Xin Chen, Lauro Langosco, Peter Hase, Erdem Bıyık, Anca Dragan, David Krueger, Dorsa Sadigh, and Dy- lan Hadficld-Menell. Open problems and fundamental limitations reinforcement learning from human feedback, 2023. URL htttps://arrii http://aaxvvv..aabbbaa
Eugene Choi, Arash Ahmadian, Matthieu Gcist, Oilvier Pictquin, and Mohammad Gheshlaghi Azar. Self- improving robust preference optimization, 2025.
Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforce- ment learning from human preferences. Advances in neural information processing systems, 30, 2017.
Xun Deng, Zhong Ha, Ai Rui, Feng Fuli, Wang Zheng, and He Xiangnan. Less is more: Improving 1lm aligument via preference data selection. arXiv preprint arXiv:2502. 14560, 2025.
Karel D'Oosterlinck, Winnic Xu, Chris Develder, Thomas Demeester, Amanprect Singh, Christopher Potts, Douwe Kiela, and Shikib Mehri. Anchored preference optimization and contrastive revisions: Addressing underspecification alignment. arXie preprint arXin:2406 08. (00:2408806666 2024.
Duanyu Feng, Bowen Qin, Chen Huang, Zheng Zhang, and Wenqiang Lei. Towards analyzing and under- standing the limitations of dpo: A thcoretical perspective. arXiv preprint arXin:2404.44620 2024.
Adam Fisch, Jacob Eisenstein, Vicky Zayats, Alekh Agarwal, Ahmad Beirami, Chirag Nagpal, Pete Shaw, and Jonathan Berant. Robust preference optimization through reward model distillation. arXiv preprint arXiv:2405. 198166 2024.
Yuxiang Guo, Yin Lu, Jiang Bo, and Zhang Jiaqi. TODO: Enhancing 1lm alignment with ternary preferences. arXiv preprint 1xXo4:241.00242, 2024.
Tan Hamilton, Nick Tawn, and David Firth. The many routes to the ubiquitous bradley-terry model, 2025. URL httppss//aarxvv.orr///////////// 13619
Jiaqi Han, Mingjian Jiang, Yuxuan Song, Jure Leskovec, Stefano Ermon, and Minkai Xu. f-po: Generalizing preference optimization with fach-fdol f-divergence minimization. arXiv preprint arXiv:2410221662, 2024.
Audrey Huang, Wenhao Zhan, Tengyang Xie, Jason D. Lee, Wen Sun, Akshay Krishnamurthy, and Dylan Foster. Correcting the mythos of kl-regularization: Direct alignment without overoptimization via chi- squared preference optimization, 2025.
Shawn Im and Yixuan Li. On the generalization of preference learning with dpo. arXiv preprint arXiv:2408.03//5 034559 2024.
Sadegh Khorasani, Saber Salehkaleybar, Negar Kiyavash, and Matthias Grossglauser. Fusing rewards and preferences in reinforcement learning, 2025. URL http:://arriv.or//aaa/22225
21506. 11363
13

---

Published in Transactions on Machine Learning Research (01/2026)
Zhihan Liu, Miao Lu, Shenao Zhang, Boyi Liu, Hongyi Guo, Yingxiang Yang, Jose Blanchet, and Zhaoran Wang. Provably mitigating overoptimization in rlhf: Your sft loss is implicitly an adversarial regularizer. In A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomczak, and C. Zhang (eds.), Advances in Neural Information Processing Systems, volume 37, pp. 138663-138611 Curran Associates, Inc., 2024.
Eric Mitchell. Direct preference optimization. github.com htgphu/oooooocctttttttt//////ton Accessed: 2024-1-01.
2023.
Eric Mitchell. A note on DPO with noisy preferences and relationship to IPO, 2024. URL https:// ericmitchell aatpaeeel.aa//ppo.oooo .df
Rafael Müller, Simon Kornblith, and Geoffrey E Hinton. When does label smoothing help? Advances in
neural information processing systems, 32, 2019.
M. E. J. Newman Efficient computation of rankings from pairwise comparisons, 2023. URL https://
arxiv. orw.orgg//a......o... htt///bbb///....22
Team OLMo, Pete Walsh, Luca Soldaini, Dirk Groeneveld, Kyle Lo, Shane Arora, Akshita Bhagia, Yuling Gu, Shengyi Huang, Matt Jordan, Nathan Lambert, Dustin Schwenk, Oyvind Tafjord, Taira Anderson, David Atkinson, Faeze Brahman, Christopher Clark, Pradeep Dasigi, Nouha Dziri, Michal Guerquin, Hamish Ivison, Pang Wei Koh, Jiacheng Liu, Saumya Malik, William Merrill, Lester James V. Miranda, Jacob Morrison, Tyler Murray, Crystal Nam, Valentina Pyatkin, Aman Rangapur, Michael Schmitz, Sam Skjonsberg, David Wadden, Christopher Wilhelm, Michael Wilson, Luke Zettlemoyer, Ali Farhadi, Noah A. Smith, and Hannaneh Hajishirzi. 2 olmo 2 furious, 2024. URL https: / /awxivaxvvvvv/////////aa///aaa hwwpg/abs2250111 00656
Arka Pal, Deep Karkhanis, Samuel Dooley, Manley Roberts, Siddartha Naidu, and Colin White, Smaug: Fixing failure modes of preference optimisation with DPO-positive. arXio preprint 24xx:2402... 13228, 2024.
Richard Yuanzhe Pang, Weizhe Yuan, He He, Kyunghyun Cho, Sainbayar Sukhbaatar, and Jason E We- ston. Iterative reasoning preference optimization. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. URL http:s:://ppperreeeeaeeeeeeeeeee wttp::/ww...ee.omrm/////aA1.11
Rafacl Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly reward model. In Advances in Newral Information Processing Systems, volume 36, pp. 13728-537448 2023.
Kashif Rasul, Edward Beeching, Lewis Tunstall, Leandro von Werra, and Omar Sanseviero. Preference tuning Ilms with direct preference optimization methods, 2024. URL pref-tuning
Noam Razin, Sadhika Malladi, Adithya Bhaskar, Danqi Chen, Sanjeev Arora, and Boris Hanin. Unintentional unalignment: Likelihood displacement in direct preference optimization. In International Conference on Learning Represenlations, 2025.
Ruichen Shao, Bei Li, Gangao Liu, Yang Chen, Xiang Zhou, Jingang Wang, Xunliang Cai, and Peng Li. Farlier tokens contribute more: Learning direct preference optimization from temporal decay perspective, 2025.
Arsalan Sharifnassab, Saber Salehkaleybar, Sina Ghiassian, Surya Kanoria, and Dale Schuurmans. Soft pref- erence optimization: Aligning language models to expert distributions. arxiv preprint (05:0:2405.00747, 2024.
Yaojic Shen, Xinyao Wang, Yulci Niu, Ying Zhou, Lexin Tang, Libo Zhang, Fan Chen, and Longyin Wen. AIPO: Improving training objective for iterative preference optimization. arXiv preprint K5i:2209000845, 2024.
14

---

Published in Transactions on Machine Learning Research (01/2026)
Nisan Stiennon, Long Ouyang, Jeff Wu, Daniel M. Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul Christiano. Learning to summarize from human feedback, 2022. URL //arxi:
btg/abs/2009.0133 01325
Xuerui Su, Yue Wang, Jinhua Zhu, Mingyang Yi, Feng Xu, Zhiming Ma, and Yuting Liu. Reveal the mystery of dpo: The connection between dpo and rl algorithms. orXiv preprinl arXiv:2502.03000 2025.
Yunhao Tang, Zhaohan Daniel Guo, Zeyu Zheng, Daniele Calandriello, Rémi Munos, Mark Rowland, Pierre Harvey Richemond, Michal Valko, Bernardo Ávila Pires, and Bilal Piot. Generalized preference optimization: A unified approach to offline alignment. arXiv preprint 2ttc:24020057443 2024.
Ben Wang and Aran Komatsuzaki. GPT-J-6B: A Billion Parameter Autoregressive Language Model. jax, May 2021.
Yue Wu, Zhiqing Sun, Huizhuo Yuan, Kaixuan Ji, Yiming Yang, and Quanquan Gu. Self preference optimization for language model alignment, 2024b. arXiv preprint arXiv:2405. 00675, 2024.
Teng Xiao, Yige Yuan, Huaisheng Zhu, Mingxiao Li, and Vasant G Honavar. Cal-DPO: Calibrated direct preference optimization for language model alignment. In The Shirty-eights Annual Conference on Neural Information Processing Systems, 2024.
Shiming Xie, Hong Chen, Fred Yu, Zeye Sun, Xiuyu Wu, and Yingfan Hu. Minor dpo reject penalty to increase training robustness. arXiv preprint Nix: 248.0990,,, 2024.
Yang Xiliang, Jiang Feng, Zhang Qianen, Zhao Lei, and Li Xiao. DPO-shift: Shifting the distribution of direct preference optimization. arXiv preprint WxXi::250200759000 2025.
Shusheng Xu, Wei Fu, Jiaxuan Gao, Wenjie Ye, Weilin Liu, Zhiyu Mei, Guangju Wang, Chao Yu, and Yi Wu. Is DPO superior to PPO for LLM alignment? comprehensive study, arXiv preprint 244.:220441111199 2024.
Binwei Yao, Zefan Cai, Yun-Shiuan Chuang, Shanglin Yang, Ming Jiang, Diyi Yang, and Junjie Hu. No preference left behind: Group distributional preference optimization, 2025.
Qingyu Yin, Leong Chak Tou, Zhang Hongbo, Zhu Minjun, Yan Hanqi, Zhang Qiang, He Yulan, Li Wen- jie, Wang Jun, Zhang Yue, and Yang Linyi. Direct preference optimization using sparse feature-level constraints. arXiv preprint 2XXi:2411.0766 2024.
Wang Yunan, Li Jijie, Zhang Bo- Wen, Wang Liangdong, and Liu Guang. Inco-DPO: Balancing distribution shift and data quality for enhancodpreference optimization. arXiv preprint arXi::2509 15880v1, 2025.
Yan Yuzi, Miao Yibo, Li Jialian, Zhang Yipin, Xie Jian, Deng Zhijie, and Yan Dong. 3D-properties: Identifying challenges in dpo and charting a path forward. In International Conference on Learning Representations, 2025.
Hanyang Zhao, Genta Indra Winata, Anirban Das, Shi-Xiong Zhang, David D. Yao, Wenpin Tang, and Sambit Sahu. Rainbowpo: A unified framework for combining improvements in preference optimization, 2025.
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhnang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 36:46595 46623, 2023.
Banghua Zhu, Michael I. Jordan, and Jiantao Jiao. Iterative data smoothing: Mitigating reward overfitting and overoptimization ion in rlhf, 2024. URL https: http///xxvvv.///./ wrgp.bs////...........g 16335
15

---

Published in Transactions on Machine Learning Research (01/2026)
A Proof of Lemmas 3.1 and 3.2
In order to prove Lemma 3.1 we will need first the following technical result. We recall that A. denotes the simplex in R" meaning the set of all vectors I= for which Σ Ps = 1 and 2 0.
Lemma A.1. Let q An and R.. Let S S [n] 21.2...., n2 be subset for which |S| 2 2. Any optimal solution of the following problem
arg max PEAR rr
E iES
ps In Pi
(19)
satisfies that (In (ps/qi) In (p5/9;)) = r holds true for any i, E S.
19 is convex and Slater's condition is satisfied; hence, the set of optimal solutions coincides with the set of KKT points. Therefore, we will consider the Lagrangian
Proof. The problem equation
Ui, A,) =
In Pi
+ Σ p: Em-1)
(20)
qi
SES
where À E R2 and E R are the Lagrange multipliers. The KKT conditions are
ri - 3 In 8i +1) + A +== ØL
VES Vi $ S, (feasibility),
op:
9:
ØL + x + +=
op:
Pi 1 0,
===
(21)
2
Vi E [n], (multipliers),
App= 0.
Vi E in]. (complementary slackness).
Let E S, after rearranging the condition we get that p; ge ( Moreover, if qi >> 0 then > 0 and hence A; = O. On the other hand, qi = 0 then = 0 and the value X does not matter. Hence, for simplicity, we can take A for all S. Thus, the KKT conditions can be equivalently written as follows
Pi ge(trtt)/8--1
e. Vi $ 5, (feasibility).
Fi + di + == 0,
E An.
(22)
A 2 0,
Vi $ S, (multipliers).
Aep = 0,
Vi $ S, (complementary slackness).
Let f = Ti. We now split the proof into the two cases:
Case T: P> 31n Eies //-11
First, for any i $ S, since X 2 0 we have that μ S -r and therefore particular / - - Thus, there must be & S for which pt > O. Indeed, if this not the case then p; 0 for all $ S and get contradiction since (recall that μ S -f)
1
Σ D: Σ P Σ geltitu)//-11
< Σ (t < <1,
(23)
iES
iES
SES
where the last incquality follows from the condition of Case I. Therefore, for simplicity take i. & S for which pt, 0. Therefore, Xi, 0.
Now, for all & S, we have r,, + As, r, + X, and thus r., = = + bg 2 ry which means that ri f. Hence μ -f, and therefore Pt qie (r. -1)3-1 for all i S.
16

---

Published in Transactions on Machine Learning Research (01/2026)
Moreover, this shows that if for some & S have < f, then p; 0. Therefore, by using the leasibility condition we obtain that any KKT point can be described by
P: gielri- ViES
P4 0, V
E
Pj = 1- - Σ gelrs-
JØS
Case II: f S Iln Eies qert/8-1
In this case, we will first prove that for all i i S. Suppose in contradiction that Pt > 0 for some & & S. Then, de = 0 and hence = Tt- Recall that re S P, we obtaín that
Σ P z Pe
Σ Pi - Pe + Σ ag 2 Pe +
Egeen 00/8-1 2
EE
iES
iES
where the third inequality follows from the condition of Case 11. Therefore, the unique KKT point is given by
qerv/B
P: = Ejes Vie
P. 0, Vi 4 S.
Finally, see that in both cases, have, for any i,j S, that
B (p/ - (p;/9))) = = Tj
required.
Now, we prove Lemma 3.1 To this end, we recall the new optimization problem that we study
Te(y)>) log [)())
max E, Eynno [ro(x,))] -B Σ
(24)
yES,
Proof of Lemma 3.1 First, note that the optimization problem equation separable in ær so we can fix some prompt x and consider the problem
max Eynro(a) [ro(z, - Σ Te(y)x)
(25)
VES.
From Lemma A.1 the optimization problem equation 24 admits closed-font solution which depends the relation between f. == max, To(x, y) and the rewards within S,, In the case where
Px > BIn
Σ yES=
neat (y x)) eetrodras-1)
(26)
policy ma is an optimal solution to equation
25 and only if
(t)
(y x) Tref (y) x) exp
Vy S.
(27)
exppppp Tref (y x) exp ( (1o(z -fz) -1)
Σ
(y) z) - Σ
(28)
((
yES.
(y) z) 0, Vy $ Sn, T (x. < fz.
(29)
17

---

Published in Transactions on Machine Learning Research (01/2026)
In the case where condition equation 26
does not hold, the unique optimal solution to equation 25 is
(y 2)) Tref (y x)) etrola))
Eyes, Tref (v' (dd Vy 2
(30)
T (y x)) = O, Vy & Sp.
(31)
We check that in both cases, for any two responses v/ E Sz, the optimal policy satisfies: To(y|=) Trorly/zrearrde brobbny To (x)
(32)
Tyrrby' xetrobe
Taking logarithms and rearranging:
B In To (v'l(2)) (re((()z)) Tret(H/ (x)
To(U)z)
r0(2, - (2,,00)) -3
(33)
For any preference triplct (x,, Vve y1) E D, applying this with y = yu and y' yt and substituting into the Bradley-Terry model p(uw un (r) or (I,, (r),)))) - (r, 31)) gives:
plu > 2/(x) = (Bln Te xx)
3 Iln To (z)
(34)
(Tr [z)
Trer(u (z)
which are exactly the probabilities used in the DPO derivation, and laking the sygalive-log-ikellhoor of equation 34 over the entire dataset yields exactly the DPO loss.
We complete this section with the proof of Lemma 3.2
Proof of Lemma 3.2 If (0 8) $ fr. for every y 3 1n(EEs,, ow is satisfied. This is because
Sz: then the condition fg
8ln Σ
Ther(ylz) :att 3In Σ yES,
S
fa +18
Trer(ylz)e
yES.
S fr + :3 Σ
Teer(y)
yES,
fr + B
-1 In Σ
(ee((
yES,
S fr - B
where the last inequality follows from the fact that Eyes, ()a(()| I This proves that the condition holds true.
Therefore, we in the proof of Lemma 3.1 in any optimal solution mo have for all y S,, that
<)(l))) = ( 12)0--11/11111----- < Trellylze = - m))y)))))
which proves the desired result.
B Derivations of Existing Algorithms via the Classification Framework
B.1
IPO
We recall that IPO can be formulated as a classification problem with the soft labels = (p = (1(1/2). and the loss given by
log nd 2
C(pe. p)
log P/
P
(35)
pd
18

---

Published in Transactions on Machine Learning Research (01/2020)
To show that we indeed recover the IPO, we first note that
p (1/2) = 1 1 exp((1122 = exp(1/2)
1 exp(1/2)
a(-1/2)
where the second equality follows from the definition of the sigmoid a(x) 1/(1 + exp(-x)). Moreover,
using the definitions of pa (see equation 10 and pb. we obtain that
Pb
Torr(Bull [z)
[e()))))) 3
(a To To (a) mer Uhe (z)
8
(36)
Pgg
Plugging these two developments to the loss in equation 35 yields
[(po, p)
log Ph 11
log
[log
To To (r (a)
1 2
log To (x) (ee((ywl2))
1 2 2,3
p¹
(o (z) Tre
2
which is exactly IPO (see Eq. (17) of Azar et al. 2024).
B.2 CDPO
Recall the CDPO Mitchell
(2024) loss is given for any triplet (x,, yues y1) by
To (vhel))) 3 Blog Tre((yllz) mo(u](z) [zc) )
3log To(Uila)) B log Teer (z:)
no (uw (xx)
- (1 - E) log
BBlg
log o
[re((Bwlz))
Tze((yr (z)
and then summed over all the triplets in the dataset D. In order see CDPD as classification with the soft labels p= (pmh, p¹) (1 c) and the CE loss, we will use the following technical fact
aB (Blog(a/b)) + 1
1
exp log B exp loga atd
1
+
6
a
+
b
Therefore, with xo(wwlx) (x) Recel a)) and b xe(u)>)) we get that log b/a)) = ptg. Therefore, the CDPO loss can be written as follows
(x) get from equation
10 that (B Pg Similarly,
-(1 <) log Pa - e log P/ = -p" log pa - -p' logr'o
which is exactly the CE loss on the vectors po = (vū.p%) .P)) and p = (1",,')
B.3 DPO (PL)
In this setting, we are given a dataset of the form D {xx Br :.e.....yN.)... In order recover, the DPO with Plackett-Luce Rafailov et al. (2023), we need to generalize the definition of the probability vector pa from pairs as in equation 10 to the following N - subsets of the list, namcly the first N, then the first N - 1, then first N - 2 and on. More precisely, for any 1 S < N we define
(2 Uns Vn-1,..., UN) = softmax (ro(I, 9n), re(z,
re(x,
In this case, the hard label vectors are defined, for any 1 S n < N, by pla = (1,0,..., E N-+11 Now, using the CE loss we get the desired result as follows
N N
N
exp [Bog 2(z)=o(2..1z))
N. log ( Bh. Bat1...., .N)) =
Tene(unl
log
iinn exp 1 log Z(z) (v. aead (1)
15
==1
N-1
exp log
To (Vi, (z) Tent (3 [=)
log
iimn exp 1 log
To To((:lz) Te (v)l)))
==1
1
exp B log
no [e( Foer(
log
i=n exp B log no(p.
[..)((]]))
19

---

Published in Transactions on Machine Learning Research (01/2026)
B.4 RPO and Distilled DPO
As we discussed in Section 3.2, RPO can be reformulated as classification with soft labels, which are defined by p = softmax(Sur: 8t). Therefore, we immediately see that
exp(sw) pd exp(sw) + exp(si) + exp( (s s())
and thus pd oss st). Similarly, we get that p' o(--sw-- si)). Thus, RPO can be seen as a generalization of CDPO where the soft labels are given by certain score and not fixed. To recover the RPO loss denote
3 log (yulz) Blog no(wi(s) and b sw st. Then, we get that
azat(tn m)
((b) a(a) + -(b)) log 10)(0) 1- o(a) = p log pd
p Pb
(b) log
+ p log
D2
By eliminating the constant terms (with respect to 8) pd log pd + p' logp', we indeed get the CE loss.
To recover the Distilled DPO (Equation (7) of Fisch et al. 2024 softmax(sw; 31) with the loss of IPO.
consider the soft labels p
log ndd 2
[pa, p) log Pø
T
Pg
Indeed, it this case we have that
p
exp(sw) pf exp(s..) + exp(sr)
exp($w) + exp(si) exp(si)
exp(sw) exp(si)
and therefore log(p" /p') 8w 81- Combining this with equation 36 yields the desired result.
C Proofs of Section 5
C.1 Proof of Proposition 5.1
Proposition 5.1. Let & :R be monotonic function and assume that equation 12 holds and that the largel labels salisfy (D" pf) e, for some < 1/2. Then, wa (ywlx) Tre(W x)) and ma- (yn (x)) < [r)((ul
Proof. For simplicity we assume that &o is monotonically increasing (the same arguments can be easily applied to the monotonically docreasing case). Since < 1/2, we have that (1 e)/s 1, and therefore from equation 13 get that
T (21(z) Tree (r)
To- (yho (z)
TTo((Unclz))
(37)
We will show that Ta- (zı|z) < (re (x) by contradiction. To this end, we assume that т (21(x) 2 (re((u) From equation 37 we immediately obtain that Tg- (wwl > (.or())l Applying the constraint function yp to the following two inequalities:
TT- (y- (x) > Tror(Uwll)) and To- (2r12) 2 [n)())
and using its monotonicity, we obtain
( (ww(x)) > (TTrr(((lla))) and ( (u])z) (
By adding both sides of the above two inequalities, we get a contradiction to equation 12 Thus, proving
that w (x) (.) Similarly, we can prove that wy- (vwllr) > (xw
20

---

Published in Transactions on Machine Learning Research (01/2026)
C.2 Proof of Lemma 5.1
Lemma 5.1. For any lao numbers and b, we have log(a + b) log log (logoloo log b).
Proof. First, write
a aabb) (
++b
which thanks to classical logarithmic rules yields that
a
log(a + b) = log log
log log
+ 6
b/a
Using the definition of the sigmoid function we get
= o(log a b)gb).
+ b/a 1 + exp(log b log a)
which proves the desired result.
D C2-DPO Implementation Details
We show implementation details of C2-DPO-Log-41, C2-DPO-Log-12, C2-DPO-I-41, and C2-DPO-I-O, below.
def dpo_loss (pi_logps. ref_logps yw yw_idxs, yl_idxs, beta, algo_name, reg_coeff):
#i_logps: policy logprobs, shape (B,)
ref _al_logp: reference model logprobs, shape (B,) yw_idxs: preferred completion indices in [0, B-1], shape (T,) yl_idxs: dispreferred completion indices in [0, B-1], shape (T,)
beta: temperature controlling strength of KL penalty
Each pair of (yw_idxs [i], yl_idxe [i]) represents the indices of single preference pair.
pi_yw_logps, pi_yl_logy
pi pi_logps[yw_idxs] pi_logps
ref_yw_logps, ref_yl_logp logps = ref sef_logpslyke [yx_ixx] ref logps [yl__dxx]
pi_logratios = -pi_y_logge - pi_yl_logps
ref_logration = ref_yy_logps ref_yl_logps
losses -F loooogggmoiibeet (pi_logratios ref (oglaaiaatss
if algo_name name 'c2dpo_log_111.
perrallzed__osses_riie squaree__urreeee
pi_yw_logps pi_yl_logps ref eef_yw_logps ref ref_yl_ll _e__y_logpss
reguralized losses (reguralized, losses
lossssesd_lsssss___eeeeeeeeeeee wi thouthot_eee thouttssuarre) ** peuuralized_loseee
losses = losses + reg_coeff
elif algo_nane _ame = @2dpo_log__d 12':
reguralized_lossee F. {
pi_yw_logn ref_yw_logps ref_yl_log
mi_yl_logpp_
losses = losses coeff reguralized losses
elif algo_nane name == 'c2dpoii_11';
reguralized_ lossed_oosss__eeeee losses_vithout http__itthttt_ptee square
pi_yw pi_yw_logps __ggpsppp___ F. Loggigmoid(pi__y__ogpp
pi_yl_logps)
rre__wwloggh F. n.logsignooddrref_y_laaaa yo_lopps ref eef_yl_logps)
Lgguralized_oossed
iigurraliad__ossse_e_ttttooo
losses = losses ree_foeff reguralized losses
21

---

Published in Transactions on Machine Learning Research (01/2026)
elif algo_name name == 'c2dpo_1__ ________:::
reguralized, losses F. (
pi_yw_logph F. r..osiigooidppi.yw.ucc. _og_pppp___ppp - [0_yl_logps)
ref ssr_y_loogs F. w..ogsigmoiddreffy__ooooo
ref ree__ll___t_ yl__ogp_s))
)
losses losses reg_ reguralized_ losses
rewards beta (pi_logpa ref sef_logps) detach() return losses, rewards
E Ultrafeedback Binarized Claude 3.5 Sonnet v2 win rate prompt and hyperparameters
In this section include the prompt. used generate win rates for the Ultrafeedback Binarized experi- ments. We Claude Somnet 3.5 v2 (AWS Bedrock model ID http:cccuuco---mm22222222222222 to generate win rates. We set the max tokens to 1024, temperature to O, and used default value 0.999 for top_p and top_k k disabled.
For the following query to a chatbot, which response is more helpful?
Query: <prompt>
Response A:
<one of the responses>
Response B: <the other response>
FIRST provide one-sentence comparison of the two responses and explain which you feel is helpful. SECOND, to indicate which response is more helpful. Your response should use the format: Comparison: <one-sentence comparison and explanation More helpful: <"A" or "B">
line, state only "A" "B"
F Reddit TL;DR Claude 3.5 Sonnet v2 win rate prompt and hyperparameters
In this section we include the prompt used to generate win rates for the Reddit TL:DR experiments. We Claude Sonnet 3.5 v2 (AWS Bedrock model ID httpppccccuba35--oee2222220-220 win rates. We set the max tokens to 1024, temperature to 0, and used default value 0.999 for top_p and top_k k disabled.
generate
Which of the following summaries does a better job of summarizing the most important points in the given forum post, without including unimportant or irrelevant details? good summary both precise and concise. Post: {prompt}
Summary A:
(baseline_ [aaiineerespossee
Summary B:
fgenerated_response) FIRST provide a one-sentence comparison of the two summaries, explaining which you prefer and why. SECOND, on new line, state only "A" or "B" indicate your choice. Your response should use the format: Comparison: <one-sentence comparison and explanation
22

---

Published in Transactions on Machine Learning Research (01/2026)
Preferred: <"A" "B">
23


Reviews:
[{"review_id": "f9nzgO6Vl5", "reviewer": "Reviewer_DN7o", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This work studies the alignment problem and presents several theoretical results. It begins with reviewing direct preference optimization (DPO), a mainstream algorithm for solving the alignment problem in reinforcement learning from human feedback (RLHF). The paper presents an alternative form of DPO derived by limiting the KL-divergence term to in-sample responses rather than to the full output distribution. The proposal of this alternative form leads to some counter-intuitive implications, including a decrease in the likelihood of both preferred and rejected responses under the optimal policy.. Next, the paper introduces Constrained Controlled DPO (C2-DPO) based on these theoretical analyses and provides experimental results that demonstrate its efficacy on several datasets.", "claims_and_evidence_yn": "Yes", "claims_explanation": "This paper is not in my research field, and I have no prior experience with RLHF or any related topics. Therefore, I will focus on my high-level understanding of this work and leave its technical rigor and merits to more experienced reviewers.\n\nThe paper is well written. I like the way the authors organized the technical sections. Even as a novice in this area, I can follow the high-level story thanks to the paper's writing.\n\nThe paper builds a theoretical framework for analyzing DPO and reveals some counter-intuitive implications. The theoretical analysis developed in this work explains existing DPO algorithms from an interesting and novel (to my knowledge) perspective. It opens new opportunities and may inspire follow-up work to conduct similar theoretical analyses in this direction.\n\nFor example, the idea of unifying several existing DPO-style algorithms under the same classification framework (Sec. 4), loaded with different objectives and data labels, seems to be a useful interpretation of these existing algorithms. I appreciate this effort.\n\nOverall, I am inclined to answer Yes to this question, based on what I have read from the paper so far. However, I have to stress that I am not a researcher in this field, and I request that the editor and other reviewers take this factor into account.", "audience_yn": "Yes", "audience_explanation": "Please see my answer above.", "requested_changes": "I have a small request for clarifying the technical exposition in the experimental results: I can read from the tables and figures that the proposed C2-DPO leads to improvements over prior works, but I do not have sufficient background information to judge whether such improvements are marginal or significant, particularly for those blue-red (DPO vs. C2-DPO) histograms in Figs. 4, 6, and 8. Again, this is primarily because this paper is not in my research field, and I very much welcome the authors' providing more context behind these numbers.", "broader_impact_concerns": "None.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "EKHzWvyzB7", "reviewer": "Reviewer_Yjk2", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper shows that the standard DPO loss can also be derived from an alternative RL-style objective with a KL penalty applied only to a subset of responses, and uses this formulation to prove conditions under which all labeled responses probabilities are systematically shrunk (\"likelihood displacement\").\nIt then presents a unified classification view of several DPO-style methods, arguing these pairwise objectives are under-specified because they only constrain probability ratios within each pair.\nTo mitigate this, the authors introduce C2-DPO, a family of DPO variants with simple pairwise constraints (e.g., preserving the sum of log-probs or probabilities of winner+loser) that provably enforce \"winner higher, loser lower\" relative to the reference and yield modest empirical improvements over vanilla DPO and a few recent baselines.", "claims_and_evidence_yn": "Yes", "claims_explanation": "Overall, most of the central technical claims are supported by correct derivations and reasonably chosen experiments, but a few parts of the narrative feel stronger than what the evidence strictly justifies.\n\nFor example, \"both the preferred and rejected responses tend to decrease in probability under its optimal policy\" is the main motivation to add the regularization/constraints.\nHowever, the formal result (Lemma 3.2) shows that if a labeled response has lower reward than some unlabeled response, then at the optimum of the in-subset KL problem its probability shrinks by at least a factor $e^{-1}$.\nThis is a conditional, per-pair statement; the upgrade to global language like \"tend to decrease\" for both responses relies on informal plausibility arguments and a few illustrative plots, rather than a quantitative probabilistic analysis of how often the lemma's condition actually holds.\n\nEmpirical claims about \"higher-quality final models\" are not sufficiently supported. The experiments do show modest, consistent gains over vanilla DPO and a few recent DPO-style baselines on UltraFeedback, TL;DR, and MT-Bench for a small set of models.\nHowever, these are relatively narrow setups, and there is no comparison to very simple practitioner baselines (e.g., DPO + explicit KL regularization), nor robustness checks across seeds or evaluators.\nSo the broad wording about \"delivering higher-quality final models\" overstates what the current empirical evidence can support.", "audience_yn": "Yes", "audience_explanation": "The derivation of the DPO loss from an alternative RL objective where the KL penalty is only on in-sample responses and Section 4 \"A Classification View of DPO-style Algorithms\" would be interesting and useful for some individuals in TMLR's audience.", "requested_changes": "## Clarify and abstract the mathematical core\n\nI strongly recommend restructuring Sections 3 to first present the underlying optimization problems and constraints in a fully abstract setting without heavy RLHF/DPO jargon and only then reintroduce the RLHF/DPO interpretation.\nThis would make the main contributions (closed-form solutions, under-specification, constraints) much clearer.\n\nThe current exposition often mixes informal phrases (\"following the recipe…\", \"in-sample prompt–response pair\") with mathematically non-essential application details, which can obscure the structure of the arguments.\nSeparating definitions from narrative and clearly distinguishing the abstract math from the RLHF interpretation would make the paper easier to read and the contributions easier to evaluate.\n\n## Make Lemma 3.1 and related derivations more precise\n\nLemma 3.1 currently states the existence of a \"closed-form solution\" but only uses a difference identity and a reference to the original DPO derivation.\nI would like to see the solution of Eq. (5) stated explicitly (with the relevant case distinction from Lemma A.1) and the steps from that solution to the DPO loss written out clearly. This would also help make transparent which properties of the underlying RL objective are preserved (or discarded) when passing to the DPO surrogate.\n\n## Connect more explicitly to learning-to-rank and preference-learning literature (non-critical)\n\nMuch of the conceptual content (e.g., pairwise Bradley–Terry modeling, non-identifiability of scores under ranking-equivalent transformations, and the need for additional regularization/constraints), is well known in the learning-to-rank and preference-learning communities.\nA more explicit discussion of how DPO-style methods fit into that landscape, and what is genuinely new in the RLHF-specific setting would significantly strengthen the positioning.", "broader_impact_concerns": "NA", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "uC6uP5rPue", "reviewer": "Reviewer_vUaj", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper provides two counter-intuitive observations about Direct Preference Optimization (DPO): first, the DPO loss can be derived from an alternative optimization problem that only applies a KL guardrail to in-sample responses. Second, this alternative formulation theoretically explains the observed phenomenon where both preferred and rejected responses tend to decrease in probability (likelihood displacement) during DPO training. Based on these insights, the authors propose Constrained Controlled DPO (C2-DPO), which introduces constraints to limit this probability displacement. C2-DPO is shown to outperform vanilla DPO and several baselines on standard alignment datasets.", "claims_and_evidence_yn": "Yes", "claims_explanation": "Mostly yes. The theoretical derivations (Lemma 3.1, 3.2, Proposition 5.1) are key and seem sound, especially the proof connecting DPO loss to the in-sample-only KL regularization. The empirical evidence shows consistent improvements of C2-DPO variants over vanilla DPO in win rates and holistic MT-Bench scores. Figures clearly support the core claim about in-sample probability reduction in DPO and mitigation in C2-DPO. The classification framework for unifying DPO-style algorithms is well-supported in the appendix.", "audience_yn": "Yes", "audience_explanation": "The alignment problem and RLHF/DPO algorithms are central topics in current ML research. The paper's theoretical dissection of DPO's subtle mechanism and its counter-intuitive side effect (likelihood displacement) is of academic interest. Furthermore, proposing C2-DPO as a simple, effective, and computationally efficient fix addresses a practical limitation of a widely used algorithm, which may be appealing to practitioners.", "requested_changes": "I generally feel this paper is well-executed. Please correct me if I'm wrong about anything I've said.\n\n1. The paper introduces $\\ell_1$ and $\\ell_2$ penalties for the regularization term in Section 5.1, but the loss equations (12 and 15) only show the squared penalty. Please clarify this inconsistency in the main text, or ensure that the generic $\\ell_1$ version is also formally presented. (The implementation details in Appendix D show both, but the main text seems ambiguous).\n2. Proposition 5.1 relies on the assumption in the proof that $\\epsilon < 1/2$ (e.g., for DPO hard labels $\\epsilon=0$). Given that CDPO and IPO use soft labels where $\\epsilon$ is non-zero, this should be explicitly stated as a condition for the proposition's claim ($\\pi_{\\theta^*}(y_w|x) > \\pi_{ref}(y_w|x)$) to hold, or confirm if the result holds for all $\\epsilon$.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)