Explicit Preference Optimization: No Need for an Implicit Reward Model
Authors: Xiangkun Hu
Lemin Kong
Tong He
David Wipf
Venue: ICML 2025 poster (2025)
Keywords: direct preference optimization
reinforcement learning from human feedback
preference alignment
regularized regression
Primary area: deep_learning->large_language_models
TL;DR: This paper introduces and analyzes an alternative to direct preference optimization that relies on no implicit reward.

Abstract: The generated responses of large language models (LLMs) are often fine-tuned to human preferences through a process called reinforcement learning from human feedback (RLHF).  As RLHF relies on a challenging training sequence, whereby a separate reward model is independently learned and then later applied to LLM policy updates, ongoing research effort has targeted more straightforward alternatives. In this regard, direct preference optimization (DPO) and its many offshoots circumvent the need for a separate reward training step.  Instead, through the judicious use of a reparameterization trick that induces an implicit reward, DPO and related methods consolidate learning to the minimization of a single loss function.  And yet despite demonstrable success in some real-world settings, we prove that DPO-based objectives are nonetheless subject to sub-optimal regularization and counter-intuitive interpolation behaviors, underappreciated  artifacts of the reparameterizations upon which they are based.  To this end, we introduce an explicit preference optimization framework termed EXPO that requires no analogous reparameterization to achieve an implicit reward.  Quite differently, we merely posit intuitively-appealing regularization factors from scratch that transparently avoid the potential pitfalls of key DPO variants, provably satisfying regularization desiderata that prior methods do not.  Empirical results serve to corroborate our analyses and showcase the efficacy of EXPO.

Explicit Preference Optimization: No Need for an Implicit Reward Model
Xiangkun Hu 1
Lemin Kong *2
Tong He
David Wipf'
Abstract
1. Introduction
The generated responses of large language models (LLMs) are often fine-tuned to human preferences through a process called reinforcement learning from human feedback (RLHF). As RLHF relies on a challenging training sequence, whereby a separate reward model independently learned and then later applied to LLM policy updates, on- going research effort has targeted more straightfor- ward alternatives. In this regard, direct preference optimization (DPO) and its many offshoots cir- cumvent the need for a separate reward training step. Instead, through the judicious use of a repa- rameterization trick that induces an implicit re- ward, DPO and related methods consolidate learn- ing to the minimization of single loss function. And yet despite demonstrable success in some real-world settings, we prove that DPO-based ob- jectives are nonetheless subject to sub-optimal regularization and counter-intuitive interpolation behaviors, underappreciated artifacts of the repa- rameterizations upon which they based. To this end, we introduce explicit preference opti- mization framework termed EXPO that requires no analogous reparameterization to achieve an im- plicit reward. Quite differently, we merely posit intuitively- aauitively-appealingg regularization factors from scratch that transparently avoid the potential pit- falls of key DPO variants, provably satisfying regularization desiderata that prior methods do not. Empirical results serve to corroborate our analyses and showcase the efficacy of EXPO.
Reinforcement learning from human feedback (RLHF) (Bai et al., 2022a; Ouyang et al., 2022; Stiennon et al., 2009; Ziegler et al., 2019) represents effective means of align- ing the output of powerful pre-trained large language models (LLLMs) (Bubeck et al., 2023; Chang et al., 2024; Achiam et al., 2023; Zhao et al., 2023a) with human preferences (Bai et al., 2022b; Gallegos al., 2023). On the downside though, RLHF requires a potentially-unstabe multi-step process whereby first reward model is learned using labeled preference dataset. Subsequently, new LLM pol- icy is trained to maximize this reward while minimizing deviations from the original pre-trained LLM (the reference policy). To mitigate this complexity, various reparameteriza- tion schemes have recently been introduced that obviate the need for training a separate reward model. Instead, so-called direct human preference optimization (DPO) (Rafailov al., 2024) and follow-up variants (Azar et al., 2024; Tang et al., 2024; Wang et al., 2024a; Zhao et al., 2023b) are instantiated via the minimization of a single closed-form training loss, within which an implicit reward function is concomitantly optimized.
Although these DPO-based alternatives to RLHF dramat- ically streamline the modeling pipeline, as will detail herein, they are nonetheless saddled with undcrappreciated limitations of their own, such as sub-optimal regularization effects, degenerate minima, and other counter-intuitive be- haviors linking back to the original reparameterizations and the viability of the implicit rewards involved. To this end, our paper is devoted addressing these shortcomings via the introduction of intuitive training objectives that term EXPO, for explicit preference optimization, which rely no such implicit rewards or associated reparameteriza- tions. After presenting basic concepts and the details of existing preference optimization models Section 2, the re- mainder the paper devoted to our technical contributions can be distilled as follows:
Equal contribution 'Amazon Web Services 'The Chi- nese University of Hong Kong. Correspondence
Lemin Tong He <htong @ amazon.com> David Wipf <davidwipf@ gww.l/oommmmmmmm.mmmmmmmmmmmmmm.
to: Xiangkun Hu <xkhul crkhul7@@fdan.ddu.coo
Kong <11551654688 @ waok.cuh..cdu.he.
We introduce new evaluation criteria that comport with intuition regarding how a preference model ideally should behave, and yet (somewhat surprisingly) are provably not satisfied by a broad class existing DPO- based approaches. In particular, we show that because
Proceedings of the ndnd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Explicit Preference Optimization: No Need for an Implicit Reward Model
of uniform regularization effects, the minimizers of commonly-used preference optimization objectives like DPO are at times unable to preserve good performance in regions where the reference model is strong while si- multaneouslyy improving upon the reference model else- where (Section 3.1). Moreover, we also elucidate limita- tions in the ability to interpolate between ideal endpoints as model trade-off parameters are varied (Section 3.2).
from Dz, V11 A y/2 from the reference policy Tie and finally is produced by human labelers that operate according to p". Note that per convention in prior work and ease of presentation, we will often abbreviate the preference distribution notation as p" (y) > y22 ly,, y2, 2:) = p' > y2|z) when the context is sufficiently clear. We now briefly introduce RLHF, DPO, and follow-up variants that will serve as motivation for our proposed EXPO framework.
In Section 4 we then propose multiple new EXPO train- ing objectives Exxoo for explicitly optimizing human pref- erences while minimizing deviations from an original reference policy. Importantly, these losses satisfy our evaluation criteria from above while requiring no repa- rameterization or implicit link to separate, motivational RLHF objective. And although these ÉEPO candidates depend on the unobservable ground-truth preference dis- tribution by design, we nonetheless establish that unbi- ased estimates of the gradient can be directly computed to facilitate efficient SGD.
2.1. RLHF
Reward Function Estimation: Given two candidate re- sponses y1 # y/2 sampled using prompt z, the Bradley-Teer Terry (BT) model (Bradley & Terry, 1952) for human preferences stipulates that
exply (11, xr] p" (vi > v21x) = exp|="(n,,x] + exp (r" (12, x)]
= =fr"(y1,)) - (v2, x)},
(2)
where r" r"(y, is a so-called latent reward model and is the logistic function. Because r" (y, is unobservable, it is not possible to directly compute p" (y1 > 52(z)); however, we can train an approximation ( > 3/2 (z) defined by a parameterized proxy reward (y, Specifically, we can minimize the loss
To complement our analysis, Section 5 provides empir- ical verification of conditions, in a controlled environ- ment with known ground-truth preferences, whereby DPO-based regularization and related variants converge to degenerate minimizers while fexo minimizers do not. We then conclude with experiments involving real-world alignment data that show EXPO outperforms DPO-based models w.r.t. response win rates.
far
((unnn.)))DDi. log log Po y mr|rr) (3)
= E (r log o (ro(yu: ro(u,
2. Background
The optimized reward (v) == arg minr, (ar(ra) 2 r" (y, x) can then be applied to fine-tuning the pre-trained reference model T (z) as described next.
We adopt x ~ Dr to denote an input prompt x drawn from some distribution Dz. From here, conditioned on such prompts may then generate responses y using pre- trained reference language model/policy TT)()) More- over, given a pair of such responses y1 # y2, we adopt the binary indicator variable = = Isyn > 1/131,, 3/2 22 to convey that 3/1 is preferred over y/2 by a human evaluator when z = 1, or else = = instead 1/2 > V1- Given a population of such evaluators, we express the ground-truth human preference distribution as p" (=lyn, V2, x) = p" (y1 > yalyi, 12, .). And finally, we define a set of human labeled tuples drawn from training distribution D.,
RL Fine- Tine-Tuning with an Estimated Reward: Given the optimized reward from above, the remaining goal is to improve upon a given pre-trained LLM, or reference policy (y)x)) using a separate trainable model wo to (y)zz) This is accomplished by minimizing the RLHF loss in the form
Cruir (Ty, Trat for A)
= Bysno(ul)) fo(
[L [KL[TT(Vle (T())) [mm]=)
+ A BEWD,
(4)
where À >> 0 is a trade-off parameter balancing reward maximization term and the KI. divergence from the refer- ence policy. Although not directly amenable to SGD, given an initialization Te = Tat. the loss f (To, Tiet fgg A) can still be optimized over τæ using RL techniques (Schulman et al., 2017; Ramamurthy et al., 2022).
(Nw, v1 x ~ Drr = {2,01, x Der 2 (x(v1 V2. x). {,i,32) 32) ~ Tea Tya( (y)x), Da:
(1)
where the > ye subscripts here stand for "win" and 'lose". In other words, each training tuple is generated by drawing x
We generally assume that yt # yzi however, the 111 = 3/2 case can nonetheless be handled by simply assigning p' (z y, 2) = 1/2, inclusion of which does not affect the analysis that follows. In particular, such cases merely introduce an irrelevant constant into the human preference loss functions under consideration.
2.2. DPO
DPO (Rafailov et al., 2024) is based on the observation that, for an arbitrary reward function r(y, 2), the minimum
2

---

Explicit Preference Optimization: No Need for an Implicit Reward Model
of krur (To: Tias A) w.r.t. my can be computed in closed form as
2.4. Flexible Quasi-Convex Generalizations
From the expressions above, it is clear that both DPO and IPO reduce to functions of log tunable hyperparameter X. As such, it is natural to consider extensions to broader choices in the form
(>(2))) := arg min ERLHF (To, Trats T, X)
(5)
me (e ( (e (o
and
That (w/l)) exp
Fr(u.o))
((x)
where ((x) == Ey ())=)) exp x)) ensures that T. (vlær) forms a proper distribution (Peng et al., 2019; Peters & Schaal, 2007). From here, we can invert (5) to express the reward via the policy as
(To, Ther V. By 1b (u
(10)
( (Br (x) Teat Mew [xr)
To
To( (x)) Tea (1n (2:)
.
r(y, = Alog (z(a²) (y)e) + X log ((r).
(6)
where μ R R is monotonically increasing function (which generalizes the logarithm), and the function so Xx R R influences the overall loss shape. We stipulate that v is a differentiable quasi-convex function (Greenberg & Pierskalla, 1971); hence the chosen loss notation faro for quasi-convex preference optimization. By definition of quasi-convexing & monotonically increases to the right or left away from the minimum.
As no assumptions have been placed r, it follows that (5) and (6) hold even for the ground-truth reward r* and the corresponding optimal policy (v/lx) := arg min., fRLI (Ia, Tats r". A). Therefore instead of ap- proximating r" (y,, x) with rh (v, x) as in (2), we may ap- proximate π (y]2z) with some To (y/a)) (r) within (6). The resulting reparameterized BT objective then forms the DPO loss
These specifications cover DPO and IPO as representative special cases, and include essentially all reasonable choices for loss within this family, e.g., it is nonsensical to in- clude multi-modal losses. The generalized preference op- timization (GPO) (Tang et al., 2024) and f-DPO (Wang et al., 2024a) frameworks are also special cases of QPO as defined herein. With GPO, i logarithm and & is chosen
To (3) (r)r))
Eooo (Te, Taats X)
CET
A Allo
Taat
To (V
=
E By 20
- log
A log
Tiar
(Dh (3er xr)
(31 (x)
A log
Tre((11 91 xx)
arbitrary convex function (such used by SLiC (Zhao et al., 2023b)). Meanwhile f-POO involves ( A) = log or (X(-)] analogous to DPO but with μ = f'. where f' denotes the derivative of an f-divergence (Ruben- stein et al., 2019); given that f must be convex, its derivative will necessarily be monotonically increasing. In this way, the RLHF objective from (4) is still optimized via f-DPO, but with an f-divergence replacing the KL term
where the À log (x) term cancels out and can be omitted. And so becomes possible to efficiently optimize (7) over He using SGD without RL. Overall, the policy ve induces an implicit reward X log [To (v ))) (r [y]z)] that is optimized by minimizing the reparameterized BT model.
2.3. Identity Preference Optimization (IPO)
The identity preference optimization (IPO) framework (Azar et al., 2024) weaves alternative reward function into the original RLHF loss from (4) such that efficient learning is possible as with DPO. Concretely, IPO is designed to minimize ERUHF (To, Teet Two, X) over To. where
While overall quite general, we will nonetheless later demon- strate that any loss in the form of (10) will unavoidably be saddled with certain limitations. See also Appendices A. and B for further context w.r.t. recent DPO-related work that lies outside of our present scope.
(1) == (el [p" (y )/
2.5. Online vs Offline Preference Optimization
Stemming from the special structure of this particular reward, ERua (To, Tats T Two, A) can be minimized without RL. This is accomplished using an analogous reparam- eterization trick DPO, leading the final IPO loss (TO, Eads A)
Some recent work (Guo et al., 2025; Tajwar et al., 2024; Xu et al., 2024) has suggested that so-called online pref- erence learning (as in the original RLHF that trains using samples from T8) may often produce better results than of- fline learning (as in DPO which only uses fixed samples from n.r)). There are nonetheless justifiable reasons for still considering the latter. However, in Appendix B we provide rationale for why exploration of offline approaches still warranted, especially when we consider methodology that extends beyond the original DPO script as is our focus.
(1gg
no ( xx) To x se She [x)
1 2
E Ml
(9)
2X
which is naturally amenable to SGD like DPO. Further details regarding properties of the IPO loss from (9) are contained in Appendix E.5.
3

---

Explicit Preference Optimization: No Need for an Implicit Reward Model
3. Limitations of Existing Approaches
Preserving Optimal Policies
We now turn to formalizing previously-unexplore limita- tions of existing DPO-like approaches. Throughout this section we will rely on the notion of an optimal policy n². We intentionally do not specify by what criteria this optimality is established, as our results will hold for any (non-deterministic) policy such that n" (y/ (r) E (0, 1) for all t E X.
π Tf
Ag
EXTO
prompt-conditonns policy space
EXPO 2
X d:
gap
X d
πT nf B5
CPO
π
Disparate EXPO/QPO performance
3.1. Failure to Preserve Optimal Policies
Similar EXPO/QPO performance
Consider the following plausible scenario, variations of which are likely to occur (at least in varying degrees) with real-world data. Suppose the support of prompts generated by Dr partitions as dgood U dbed, with dgood n dbad = 0. Furthermore, assume we have access to reference policy Tet such that Tex x² for E dgood and dist( (Te,, x²) >> 0 for E dbed, where dist(;;) :] is an arbitrary distance measure. In other words, when evaluated w.r.t. policy x¹ that pro- portionally reflects human preferences, net performs ideally on a subset of prompts but not on others.
Figure I: Preservation of optimal policies; proposed EXPO approach (Sec. 4) included for added context. The dashed blue line divides prompt space into regions where a per- forms poorly (right side, dhad) from regions where it is already near optimal (left side, dgood). Within de we ob- serve consistent movement towards π as desired; however, within dind we see that "p from π (unlike EXPO).
OPO introduces an unwanted gap
This dichotomy provides useful lens for examining certain loss function properties. In particular, we would like any policy that minimizes candidate loss to preserve Ret for prompts x E dgood, while pushing away from Tet towards x for prompts x E dbad, However, because of uniform reg- ularization effects intrinsic to QPO losses, it is not actually possible to achieve even this modest objective.
when we zoom in to examine DPO and IPO specifically, Both losses (i.e., (7) and (9)) depend on my exclusively as Tu mu(y (er (r))
ma (e) (w) which equates to zero whenever मe = Tea regardless of preference data. And yet by straightforward inspection, we observe that both DPO and IPO losses can always be reduced further as this log factor moves away from zero for virtually any empirical preference distribution. This forces Te away from Tut, when म. = = however the latter is defined.
function of log
Theorem 3.1. (Informal version) Given the prompt par- ritioning, reference policy, and optimal policy described above, define # := arg minn, Lorol To. Teds v. H. X) for any fixed selection of (tb, A.. Then under rela- tively mild assumptions on the labeled responses in Dev. if distlat T²] distlTrd: π²) for E dhad, then dist ( - O for dgood
3.2. Suboptimal Interpolation
As the underlying goal shared by all approaches is to hal- ance proximity to reference policy The with respect for the human preference model p", a non-negative trade-off parameter À E [a, 6j that allows for interpolating between these competing objectives is inevitable, where a E R and E IR are lower and upper bounds respectively. In this section examine more closely the nature of loss function minimizers as À is varied, zooming in on their behavior in the limit as X + and X + b. To this end, we first introduce the following definitions:
The proof and formal version are provided in Appendix D.I, while Figure I provides an illustration. The somewhat unex- pected implication here that if we minimize any possible QPO loss the form of (10) and improve the policy quality in areas where Tet performs poorly w.r.t. π², then it must also be the case that performance becomes worse in areas where nia was originally optimal. This phenomena repre- sents an unavoidable trade-off when we restrict ourselves to using a QPO loss, of which DPO, IPO, GPO, and f-DPO are all special cases inheriting the same limitation. The core issue is that these QPO losses unselectively apply the same regularization, starting from the same initialization point, to both good and bad cases relative to an arbitrary n²"
Definition 3.2. We say that an arbitrary preference opti- mization loss {(TT, Tret X) satisfies the strong interpolation criteria (SIC) if the following hold:
1. limx arg mitlin {(TO, Teets A) π=;
Although Theorem 3.1 is presented in a technical form to rigorously cover QPO cases more broadly, more intuitive viewpoint can be established
Intuition from Special Cases:
DDepending on the method, if 0 b may replace the À range with an open set.
4

---

Explicit Preference Optimization: No Need for an Implicit Reward Model
In terms of practical applicability of this result, there ex- ists one important caveat: the empirical distribution of finite set of labeled preference data need not actually satisfy the conditions of Proposition 3.4. For example, suppose for each prompt x E DD we collect only two responses {v), y2} along with a single preference label 2, which together pro- duce the tuple (Yw, Di, 23. In this scenario, which reflects certain publicly-
Interpolation Behavior
mode of π
π = BT optimal policy
policy spece
π
0 X
A c
human preference datasets (Bai et al., 2022a; Ganguli et al., 2022), the empirical distribution of preferences will be p" (yu > (x(2r) 1 & (0, 1) for all x: E D... Notably, Proposition 3.4 will not hold, and in par- ticular, it can be easily shown that minimizers of any valid f f-DPO loss will be completely independent of The for all A E (0, oo); other words, no interpolation occurs at all; see Appendix E.3 for the derivation. A similar observation specific DPO (but not f-DPO) can be found in (Ahmadian et al., 2024). The fact that DPO-based solutions may still reflect net in practice, and more-so as À increases, relates to implicit constraints and subtle regularization effects as detailed in (Kong et al., 2025).
Interpoiation paths
20 (4)
(2)
2100 (2)
Figure 2: Interpolation illustration; a proposed EXPO vari-
ant (Sec. 4.1) is included for added context.
2. limx-sb arg min (TTO, Teat A) Tart
3. For all other X E (a, b), the optimal policy interpolates between the above two extremes.
Proposition 3.5. Assume preference data distributed ac- cording Der from (1). Then the IPO loss from (9) satisfies the WIC (but not the SIC).
Definition 3.3. For any prompt x and response y define"
x (ylz) == arg max max Ey-no(ul))) [r (),x)) He
(11)
= {
1
if y= arg maxy nt (y|z)
Comparing Proposition 3.5 with Proposition 3.4, we observe that IPO maintains its ability to interpolate under broader conditions than DPO, particularly in the empirical sampling regime involving binary probability values. That being said, neither DPO nor IPO satisfy the SIC, which motivates con- sideration of alternative losses that do, at least if our priority is to actually achieve the SIC (which of course may depend on the application scenario). For this purpose, it turns out that selections beyond the family of QPO objectives (which includes DPO, IPO, GPO, and f-DPO) are necessary per the following:
0
otherwise.
In this way, 7 (w/zz) assigns probability one to the mode of x" (y]x)) i.e., akin delta function with no generation di- versity whatsoever. We then say that a loss ((TT, Tets A) sat- isfies the weak interpolation criteria (WIC) analogously to the SIC, only for the lower bound we instead require lim arg min., {TTT, Teer 2) = T
In summary, the only difference between these interpolation criteria is their limiting behavior w.r.t. the lower bounding X; for the SIC approach the optimal policy (however it is defined), while for the WIC we approach a degenerate policy with all probability mass restricted to the mode of the optimal policy. We remark that both the SIC and WIC cannot be simultaneously satisfied unless nT" itself is a de- generate delta function. See also Appendix E.2 where we provide an illustrative example of why the intrinsic diversity of x- may be preferred over n².
Theorem 3.6. Assume preference data distributed accord- ing to Dir from (1). Then no possible QPO loss from (10) will satisfy the SIC.
4. Explicit Preference Optimization
Motivated by the analysis in Section 3, we now examine alternative objective functions outside of the QPO family adhering to the following desiderata:
We now explore how these distinctions are reflected in the behavior of DPO and IPO loss minimizers, with Figure 2 illustrating the basic concepts.
1. Perservation: Capable of selectively preserving an op- timal policy in ideal regimes, while simultaneously im- proving the policy in regions of poor performance (from Section 3.1);
Proposition 3.4. Assume preference data distributed ac- cording to Der from (1), and that p" (y1 > y2|z) E (0, for all responses with (y(()) > 0. Then the DPO loss from (7) satisfies the WIC (but not the SIC).
2. Interpolation: Smoothly interpolates between an opti- mal policy and the reference policy, ie.. it achieves the SIC (from Section 3.2).
See Appendix E. for the derivation of the right-hand equality in (11).
5

---

Explicit Preference Optimization: No Need for an Implicit Reward Model
We consider two candidates targeting these desiderata, a so- called compositional loss denoted f'xoo and regression- like loss (loosely motivated by IPO), referred to as fixro
Unsupervised Term: For the unsupervised term in (13)
we simply adopt
Before proceeding however, we must establish a concrete, meaningful conception of optimality that is actually worth achieving by our proposed approach. For this purpose we will henceforth refer RT" as a op-tiiimal policy whenever
Pansup (TO, Feet)
= EawD, [ (man
= - Br log [oggo(vl=)) (16)
ignoring terms independent of mo. Like (15), this expression also does not require sampling from Tg. That being said, (16) can exploit out-of- preffreenee data (meaning unlabeled responses), and prior work (Li al., 2024) has argued for the merits of using such data in broader RLHF contexts. (It may also be reasonable to consider switching unn Teet) to a reverse-KL Keeerse-KLLs term and optimize with REINFORCE per general observations from (Ahmadian et al., 2024); however, we do not pursue this direction further here.)
π [yn [z)
p" (v1 > 82|z) " (M) + x (y2/z)
(12)
In general, any preference distribution expressible via (2) can be uniquely formed w.l.ogg. using (12); see Appendix E.1 for further details. Note also that from intuitive standpoint, a BTToppimal policy so-defined is such that p"(y1 > /2|z)) > 1/2 implies that x² [xr > T" (z:) Overall then, our notion of optimality is not arbitrary as it reflects the unique policy consistent with the assumed BT preference model and associated ground-trutt p".
4.2. Regression-based EXPO Objective
We next turn to our second proposed EXPO loss Exoo with structure more closely related to IPO, but a completely inde- pendent (and simpler) derivation and notably different final attributes to be discussed below and in Appendix E.5.
4.1. Compositional EXPO Objective
Consider a loss, composed of separable supervised and unsupervised factors, in the general form
Exro (TO, Tret, A) = (TO) + Xlunop(T.. Tatt)
Given that hu- man preference optimization seeks to find policy reflecting both the human preference distribution p" and a pre-trained reference policy Tet, it is natural to form a loss that simply penalizes deviations from the average of two factors, one representing the true preference distribution, the other rep- resenting the reference policy. The only unresolved issue how frame this averaging in tractable apples-to-apple manner, noting that pairwise preference distributions and reference policies operate in different probability spaces.
Establishing a Shared Probability Space:
E [Jw J-D DDr
dhup (1ollul [o(()(=))]
(13)
[dumoup [ [w) [v)]
+ AIE AlBymmmatllla)
where dup serves as a supervised penalty over labeled train- ing tuples {a, Nwr 313 while dunsup represents an additional regularization term independent of labeled preferences. We remark that objectives in the form of (13) are natural candi- dates for SGD given that all sampling is independent of A, unlike the regularized loss adopted by RLHE, which requires samples from To (y)
Fortunately, in deriving Lixo we already motivated the use of Ty to induce the parameterized preference distribution po(2|y1, x) = Po(111 > V/ 12lzz). Analogous to (14) we may then also define (1) > y2|=)) to facilitate averaging in a shared probability space with p" (vı > y2(2z), From here, revised goal is simply to produce policy such that the induced preference distribution po(n > y2 is close to a weighted average of p" (y1 > y2, and (1r > y2 (z), all of which are directly comparable as preference distributions in the same probability space.
Supervised Term: After first defining
(a Re (c) + (e)
if = =1 1
po(2|y1, 22)
(14)
Na
if 2 = 0
(an to
we then consider the supervised term (T))
E yn p
KKLL [p" (z|8) 1/, x))opp /2,
To(H)(z) (x) )]
= E (vw.th --r
log 1
(15)
Regression to Weighted Average: Per the developments from above, we explicitly penalize deviations of pe from a weighted average of p" and pert via the EXPO loss
Te(
Please see Appendix E.4 for the derivation of this key equiva- lence. Importantly, because the KL-ddieerennc is minimized iff p" (2|g11 y2. æ) = pe (=(2/1 V2, IT). the optimal solution to (Tg) will necessarily recover the BT-optimal distribution unlike minimization of an arbitrary reward; Section 4.3 will formalize this notion.
Coxoo ( Teat A) = E. E (( (
(polum  > m1/z))
[Daallhe > w/(z) + (1 - >)pp (vro > mlbi))
27 (17)
6

---

Explicit Preference Optimization: No Need for an Implicit Reward Model
with À E [0, 1]. Therefore, by design this loss favors solu- tions such that (11 > y2(z)) 2
or explicitly interpolating between an optimal policy and reference policy. Moreover, these limitations persist re- gardless of how optimality is defined. In contrast, we have herein derived new EXPO preference objectives (outside the broad, existing QPO family) that not only navigate around the aforementioned shortcomings, but do so w.r.t. a prin- cipled specification of the optimal policy x². Namely, by design our EXPO objectives preserve the unique policy aligned with the ground-truth preference distribution p", and likewise interpolate between this policy and Tet.
(M > y2 ) + (1 - 2)p > =222), (18)
where the relative importance of Pier versus p" is modulated by A. From here, although we do not have access to '², and therefore cannot directly compute (17) in its current form using (12), the following result provides a workaround: Proposition 4.1. The loss from (17) satisfies
(EXPo(TO: (To Teg. 2)
We conclude with a remark regarding the underappreci- ated role of learning constraints when interpreting many of the most popular QPO family members. Specifically, the core reparameterization from (6) that establishes DPO as an idcalized instantiation of RLHF is based on the uncon- strained optimization problem from (5); likewise for the analogous reparameterizatioos adopted by IPO and S-DPO. However, as rigorously investigated in (Kong et al., 2025), once widely-adopted learning constraints are introduced dur- ing preference optimization (e.g., early stopping, weight de- cay, etc.), the once-transparent motivational association with RLHF can be obscured. In this regard, we emphasize that none of the derivations used to motivate EXXPO (To; That, A) variants rely on unconstrained optimization to form a repa- rameterized objective. As such, the inevitable introduction of such constraints in practice does not compromise the EXPO origin story. In other words, since EXPO is not based on any implicit association with RLHF in the first place, adding constraints that might otherwise compromise such an association poses no issue.
E a0) (po(uw > wx(r)
(19)
(XPrrtir > yi ve(z) + (1 (1-2))) 10)) 21
As all quantities within the revised expectation from (19) are known in closed form, we can now easily obtain unbiased estimates of the gradient via sampling as needed for SGD optimization. While E'xo bears some resemblance to the quadratic IPO loss from (9), the EXPO derivation is more transparent, requiring no reparameterizatioon tricks nor de- pendencies on unstable limiting hyperparameter behaviors; see Appendix E.5 for further details. And importantly, both fExoo and fExoo possess key attributes that IPO (and other related DPO-like methods) fail to achieve as discussed next.
4.3. Shared Properties of EXPO Objectives
In this section we denote ( Tirr X) as either EXPO loss, namely, fixoo instantiated using (15) or (16) or tExo as defined via (19); the two main results we now present apply equally to either.
5. Empirical Validation
Proposition 4.2. Assume a n² satisfying (12), and un- der the same setup Theorem 3.1, let #xx00 == arg minz, LExPo( Taf. " 2). Then #gxPo x² for all x E dgood including in cases where dist[aparo, m²1 dist(T x²) f') x E dbed
We first present a series of synthetic experiments adapted from (Azar et al., 2024) (which in deriving IPO served as initial motivation for our work) to highlight aspects of EXPO behavior vis-à-vis our proposed interpolation and preservation desiderata. As the most relevant published points of reference, we contrast with DPO, IPO, and S-DPO; for the latter we choose the Sensen-Shannon divergence, which next to the reverse-KL implicitly assumed by DPO, performed well in prior experiments (Wang et al., 2024a). We then push beyond (Azar et al., 2024), which presents no real-world validation of IPO or other methods, and compare our EXPO framework using the Anthropic Helpfulness and Harmlessness (HH) real-world preference dataset (Bai et al., 2022a; Ganguli et al., 2022). For space considerations, some experiment details, including hyperparameters and training setups, are deferred to Appendix C. We note here though that in producing EXPO results on synthetic data we adopt f results with Exxo converge similarly.
Per this result, minimizers of Expoo (Tg, Tea A) are capable of preserving zet in regions dgood where performance is strong relative to a BT-optimal n², while concurrently im- proving performance in other areas where it is not. Figure visualizes this unique EXPO capability.
Proposition 4.3. LExPo (wo; Tags X) satisfies the SIC with n. satisfying (12).
Figure contrasts the EXPO-obtainable SIC property with the WIC achieved by prior methods. Note that the figure illustrates EXPO instantiated via E but reflects fExo equally well for the revised interpolation range À [0, 1].
4.4. Final Contextualization w.r.t. Prior Models
In Section we established that a broad class of DPO-related models are incapable of either preserving optimal policies
Interpolation Tests: As in (Azar et al., 2024) we con- sider the bandit setting with a discrete space of three re-
7

---

Explicit Preference Optimization: No Need for an Implicit Reward Model
DPO
IPO
f-DPO
EXPO
1.0
π (va)
n (vo)
n (yc)
melya)
molyo)
nolyc)
5000 10000
5000
10000
10000 20000 30000
0
5000
10000
Training Steps
Training Steps
Training Steps
Training Steps
Figure 3: Support for Sections 3.2 and 4.3 interpolation analysis. Dashed lines represent BT-optimal preference probabilities π², while solid lines are model learning curves for X = 10 (small). Only EXPO converges to πT,, others converge to T degenerate solution with no generative diversity; instead all mass concentrated on just a single response at odds with the ground-truth, BT-optimall policy underlying the data.
DPO
IPO
f-DPO
EXPO
1.00
0.75
π ref
T ref
Tef
ref
nolya)
0.50
π
nolys)
nolyc)
0.25
0.00
5
0
-5
5
-5
(oy(A)
log(A)
log(A)
log(A)
Figure 4: Further support for interpolation analysis. Each plot displays the final converged probability distributions Ta (v) across varying Å (small to large) under the same conditions as in Figure 3. As X becomes small, only EXPO converges to the BT-optimal policy n"; see left-hand side of each plot. The others converge to the mode the optimal policy consistent with expectations and Figure 3. Meanwhile, as grows large all methods converge to Wari right-hand side each plot. See also Appendix A.4 for the corresponding convergence curves with large fixed A. For intermediate values EXPO naturally interpolates between '² and The unlike the other approaches.
sponses/actions {Va, Vb, ye} and create dataset of la- beled response pairs as (Ma, }, (3h, NeJ. {Me, No} }. i.e., a total ordering consistent with the BT model. Preferences are assigned via ground-truth p" (y1 > 32) computed using (12) with x (Ba) = 0.6, π (16) = 0.3, and T" TVe)) = 0.1. Furthermore, again following (Azar al., 2024) we form our trainable policy (3)) softmax [0.] with 0 e R optimized using Adam over each different preference loss. Results using small À 10-5 are shown in Figure 3, where we observe that EXPO (we use closely converges to the BT-optimma ground-truth solution, while DPO and IPO converge to T" (the mode of π²) consistent with Proposi- tions 3.4 (DPO), 3.5 (IPO), and 4.3 (EXPO), as well The- orem 3.6 which applies to f-DPO. Additional interpolation results convergence traversing different À are depicted in Figure 4; here observe that only EXPO smoothly interpolates between x² and met-
Preservation Tests: above to include two input prompts {Tg. 2s chosen such that Hg E dgood and 2n E dind sampled with equal proba- bility. We then specify the corresponding response space Vxag) = {yga, Bgb: yge M(rx) = {ybas V66- Ybe} and prompt-dependent probabilities (see Appendix C.1). For the reference policy we )π(()/=,,) *)((x,) and (u)())))))))))))))))))))))))))) * n" (y]ax)) We generate pair-wise preference data before, only Results shown in Figure are in direct accordance with Theorem 3.1 and Proposition 4.2, whereby EXPO is the only approach that preserves strong policy with prompt Ty E dgood while at the same time improving performance relative Re for I4 E dbad over all
We next modify the setting from
with prompt-dependen responses.
Finally, to ex- plore EXPO in a real-world scenario, we train a Pythia 2.8B
Testing on Real-Worle Preference Data:
8

---

Explicit Preference Optimization: No Need for an Implicit Reward Model
Good Cases
Anthropic-HH Dialogue Win Rate vs Chosen
0.25
0.20
0.15
0.10
IPO
0.00
EXPO (comp)
-5
-3
-2
-1
logA
0.50
Sampling
Bad Cases
IMDb Win Rate Chosen
dist(n" (y/t)) MedlyHH)]]
0.30
DPO distin (yjo) mely(x0]
0.95
distin" (yje), matypel
ADPO distim moype)
0.50
0.20
EXPO distir" (y(x), molpx0]
Bete
0.80
0.10
0.75
0.05
0.70
DPO
0.00
0.65
IPO
EXPOIComp)
EXPO6)e))
log
0.60
0.50
100
Sampling Temperature
Figure 5: Support for Sections 3.1 and 4.3 preservation analysis while varying x. In the top plot prompts are drawn from dgood where πe = =,,, and yet as À is reduced existing methods produce policy that increasingly deviates from n.. In contrast, in the bottom plot prompts are drawn from dead, here improved performance over Ter occurs only at values below the dashed blue line. Unlike prior approaches, broad Ả range allows EXPO to both improve upon Thr on these bad cases, while also simultaneously preserving n on the good cases.
Figure 6: Win rate comparisons on real-world datasets An- thropic HH and IMDb; see Appendix A for other examples and additional baselines.
dialogues. Results in Figure show a significant improve- ment using both EXPO variants. We emphasize that our comparisons cover both helpfulness and harmlessness (see Appendix C.2), whereas the original DPO paper (Rafailov et al., 2024) only tests the former.
model (Biderman et al., 2023) on the Anthropic Helpful- ness and Harmlessness (HH) preference dataset (Bai al., 2022a; Ganguli et al., 2022) and the IMDb dataset (Maas et al., 2011; Wang et al., 2024a). Notably, the Anthropic HH dataset is the largest benchmark used in (Rafailov et al., 2024). Following their settings, we first execute supervised fine-tuning (SFT) using you values as the target response. We then use this SFT model as Tet for training DPO, IPO and EXPO. Given that alignment results (our focus) from (Wang al., 2024a) already show that reverse KL (i.e., DPO) works best among /-divergences, we do not com- pare with more f-DPO selections here and other recent unpublished baselines (see Appendix B). We use GPT-4 to evaluate the win rate of the generated responses from each model against the chosen the on the test set for single turn
Additional Real-World Results: In Appendix A present additional experimental testing covering other base- line models outside of the QPO class, real-world data, and larger LLM architectures. We also evaluate the sample diver- sity of model outputs after preference optimization, where we might expect EXPO to have some advantage.
6. Conclusions
In this work we have introduced EXPO convenient sub- stitute for DPO and related preference optimization schemes. Rather than relying on RLHF-based reparameterizztions and implicit rewards, EXPO is predicated on explicit ob- jective functions satisfying intuitive desiderata that prior approaches do not achieve.

---

Explicit Preference Optimization: No Need for an Implicit Reward Model
Impact Statement
model. In International Conference on Machine Learn- ing, pp. 2160-2177. PMLR, 2022.
Aligning the output of LLMs with human preferences has obvious, well-documented benefits. However, there nonethe- less remains the risk that tools designed to improve LLM responses could be repurposed for nefarious aims. For ample, preference data labels could potentially be modified to train models, using preference losses such as ours, that intentionally produce toxic content.
Bradley, R. A. and Terry, M. E. Rank analysis of incom- plete block designs: L. the method of paired comparisons. Biometrika, 39(3/4):324-345, 1952.
Bubeck, S., Chandrasekaran, V., Eldan, R., Gehrke, J., Horvitz, E., Kamar, E., Lee, P., Lee, Y. T., Li, Y., Lund- berg, S., Nori, H., Palangi, H., Ribeiro, M. T., and Zhang, Y. Sparks of artificial general intelligence: Early experi- ments with gpt-4, 2023.
References
Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S.,
Chang, Y., Wang, X., Wang, J., Wu, Y., Yang, L., Zhu, K., Chen, H., Yi, X., Wang, C., Wang, Y., al. A survey on evaluation large language models. ACM Transactions Intelligent Systems and Technology, 15(3):1-45, 2024.
Anadkat, S., et GPT-4 technical report. arXiv preprint arXiv:2303.08874 2023.
Ahmadian, A., Cremer, C., Gallé, M., Fadace, M., Kreutzer, J.. Üstün, A., and Hooker, S. Back to basics: Revisiting REINFORCE style optimization for learning from human feedback in LLMs. arXiv preprint 24Xv:24022.14400 2024.
Chen, C., Liu, Z., Du, C., Pang, T., Liu, Q. Sinha, A.. Varakantham, P., and Lin, M. Bootstrapping language models with DPO implicit rewards. arXiv preprint AKKY::24060997600 2024.
Cui, G., Yuan, I.., Ding, N., Yao, G., He, B., Zhu, W., Ni, Y., Xie, G., Xie, R., Lin, Y., et al. Ultrafeedback: Boosting language models with scaled ai feedback. In International Conference on Machine Learning, pp. 9722- 9744. PMLR, 2024.
AI@ Meta. Llama 3 model card. 2024. URL hettpa://githie coommmet--i 11ama/ /1lama3/ blob/maing MMM//MME_CAAAAA
Amini, A., Vicira, T., and Cotterell, R. Direct pref- erence optimization with an offset. arXiv preprint a0Xiv:2402.10571, 2024.
Ding, N., Chen, Y., Xu, B., Qin, Y., Hu, S., Liu, Z., Sun, M., and Zhou, B. Enhancing chat language models by scaling high-quality instructional conversations. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 3029-3051, 2023.
Azar, M. G., Guo, Z. D., Piot, B., Munos, R., Rowland, M., Valko, M., and Calandriello, D. A general theoretical paradigm to understand learning from human preferences. In International Conference on Artificial Intelligence and Statistics, pp. 1447-4455 PMLR, 2024.
Dubois, Y., Galambosi, B., Liang, P., and Hashimoto, T. B. Length-controlled alpacaeval: A simple way to debias automatic evaluators. arXiv preprint 3xXxv:440400447. 2024.
Bai, Y., Jones, A., Ndousse, K., Askell, A., Chen, A., Das- Sarma, N., Drain, D., Fort, S., Ganguli, D., Henighan, T., et al. Training helpful and harmless assistant with rein- forcement learning from human feedback. arXiv preprint 2XX4:2204400866.. 2022a.
Ethayarajh, K., Xu, W., Muennighoff, N., Jurafsky, D., and Kiela, D. Kto: Model aligmment as prospect theoretic optimization. arXiv preprint arXiv:2402.01 2024.
Feng, D., Qin, B., Huang, C., Zhang, Z., and Lei, W. Towards analyzing and understanding the limitations of DPO: A theoretical perspective. arXiv preprint arXiv:4404.06266 2024.
Bai, Y., Kadavath, S., Kundu, S., Askell, A., Kernion, J., Jones, A., Chen, A., Goldie, A., Mirhoseini, A., McKin- non, C., et al. Constitutional ai: Harmlessness from ai feedback. arXiv preprint a7X1v:2212.08073, 2022b.
Gallegos, I. O., Rossi, R. A., Barrow, J., Tanjim, M. M.. Kim, S., Demnnncurr, F., Yu, T., Zhang, R., and Ahmed, N. K. Bias and fairness in large language models: A survey. arXiv preprint
Biderman, S., Schoelkopf, H., Anthony, Q. G., Bradley, H., O' Brien, K., Hallahan, E., Khan, M. A., Purohit, S., Prashanth, U. S., Raff, E., al. Pythia: A suite for ana- lyzing large language models across training and scaling. In International Conference Machine Learning, pp. 2397 2430. PMLR, 2023.
2023.
Ganguli, D., Lovitt, L.., Kernion, J., Askell, A., Bai, Y., Kadavath, S., Mann, B., Perez, E., Schiefer, N., Ndousse, K., et al. Red teaming language models to reduce harms: Methods, scaling behaviors, and lessons learned. arXiv preprint www.iv:209.078... 2022.
Bong, H. and Rinaldo, A. Generalized results for the exis- tence and consistency of the mle in the bradley-terry-Ine
10

---

Explicit Preference Optimization: No Need for an Implicit Reward Model
Gorbatovski, A., Shaposhnikov, B., Malakhov, A., Sur- nachev, N., Aksenov, Y., Maksimov, Balagansky, N., and Gavrilov, D. Learn your reference model for real good alignment. arXiv preprint arXiv:2444.09656, 2024.
Meeting of the Association for Computational Linguis- ties: Human Language Technologies, pp- 142-150, Port- land, Oregon, USA, June 2011. Association for Com- putational Linguistics. URL http:////// /wwp::///wwwwwwwww.ww/./ aclweb.
Greenberg, H. and Pierskalla, W. A review of quasi-convex functions. Operations research, (9(7):153--1570 1971.
Meng. Y., Xia, M., and Chen, D. Simpo: Simple preference optimization with a reference-free reward. arXiv preprini arXiv:2405.1477 2024.
Guo, D., Yang, D., Zhang, H., Song, J., Zhang, R., Xu, R., Zhu, Q. Ma, S., Wang, P., Bi, X., et al. Deepseek-R1: Incentivizing reasoning capability in LLMs via reinforce- ment learning. arXiv preprint arXiv:2501.12944 2025.
Muldrew, W., Hayes, P., Zhang, M., and Barber, D. Active preference learning for large language models. arXiv preprint arXiv:2402.08111 2024.
Hong, J., Lee, N., and Thorne, J. Orpo: Monolithic pref- erence optimization without reference model. arXiv preprint ar XrXv:22433.07611 2024.
Ouyang. I.., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A.. al. Training language models follow instructions with human feedback. Advances in neural information processing systems, 35:27730-2777 277444 2022.
Im, S. and Li, Y. Understanding the learning dynam- ics of alignment with human feedback. arXiv preprint arXiv:2403. 18742, 2024.
Pal, A., Karkhanis, D., Dooley, S., Roberts, M., Naidu, S., and White, C. Smaug: Fixing failure modes of pref- erence optimisation with DPO-positive. arXiv preprint arXiv:2402.132288, 2024.
Kashif.
2024.
Kingma, D. P. and Ba, J. Adam: A method for stochastic optimization. arXiv preprint arXiv: 14Xx:::412.698) 2014.
Park, R., Rafailov, R., Ermon, S., and Finn, C. Disentan- gling length from quality in direct preference optimiza- tion. arXiv preprint arXiv:2403. 19/59, 2024.
Kong, L., Hu, X., He, T., and Wipf, D. Common learn- ing constraints alter interpretations of direct preference optimization. In The 28th International Conference on Artificial Intelligence and Statistics, 2025.
Peng, X. B., Kumar, A., Zhang, G., and Levine, davaatage--eiihter regression: Simple and scalable off-policy reinforcement learning. arXiv preprint arXi::9910001177 2019.
Kwon, W. Li, Z., Zhuang, S., Sheng, Y., Zheng, L., Yu, C. H., Gonzalez, E., Zhang, H., and Stoica, I. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles, 2023.
Peters, J. and Schaal, S. Reinforcement learning by reward- weighted regression for operational space control. In Proceedings of the 24th international conference on Ma- chine learning, pp. 745-750, 2007.
Li, X., Zhang, T., Dubois, Y., Taori, R., Gulrajani, I., Guestrin, C., Liang, P., and Hashimoto, T. B. Alpacae- val: An automatic evaluator of instruction-folloiinng models. https:/ htt///gttubbb com/tatsu lab/
Rafailov, R., Sharma, A., Mitchell, E., Manning, D. Er- mon, S., and Finn, C. Direct preference optimization: Your language model is secretly a reward model. Ad- vances Neurul Information Processing Systems, 36, 2024.
Alpaca_eval,, 1, 2023.
Li, Z., Xu, T., and Yu, Y. Policy optimization in rlhf: The impact of out-of-preference data. arXiv preprint axXiv:2112.1558411, 2024.
arXiv:2312
Ramamurthy, R., Ammanabrolu, P., Brantley, K., Hessel, J., Sifa, R., Bauckhage, C., Hajishirzi, H., and Choi, Y. Is reinforcement learning (not) for natural language processing: Benchmarks, baselines, and building blocks for natural language policy optimization. arXiv preprint arXiv: 2r.0022210.01241 2022.
Lin, Y. Seto, S., Ter Hoeve, M., Metcalf, K., Theobald, B.-J., Wang, X., Zhang, Y., Huang, C., and Zhang, T. On the limited generalization capability of the implicit reward model induced by direct preference optimization. arXiv preprint arXiv:2409. 03650, 2024.
Rubenstein, P., Bousquet, O., Djolonga, J., Riquelme, C., and Tolstikhin, I. O. Practical and consistent estimation f-divergences. Advances Neural Information Pro- cessing Systems, 32, 2019.
Maas, A. L., Daly, R. E., Pham, P. T., Huang, D., Ng, A. Y., and Potts, Learning word vectors for sen- timent analysis. In Proceedings of the 49th Annual

---

Explicit Preference Optimization: No Need for an Implicit Reward Model
Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O. Proximal policy optimization algorithms. 2017.
Zhao, Y., Joshi, R., Liu, T., Khalman, M., Saleh, M., and Liu, P. J. SLiC-HF: Sequence likelihood calibration with hu- man feedback. arXiv preprint arXiv:2305. 10425, 2023b.
arXiv preprint
Shao, Z., Wang, P. Zhu, Q., Xu, R., Song, J., Bi, X., Zhang, H., Zhang, M., Li, Y., Wu, Y., et al. DeepseekMath: Push- ing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.05300 2024.
Ziegler, D. M., Stiennon, N., Wu, J., Brown, T. B., Radford, A., Amodei, D., Christiano, P., and Irving, G. Fine-tuning language models from human preferences, arXiv preprint arXiv: 1909: 080.9, 2019.
Song, Y. Swamy, G., Singh, A., Bagnell, J., and Sun, W. The importance online data: Understanding preference fine-tuning via coverage. Advances in Neural Information Processing Systems, 37:12243-1222 12270, 2024.
Stiennon, N., Ouyang, L.,, Wu, J., Ziegler, D. M., Lowe, R., Voss, C., Radford, A., Amodei, D., and Christiano, P. Learning summarize from human feedback, 2020.
URL https://arxiv. org/abs, 2009.
Swamy, G., Choudhury, S., Sun, W. Wu, Z. S., and Bag- nell, J. A. All roads lead to likelihood: The value of reinforcement learning in fine-tuning. arXiv preprint arXiv:2503.01067, 2025.
Tajwar, F., Singh, A., Sharma, A., Rafailov, R., Schneider, J. Xie, T., Ermon, S., Finn, C., and Kumar, A. Prefer- ence fine-tuning LLMs should leverage suboptimal, on-policy data. arXiv preprint arXiv:2404. /4367, 2024.
Tang, Y., Guo, Z. D., Zheng, Z., Calandriello, D., Munos, R., Rowland, M., Richemond, P. H., Valko, M., Pires, B. Á., and Piot, B. Generalized preference optimization: A unified approach to offline alignment. arXiv preprint arXiv:2402.05749. 2024.
Wang, C., Jiang, Y., Yang, C., Liu, H., and Chen, Y. Beyond reverse KL: Generalizing direct preference optimization with diverse divergence constraints. International Con- ference on Learning Representations, 2024a.
Wang, F., Zhou, W., Huang, J. Y., Xu, N., Zhang, S., Poon, H., and Chen, M. mDPO: Conditional preference opti- mization for multimodal large language models. arXiv preprint XrXiv:2406. 11839, 2024b.
Xu, S., Fu, W. Gao, J., Ye, W., Liu, W., Mei, Z., Wang, G., Yu, C., and Wu, Y. Is DPO superior PPO for LLM alignment? A comprehensive study, arXiv preprint arXiv:240..10719 2024.
Zhao, W. X., Zhou, K., Li, J., Tang. T., Wang, X., Hou, Y., Min, Y., Zhang, B., Zhang, J., Dong, Z.,, Du, Y., Yang, C., Chen, Y., Chen, Z. Jiang, J., Ren, R., Li, Y., Tang, X., Liu,
Z., Liu, P., Nie, and Wen, J.RR. A survey of large lan- guage models. anXiv preprint 203.2. 2023a. htta:://arriv.orrg/////////00
URL wwt.thtp://// rxiv. wwwpvii..rgrb///ooeeo
12

---

Explicit Preference Optimization: No Need for an Implicit Reward Model
A. Additional Experiments
In this section we present experimental results covering additional baseline models, real-world datasets, and larger LLM architectures. We also include testing of the sample diversity of model outputs after preference optimization.
A.1. Comparison with More Baselines on Anthropic HH Dataset
The SimPO approach (Meng et al., 2024) proposes what amounts to setting Tear to a constant in the DPO loss (i.e., the reference policy is not actually used at all). while also introducing two additional modifications: inclusion of margin offset analogous to the procedure from (Amini et al., 2024), and (ii) use of length normalization to DPO-like models even if not explicitly stated (Kashif, 2024). Such orthogonal modifications can be equally applied to our proposed EXPO framework well in future work to boost performance, a downside being additional hyperparameter tuning to handle the margin offset. Moreover, by excluding the reference policy completely, special care must be taken to avoid overfitting, i.e., training until convergence is never feasible with SimPO.
commonly been applied
Meanwhile, the KTO model (Ethayarajh et al., 2024) was originally designed to target scenarios where labeled data pairs are not necessarily available at all. but instead, only individual examples labeled as either good or bad. Using ideas from prospect theory, KTO can be effective in practice, although like SimPO requires an additional tunable hyperparameter. Moreover, the proposed KTO training loss is not explicitly optimized, pass- mass-throuuh gradients are used for certain factors, which could in principle compromise convergence.
Collectively, SimPO and KTO fall outside of the QPO family we investigate, and are largely outside of our scope. However, we nonetheless perform additional experiments using these baselines for context given their growing influence. We adopt the same setup the main paper for the Anthropic HH dataset. Results are shown in Figure 7, where our EXPO approach remains the top performer, even without the benefit of additional byperparamettes/penaall factors as with SimPO and KTO.
Anthropic-HI Dialogue Win Rate vs Chosen
0.6
Win Rate
0.4
SFT
SIMPO
KTO
(camp)
EXPO (reg)
0.2
0.25
0.50
0.75
1.00
Sampling Temperature
Figure 7: Comparison with additional baselines (outside the QPO family). Both KTO and SimPO benefit from the inclusion of additional hyperparameters (i.e., tunable degrees-of-fie offrreedomm)))mmmm that define the corresponding training objectives. And yet EXPO still remains competitive, even with just single hyperparameter associated with its loss.
A.2. Diversity Results with Anthropic HH Dataset
To evaluate response diversity, we followed the protocol from Section 5.3 of (Tang et al., 2024). For each prompt in the Anthropic HH test set, we generated 25 responses using nucleus sampling (p = 0.95, temperature=1.0), Since certain experimental details such as the maximum number of tokens-tve were not reported in the original paper, we adopted our own settings for these parameters. Diversity was measured using three complementary metrics: predictive entropy (normalized by token length to avoid bias toward longer responses), self-BL.e self-BLEU,, and distinct-n scores. As shown in Table 1, EXPO
13

---

Explicit Preference Optimization: No Need for an Implicit Reward Model
variants consistently demonstrate higher diversity compared to DPO across all metrics.
Table I: Diversity results on Anthropic HH dataset.
Method
Normalized Entropy 1
Self-BIEU 4
Distinct-1 f
Distinct-2
DPO
0,033
0.93
0.018
0.29
EXPO (comp)
0.036
0.90
0.025
0.35
EXPO (reg)
0.035
0.88
0.023
0.33
A.3. Testing Larger Models with AlpacaEval 2
We evaluate models using the Llama-3-Base-8B (AI@ 2024) on the widely-used open-ended instruction-foolloinng benchmark AlpacaEval (Li et al., 2023). AlpacaEval 2 consists of 805 questions from five datasets and employs GPT-4 Turbo as both the baseline model and the judge model. Following AlpacaEval 2's evaluation protocol, we report results for both the raw win rate (WR) and the length-controlles win rate (LC) (Dubois et al., 2024), the latter designed to reduce the influence of model verbosity. Detailed experiment settings are provided in Appendix C.4. Results in Table 2 demonstrate that EXPO (reg) achieves a notable improvement in congth-contoolled win rate compared to other methods.
Table 2: Win rate and length- controlon (LC) win rate results on AlpacaEval 2. Note that results here may not be directly comparable with prior work for two reasons: (i) differences in batch size due to computational constraints, and (ii) updates to AlpacaEval upon which all evaluations depend.
Method
Win Rate
Win Rate (LC)
SFT
6.33
4.04
DPO
14.62
16.71
IPO
11.16
13.31
EXPO (reg)
14.64
20.32
A.4. Additional Results with Synthetic Data
DPO
IPO
F-DPO
EXPO
n ("(a))
n (yb)
0.4
n lycc
molys)
nolys)
0.2
nolyc)
5000
10000
5000
10000
0
10000 20000 30000
5000
10000
Training Steps
Training Steps
Training Steps
Training Steps
Figure& 8: Converged probability distributions of Ta (y) for DPO, IPO, f-DPO and EXPO with large À (here = 100). All methods immediately stabilize around wen the initialization point, as expected. This figure can be viewed as the complement of Figure 4 in the main text; the only difference the size (small vs large).
14

---

Explicit Preference Optimization: No Need for an Implicit Reward Model
B. Extended Related Work
In this section we call attention to additional recent work proposing modifications of the original DPO paradigm, providing relevant analysis of DPO and other preference model properties, and/or comparing online versus offline preference optimization. We believe these efforts to be complementary to our contribution, as well as the many existing DPO-like extensions by others discussed in the main body of our paper (i.e., within the broad QPO family we define) and in Appendix A.I above (i.e., SimPO and KTO).
Algorithmic Enhancements to DPO: There exist multiple DPO extensions that involving supplementing the original loss from (7) with additional penalty factors targeting potential failure modes. For example, based on the observation that DPO may exhibit a decrease in accuracy when applied to preference data with small edit distances between responses, the Smaug framework (Pal al.. 2024) augments the DPO loss with an additional factor designed to maintain high ligeliiessoooss in such cases. Meanwhile, sensitivity response lengths are investigated in (Park et al., 2024), where as a counter- measure, the DPO loss is supplemented with a penalty on length differences between winning and losing responses. It has also been observed that not all preference pairs in a training data set are equal, with some preference gaps larger than others, As a mitigation strategy for this discrepancy, the ODPO approach (Amini et al., 2024) introduces a preference offset term during model training. While all of these methods have their merit, they each require an additional key hyperparameter that must be tuned.
Somewhat differently, the ORPO algorithm (Hong et al., 2024) proposes an alternative to DPO that combines an odds ratio-based penalty with conventional negative ligeliieldoood SFT (i.e., supervised fine-tuning) loss. The appeal here is that separate SFT and preference alignment phases are no longer required. Another deviation from DPO is proposed in (Gorbatovski et al., 2024), whereby the reference policy itself is no longer fixed, but iteratively updated during training. And finally, DPO has recently been extended to handle multimodal data (Wang et al., 2024b).
Analysis of DPO: Topics addressed by recent work include analysis of DPO learning dynamies (Im & Li, 2024), the impact of ofteff-preference data on estimation errors (Li et al., 2024), and the disproportionate rates with which the DPO loss gradients favor reducing the probability of dispreferred responses relative to increasing the probability of desired responses (Feng et al., 2024). Broader consideration of preference optimization spanning various DPO-based and RLHF-based approaches is presented in (Tajwar et al., 2024).
Online vs Offline Methods: While some recent work indicates that online preference learning (as in the original RLHF that trains using samples from ng) may at times produce better results than offline learning (as in DPO or EXPO which only use samples from Rg)) we consider three reasons to justify ongoing exploration of the latter:
The inference that online preference learning is generally superior is to date often derived based on testing with QPO approaches, and sometimes only DPO or even earlier techniques alone; see for example (Guo et al., 2025; Shao et al., 2024; Song et al., 2024; Swamy et al., 2025; Tajwar et al., 2024: Xu et al., 2024). Hence the open possibility remains that offlinn methods that mitigate DPO deficiencies (like our EXPO) might reset the scales in certain settings relative to online alternatives. And crucially, some of the specific arguments provided for why DPO outperformed by online approaches like PPO do not apply to EXPO, For example, in (Xu et al., 2024) it is argued that a key DPO limitation is that it does not exploit unlabeled prompt-only data, which can then lead to undesirable biases. However, as we point in Section 4.1, EXPO (the compositional variant (ExP)) can naturally incorporate such unlabeled prompt-only data.
Many references that argue in favor of online preference learning nonetheless suggest that DPO is still valuable when reformulated as an online or related on-policy iterative approach; see for example (Chen et al., 2024; Lin et al., 2024; Song et al., 2024; Swamy et al., 2025: Tajwar et al., 2024). Some even explicitly advocate for generalizing beyond DPO to online/iterative versions of IPO and the like (Chen et al., 2024; Tajwar et al., 2024). Hence our analysis of QPO models in general, and EXPO in particular, remains quite relevant in wide variety of revised online or otherwise iterative settings.
Even if we concede that offline preference learning is at times sub-optimal, in many scenarios it is still considerably simpler, possibly even with convergence guarantees. As such, for prototyping or resource- constrained environments offline learning may enjoy some durable advantages.
15

---

Explicit Preference Optimization: No Need for an Implicit Reward Model
C. Experiment Settings
This section describes relevant experimental details not covered the main text. Code is available at com/
C.1. Synthetic Data Testing Details
For the preservation and interpolation tests, we train models with the Adam optimizer (Kingma & Ba, 2014) and clip the gradients via a max norm of 10. Experiments are run on a single A10 GPU. As models are trained until convergence in a synthetic environment, there is negligible trial-to-trial variability. We also adopt the Exro loss from Secion 4.1 for obtaining EXPO performance; results with Exoo converge similarly.
For the interpolation tests, we use batch size of 20 and choose (wr( = 0.4, (Wb) 0.4, and (Br( ) = 0.2. We use learning rate of 0.001 for DPO, IPO and f-DPO and 0.0005 for EXPO; we train DPO, IPO and EXPO for 1000 epochs and f-DPO for 3000 epochs as we observed that it converges more slowly.
For the preservation test, we choose
{xg)) = {yga Mob, Noot: y(Ib) = {Yaas yob, ybo
π (Uga (gg) = 0.6; x (Ugelzzg) = 0.3; x' (Vyel [gg) = 0.1;
(20)
(Ybal (26) = 0.4; x (866l17)) 0.2; ' (yoe = 0.4.
And for the reference model we select The (2r6) = 0.6, merr(uns = 0.2 and (ur( = 0.2. We randomly sample examples for good and bad prompts respectively. The model parameters are 0 E R2x3 and we set the values Tg and zy to the vectors [1, 0) and [0, 1] respectively.
C.2. Anthropic HH Testing Details
For the results in Figure 7, we train the base SFT model for 2 epochs and all the other models for I epoch, using a learning rate of x 10-6 and a batch size of 40. We emphasize that the precise role of the hyperparameter differs for DPO, IPO, and EXPO (with two variants based on e E and fE see Sections 2.2, 2.3, and 4 respectively. We set X 0.1 for DPO and IPO. For EXPO (reg), we set À = 0.2; also found that increasing À to 0.5 did not substantially alter EXPO performance. For EXPO (comp) we used X = 0.05 since again, its influence is different between the two variants. Regarding the expanded results in Figure 7, for SimPO we use À = 2 (denoted 3 in the SimPO paper) and an offset of y = 1. For KTO, we set their 3 = 0.1 and use ÅD = Au = 1 for the desirable coefficient and undesirable coefficient.
All training was conducted using an 8x A 100 40G GPU instance and the Adam optimizer (Kingma & Ba, 2014). Our code for these experiments is based on a simple adaption of the official DPO repo from GitHub.* Additionally, we note that the Anthropic HH benchmark upon which these experiments are based has been released under MIT License agreement."
To evaluate the performance, we calculate the win rate single-turn dialogues in the test set using GPT4 (gpt-40). For this purpose we adopt version of the prompt used in Rafailov et al. (2024) modified to consider both helpfulness and harmlessness as shown in Figure 9. Additionally, to avoid inadvertently introducing potential bias of GPT-4 towards selecting "Response A" or "Response B", we uniformly at random set the former as the generated response associated with Yeu Additionally, for assessing win rate, we apply vLLM (Kwon et al., 2023) to randomly sample responses from the Anthropic HH dataset by setting top_k = 60, top_p = 0.8, max_tokene = 256, and repetition_penalee = 1.1.
C.3. IMDb Testing Details
We train all models for one epoch, using a learning rate of 1 x 10-6 and a batch size of 64. For DPO, IPO, and EXPO (comp), we set À = 0.1, while for EXPO (reg), À 0.01. All training was conducted using an Ax 40G GPU instance and the Adam optimizer (Kingma & Ba. 2014).
To evaluate the performance, we calculate the win rate on single-turn dialogues in the test set using GPT-4 (gpt-4o) For this purpose, we adopt version of the prompt used in (Muldrew et al., 2024). All other evaluation details follow those used for the Anthropic HH dataset.
prtttpa://gthhee com/ Phttps: : htt.ttbbtttttt/////g com/
zat in
www//ott//tteeeeeeee htt///ttrropcccsc cetthhrppcs/hhh-riiie http:/hhhhhhhhhhhhoooc
16

---

Explicit Preference Optimization: No Need for an Implicit Reward Model
I have question and two responses from different AI models. Please analyze the responses and provide your assessment of which response is nore helpful and harmless overall, Consider factors such as:
How well each response answers the original question
The accuracy truthfulness of the information provided
Potential blases, stereatyping, or controversial views expressed
Any potential for harm or dangerous suggestions
Clarity, coherence
usefulness
response
Evaluate the responses their merits, ignoring superficial features like length ar writing style, After analyzing both, please explain which response you believe is better and why.
The original question was: (query)
Response {a>
Response
(b)
provide a one-sentence comparison the two responses and explain which you feel is more helpful and harmless. SECOND, on a new line, state only "A" or "B" to helpful and harmless. response should use
response
fornat:
Comparison: <one-sentence comparison and explanation> More helpful and harmless: <"A" or "B">
Figure 9: The prompt used for evaluating the win rates the generated responses against the chosen responses for single turn dialogues on the test set of Anthropic HH dataset. This prompt is modified from Rafailov et al. (2024) to consider both helpfulness and harmlessness.
C.4. AlpacaEval 2 Testing Details
For the results in Table 2, the UltrCChat-20kk dataset (Ding et al., 2023), the reference model. Preference optimization is then conducted on the Ulrraeeedback dataset (Cui et al., 2024), initializing from the SFT model. For DPO and IPO, we adopt the hest-performing hyperparameters reported in (Meng et al., 2024). Specifically, for DPO, we set À = 0.05 and use a learning rate of 5 x 10- For IPO, we set = 0.5 with the same learning rate. For EXPO (reg), we use À = 0.2 and a learning rate of 5 x 10 10-7, To generate responses, we use vLLMM with top_p= 1.0, temperature 0.9, and max_ _ew___e_____ _el_eee__ttle _e____et_tttesssstetteee 4096.
the SFT model from (Meng al., 2024), fine-tuned from the Llama-3- B-Be model on
D. Technical Proofs
D.1. Proof of Theorem 3.1
Definition D.1. We define labeled human preference data D., some Drr as introduced via (1), satisfying the following additional properties:
1. The prompts drawn from Der split between two disjoint support partitions dgood and dhadd i.e., E dgood U dbodd with probability one, with dgood n /baa = 0.
2. For each prompt dgood U dbad within Der. the preference distribution filling Der maintains support over single (prompt-dependent) response pair y.).
3. Pair-wise preferences dictated by ground-truth BT model satisfying p" (y1 y2(2) (0, 1) for all dgood U dded
4. We have p² x y2 (ooodd = (y1 val for at least one good E doood and bud E abad.
Although the second specification above can naturally be relaxed to address more general scenarios, doing so unnecessarily complicates the presentation without providing sufficiently compelling additional insight. Additionally, for convenience below we adopt dist(..)) indicate an arbitrary distance measure.
17

---

Explicit Preference Optimization: No Need for an Implicit Reward Model
Theorem (Restated formal version) Assume preference data Der that satisfies Definition D.I. Farthermore, assume reference policy ang such that aig = x²" for E dgood and dist(T x²) > 0 for x: E dbad, where xt" is BT BTTootimal policy. It follows that for any selection of (v, ,4 X), if
dist m9 (
dist(Tedt H²) for a E d
(21)
then
dist " > for x E dgood
(22)
where =g=0= arg ming, Copo(TT (T,, Trds 1 pc A).
The proof proceeds as follows. With some abuse/imprecision of notation, we first define
(v) (2:
no (122 (V2(12))
u(y 1/2 2) =
Ter (V1 (x))
/4
(23)
Next, per the assumptions of the theorem statement and Definition D.1, we have that the QPO loss decouples as lopo( (To Tee, v/, X)
E (w 1 (N To (T((wll))
To (x) Teet (31 x:) .1)
11 (Vol(
(24)
= EgnD, (p"(xx (v1 > w2 (uV1 32. x), A
+ p" (w2 > 21 k u(y2 M1, x)) 2))
E ggodd (01 > y2 (o)r))(((( 82. x), A] + p" (yy2 - yn
(vr > V2l=) V2, -), A] (v2 + vn x V2. V2, 1)) 21].
B Bewdd
Now consider a single prompt aoo drawn from dea and other good drawn from dgood, where p" p²(1 > Ve (000d) p" (yy y22 p.bad). In order to find a mo such that dist|mo, x²] < dist(mer x²), it must be the case that no To (y)ab * (y bad), ), which then implies that u(yi, 3/2 bad) of O. To achieve this, (1b, u, A) must be chosen such that
arg
min u( Shoady
(v1 > 1/2 baddd jolu (M(1 42, badd, 2] + p= (1/2 > 1/1 (bl M2 x bad) >]
#
(25)
However, to simultaneously maintain To (y02000 marlyy Tr" (yola it must also be true, for the same fixed (1,,u u, A) tuple, that
arg
min a(n (ooodd
(v1 yal good ( J y2, good) A1 + p" (y22 > 8ı (ood b uu(1, y2, good) x1 = 0. (26)
p2
But this a contradiction, as the respective arguments that minimize (25) and (26) will be identical. Hence (25) true then dist[# x²1
0 when computed over dgood
D.2. Proof of Proposition 3.4
DPO lower limit: Given our assumption that < (111 > 82/z)) 1, it follows that an optimal finite reward r" x) (-oe, oo) exists. Moreover, given that & and y are drawn from finite sample spaces, there will exist finite maximum and Furthermore,
minimum optimal rewards, i.e., r" (y, E (-B for some B
lim arg min bRLHE (To, Tit r", A) arg max Ey-so (ul=x) [r" (y. z)] n (g/z). me
(27)
na
Additionally, given that the data are generated by (1), we also know that the same optimal reward satisfies
arg min CoT (ro),
(28)
To
which is independent of Ta.. However, without constraints on Me. there also exists bijection between policy and reward such that
To (3(22) A log arg min Eur (Xlog
}
X log Ter(ylz)) = r.
(29)
Teat ( (z)
18

---

Explicit Preference Optimization: No Need for an Implicit Reward Model
Hence the DPO reparamettrrization produces the policy given by (5) with r". From this point then observe that
lim
(v, = x (v(lr)
(30)
(xx)
noting that for any > 3 we have exp
[
exp
X
exp
(a-0)
X - O. Hence have fulfilled the
requirements of the lower limit.
DPO upper limit:
The upper limit follows trivially from the fact that for any bounded reward
lim 1 ((x)
[
1
Tea (y)(y) exp
rgy, x)
(y)(y) exp(0) Tea-
(31)
X
Z(x)
D.3. Proof of Proposition 3.5
Establishing the upper and lower limiting values for IPO follows similar pattern to the proof of Proposition 3.5. However, because the IPO reward is bounded between zero and one by definition, we ultimately do not require any constraint on p" (111 > (z) as we did for DPO.
D.4. Proof of Theorem 3.6
We first define
ò = arg min E (vw a us p M1 B. no, Trer); 2).
(32)
P
Now suppose that for given tuple (ym, n1 x) we observe
no (Yar (2) TT B(A)
p(y yt, I, na, Tnd)) log
(33)
To x (t) [x)
for some optimal ñ and fixed X E (0, oc), where B(A) E (0, 00) is a finite value dependent on through the definition of vb. Therefore, have that
* exp B(A) + log (Mer fo(w]z)
(34)
Tra(y)
Obviously this ratio will depend on wer for any fixed B()). To satisfy the SIC though, in the limit À - the optimized policy de must be independent of wet and converge x². However, the only way for âee be independent of गंत is if limx-+0 (() toc. But if so, only the WIC is achievable, not the SIC.
D.5. Proof of Propositions 4.2 and 4.3
These results both follow directly from the original design of EXPO losses. First consider the case where lexPo (To, Thet A) ge Ex (To, aa, A). Regarding Proposition 4.2, given that Teat π² for all dgood then for the unsupervised term we have
arg min E, me (
KTL (m)](((())))))]])((())))
(35)
And for the supervised term we have
arg min E
KLLpp" (2/81, U2. 2))opo U2, x]]]
(36)
Hence overall, for any x E dgond no = xt will be optimal for any X, as this selection independently optimizes the constituent terms. Moreover, this optimality is independent of optimization over x E badd which retains the flexibility to achieve
19

---

Explicit Preference Optimization: No Need for an Implicit Reward Model
solutions with dist[=p >² < dist[Tets J" From this Proposition 4.2 immediately follows. Additionally, Proposition 4.3 stems from the same basic line of reasoning. For completeness, note that when À - 0, only the supervised term will be minimized (which recovers the BT-optimal policy above), while when X + co, the unsupervised term will dominate the optimization (which transparently produces Teat).
And finally, both Propositions 4.2 and 4.3 fEXPO (To, Tat, A) = Exoo (To, Tet X) transparently follow from the construction of (17) and an analogous line of reasoning as detailed above, only now with lambda E [0, 1].
D.6. Proof of Proposition 4.1
Recall from Section that the expectation over tuples (we yr, 2) ~ Der is equivalent to an expectation over the revised generative process
(2, Der ~ (=ly), 22, (91,32) (u((v))) x Dz.
(37)
From this expression, first z is drawn from some prompt distribution Pz, then conditioned on this prompt, two responses {1h, are drawn from training policy Te (which could be equal to Tua but the specific choice does not impact the derivations or conclusions below). And finally, the indicator variable = = Ily > y21 1/2 x] is adopted to reflect the conditional preference distribution provided by human annotators; namely, 2 indicates 1/1 > y/, while & = 0 if y/ > y1. In this way, p" (z|y1, 3/2 .) = p" (y) > y2(z)) per the original specification Section 2.
Given (37), our task reduces to showing that, for any fixed {y1, 1/2 2)., the remaining expectation over = is such that an equivalence between (17) and (19) is maintained. To demonstrate this, introduce the simplifying notation
ug == py > y2|z), Une == Perl > 12(zz), u := p > 121z), and
(38)
v/g = (02 > v1|0) = uo
Waar (re > yılar) = thes xt = p" (U2 + va(a) 11 u².
(39)
Note that for non-trivial generative process, we are implicitly assuming that y1 * y2: we will return to this assumption below.
Now for any fixed (y1, (31,8/22)) 2) within (17), the remaining expectation over = given by
u (ua Duther + (1 - ))²² + v' (ubo - Deuher + (1 - ))²(=1)
(ug- /Awer + +11 X)u"))
(40)
ii 2 Auuee (1 X)u"l us + [Auler + (1 -A)""(?
Meanwhile, analogously for (19), we have the expectation
u" (us- DAuted (1 2)))² + u" (u/g DAuler + (1 -)]))²
u² (uo- AuGe + 1)² 2 + (1 u²) (1 ug Xeeeet 1+ 1)
u² (ug- Auter + 1)² + (1 u²) (ug - > 2²
ug + [2u" (-Xuee + 1) + (1 u²) (-2Auuz)) ug C
up 2 [11 A)u" + Ausal ug C,
(41)
where C' is constant. As this expression is equivalent to (40) irrelevant constants notwithstanding, we have completed the proof, with the exception of addressing the possibility that yı = y2. However, if y/2 then it follows by definition that Ug = u/g = Uher = u/g = "' = x² = 1/2. This scenario does not impact (40) and results in only an inconsequential constant being added to (41) which can be absorbed into C.
20

---

Explicit Preference Optimization: No Need for an Implicit Reward Model
E. Additional Supporting Derivations and Analysis
E.1. BT-Optimal Policies and the Derivation of (11) Note that
explr" (vi x))
(v1,
(v1 821z)
[(x)
explr (y1 x)] + expl (V2,
explr" V1
expl (22
((2)
2(2)
x² (11 (z) n" イ" ()) + म (322 x))
(42)
where n² (2(22)
(V1 and ((x) explr (u.z)). The policy x² so-defined is necessarily BT-optimal by ((x)
construction. We also remark that an optimal reward r* is generally unique when conditioned on some form of normalization (Bong & Rinaldo, 2022), hence X² will also be unique by virtue of dividing by (zx).
From here then we have
arg max Eyrnr (V" [v,
arg max E
yyrro (uli)) [r" (v,
|expfr" (11, x)]
arg max E ( To
((x)
arg max E (y) (u|xr] Re
1 if arg maxy x (v' (x) otherwise
(43)
which is the definition of .
E.2. Illustration of Key x O Distinction
Per the discussion in Appendix 1. we know that optimizing a policy solely to maximize rewards will trivially produce T.. namely, a degenerate policy assigning all mass to the single response with maximal reward. While there may exist cases where this desirable, the goal is an actual generative model capable of diverse output responses reflecting gradations of human preferences, is generally not.
As simple hypothetical example, suppose we have three candidate responses {ya yo, ye3 the bandit setting, where yo and y/ similarly preferred by humans, but yo very slightly more so, while y/ heavily dispreferred. To align with this preference distribution, optimal policy x" may ideally generate y/ and y/ roughly equally, while ys will be avoided. Meanwhile, reward maximization will generate yn with probability one and both and ys will be excluded, i.e., n produces no diversity at all.
E.3. Further /-DPO Analysis
f-PDO represents a novel generalization of DPO, but there remain certain aspects worth considering.
Minima that ignore the reference policy: cases QPO are expressible in the form
Consider general f-DPO losses as described in Section 2.4, which as special
Eyeo(To (To, Tet log (A(())..'AA)
(44)
(v.. log (x8'
(o((u)lz)) Xf' Tepe (ww (x))
no (3) Toer (31 (x:) .1)
E
In addition to the requirements on f to form an f-divergence, to produce a valid f-DPO loss per Theorem 1 from (Wang et al., 2024a) must be that f invertible with 0 & domain of f.. Therefore the domain of f will be (0, oc) and (u( 00 as u because of convexity. But if this is the case, upon inspection of (44) we observe that when
21

---

Explicit Preference Optimization: No Need for an Implicit Reward Model
T (x) + O, then for any fixed my (Nw (r) > 0 the input argument the logistic function a()) 1+expl [.]] infinity, pushing the output to and subsequently minimizing the corresponding negative-log factor. And the global
will converge
optimum can be achieved independent of the value of Tea.
E.4. Derivation of (15)
dup ( (To Tedt)
E (tt.ma))mmr((ll)))))
KLL (p" (zly1, U2, =))lpo V2. 22)
(11.22) www
"xlp"((l log (=|v) 122 x)) + C
[y 32 (vll))
pr (= = 1 22. x)) log pa = 1l11, U2,
EE (2n.
p² 0/ U2 x)) log (a = 0/8/ y2 xx)
- E (v wa m ( aw p 22= 1ly1, U2 x) log po 2 = 1 x)
=
+ p² = 1/ vn, log = 1/122 1/, 2))
= - -Ey (yw ar)
log pa (z = 1/ Mt: xt)
Te (20)
- -E (B
log
(2( + Te(ull))
To (30 (x))
E (y in 20 Dan
log 1
(45)
To (1 (2)
where C' is constant independent of 0. Additionally, the third-to-last equality stems from the definition of how tuples
{Mh Vi, x } are sampled. In particular, for given pair {y1, }, by definition proportion (² = 1/311 82,2) of the time Yw = y1, while a proportion p" (= = 0/ V2, x) = p² (2 = 1 [82, U1, x) of the time you = y2. Hence
p" (z = 1 122 po = 11 x2.x) x) + p"( (= 1/22, W1, x) log po = 1l v1
log = W/r 2:
(46)
when the latter is averaged over the preference distribution.
E.5. Further Comparisons between IPO and EXPO
Although originally designed to address certain DPO shortcomings, IPO introduces other potential sources of instability. In particular, the inherent dependency factors log (re((((z=)) [z)] and (2A)-1, both of which may have arbitrary magnitudes, entails that the Lipschitz constant of (9) may be arbitrarily large, destabilizing SGD. Moreover, if hypothetically Tee already closely approximates an ideal policy, minimizing (9) can actually lead to a degradation in quality (see Section 3.1).
In this regard, may be tempting to consider reparameterizing the IPO loss analogous to f so as to accommodate À E [0, 1] and reduce potential pathways for instability. But this is not viable option for multiple reasons. First, IPO is based on reparamcterizations analogous to (5) which, for pair of responses y1 * y2 and the reward from (8) leads the key equivalence
Two (y1 Ta (V2(2))
1 (vr) (12 x)].
log
(47)
Tiro (2/2 x)) na [z]
The entire motivation for IPO is then to approximate this equivalence by minimizing the loss
(1os
Te( (T) xx)
1 (rno(11, 2)) - (voo(
Big vx (
(48)
no To (z:
(r (x))
22

---

Explicit Preference Optimization: No Need for an Implicit Reward Model
which equates to minimizing (9) per results in Azar al. (2024). However, if we modify (9), this equivalence is compromised, L.., unavoidably break the foundational connection with an RLHF--iike loss and implicit reward upon which IPO is entirely based in the first place.
Secondly, even if we are willing to jettison the original IPO motivation, and prefer to adopt an IPO-related hybrid loss, minimicking the weighted average of EXPO form such as
To (We (ar) ( xx)
Tea (Muw (x)
x (yw (x) (yn (x) |1)
[TTO; mett A)
E (ym un Dn
log
A log
(1 X) log
(49)
Teat (31 (z)
practical implementation remains problem. In particular, expectations, it is no longer possible to establish the equivalent of Proposition 4.1 when using (49). Therefore an unbiased practical implementation becomes problematic, unlike the original IPO or EXPO-
consequence of Jensen's ' inequality and properties of
23


Reviews:
[{"review_id": "xV3wxx4lX0", "reviewer": "Reviewer_zMiU", "summary": "This paper proposes a framework for aligning language models with human preferences without relying on implicit reward models. EXPO addresses limitations of existing methods like DPO and IPO, which suffer from suboptimal regularization and interpolation issues. The authors propose two variants: a compositional loss and a regression-based loss, which explicitly balance human preferences and reference policy adherence. Theoretical analysis shows EXPO preserves optimal policies in regions where the reference model performs well while improving elsewhere, and it satisfies strong interpolation criteria (SIC). Experiments on synthetic and real-world datasets (Anthropic HH, IMDb) demonstrate EXPO outperforms DPO, IPO, and other baselines in win rates and policy preservation.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for checking many of the technical details of our paper, including proof and empirical materials, while acknowledging the novelty of our approach in addressing underappreciated limitations in prior preference optimization methods.  We respond to points of critique as follows.\n\n**Comment:**\n*Limited experiment results on real world tasks. Would like to see its performance on Llama-3-8B on AlpacaEval 2.*\n\n**Response:**\nDuring the short rebuttal window, we tried fine-tuning a Llama-3.1-8B model using Lora to reduce the computational burden, and indeed EXPO is better than DPO in this limited setting.  However, complete testing with further baselines or a full parameterization is unfortunately not feasible at this time.\n\n**Comment:**\n*The same algorithm name ExPO has already been used in \"Weak-to-Strong Extrapolation Expedites Alignment\".*\n\n**Response:**\nThanks for the notice, we could easily change the name to avoid any confusion.\n\n**Comment:**\n*Can EXPO handle non-BT preference models?*\n\n**Response:**\nGood question.  In principle yes, but some theoretical results would need to be reconsidered.", "claims_and_evidence": "EXPO avoids suboptimal regularization is validated by synthetic experiments showing EXPO converges to BT-optimal policies, while DPO/IPO converge to degenerate solutions.\n\nEXPO satisfies SIC is validated by empirical results show EXPO interpolates smoothly between $\\pi_{ref}$ and $\\pi^*$, unlike DPO/IPO.", "methods_and_evaluation_criteria": "EXPO has been tested on Anthropic Helpfulness and Harmlessness (HH) preference dataset and IMDb dataset, as well as synthetic experiments with controlled preference distributions.", "theoretical_claims": "Theorems 3.1 and 3.6 are derived, showing QPO methods (DPO/IPO) cannot preserve optimal policies or satisfy SIC. Propositions 4.2–4.3 demonstrate EXPO’s advantages. I checked the proof of Theorem 3.1.", "experimental_designs_or_analyses": "I checked both the LLM tasks and the synthetic data tasks, and the ablation studies and did not find any issues.", "relation_to_broader_scientific_literature": "EXPO builds on DPO/IPO but eliminates their dependency on implicit rewards via explicit regularization. It connects to RLHF’s KL-regularization but avoids multi-stage training. The work fills a gap in ensuring policy preservation and interpolation, addressing underappreciated limitations in prior preference optimization methods.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "Strengths:\nNovel explicit regularization approach enhances interpretability and control.\n\nWeaknesses:\nLimited experiment results on real world tasks. Would like to see its performance on Llama-3-8B on AlpacaEval 2.", "other_comments_or_suggestions": "The same algorithm name ExPO has already been used in \"Weak-to-Strong Extrapolation Expedites Alignment\".", "questions_for_authors": "Can EXPO handle non-BT preference models?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "oNvVhjdl6d", "reviewer": "Reviewer_udv5", "summary": "This paper works broadly on offline preference optimization methods, the most canonical of which is DPO, discusses a common weakness shared by all of these methods, and proposes a method that fixes this problem. The paper argues that DPO's uniform regularization to the reference policy creates problems: assume the space of prompts can be divided into two subsets, **set A**: one where the reference policy (also usually the policy one starts with during fine-tuning) is already close to optimal, and **set B**: one where the reference policy performs poorly. The paper then shows that DPO and its variants in general improves performance of the trained policy on $B$, but it inevitably comes with performance degradation on $A$. The paper then proposes their method, EXPO, that removes the reparaterization trick (designing an implicit reward function based on log probabilities), and shows that their method can improve performance on set $B$ while preserving performance on **set A**.\n\n\n# Updates after Rebuttal\n\nThe authors have answered my concerns. Thanks for the additional experiments, specially the large scale experiments on Llama-3-8B.\n\nI maintain my score at 3, and recommend acceptance of this paper.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for pointing out that our paper is clear, based on convincing evidence, and supported by interesting and relevant theory.  The reviewer also provided many constructive comments; with limited space to reply, we prioritize the main critiques as follows:\n\n**Comment:**\n*Methodology is sound, but warrants testing on larger scale models, e.g., in 8B parameter range as opposed to 2.8B range.*\n\n**Response:**\nWe agree with the reviewer that the community is trending towards experimentation with larger models, particularly on less theoretically driven papers.  But even so, many recent papers still lean heavily on 2.8B-sized models or smaller for analyzing preference methods and drawing comparative conclusions between approaches.  As representative examples, the DeepSeek GRPO paper [14, Figure 5] uses a 1.3B model to compare online vs offline preference learning; likewise 1.4B-2.8B models are used for a similar purpose in [9, Table 1] and [13, Figure 3].  We also note that other more theoretical works such as the IPO paper (Azar et al.,2024) contain no real-world experiments at all and yet remain highly influential to the field.  In any event, we are not disputing the value of larger models; instead we are merely advocating that high quality contributions are still possible without them, especially when granting that compute resources and timing constraints are often limiting factors (as is the case for us within the rebuttal period).\n\nEven so, during the rebuttal window we tried fine-tuning a Llama-3.1-8B model using Lora to reduce the computational burden, and indeed EXPO is better than DPO in this limited setting. However, complete testing with further baselines or a full parameterization is unfortunately not feasible at this time.\n\n**Comment:**\n*Recent evidence suggests that online preference learning generally performs better ... so relevance of offline approaches may be waning.*\n\n**Response:**\nThis is a very reasonable point to consider, and addressing it helps to further advance the relevance of our paper.  In this regard, the suggested references from the reviewer are extremely helpful.  Our overall response is three-fold:\n\n1) The inference (from references [5,6,12,13,14] and others) that online preference learning is generally superior is to date mostly derived based on testing with QPO approaches, and usually DPO alone (or in the case of [6], an even earlier rejection-sampling approach).  Hence the open possibility remains that offline approaches that mitigate DPO deficiencies (like our EXPO) might reset the scales relative to online alternatives.  And crucially, many of the specific arguments provided for why DPO is outperformed by online approaches like PPO do *not* apply to our EXPO.  For example, in [5] it is argued that a key DPO limitation is that it does not exploit unlabeled prompt-only data, which can then lead to undesirable biases.  However, as we point on on Lines 322-324, EXPO can naturally incorporate such unlabeled prompt-only data.\n\n2) Many references that argue in favor of online preference learning nonetheless suggest that DPO is still valuable when reformulated as an online or related on-policy iterative approach; see for example [6,9,10,11,13].  Some (e.g., [11]) even explicitly advocate for generalizing beyond DPO to online versions of IPO and the like.  Hence our analysis of QPO models in general, and EXPO in particular, remains quite relevant in a wide variety of revised online settings.\n\n3) Even if we concede that offline preference learning is sub-optimal, in many scenarios it is still considerably simpler, possibly even with convergence guarantees.  As such, for prototyping or resource-constrained environments offline learning may enjoy some durable advantages.\n\n\n**Comment:**\n*Relation to literature, missing references ... [10] should be cited and discussed, but it is not. [11] should also be cited ...*\n\n**Response:**\nReferences [10] and [11] are empirically-driven studies that advocate strongly for iterative versions of DPO, e.g., exploiting DPO implicit rewards over multiple on-policy rounds.  See also our comments above that include reference to [10,11].  Overall, these works are quite complementary to our own, and we can definitely cite them in our revision.  Likewise for other references the reviewer has kindly provided.\n\n**Comment:**\n*Questions 1-5 ...*\n\n**Response:**\n1. Memory consumption is the same as DPO.\n2. Using unlabeled samples for (16) may help some, but further study is needed.\n3. SIC and WIC differ strongly as $\\lambda$ becomes small.  For SIC the minimal loss equates to an optimal preference distribution, while for WIC we only obtain the *mode* of an optimal distribution.  Appendix E.2 contains further details.\n4. We have not tested this (and it could depend on the implementation), but intuition suggests that EXPO may still provide a benefit in some cases.\n5. Multi-turn extensions of EXPO represent an interesting direction for future work.", "claims_and_evidence": "The claims made in this paper is clear and has convincing evidence.", "methods_and_evaluation_criteria": "The proposed methods make sense to me. I think the paper is lacking in benchmark datasets and models that it shows results on. The paper mentions multiple times that they use the same model and same datasets used in the original DPO paper. While that is great, the DPO paper is 2 years old at this point, and the same level of evaluation is no longer sufficient in my opinion. The LLM used in real-world experiments is Pythia-2.8B, which is very outdated by this point. \n\nI would request the authors to show evidence that their method works with:\n\n1. At least one more dataset, say UltraFeedback [1], which has ~60K prompts and (preferred, dispreferred) pairs, so it has reasonable scale.\n\n2. At least one more LLM of 8B parameter size, say Llama-3.1-8B [2]. (It is possibly good to fine-tune the base model instead of the instruction tuned model, to show improvements resulting from EXPO).\n\nThe paper mentions that they use the same evaluation setup as the original DPO paper. But I would like to point out that the original DPO paper is around 2 years old now, so a paper submitted for review now should use more recent benchmarks/models for it to be accepted.", "theoretical_claims": "I took a brief look over the theoretical proofs, but since my expertise is not theory, I cannot comment on the correctness of the proofs. However, the results surely are interesting and relevant.", "experimental_designs_or_analyses": "Yes, I checked the soundness of the experiments, they seem correct to me and follows what is common in relevant literature.", "relation_to_broader_scientific_literature": "The paper deals with offline preference optimization methods for fine-tuning language models, a canonical example of which is direct preference optimization or DPO [3], or more broadly other direct alignment methods [4]. The paper studies specific problems with this class of methods, some of which were discussed by these papers:\n\n1. DPO, even the online variant, underperforming PPO due to poor regularization [5]\n\n2. DPO often reduces the log-likelihood of the preferred response, leading to unintentional unalignment [6, 7, 8]\n\n3. DPO's offline regularization to the reference policy can be problematic and fixed using an online way of regularization [9]", "essential_references_not_discussed": "There have been prior work that has discussed the problems with DPO's implicit reward formulation. For example, [10] should be cited and discussed, but it is not. [11] should also be cited. I would request the authors to find more papers that talk about implicit reward formulations.\n\nAlso, all the papers mentioned in **Relation To Broader Scientific Literature** section above should be cited in related works, but it does not seem to be case.", "other_strengths_and_weaknesses": "# Strengths\n\nMany other recent works have also found out different problems with DPO and similar methods. This paper uncovers a major problem with DPO-like methods, shows it both theoretically and empirically. I like the paper's theoretical results.\n\n# Weaknesses\n\n1. The most major weakness is the scale of LLM experiments in this paper. A 2.8B old LLM does not tell practitioners enough about how much to trust this paper's results. \n\n2. It is pretty clear from recent papers such as Deepseek [12] and other works [5, 6] that offline preference learning algorithms simply don't perform as well as their online counterparts. Another recent work such as [13] confirmed this for RLHF, where we do not have access to gold reward during training (unlike Deepseek, which trains on verifiable problems with gold reward). It is very unclear what would be the value of this paper (or in general any other paper that gives an n-th variation of an offline preference learning algorithm), given that the community might shift to more online methods like GRPO [14] anyway.\n\n**If the authors can show that their method works with a newer model with around 8B parameter count, I would be very happy to increase this paper's score to 4.**", "other_comments_or_suggestions": "No other comments that I can think of.", "questions_for_authors": "# Questions\n\n1. How does the memory and computational cost compare between EXPO, DPO and other key variants? It would be good to report these numbers on an 8B model against a standard dataset.\n\n2. How does the log-probability of preferred and dispreferred responses change throughout training? Essentially I am interested in Figure 17 of [6] but for EXPO.\n\n3. Is there any intuitive explanation of the difference between WIC and SIC conditions? I am not entirely sure if I grasp their difference and significance.\n\n4. Does this paper's main observation, that performance improvement on set of bad prompts also result in performance decrease on set of good prompts, also happen for online preference optimization techniques? Why or why not? (**Note: this is outside the scope of this paper and the authors should not be penalized for discussing this in their paper, but I am curious and would love it if they have any explanation.**)\n\n5. Most preference learning algorithms work on the single-turn setting. However, LLMs are being increasingly used in more and more multi-turn setting. Algorithms like DPO has a simple multi-turn extension: one can just take loss over the agent action tokens and discard the log-probabilities of the environment tokens, see [15]. Is there a way to extend EXPO to the multi-turn setting?\n\n\n# References\n\n[1] UltraFeedback: Boosting Language Models with Scaled AI Feedback, https://arxiv.org/abs/2310.01377. **Dataset link: https://huggingface.co/datasets/HuggingFaceH4/ultrafeedback_binarized**\n\n[2] The Llama 3 Herd of Models, https://arxiv.org/abs/2407.21783.  **Model link: https://huggingface.co/meta-llama/Llama-3.1-8B**\n\n[3] Direct Preference Optimization: Your Language Model is Secretly a Reward Model, https://arxiv.org/abs/2305.18290\n\n[4] Scaling Laws for Reward Model Overoptimization in Direct Alignment Algorithms, https://arxiv.org/abs/2406.02900v1\n\n[5] Is DPO Superior to PPO for LLM Alignment? A Comprehensive Study, https://arxiv.org/abs/2404.10719\n\n[6] Preference Fine-Tuning of LLMs Should Leverage Suboptimal, On-Policy Data, https://arxiv.org/abs/2404.14367\n\n[7] Unintentional Unalignment: Likelihood Displacement in Direct Preference Optimization, https://arxiv.org/abs/2410.08847\n\n[8] Iterative Reasoning Preference Optimization, https://arxiv.org/abs/2404.19733\n\n[9] The Importance of Online Data: Understanding Preference Fine-tuning via Coverage, https://arxiv.org/abs/2406.01462\n\n[10] On the Limited Generalization Capability of the Implicit Reward Model Induced by Direct Preference Optimization, https://arxiv.org/abs/2406.01462\n\n[11] Bootstrapping Language Models with DPO Implicit Rewards, https://arxiv.org/abs/2406.09760v2\n\n[12] DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning, https://arxiv.org/abs/2501.12948\n\n[13] All Roads Lead to Likelihood: The Value of Reinforcement Learning in Fine-Tuning, https://arxiv.org/abs/2503.01067\n\n[14] DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models, https://arxiv.org/abs/2402.03300\n\n[15] From $r$ to $Q^*$: Your Language Model is Secretly a Q-Function, https://arxiv.org/abs/2404.12358", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "utwvn7MuBV", "reviewer": "Reviewer_tgLB", "summary": "This paper proposes a new direct preference optimization (DPO) method. The authors first formulate quasi-convex generalizations to unify some of existing DPO based methods. Then, they identify two limitations of existing DPO based methods under this formulation. One limitation is the failure to preserve optimal policies and the other one is suboptimal interpolation. To address these two limitations, the authors propose two new preference optimization objectives. Experiments on synthetic and real-world datasets validate the effectiveness of the proposed methods.\n## update after rebuttal\nThe authors' responses address my concerns. I am leaning towards accept.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the constructive comments, and address the main points as follows (grouping where appropriate).\n\n\n**Comment:**\n*First/Main limitation: I am not convinced by the first limitation of existing approaches. The considered setting seems a bit synthetic to me as it only considers two types of prompts: good and bad, where* $\\pi_{ref} = \\pi^*$ *for good prompts. However, it is very likely that $\\pi_{ref}$ is not optimal for all the prompts in the datasets. A more realistic setting is that the authors can prove the proposed methods can improve over over all the prompts without requiring* $\\pi_{ref} = \\pi^*$ *for good prompts.*\n\n\n**Response:**\nWe remark that it is commonplace for theoretical results to involve simplifying assumptions, otherwise many analytical steps become infeasible.  That being said, in the present circumstance the point is not necessarily to reflect the most realistic scenario possible.  Instead, the goal is to show that even in an idealized case DPO-like methods perform below expectations, the implication being that such subpar performance is unlikely to disappear even under broader conditions.  We also reiterate that our assumption (for Theorem 3.1) is not that $\\pi_{ref} = \\pi^*$ for all prompts, but rather, only for so-called ideal good cases.\n\n\n**Comment:**\n*Second limitation:  The synthetic evaluations support the limitation claim and the advantage of the proposed method over the existing method. However, more evidences are needed to demonstrate using the real-world dataset. For example, to address the second limitation. the authors can show the diversity of the proposed methods over existing methods while ensuring good generation quality.*\n\n**Response:**\nAs demonstrated in Figure 5 of the submission, our proposed method achieves high-quality generation on multiple real-world datasets. However, to quickly explore diversity per the reviewer's suggestion with limited time during the rebuttal period, we evaluate using the basic setup, sampling method and metrics from Section 5.3 of [this paper](https://openreview.net/forum?id=2cRzmWXK9N) (although some experimental details, like max_token are not reported in the paper; for these we adopt our settings).  Results are shown in the new table below, where generally EXPO displays higher diversity relative to DPO.  We also normalize the entropy by token length to avoid bias towards longer responses; other metrics are already implicitly normalized w.r.t. length.\n\n|     | **Normalized Entropy** $\\uparrow$ | Self-Bleu $\\downarrow$ | Distinct-1 $\\uparrow$ | Distinct-2 $\\uparrow$ |\n| --- | --- | --- | --- | --- |\n| DPO | 0.033 | 0.93 | 0.018 | 0.29 |\n| EXPO (comp) | **0.036** | 0.90 | **0.025** | **0.35** |\n| EXPO (reg) | 0.035 | **0.88** | 0.023 | 0.33 |", "claims_and_evidence": "1. I am not convinced by the first limitation of existing approaches. The considered setting seems a bit synthetic to me as it only considers two types of prompts: good and bad, where $\\pi^*=\\pi_{ref}$ for good prompts. However, it is very likely that $\\pi_{ref}$ is not optimal for all the prompts in the datasets. A more realistic setting is that the authors can prove the proposed methods can improve over $\\pi_{ref}$ over all the prompts without requiring $\\pi^*=\\pi_{ref}$ for good prompts.\n\n2. The synthetic evaluations support the limitation claim and the advantage of the proposed method over the existing method. However, more evidences are needed to demonstrate using the real-world dataset. For example, to address the second limitation. the authors can show the diversity of the proposed methods over existing methods while ensuring good generation quality.", "methods_and_evaluation_criteria": "The proposed methods are well motivated by the identified two limitations. However, my major concern is that the first limitation seems unreasonable to me. In practice, it is very likely that $\\pi_{ref}$ is not optimal for all the prompts in the datasets. A more realistic setting is that the authors can prove the proposed methods can improve over $\\pi_{ref}$ over all the prompts without requiring $\\pi^*=\\pi_{ref}$ for good prompts.", "theoretical_claims": "I didn't check the proof.", "experimental_designs_or_analyses": "The synthetic evaluations effectively highlight the advantages of the proposed methods in overcoming the two identified limitations. However, additional evidence is required to demonstrate these benefits on real-world datasets. For instance, to address the second limitation, the authors could showcase the diversity of the proposed methods compared to existing approaches while maintaining high-generation quality.", "relation_to_broader_scientific_literature": "The paper is broadly related to AI alignment.", "essential_references_not_discussed": "NA", "other_strengths_and_weaknesses": "NA", "other_comments_or_suggestions": "NA", "questions_for_authors": "Please refer to the above questions for comments.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "hxSZqZPu2x", "reviewer": "Reviewer_BiLh", "summary": "The paper presents a new objective to use for preference optimization that replaces DPO-like objectives. The new objectives (EXPO) are designed to address two issues with DPO-like objectives, (1) shifting the learned policy away from the reference policy when the reference policy closely matches the optimal policy and (2) smoothly interpolating between the optimal and the reference policy where the model's behavior should more closely match that of the reference policy as lambda increases. The analysis presented a generalized Quasi Preference Optimization framing for DPO-like objectives, and then breaks down how the structure of the objective contributes to the two limitations. Then the two EXPO algorithms are introduce with one a combination of multiple objectives and the other a regression objective. An empirical study is conducted to evaluate how well EXPO aligns with the objectives of not moving the learned policy away from the reference policy when the reference policy is of a high quality and smoothly interpolating between the optimal and reference policy based on the value of lambda. The evaluations are conducted on a synthetic dataset and two real world preference datasets. The conclusions are that EXPO better exhibits the two properties than the QPO objectives, and has a higher win rate on the real world tasks.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are appreciative of the reviewer's helpful comments.\n\n**Comment:**\n*For the right $\\lambda$, QPO can closely match EXPO ... results of Figure 4 support this ... so how beneficial is EXPO in real-world cases with tuned $\\lambda$.*\n\n**Response:**\nActually, there are no values of the QPO hyperparameter that can *simultaneously* match EXPO in both the good and bad input prompt regimes as depicted.  For example, consider IPO: for the good cases $\\log \\lambda > 0$ is optimal, while for the bad cases $\\log \\lambda \\approx -1$ is optimal.  We provide another illustration of this point at this  [link](https://anonymous.4open.science/r/ICML2025_rebuttal-15908/Figure4_extention_preservation.pdf); only change is adjusting $\\pi^*$. And overall, we regard reduced sensitivity to $\\lambda$ as a positive even in real-world scenarios, as hyperparameter tuning can be expensive.\n\n\n**Comment:**\n*Ambiguity of Figure 8 ... it looks like all methods converge to the optimal policy, not reference.*\n\n**Response:**\nFigures 7 and 8 are meant to be viewed in tandem (and we can add explicit text to better convey their relationship).  From Figure 7, when $\\lambda$ is large we observe training convergence to $\\pi_{ref}$ values (0.2 and 0.4).  Meanwhile in Figure 8 we show fully converged solutions for a range of $\\lambda$ values varying from small to large.  For large $\\lambda$ (right side of each plot) we observe that all models produce $\\pi_{ref}$ as expected.  Conversely, for small $\\lambda$ (left side of each plot) we see that only EXPO maintains the optimal policy.  Again, we can easily clarify this in a revised appendix.\n\n\n**Comment:**\n*What exactly about the QPO objectives is problematic and exactly how EXPO addresses the issue ... higher-level intuitive explanation would be helpful.*\n\n**Response:**\nGood suggestion.  We agree that accessible explanations are valuable for conveying some of the subtle messages associated with our work.  In this regard, the paragraph beginning on Line 172 (righthand column) may be helpful.  There we provide a more intuitive perspective for the specific QPO cases of DPO and IPO; similar ideas hold more broadly, but are decidedly more involved to present, hence the technical aspects of our submission.  There is also another natural entry-point for understanding where QPO objectives begin to fall short.  For simplicity, consider only the special case of DPO, which is advertised as producing the minimal solution to equation (4) instantiated with an optimal reward.  Next observe how (4) behaves as $\\lambda \\rightarrow 0$.  Basically, once the KL term is ignored, the remaining loss will be trivially optimized by a degenerate policy assigning *all* probability mass to a single response; see equation (43) for the derivation.  In contrast, the EXPO loss is explicitly designed to reflect the *full* optimal BT policy in this same limit (not just a mode).  Appendix E.2 provides further details regarding why this distinction is important.\n\n**Comment:**\n*... it would be beneficial to explicitly spell how IPO and DPO are expressed in terms of the QPO objective.*\n\n**Response:**\nGreat suggestion, and this is easy to include in a revised appendix.\n\n**Comment:**\n*Given that the similarity in performance between the QPO and EXPO methods varies with lambda, it would be beneficial to show how the different lambda values  impact performance on the real world tasks.*\n\n**Response:**\nAs we have clarified in a previous response above, QPO and EXPO performance is quite distinct as $\\lambda$ is varied across our synthetic experiments.  And while we do agree that further ablations with real-world data would be interesting, such testing incurs significant time and computational cost that we unfortunately cannot accommodate. That being said, in Appendix B.3 we mention $\\lambda$ stability for EXPO under a limited testing range.\n\n**Comment:**\n*It is not clear to me how Figure 3 addresses the interpolation goal ...*\n\n**Response:**\nIndeed Figure 3 only shows the interpolation extreme on the optimal policy side (i.e., when $\\lambda$ is small).  However, Figures 7 and 8 in Appendix B.2 complete the full picture, as we ran out of space in the main text.  For reference, there is also a pointer on Line 346 (righthand column).\n\n**Comment:**\n*Good to add the distance between the reference and optimal policies to the \"Good Cases\" plot in Figure 4 ...*\n\n**Response:**\nBy design, the distance between $\\pi_{ref}$ and $\\pi^*$ is zero in Figure 4, top plot.  Please also see Line 355 (righthand side) in the text.\n\n**Comment:**\n*How well tuned is lambda for Anthropic-HH and IMDB experiments ... relationship with Figure 4.*\n\n**Response:**\nPlease see our responses above regarding Figure 4, and the distinct performances of QPO vs EXPO once both good and bad cases are considered collectively.  In terms of tuning $\\lambda$ on real-world data, we only explored a few distinct values because of computational cost; see also Section B.3.", "claims_and_evidence": "The main claim of the paper is that the EXPO preference objective better meets the two goals of preserving the things the reference model already does well and smoothly interpolating between the reference and optimal policy that the QPO objectives. Experiments on synthetic data are used to specifically test how well EXPO meets the two goals relative to a subset of specific QPO objectives. The EXPO and QPO methods are then assessed using real data and win rate compared to the chosen responses in the corresponding dataset's test split.\n\nNot something mentioned in the paper, but from the results it looks like the benefit of the QPO objectives is they are less sensitive to the value of the lambda hyper-parameter than the QPO objectives. This means that there are values of lambda for which the QPO results closely match of the EXPO learned models. The results in Figure 4 support this. However, the results in Figure 5 (win rate comparisons) do not account for the performance differences with respect to the value of lambda in the case of \"real world\" data. So it is unclear how beneficial the additions of EXPO are in real world scenarios where lambda is carefully tuned.\n\nFollowing the paper and the claims requires following a lot of detailed proofs. It would be beneficial to include a higher-level intuitive explanation for the differences between the QPO and the EXPO objectives. \n\nIt is difficult to assess the claims made around Figure 8 without lines marking the optimal policy. If the lines for the optimal policy match those in Figure 3 then it looks like all methods converge to the optimal policy, not reference.", "methods_and_evaluation_criteria": "It is not clear to me what exactly about the QPO objectives is problematic and exactly how EXPO addresses the issue. \n\nIn motivating the QPO generalization, it would be beneficial to explicitly spell (can be in the appendix) how IPO and DPO are expressed in terms of the QPO objective.", "theoretical_claims": "I did not. My background is more empirical than theoretical", "experimental_designs_or_analyses": "- Given that the similarity in performance between the QPO and EXPO methods varies with the value of lambda, it would be beneficial to show how the different values of lambda impact performance on the real world tasks.\n- It is not clear to me how Figure 3 addresses the interpolation goal. The results are only provided in relation to the optimal policy, but the interpolation is in relation to the reference and the optimal policies.", "relation_to_broader_scientific_literature": "The paper attempts to address issues faced with DPO-like objectives.", "essential_references_not_discussed": "The correct papers appear to be discussed.", "other_strengths_and_weaknesses": "- The paper is well written and the text is easy to follow.", "other_comments_or_suggestions": "- It would be beneficial to either add the distance between the reference and optimal policies to the \"Good Cases\" plot in Figure 4 or describe where it is.", "questions_for_authors": "1. How well tuned are the lambda parameters for the Anthropic-HH and IMDB experiments? What values were used and how were they selected. This question pertains to my concerns around the QPO and EXPO methods having similar performance in Figure 4 for certain values of lambda.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper introduces EXPO, an explicit preference optimization method that addresses key limitations of DPO and IPO, such as suboptimal regularization and poor interpolation behavior. The theoretical analysis is strong, and reviewers appreciated the novel framing and provable advantages. While the initial experiments were limited to smaller models, the authors provided additional results with LLaMA-3.1-8B during the rebuttal, which helped alleviate concerns. All reviewers gave weak accept scores, citing strong theoretical contributions and clear writing. The paper is therefore recommended for acceptance.
Decision: Accept (poster)