Why DPO is a Misspecified Estimator and How to Fix It
Authors: Aditya Gopalan
Sayak Ray Chowdhury
Debangshu Banerjee
Venue: ICLR 2026 Oral (2026)
Keywords: Direct Preference Optimization
Reinforcement Learning
Reinforcement learning with human feedback
Primary area: foundation or frontier models, including LLMs
TL;DR: DPO is not sound by design and can fail due to misspecification, we fix it with careful analysis.

Abstract: Direct alignment algorithms such as Direct Preference Optimization (DPO) fine-tune models based on preference data, using only supervised learning instead of two-stage reinforcement learning with human feedback (RLHF). We show that DPO encodes a statistical estimation problem over reward functions induced by a parametric policy class. When the true reward function that generates preferences cannot be realized via the policy class, DPO becomes misspecified, resulting in failure modes such as preference order reversal, worsening of policy reward, and high sensitivity to the input preference data distribution. On the other hand, we study the local behavior of two-stage RLHF for a parametric class and relate it to a natural gradient step in policy space.  Our fine-grained geometric characterization allows us to propose AuxDPO, which introduces additional auxiliary variables in the DPO loss function to help move towards the RLHF solution in a principled manner and mitigate the misspecification in DPO. We empirically demonstrate the superior performance of AuxDPO on didactic bandit settings as well as LLM alignment tasks.

Published as a conference paper at ICLR 2026
WHY DPO IS A MISSPECIFIED ESTIMATOR AND HOW TO FIX IT
Aditya Gopalan
Sayak Ray Chowdhury Indian Institute of Technology Kanpur, India sayakre @ itt..ac.in
Debangshu Banerjee HP AI Research Bangalore, India
Indian Institute of Science Bangalore, India aditya @ issc.ac.in
debangshu.banerjeeee waw.rjee@hp.com
ABSTRACT
Direct alignment algorithms such as Direct Preference Optimization (DPO) fine- tune models based preference data, using only supervised learning instead of two-stage reinforcement learning with human feedback (RLHF). We show that DPO encodes statistical estimation problem over reward funetions induced by parametric policy class. When the true reward function that generates preferences cannot be realized via the policy class, DPO becomes misspecified, resulting in failure modes such as preference order reversal, worsening of policy reward. and high sensitivity the input preference data distribution. On the other hand, we study the local behavior of two-stage RLHF for parametric class and relate to a natural gradient step in policy space. Our fine- theeegraied geometric characterization allows to propose AuxDPO, which introduces additional auxiliary variables in the DPO loss function to help move towards the RLHF solution in a principled manner and mitigate the misspecification in DPO. We empirically demonstrate the superior performance of AuxDPO on didactic bandit settings as well as LLM alignment tasks.
INTRODUCTION
Preference-bacee based alignment key part of the training process of large language models (LLMs). aims to steer a pretrained model's conditional distribution toward outputs that humans (or carefully calibrated annotator models) prefer. Formally, given comparison data (s, awn a.). the goal is to shape policy π whose induced responses align with latent reward model that generated those preferences.
Two-stage RLHF the standard way of carrying out preference-basee 2019). However, it is computationally demanding (it requires training separate reward model) and complex due two-stage pipeline (supervised learning for the reward model followed by RL policy optimization based the learned reward model). Concretely, the reward model (s a) is trained on preference pairs via a Sradley-Terry/Logiitie objective (Bradley and Terry 1952), maximizing log (f aw) r ai) over (s, aw, a1). The second stage then optimizes KL-regularized Dxi(=(- s) Tree(- s)). typically implemented with PPO-style updates. This stage is on-policy and rollout-heavy: the model must repeatedly generate samples to estimate advantages under T&, maintain stable KL the reference policy rel (often the SFT model), and tune sensitive hyperparameters (e.g., 3, clip ranges, learning rates). In practice, this entails nontrivial engineering (reward hacking mitigation, variance reduction, response-length control) and significant compute for both reward-model training and RI. updates, which motivates interest in highter-wiight alternatives.
alignment (Ziegler al.)
objective of the form max, Ex-p,a-r((e) a~*(1 (r (sssaa)
The introduction of direct alignment algorithms such as Direct Preference Optimization (DPO) (Rafailov et al., 2023) was landmark step that paved the way for lightweight alignment of a base model using preference data and only single supervised training phase. DPO operates by explicitly solving the second, KL-regularized, policy optimization phase RLHF and using it to reparameterize the first phase of reward learning in terms of the optimized policy, in effect achieving one-step equivalent to the original two-step pipeline. This has been instrumental in enabling both industrial players and the open-source AI community carry out fast alignment of models without the burden

---

Published as a conference paper at ICLR 2026
of additional resources. Many variants of DPO have since been developed catering to various aspects
of direct aligmment.
Despite its widespread appeal, however, the design of DPO rests the idealized assumption that the policy class is tabular, i.c., it includes every possible input-output conditional probability distribution (Iaa s)),,a where and denote prompt and response strings, respectively. This assumption enables the KL-regularized policy optimization problem to be solved in closed form and used explicitly derive the equivalent supervised DPO loss (Rafailov et al. 2023 Appendix A.1).
In contrast, real-world LLMs are far from tabular and are, in fact, parametric policy classes, resulting naturally from the use of neural architectures (e.g., Transformers) with only finite number of parameters. One may then ask: does minimizing the DPO loss over a non-tabular policy class still preserve the claimed equivalence with full two-stage RLHF? If not, then how does it differ from the ideal RLHF--optima policy? Does it enjoy any guarantees with respect to the performance of the latter, and, if not, is there a principled fix?
We address these questions by introducing a systematic frame- work to uncover the geometry of direct preference optimiza- tion in parametric policy classes. Our study helps show how DPO essentially solves misspeci- fied statistical estimation prob- lem in the space of reward func- tions that are implicitly param- eterized by the underlying pol- icy class. Misspecified estima- tion problems have been ana- lyzed, in the statistics literature, for exhibiting undesirable phe- nomena such as inconsistent and arbitrary estimates that are sen- sitive to the input data distribu- tion (White| 1982); we show that such phenomena also manifest in the DPO setting. Our anal- ysis framework also allows to modify DPO in a principled manner, yield new algorithm (AuxDPO) towards achieving the performance of two-stage RLHF in parametric models. More specifically, we make the following contributions:
AmdDPO elde degroox of Froddom
DPO solutions based prefermez distribution
7%
Yhear
Ideal KLHP solution
Locally linearized
DPO's implicit reward
muniffold
The geometry of DPO for parametric policies. (Left) DPO essentially performs projection of the true preference- generating reward function (r" black) onto the manifold reward functions implicitly expressed by the policy class. If is in the manifold, then DPO finds the correct KL-regularized RLHF policy. but otherwise, the policy found (any orange point) is unreliable. (Right, zoomed inset) I Locally lincarizing the manifold around the base policy's implicit reward function (ro,) uncovers geometric insights. To reliably drive the solution to the reward function corresponding to the ideal RLHF solution (Faguu in blue), AuxDPO introduces additional controlled degrees of freedom, along the null space of a base-policy dependent matrix to sidestep misspecification.
Figure 1:
preference-generating
1. We show that for general parametric policy classes, there is a misspecified statistical estimation problem the core of the DPO algorithm by design: DPO loss minimization is equivalent to a weighted KL-projection of the truc reward function r* onto the (parametric, lower-dimensional) manifold of reward functions induced by the policy class. The weights of the projection are governed by the preference data collection frequencies (Fig. [ left).
2. We show that DPO, in the misspecified setting, can suffer from various failure modes such as order reversal of preferences, overall reward reduction, sensitivity preference data frequencies, etc. These failure modes occur even with 'clean data', i.e., infinite preference data generated using a BTL model based on an underlying true reward function y* and fed DPO. Our analysis is based on taking a local, linearized view of DPO's implicit reward function manifold, which is accurate in the large-f regime.
3. On the other hand, studying the local geometry of two-stage RLHF for general parametric policy classes yields new insights about linear equivalence classes reward functions. We use these insights to design AuxDPO, a direct preference optimization algorithm that effectively mitigates the misspecification issue by introducing auxiliary controlled degress freedom in reward space (Fig. [ right). We demonstrate the effectiveness of AuxDPO in experiments. On
2

---

Published as a conference paper at ICLR 2026
real-world LLM preference tuning tasks, AuxDPO consistently outperforms DPO in aligning to held-out human preferences, confirming its practical value.
Related work. A recent line of work focuses on studying the insufficiency and implications of the tabular policy class assumption. Gao et al. (2024) and Swamy et al. (2025) call into question the tabular policy class assumption in the context of original two-Stage RLHF Tajwar et al. (2024) earry out empirical investigation and note that the standard DPO loss can inadvertently reduce the model's 's absolute likelihood of chosen responses as long as the relative probability between chosen and rejected responses increases. MMeg al. (2024) Xu al. (2024a) propose fixes based on considerations of margin and length normalization, and elimination of the reference policy.
Xu et al. (2024b) and [Song al.] (2024) study the shortcomings of DPO arising from lack of coverage, arguing that DPO can fail if strong coverage condition is not met. The latter provide counterexample to this end, showing the existence of implicitly expressible reward function that is e-approximately close to the true reward function but corresponds to policy not in the KL neighborhood of the base policy. It is, however, unclear if such an implicit reward function can actually be output by DPO. Our fine-grained analysis in this paper shows that even with perfect coverage (uniform base policy), the policy returned by DPO can suffer from pathologies such as preference reordering and a decrease of overall expected reward (Proposition 3)
separate line of work focuses on the gradient dynamics of DPO loss optimization and its impact on policy probabilities (Pal et al. 2024 Razin et al.) 2024 Jian et al. 2025). It is shown that an individual gradient step on the standard DPO loss can result in likelihood displacement, where the probability of preferred responses can drop relative to the base policy for a gradient step. Our approach eschews assuming any specific optimization algorithm such as gradient descent and considering individual gradient steps, and instead focuses showing failure modes such likelihood displacement, preference reversal, reward reduction, etc. by studying the minimizer of the DPO loss.
Shi et al. (202)), perhaps the closest in spirit to our study, demonstrate multi-armed bandit example that, when subjected to DPO with a log-linear policy class, does not cause any movement from the base policy. However, this example relies on degenerate and symmetric reward function; we are able to demonstrate, in fine- mee-ggraied manner, that the policy can strictly worsen from the base policy in manner that is highly sensitive to the preference data distribution (Proposition 3).
2 PRELIMINARIES
Let Let ((00 al aa))) i E [ml] be a dataset of n samples, where each sample has a prompt E S, two responses aws aj E A such that au > aj, i.e., aw is a preferred response over a. We assume both S and A are finite sets, with |S]- = m. The prompt is sampled from a distribution over S. The pair of responses (aw. at) is sampled from some base (reference) policy Teet conditioned on 8, i.e., aw. aj (s). The preference ordering between pair of responses is assumed to be sampled according to a Bradley-Terry-Lan
(BTL) model: the probability of a being preferred to a' is given by PETI, (r*) = (r (s, a) r d')), where r* SS x A + R is (latent) reward function and or (2) 1+e- is the sigmoid function.
Let no : :S + ((A) be policy (e.g.. language model) smoothly parameterized by a d-dimensional vector 0 E d (e.g., the weights of a transformer), where ((A) denotes the probability simplex over A. Let 0, E RRd be the parameter for the base policy neef so that गee = T&. A special case is the tabular policy class, where d = generality that E.. 0s,a= = 1). However, LLM policy classes are structured and non tabular with exp(fe(s.o) parameter dimension d < m, e.g., the neural softmax policy Te(a s) = En'es (x, where fo is, say, neural network.
|S| |A| and (a(a))) = 0z.a (assuming without loss of
For a given reward function r", the optimal policy in a KL-regularized sense is obtained by maximiz- ing the following objective:
(a(a))) To, (a|s)
J(Orr") Eppr r¹ (s, -Blog
Ep (s, -DDx (-))s]] (1)
3

---

Published as a conference paper at ICLR 2026
where Ep.r, denotes expectation taken over p(.) and TT(- s), and B > 0 is parameter that controls the amount of deviation from the base policy. We assume that 0" E. R2 is the unique minimizer of (T). For our analytical results, we will focus on the 'local' case 3 >> 1, meaning that the
policy is not allowed move beyond local neighborhood of To.- When the policy class is tabular, follows that the optimal policy тg- and the latent reward r" satisfy
To- (a|s) = 1
(s) Too (als) exp(r" (s, = r"(ssa) = -Blog To- (als) no, (als)
+ B log Z" (s), (2)
where (s) Eaea Ta, (a|s) exp(r" (s, a) is the normalizing partition function (Rafailov et al. 2023). Under this neward-policy equivalence, the preference probabilities under the BTL model can be expressed using the optimal policy Te- and the base policy Too as follows.
TT- (als)
TT- (a'ls)
BIL (r*) = o 1og
3 log
Te, (a'ls) = a ((..))
Te, (als)
where, for any 0 E θ 8 rg
(s, a) = Blog Blog=e, (als) denotes the implicit S x A + R defined via a
mo(als)
reward function corresponding to the policy my deviation level |3 (note that roo 0 by definition).
Let RD = -(re 0 Rd ç I be the set of all implicit reward functions induced by the policy parameters 0 at deviation level B. Given the dataset D, DPO (Rafailov et al. 2023) finds the minimizer of the empirical DPO loss
12
LD( - -log log a (r) -0) (0
ENa,a No,aw. log o (r" (s) (r ai).
(s.aa).
9,0w.
where on is the total number of pairwise preferences for which aw 0₁ at If ne.e,aa denotes the total number of pairwise preferences for the triplet (s, a, a') (assumed non non-random and fixed in
advance), then No Najawaa ~ Biiooiill (Ne .a Pa (r*)), yiclding the populatioon DPO loss
I3TI.
((0)= -- Σ ns.a,.
peja,a (r") log BTL (rg) + (1 - (1-Pa,a, (r")aa(("""))) (r") (1-
PTT -p (3)
5,a,a'
We take up this loss for DPO as our main object of study in the sequel.
3 REWARD MISSPECCIIICTTION IN DPO
Let nr denote the m-dimensional vector of latent rewards (r"(s, by slightly abusing notation. Proposition (DPO is weighted KL-projection), Assume that the pairwise preference data are drawn from ps,au.aa "areaaa("""
(r*) for some r" Imm with ns.a.a preference pairs drawn for each triplet (s, a, ) If Opro minimizes the DPO loss (3) then its corresponding implicit reward function satisfies
B
arg min Σ Tisaaaa dga (VBTL ("")lPPBT.
T
Copo
FER
8,a,a!
where dxs(pllg) is the Kullback-Leiblee divergence between two Bernoulli random variables with parameters p,
The result establishes that DPO projects (according reverse-Kl. KL divergence weighted by pairwise preference counts ) the true reward function r' onto the set of implicit reward functions R (with the corresponding policy being returned after fine-tuning). implies that if r" is realizable, i... r* = r for some 0, then this projection (trivially) finds r and hence the policy ng. which coincides with the RLHF policy, i.e., 0= 0"
However, if r* & RD (which typically the case since RD a lower-dimensionall (d-dimensional) in the misspecified estimation setting. In this case, the result of the
manifold of ")" then KL-projection will, in general, be dependent on the exact weighted projection which is determined by the preference data frequencies (Ms,a,a') ).a.ar We demonstrate, in the next section, that the policies resulting from DPO's misspecified estimation enjoy guarantees; they could suffer from preference reversal, or even worse, yield a lower average reward than even the base policy (contrary to two-stage RLHF, where the average reward can never decrease).
4

---

Published as a conference paper at ICLR 2026
3.1 LOCAL GEOMETRY OF DPO
Let us locally approximate the implicit reward r (s, a) via its first-order Taylor expansion around 0o:
r (s, a) Re rig (s, a) + ( (Vra, (s, a), - 0o) = ( Via, (als)
0 0o) 3V log THo (als) T(0 0o)
x (a|s)
Define the d
Jacobian matrix Ag, the matrix containing V log Te, (als) in its (s, a)-th
column. With this, the local linear approximation of rg takes the form To BAg (0 0o).
Since (r) (ro:0 E ( is the column space of AJ, we arrive at the linear manifold approximation RR s3 C(A))), for 0 in the local neighborhood of On- Note that this linear approximation is
independent of the choice of B and is a function only of the policy class and base policy To.-
Remark 2. The error in the linear approximation of the manifold R by C( (()))) can be controlled to within any arbitrary tolerance by taking the policy deviation parameter 3 to be sufficiently large. Proposition & formally controls the upproximation error. In the sequel, we only work with the linear approximation to develop our results.
Armed with this local linearization of the implicit reward manifold, we now show an example of a single-prompt, 3-response setting, with 1-dimensional policy parameter, in which DPO exhibits counterintuitive and unexpected behavior, including preference reversal, reward reduction, and high sensitivity the pairwise preference data counts na,,a,a"
Proposition 3 (Fxample of DPO with preference reversal and reward decrease). There exists promptless policy optimization problem with three responses and linear softmax policy class parame- terized with a 1- 1-dimensional parameter O and true reward vector r" E IR3 such that DPO, carried out with pairwise preferences generated according to BTL(r"), ), sufficiently large 3 and pairwise counts {ns.gh, h. yields policy Te such that (i) no favors the response with second highest reward, (ti) Ty decreases (resp. increases) the probability of the action with the highest reward (resp. second highest reward) with respect to the base policy Teg and (iii) To r < To r"
Proof. Consider the following example with 3 responses a1 02 03 with latent rewards
[2. 1]. which yields the preference order 02 > > (33- The pairwise pref- erence counts in the dataset are highly imbalanced in a
way that 13,11 >> m n2 The policy is given by
my = [e,, e-0 1]. where Z = +e + - We take uniform base policy (i.e., 0.- = 0). Under the BTL model, r" 21,2,0 0] since both induce the same preference distri- bution. Hence, we will work with this equivalent r*. The setting is depicted in Figure 2.
<<0
*
In this setting, the policy gradient matrix takes the form Ag
C(A
ee"),e=* 9 c²]. Hence Ae, = [1, 1, 0] and thus R xs span ([L, -1.--1,10 0)) in the local neighborhood of f = 0. Now, since 11.3 domi- nates the other two comparison counts, the solution to the optimization problem in (4) e.g..
11-2e- -0 (1
An example with 3 responses and 1-d policy parameter showing failure modes of DPO. r" is the latent reward. The red line denotes the linear approximation C(AA)) of the implicit reward manifold Rº The region shaded in orange represents all possible implicit reward functions that DPO can possibly project onto, depending the relative proportion of pairwise preference counts 1.2, 722,3 m3,1. If mg dominates the rest, then the projection ri induces post- optimized policy parameter > O, leading to preference reversal and reduction of expected reward, causing DPO to fail.
Figure 2:
arg min Σ de (p (BBL BTL (r)
FERS
will push the probability that a is preferred over 0z, i.e.
(r3 close to pT. yiclding r (a1) (a3) = O(a) for some a 0. Moreover, since r should lie in (pa -,, 0)), DPO will end up with the reward func- tion To R (a, -a, 0J. This, in turn, will make to be preferred over 02 by the learned reward, indicating a pref- erence reversal from that given by r- Furthermore, from the equivalence relation (.) get the post-optimized policy parameter 0 = O(a). This yields (i) To(a)) > TT((2) (the policy favors a sub- optimal response) (ii) mo (az) no, (a2) and no (a1)) ) > no, (a1) (likelihood of the optimal response
PI.3
5

---

sufficient for DPO Published as a conference paper at ICLR 2026
decreases and that of suboptimal response increases) and (iii) ng) r/ (average reward decreases relative to the base policy).
22426 Ite-+e-a TT.
The example exhibits the following aspects:
1. There are pairs (i, (e.g., i = = 1) where a; > ag with respect to r (i.e., r" (a:) > r' (a)))
but TT((;)) decreases and (T{a,) increases with respect to the base policy. This is more extreme than likelihood displacement, where wo (a,)) and To(a;) both decrease increase together but their difference (a)),) - (a(a)) is presumed to increase (Razin et al. 2024 Pal et al. (024).
2. The expected reward with respect r' decreases from To, whereas two-stage RLHF would have increased the expected reward on any policy class (assuming r" is learnt accurately in the first stage).
3. Our example is based the DPO population loss, which is effectively DPO operating in the data-rich regime where unlimited pairwise preference data from a BTL model are used as input. This circumvents the issue of failure modes of DPO known occur due scarce data sampling (Tajwar et al. 2024). We show that failure modes arise due to the inherent misspecified geometry induced by the lack of model capacity, interacting with the frequencies of pairwise sampling.
4. Our example applies in the strongest possible 'oracle' optimization model where we assume that the (population) DPO loss can be optimized exactly. Our results are not dependent on the idiosyncrasies
specifics of what algorithm is used to optimize the DPO loss (c.g.. gradient descent and variants as explicitly considered in other works (Razin et al. 2024 Pal et al. 2024 )
as long as it is (near) optimal.
5. Sensitivity of DPO with respect to the preference data distribution: In the example, if the pairwise preference counts are such that n1,2 > {M22,3 13,11, then DPO would learn the desired reward function rg R [-ar or, OJ. which would, in turn, induce a policy parameter 0 < O, escaping the failure modes effectively. Therefore, depending on the relative proportions of pairwise preference counts {n4. Jr.j. one could either get the desired result or the opposite one, as in the example, or perhaps no movement at all, making DPO sensitive to preference data sampling distribution. This sensitivity to the exact preference distribution also represents failure mode of DPO.
Remark (Global coverage is not sufficient for optimality). Song et al. (2024) argue that global
coverage, ie.. maxs,a Teg [a](a]s] C vo E Rdd, is necessary for DPO to converge to the optimal policy 0r In our example, Te, satisfies this with C' == However, this condition
Na(a|s]
to perform optimally, since, as we have seen, DPO can learn an unaligned reward model depending on the relative frequency of preference counts.
TOWARDS MITIGATING DPO's PITFALLS
We propose to address the misspecification issue of DPO by first studying the nature of the RLHF policy optimization step in a suitably local sense (assuming ideal reward learning), and then using the insights gained to encourage movement towards this solution.
4.1 LOCAL GEOMETRY OF RLHF OPTIMIZATION
We locally approximate the objective ((;rr*) in T around the base policy To. To do so, we approximate the expected reward using first-order Taylor series expansion and the KL penalty using a second-orde Taylor series expansion:
Ep.ro (s, a)] = Ep, (s,, (0-))) Ao, Dp..r" (mollTl ( (00-0)FF,.o. F0 (0-
(0-80)
where Dp. is a diagonal matrix with scaled base policy-probabilitce p(s)=o, (als) the diagonal entries, and Fp.0g E.p,,o [V log Te, (a|s) (/s)V log ne, (a ()) denotes the Fisher information matrix Te, (Amari 2016 Introducing the d x m matrix Ap, Ag, D,. containing the scaled gradients p(s) V To [a]s) in its columns, we arrive at a local quadratic approximation of the objective
B (0 000 Fp.0, (0 0o)
J(O:r") R By.,, (r"(s, a)) + (0- 0o)T Ap.
2

---

Published as a conference paper at ICLR 2026
Remark 5. Just described in Remark 2 the error in this local quadratic approximation of [ can be controlled to a desired level of accuracy by taking 3 to be sufficiently large: see Proposition
Based on this local quadratic approximation of the RLHF objective function, we can deduce (via first-order optimality conditions) relation that suitably generalizes ( Tg- and the latent reward r", to parametric policy classes:
between the optimal policy
1 Ap,0r p...0 8o) 0- 9o
3 F. Ap,or""
(5)
This has the form of a natural policy gradient update (Kakade, 2001). Moreover, it partitions the set of all reward functions into equivalence classes as follows. For each policy parameter 0, define
Rey (0) = fr E R Ap..o = 3F....00 00]}
(6)
Lemma (Equivalence classes induced by RLHF). For a fixed 0 R., two reward vectors T2 R belong to (0) if and only if they differ by vector d E =A(A),)), i.e., null
space element of Ap.0o
Note that for tabular policies, the class Reg (0) essentially reduces the singleton rg. the "implicit sofals)
reward" of DPO, i.e., r a) = =3 log Tao (a[s)
Interestingly,
show that DPO's linearized reward functions og are in bijective correspondence with RLHF's equivalence classes, with each linearized reward function being the minimum-norm representative of its equivalence class:
Proposition 7 (Relationship between RLHF equivalence classes and DPO linearization). For a base policy Te, and KL penalty 3 > O, the reward vector r E R.y (0) which has the minimum Mahalonobis- norm [r||p,,.o /rl D, i.e., argmin,eR" (r.90, such that Ap. 3 FF (0 0o).
is given by F B.A, (A- 8o), the local linearization of (s, a) 3 log Tap (als))
sola s)
The following result helps to justify the local linear approximations made to (i) the DPO implicit reward function manifold and (ii) the RLHF policy optimization objective, by showing that error between the local approximations and the original functions can be controlled to an arbitrarily prescribed level by taking the deviation parameter essentially reduce to optimization over policies in a neighborhood of k.,)-
be suitably large (so that DPO and RLHF
Proposition 8 (Approximation errors). Fix > 0. There exists a bounded neighborhood E C Fdd containing 9g. bounded set R C R", and Smin > such that for every de- viation parameter > Bain we have (i) re E R. for each 0 E, (ūi) rrro
r for some O E E, (ii)) rg a) BV VV Te, (als) -0)) S E each O E E, and (iv) J(O:r") (Ep,zo, (s a)] + (0 A r" - (0 - 0o) Fp., - (0-))) I E for each E E.
4.2 THE AUXDPO ALGORITHIM
We introduce a new direct alignment algorithm, AuxDPO, which leverages our insights from the analysis of the local geometry of DPO and RLHF policy optimization mitigate the failure modes of DPO in a principled manner.
Recall that, in the general setting where the reward function r is misspecified (outside the implicit reward manifold R$), then DPO finds the optimal policy 6* only if the reverse-KL projection of r' on RD fortuitously lands on rig (Proposition I.. This depends crucially on relative proportions of pairwise preference counts { the dataset and, as such, is beyond the learner's control.
Instead, we take the following approach to encourage the optimization to move towards r Note that by our local analysis of the RLHF optimization step (Sec 4.1), the true (misspecified) reward function yx and & differ by an element of the nullspace of Ap.o i.e., rx = rg + 5, where
Mooe formally, (0) contains functions of the form rf( + log (8).)
7

---

Published as a conference paper at ICLR 2026
8 E (V(Ap.0)) RR.. Therefore, if we allow the search in the reward space RT to utilize additional degrees of freedom along this nullspace (in addition to the usual degrees of freedom via f), then r" is no longer misspecified in this augmented representation. This should ideally result in the variables 0 E E2d and å (A(,,,))) settling in manner that achieves r" rg. 8
Observe that since Ap. - Aa, Dp. both (Ae,) and ((Ap),))) have the same dimension. Moreover, C(A and ((Aa,) also have same dimension. Thus, by the rank- nullity theorem, by varying both 0 and 6, can search over the entire space of the rewards IRm,, contrary to DPO which searches only over C(A (under linear approxima- tion), a manifold in R" with dimension at most d.
(Anm))
e
rd.
(2((0")
To this end, we introduce auxiliary variables 8 1 into the population loss of DPO (3), and minimize it jointly over A E Rd 5 while enforcing the nullspace constraint 5 E (Ap,.)). This gives us the AuxDPO procedure"
(2y
C(AT.
minimize DER (ER",,E
L(,,8),
where
Figure 3: AuxDPO fixes DPO's 's misspeci- fication. r" is the latent reward. The blue (7) line denotes the equivalence class R2y (8") of all reward functions that yield the RLHF- optimal policy Æg- The red line denotes the linear approximation C(A of the implicit reward manifold Ra. The region shaded in orange represents all possible implicit reward functions that DPO can possibly project onto, The green line depicts the domain of opti- mization over AuxDPO's auxiliary variables 8 E (Ap.o.) for fixed (the line shifts in parallel for other 0). 8 introduces addi- tional degrees of freedom. which help push the KL projection of r" to lie in the equiva- lence class Rg., The projection induces the optimal policy To..
C(0,6) = = - n
BTL Pa (r") log Ps... (T
BTL
5,a,a'
BTL + (1 (1-P5,a,a (r**) log (1-p5.g. (_- (r
BTL
Proposition 9 (Auxiliary variables bypass misspecifica- tion). Let the hypothesis of Proposition / hold. Fix tolerance > O. Then, for sufficiently large B > 0, the optimization (7) is minimized at 0 0* up error Oss).
We develop AuxDPO's corresponding empirical loss ver- sion, which can be implemented with a finite preference dataset. We convert the constrained optimization over the set (Apoo)) C R to an unconstrained one over Ihm by
adding the penalty term ||A 2 to the log-loss. Note that Ap.00 = Ep,re, (0(s, a) V log te, (a|s)] in Monte-Carlo fashion. This leads to the empirical AuxDPO loss Lo over variables 0 E Rd and 8 E R2n.
We approximate it with given dataset D (5 dew , ag
(2) (6)
(D( = n Σ logo (r8() aL - rg (s aa)) + 8(5() -(0(0) aa()))
1
2 (5(0(0), du ( loggon To (a (0))) + 8()(0) ai (0) log To (a 8 (0))
+
2n
where 8 = { 8(s(0) a )(0)(0 )2 RR denotes the vector of auxiliary variables (typically
that is responsible for enforcing the nullspace constraint on Note that the total number of trainable parameters is d 2n ((d), since typically, < d.
2n < m) and A > 0 is
EXPERIMENTS
Datasets. We conduct evaluations on two benchmark datasets: REWARDBENCH v2 and MMLU- PRO. REWARDBENCH v2 Malik et al.) (025) contains 1.87K prompts covering categories like factuality, precise instruction following, and focus, with each prompt containing a chosen and a rejected response. MMLU-PRO (Wang et 2024b) is a multi-task understanding dataset containing 12K complex questions across various disciplines. Each question has 10 possible answers and a
AuxDPO can be run any dataset, irrespective of any approximation needing be assumed. Its theoretical guarantee (Prop. T. however, valid in the local sense, which manifests sufficiently high B8.
8

---

Published as a conference paper at ICLR 2026
Model
Dataset
Method
DPO
AuxDPO
IPO
DPOP
1. allama... 3. 888
MMLU-PRO
ID
57.14
63.26
59.18
61.22
MMLU-PRO
OOD
8.16
14.28
10.20
6.12
REWARDBENCH V2
ID
56.01
66.72
61.34
62.27
REWARDBENCH
OOD
14.31
32.44
20.17
19.87
Llama3. 3.2-BB 22-1B
MMLU-PRO
ID
39.58
45.83
43.75
44.21
MMLU-PRO
OOD
6.25
12.52
14.58
4.16
REWARDBENCH V2
ID
77.21
86.37
69.72
71.21
REWARDBENCH V2
0OD
14.11
43.27
20.42
18.76
3wen3-0. 6R
MMIU-PRO
ID
53.12
61.78
47.48
56.67
MMLU-PRO
OOD
11.34
22.22
15.56
17.78
REWARDBENCH V2
ID
55.10
65.31
53.06
51.02
REWARDBENCH V2
OOD
-8.16
18.36
-8.23
6625
Table Algorithm comparison. Values show percentage change in mean accuracy relative to the base policy, across MMLU-PRO and REWARDBENCH v2 under in-domain (ID) and out-of-domain (OOD) settings. Best gains in bold, second-best underlined. Accuracies which degrade from the base policy marked in red.
correct answer. We use ULTRAAFEDBBAK (Cui et al. 2024) training dataset. Specifically, its pre-processed and binarized version (Dong et al. 2024). which generates higher-quality reward
models (Wang et al. 2024a; Xiong
2024
Banerjee and Gopalan 2024).
Subject
Base
DPO
AuxDPO
IPO
DPOP
OOD
ID
OOD
ID
OOD
ID
OOD
ID)
Overall
25.37
27.06
46.60
39.26
51.95
31.73
47.17
32.64
48.25
Biology
55.93
52.44
76.50
75.07
86.87
60.67
79.12
60.39
81.50
Business
13.18
20.03
35.67
29.96
43.49
24.21
32.56
24.08
37.96
Chemistry
10.42
16.25
37.12
26.78
40.86
21.64
34.56
21.73
35.86
Comp. Sc.
24.39
26.59
42.78
38.93
48.35
31.46
43.57
31.22
47.08
Economies
38.98
37.80
59.24
55.12
68.30
44.55
62.58
43.60
64.50
Engincering
10.22
17.75
46.06
32.18
48.60
26.01
41.46
28.48
43.82
Health
41.81
41.32
62.41
55.67
68.58
44.99
64.08
44.13
66.23
History 38.06
32.55
54.96
47.41
64.58
38.32
59.56
40.42
57.29
Law 27.25
23.61
42.20
34.50
48.01
27.88
45.25
30.43
42.31
Math 11.77
16.51
28.97
22.26
30.80
17.99
29.34
20.28
30.24
Table 2: Per-subject accuracies (top 10 subjects alphabetically) and overall win-rates across baseline (Llama3. 8B) and preference optimization methods. For each method, two settings are shown: OOD transfer) and ID (in-domain learning). along with the reported results. In each the best accuracy shown in bold, the second- best is underlined. Accuracies which degrade from the base policy are marked in red.
Evaluation and Methodology. We compare AuxDPO with DPO, IPO (Azar et al., 2023), ), and DPOP (Pal al. 2024). Table I reports accuracies in terms of whether the logits of the chosen response were higher than those of the rejected response. While REWARDBENCH v2 provides chosen and rejected responses directly, we make MMLU-PRO the correct answer as the chosen response and any incorrect response as the rejected response. We consider both in-distribution (ID) and out-of-distribution (OOD) evaluation settings. In the IDD setup, each dataset is split 80/20 into training and evaluation subsets, ensuring IID comparisons. In the OOD setup, models are trained on cleaned ALKKAFEEDBACK and evaluated on the preference datasets. report full finetuning results on the models. Table 2 presents overall and subject-wise accuracies on MMLU-PRO. Accuracy is measured by comparing the finetuned model's 's generated answer with the correct answer provided in the dataset. We compare all methods on Llama3. 1-8B under both OOD and ID settings. Across all three models, we see that AuxDPO outperforms other finetuning methods. Ablation studies, implementation, and dataset details are presented in Appendix B.2
preference dataset by filtering

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGMENTS
SRC would like thank an early-career research grant from ANRF, India. DB would like thank HP AI Labs for providing the necessary computational infrastructure to conduct the experiments.
REFERENCES
Shun-ichi Amari. Information geometry and its applications, volume 194. Springer, 2016.
Mohammad Gheshlaghi Azar, Mark Rowland, Bilal Piot, Daniel Guo, Daniele Calandriello, Michal Valko, and Rémi Munos. A general theoretical paradigm to understand learning from human preferences. arXiv preprint arXiv:2310. 12036,, 2023.
Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.0586) 2022.
Debangshu Banerjee and Aditya Gopalan. Towards Reliable Alignment: Uncertainty-aware RLHF, 2024.
Ralph Allan Bradley and Milton E Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39(3/4):324-345, 1952.
Ganqu Cui, Lifan Yuan, Ning Ding, Guanming Yao. Bingxiang He, Wei Zhu, Yuan Ni, Guotong Xie, Ruobing Xie, Yankai Lin, Zhiyuan Liu, and Maosong Sun. UltraFeedback: Boosting Language Models with Scaled AI Feedback, 2024. htttts:s//aaiii.cttoto
2211 2310001377
Hanze Dong, Wei Xiong, Bo Pang. Haoxiang Wang. Han Zhao, Yingbo Zhou, Nan Jiang, Doyen Sahoo, Caiming Xiong, and Tong Zhang. RLHF Workflow: From Reward Modeling to Online RIHH, 2024. URL
07863
Zhaolin Gao, Jonathan Chang, Wenhao Zhan, Owen Oertell, Gokul Swamy, Kianté Brantley, Thorsten Joachims, Drew Bagnell, Jason D Lee, and Wen Sun. Rebel: Reinforcement learning via regressing relative rewards. Advances in Neural Information Processing Systems, 37:52354-52400 2024.
Chengtao Jian, Kai Yang. Ye Ouyang, and Xiaozhou Ye. Stable preference optimization for LLMs: A bilevel approach beyond direct preference optimization. arXiv preprint arXiv:2507.07723, 2025.
Sham M Kakade. A natural policy gradient. Advances in neural information processing systems, 14, 2001.
Saumya Malik, Valentina Pylkkin, Sander Land, Jacob Morrison, Noah A Smith, Hannanch Hajishirzi. and Nathan Lambert. Rewardbench 2: Advancing reward model evaluation. arXiv preprint arXiv:2506.01937, 2025.
Yu Meng, Mengzhou Xia, and Danqi Chen. Simpo: Simple preference optimization with reference- free reward. Advances in Neural Information Processing Systems, 77:124198-124235 2024.
Arka Pal, Deep Karkhanis. Samuel Dooley, Manley Roberts, Siddartha Naidu, and Colin White. Smaug: Fixing failure modes of preference optimisation with dpo-positive. arXiv preprint arXiv:2402.13228. 2024.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D Manning, and Chelsea Finn. Direct preference optimization: Your language model is secretly reward model. arXiv preprint arXiv:2305./82 2023.
Noam Razin, Sadhika Malladi, Adithya Bhaskar, Danqi Chen, Sanjeev Arora, and Boris Hanin. Unintentional unalignment: Likelihood displacement in direct preference optimization. arXiv preprint arXiv:2410.088. 2024.
Ruizhe Shi, Minhak Song. Runlong Zhou, Zihan Zhang, Maryam Fazel, and Simon S Du. Under- standing the performance gap in preference learning: A dichotomy of RLHF and DPO. arXiv preprint arXiv:2505. 19770, 2025.
10

---

Published as a conference paper at ICLR 2026
Yuda Song, Gokul Swamy, Aarti Singh, Bagnell, and Wen Sun. The importance of online data: Understanding preference fine-tuning via coverage. Advances in Neural Information Processing Systems, 17:12243-12270. 2024.
Gokul Swamy, Sanjiban Choudhury, Wen Sun, Zhiwei Steven Wu, and J Andrew Bagnell. All roads lead to likelihood: The value of reinforcement learning in fine-tuning. arXiv preprint
arXiv:2503.01067 2025.
Fahim Tajwar, Anikait Singh, Archit Sharma, Rafael Rafailov, Jeff Schneider, Tengyang Xie, Stefano Ermon, Chelsea Finn, and Aviral Kumar. Preference fine-tuning of Ilms should leverage suboptimal, on-policy data. arXiv preprint arXiv:2404. 14367, 2024.
Haoxiang Wang, Wei Xiong, Tengyang Xie, Han Zhao, and Tong Zhang. Interpretable preferences via multi-objective reward modeling and mixture-of-experts, In The 2024 Conference Empirical Methods in Natural Language Processing, 2024a.
Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Aruiraj, Xuan He, Ziyan Jiang, et Mmlu-pro: more robust and challenging multi- task language understanding benchmark, Advances in Neural Information Processing Systems, 37: 95266-95290, 2024b.
Halbert White. Maximum likelihood estimation of misspecified models. Econometrica: Journal of the econometric society, pages 1-25, 1982.
Wei Xiong, Hanze Dong, Chenlu Ye, Ziqi Wang, Han Zhong, Heng Ji, Nan Jiang, and Tong Zhang. Iterative preference learning from human feedback: Bridging theory and practice for rlhf under kl-constraint. ICML, 2024.
Haoran Xu, Amr Sharaf, Yunmo Chen, Weiting Tan, Lingfeng Shen, Benjamin Van Durme, Kenton Murray, and Young Jin Kim. Contrastive preference optimization: Pushing the boundaries of Ilm performance in machine translation. arXiv preprint arXiv:2401.08411 2024a.
Shusheng Xu, Wei Fu, Jiaxuan Gao, Wenjie Ye, Weilin Liu, Zhiyu Mei, Guangju Wang, Chao Yu, and Yi Wu. Is dpo superior ppo for IIm alignment? comprehensive study. arXiv preprint arXiv:2404. 107 2024b.
Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B Brown, Alec Radford. Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences. In arXiv preprint arXiv: 2019.


Reviews:
[{"review_id": "g1If48lwYQ", "reviewer": "Reviewer_zAc8", "summary": "The paper investigates a fundamental limitation of Direct Preference Optimization (DPO), a widely used direct preference alignment method. It shows that when using parametric policy classes (as opposed to the tabular assumption underlying DPO's derivation), DPO suffers from a misspecified statistical estimation problem. This misspecification arises when the true reward function that generates preferences cannot be represented by the chosen policy class, leading to undesirable outcomes. To address this, the authors propose AuxDPO, which augments the DPO loss with auxiliary variables to mitigate misspecification. They demonstrate the empirical effectiveness of AuxDPO on LLM preference alignment tasks.", "questions": "(Related to the constructive example in Proposition 3): If preference data are balanced (uniform) across $(s, a, a')$, would DPO still exhibit the preference reversal issue shown in the example?\n\nFor the datasets used in the experiments, were the preference frequencies balanced or imbalanced?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper is well written and clearly structured.\n\nProvides an insightful theoretical analysis of DPO and RLHF via Taylor approximation, revealing:\n- the local geometry of DPO under parametric policies,\n- the local geometry of RLHF optimization, and  \n- the relationship between RLHF equivalence classes and DPO linearization.\n\nProposes a novel and principled solution (AuxDPO) to address the identified misspecification issue.", "weaknesses": "Could oversampling or undersampling preference pairs to balance frequencies before DPO training mitigate the misspecification issue and yield comparable performance?\n\nThere is no discussion or experimental analysis of AuxDPO's sensitivity to its core hyperparameters ($\\lambda$ and $n$). How should these values be chosen in practice?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qGwqLv7KZm", "reviewer": "Reviewer_7Hcd", "summary": "This paper first demonstrates that DPO implicitly projects the weights of the optimal solution to the manifold of possible reward functions under the given policy class. The authors prove that this can lead to mis-specifications where the reward function does not actually obey preferences.", "questions": "See weaknesses section.", "limitations": "", "rating": 8, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* the paper is mathematically rigorous in demonstrating its claims. \n* the work does a good job demonstrating why the mis-specification is a problem through a useful example and follow-up points.\n* the results demonstrate strong performance in both in distribution and out of distribution settings.", "weaknesses": "* the derivation of the aux DPO objective makes sense, but is justified using a local approximation. While this makes sense, it could be worth pointing out the the AuxDPO solution (at least to my understanding) holds under these approximations only. \n* The paper is a bit hard to follow at times as it is particularly dense. I think at various points in the manuscript having more motivation and explanation would be helpful. Why do we want to do a first order approx? What is the meaning of the A matrix? \n* In a similar manner, the actual instantiation of AuxDPO is a bit unclear. AuxDPO adds variables in the null space of the A matrix, but how are these variables actually represented in code? Is $\\delta$ just predicted as another head of the LLM? \n* I understand why the authors used MMLU etc. as a preference learning benchmark, but it (in spirit) does not seem exactly like one since its more of a QA dataset. While DPO can be used in this case I think more benchmarks actually based on human preferences woudl be preferred.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Zy2BjeU5vc", "reviewer": "Reviewer_egJi", "summary": "The paper studies the misspecification of the parametric reward model in the DPO framework. \nThe standard DPO framework assumes that the LLM is expressive enough to be able to fit the ideal solution characterized by RLHF.\nThe authors raise critical questions that the LLM models can be misspecified with a concrete example (Section 3.1). This example shows that such misspecification may lead to undesirable behavior. \nBased on the local analysis of the geometry, they propose to add additional optimization variables which allow enough slackness to the model to fit to the desired RLHF solution. \nThe experiment results show clear advantage of the proposed method AuxDPO.", "questions": "- I cannot understand the implication of Remark 4. Do the authors want to argue that Song et al. (2024)'s argument is wrong? If the authors can end up with a different conclusion, what is the difference? I believe that the difference comes from the misspecification, but I appreciate if the authors can clarify.\n- I have a question about the notation in Section 4. The authors use $\\mathcal{N}(A)\\subset\\mathbb{R}^m$ to denote the nullspace of $A$. Given this, I do not know how to parse $\\\\|\\\\mathcal{N}(A)\\\\delta\\\\|^2$, and how it can manifest as in the final objective function. I think this paragraph deserves a further expansion.\n\n#### **Suggestions**\n- The notation for dataset is a bit weird. For example, in line 138, consider $\\mathcal{D}=\\\\{(\\text{tuple}\\_i)\\\\}_{i=1}^n$.\n- The shorthand b/w and w.r.t. read weird. Please expand them in the final version.\n\n---\n\nOverall, I find the core idea compelling and the paper a solid contribution. However, a thorough revision to streamline and simplify the notation would greatly improve the manuscript’s readability and its accessibility to a broader audience.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 4, "presentation": 2, "contribution": 3, "strengths": "This paper proposes an interesting idea to further improve the quality of DPO, based on principled local information geometry.\nThe proposed technique is not only principled but also effective in practice.", "weaknesses": "The only downside of the paper is that it is very notation heavy and not easy to follow in the first read.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Oral)