Mitigating Mismatch within Reference-based Preference Optimization
Authors: Suqin Yuan
Xingrui Yu
Jiyang Zheng
Lei Feng
Dadong Wang
Ivor Tsang
Tongliang Liu
Venue: ICLR 2026 Poster (2026)
Keywords: machine learning
language models
alignment
preference optimization
offline preference alignment
Primary area: alignment, fairness, safety, privacy, and societal considerations

Abstract: Direct Preference Optimization (DPO) has become the de facto standard for offline preference alignment of large language models, but its reliance on a reference policy introduces a critical tension. 
DPO weighs each update relative to a reference, which stabilizes the training by regularizing the updates within a trusted region. This reliance becomes problematic for pessimistic pairs, where the reference model prefers the rejected response. For these pairs, DPO prematurely attenuates the gradient as soon as the policy margin ($\Delta_\theta$) merely beats the reference margin ($\Delta_{\mathrm{ref}}$) even if the policy is still wrong ($\Delta_{\theta}<0$). We name this failure premature satisfaction, which is a concrete form of the training–inference mismatch.
Reference-free objectives remove this mismatch by optimizing the absolute margin, but at the cost of discarding the stabilizing signal of the reference. We mitigate this tension with Hybrid-DPO (HyPO), a drop-in modification to DPO that applies reference conditionally: HyPO behaves exactly like DPO when the reference is optimistic or neutral, and it treats the reference as neutral when it is pessimistic by replacing $\Delta_\theta-\Delta_{\mathrm{ref}}$ with $\Delta_\theta-\max\{0,\Delta_{\mathrm{ref}}\}$. This one-line change strictly strengthens per-example learning signals on pessimistic pairs while preserving DPO’s objective form and computational cost. By conditionally debiasing the pessimistic reference signal, HyPO mitigates premature satisfaction; empirically, across preference alignment, HyPO improves inference-aligned metrics and achieves higher pairwise win rates. Our results provide evidence that direct preference alignment could be enhanced by conditionally debiasing the reference signal, rather than discarding it.

Published as a conference paper at ICLR 2026
MITIGATING MISMATCH WITHIN REFERENCE-BASED PREFERENCE OPTIMIZATION
Suqin Yuan' Xingrui Yu² Jiyang Zheng!.3
Lei Feng** Dadong Wang' Ivor Tsang Tongliang Liu'*
Sydney AI Centre, The University of Sydney
2 CFAR and IHPC, Ageney for Science Technology and Research (A STAR), Singapore
3 Data61, CSIRO Southeast University CCDS, Nanyang Technological University
suqi nyuan, es gma 1 I. cem, feng le sésu.. edu. co, tongl ang. 1 fugsydneyy edu..a
ABSTRACT
Direct Preference Optimization (DPO) has become the de facto standard for offline preference alignment of large language models, but its reliance on a reference policy introduces critical tension. DPO weighs each update relative to reference, which stabilizes the training by regularizing the updates within a trusted region. This reliance becomes problematic for pessimistic pairs, where the reference model prefers the rejected response. For these pairs, DPO prematurely attenuates the gradient soon the policy margin (Ao) merely beats the reference margin (Aret) even if the policy is still wrong (Ag < O). We name this failure premature satisfac- tion, which is a concrete form of the moiniing-infeeree mismatch. Reference-free objectives remove this mismatch by optimizing the absolute margin, but at the cost of discarding the stabilizing signal of the reference. We mitigate this tension with Hybrid-DPO (HyPO), a drop-in modification to DPO that applies reference conditionally: HyPO behaves exactly like DPO when the reference is optimistic or neutral, and it treats the reference as neutral when it is pessimistic by replacing Ag - Aref with Ae max(0, Aret). This one-line change strictly strengthens per-example learning signals pessimistic pairs while preserving DPO's objective form and computational cost. By conditionally debiasing the pessimistic refer- ence signal, HyPO mitigates premature satisfaction; empirically, across preference alignment, HyPO improves inference-aligned metrics and achieves higher pairwise win rates. Our results provide evidence that direct preference alignment could be enhanced by conditionally debiasing the reference signal, rather than discarding it.
INTRODUCTION
Aligning Large Language Models (LLMs) with human preferences and values is paramount for developing safe, helpful, and reliable AI systems (Askell et al., 2021). The pioneering Reinforcement Learning from Human Feedback (RLHF) paradigm (Christiano et al., 2017) achieved this by first training a reward model on human preference pairs (Ziegler et al., 2019; Stiennon et al., 2020; Ouyang et al., 2022; Bradley & Terry, 1952; Schulman al., 2017; Bradley & Terry. 1952), and then using reinforcement learning (Schulman et al., 2017) to optimize the LLM for reward maximization, usually with KL--divereence penalty to regularize deviation from reference policy (Ouyang et al., 2022). Despite the widespread success, RLHF's multi-stage process suffers from complexity, instability, and high computational cost (Sharma al., 2024; Gao al., 2023). Addressing these challenges, Direct Preference Optimization (DPO) (Rafailov et al., 2023) emerged as a compelling alternative, offering significant simplification by analytically deriving a direct policy loss. This loss allows for direct optimization on pairwise preference data (I,, yw: v1). bypassing explicit reward modeling. Its mechanism implicitly enforces KL regularization by comparing the policy to a reference model (Trer). making the loss dependent on the log- ligeeiill margin AS A DPO has gained widespread adoption due to its stability, efficiency, and strong empirical performance (Rafailov et al., 2023; D' O'Oosterlinck et al., 2024; Zhao et al., 2023; Yuan et al., 2023b; Rashidinejad & Tian, 2025).
"Corresponding authors.

---

Published as a conference paper at ICLR 2026
ol Blåg- Auwi)
W o( -(- [.)
W o( (BB(d ma 01)
1.0
Ireference margin)
40
Dar Ireforrenece margin)
40-
+-321
Aur Ireferenncce margin)
40
0.0
20
20
20-
0.6
0-
0-
0.4
-20
-20
-20
0.2
0.2
0.2
-40
-40
-40
0.0
0.0
0.0
-50
-25
-50
-5
50
-25
25
da (policy margin)
4o (policy margin)
Ao Ipolicy margin)
(a) DPO
(b) Reference-free Method
(c) HyPO (Ours)
Figure 1: Per-example gradient weight heatmaps. The weight, plotted over the policy margin A (z-axis) and the reference margin Aeef (y-axis), indicates training signal strength. DPO's reliance on the relative margin leads to premature satisfaction: on pessimistic examples (blue dot, Aref < 0), the signal is heavily attenuated even though the policy is still wrong in absolute terms. Reference-free methods provide strong signal but discard the reference entirely. HyPO mitigates this by mirroring the reference- free behavior on pessimistic examples to ensure a strong signal, while reverting to DPO on optimistic examples (red dot) maintain proximity to the reference policy.
While DPO is simple and effective. its reliance on reference model introduces tension that has led to two seemingly opposing lines of work. One argues that the reference should be removed due to training-inference mismatch (Chen et al., 2024; Meng et al., 2024): DPO optimizes reference-relative margin (e.g., As Doo)) whereas inference depends on absolute log-likelihoods with no reference. Empirically, after DPO alignment, the agreement between the implicit-rewan ordering and the likelihood ordering is only ~50%, motivating reference- free objectives that directly optimize the inference metric and often outperform DPO (Meng et al., 2024). The other line takes the opposite view: the reference can be beneficial when strengthened or made adaptive. Updating Te alongside the policy helps mitigate over-optimization (Gorbatovski et al., 2025; He et al., 2026), and employing stronger or better-matched reference improves stability and final performance, sometimes surpassing reference-free methods (Liu al., 2024b; Azar al., 2024; Pan al., 2025). This leaves practical dilemma: removing Tref eliminates metric mismatch and can enhance performance, whereas retaining Tet improves training stability and can also enhance performance.
To mitigate the tension between reference- enference-bbsed and reference- free direct aligmment, we propose a unifying principle: the proximal stability offered by the reference model should be applied condition- ally. We argue that the reference provides constructive signal for stable learning only when it aligns with the preference data. When it pessimistic (Aref < O), its stabilizing pull becomes a misleading bias, making the learning objective unnecessarily difficult and causing the gradient to attenuate even when the absolute ordering (Aa) is still incorrect. We term this effect premature satisfaction. Prior approaches either accept this bias (standard DPO; Rafailov et al., 2023) or discard potentially useful reference signal altogether (reference- free methods; Yuan et al., 2023b; Meng et al., 2024).
To realize this principle, we introduce Hybrid-DPo (HyPO), minimalist and efficient modification to the DPO objective. HyPO behaves identically to DPO when the reference is helpful (optimistic or neutral) but treats a pessimistic reference as neutral (Aref = 0). This is achieved with minor change to the DPO loss: the log-likell likelihoood margin A Aref becomes An max (0, Arot), ). As illustrated in Figure standard DPO (a) prematurely attenuates the learning signal on pessimistic examples; typical reference-free method (b) discard the reference signal entirely; our proposed HyPO (c) uses the stabilizing reference where it helps and nullifies it where it is detrimental, thereby continuing to push the policy toward the correct absolute ordering.
We conduct comprehensive empirical evaluation on preference datasets, aligning both base and instruction tuned variants of two widely-used LLM family (Jiang et al., 2023; AI Meta, 2024). Our proposed method, HyPO, demonstrates superior performance on general instruction-followinn benchmarks like AlpacaEval 2.0 (Li et al., 2023) and Arena-Hard-vll 1 (Li et al., 2024b). Notably, these results translate to a 41.2% average relative improvement over the original DPO. Further analyses and ablations corroborate the robustness of our approach, showing that H/yPO maintains its advantage when scaling to larger models and transferring to different datasets, all while preserving performance wide array of downstream tasks. These findings empirically validate thesis: conditionally debiasing the pessimistic reference signal, rather than discarding it entirely, provides more robust and effective path toward stable, high-performing direct preference alignment.
central
2

---

Published as a conference paper at ICLR 2026
Our main contributions can be summarized as follows:
1. We identify and formalize premature satisfuction: for pessimistic pairs (Aret < 0), the DPO weight attenuates once the relative margin turns positive even when the absolute margin remains wrong (Aa 0), revealing tension between correctness and stability.
2. We propose Hybrid-DPO (HyPO), minimalist modification to the DPO objective that clips the reference margin in the DPO loss, thereby preserving the DPO-style logistic loss and, on pessimistic pairs (Arel 0), reducing to an absolute margin update, with no additional computational cost.
3. Across large-scale preference aligument, HyPO improves inference-aligned metrics and raises pairwise win rates on general instruction-follooinn benchmarks (e.g., AlpacaEval, Arena-Hard).
2 RELATED WORK
We situate HyPO within the off-policy branch of RLHF: methods trained on fixed human preference pairs without on-policy rollouts. Accordingly, we do not compare to on on-policy RLHF (Christiano et al., 2017; Shao et al., 2024), and instead focus on recent direct preference alignment (DPO-style) objectives and extensions, underscoring HyPO's distinct features.
Manipulating the Reference Signal. DPO's reliance on Feef has inspired spectrum of solutions. One direction focuses on refining the reference: performance is sensitive to reference choice and strength (Liu et al., 2024b; Azar et al., 2024), leading to Trust-Region DPO (TR-DPO) (Gorbatovski et al., 2025), which dynamically updates Tyee during training to keep it aligned with the policy. RainbowPO (Zhao et al., 2025) mixes the standard reference with an idealized constant-margin policy. Pre-DPO (Pan et al., 2025) further refines the reference model's 's role as an implicit sample reweighter. At the other extreme, reference-free methods like RRHF (Yuan et al., 2023b), SLiC-HF (Zhao et 2023), CPO (Xu et al., 2024), ORPO (Hong et al., 2024a), and SimPO (Meng et al., 2024) eliminate the reference directly optimize absolute log-likelihoods, addressing the training-inference inference mismatch at the cost of removing a stabilizing prior. The core problem of reference mismatch has also been identified in other modalities; for example, MaPO (She et al., 2024) removes the reference to improve multimodal alignment. HyPO offers a principled middle ground: it keeps the DPO-style logistic loss, but clips the reference margin, replacing Aref with max(0, (0, Aret This removes pessimistic anchoring while preserving proximality when the reference is pessimsiimitt..
Explicit Reweighting and Robust Optimization. A parallel line of work makes the enfrrence-induedd sample weighting explicit to improve training dynamics and robustness. Several distributionally robust DPO variants (e.g., rDPO (Chowdhury et al., 2024), Dr.DPO (Wu et al., 2024a), WDPO (Xu et al., 2025)) reweight examples to minimize the impact of label noise (Yuan et al., 2023a; 2024; 2025b;a) or worst-case distributional shifts. Others use data-driven signals; for instance, 3-DPO (Wu et al., 2024b) dynamically adjusts the temperature /3 based on data quality, while WPO (Zhou et al., 2024) reweights pairs to better simulate on-policy optimization from off off-policy data. Some methods operate at a finer granularity, such as TDPO (Zeng al., 2024) and TIS-DPO (Liu et al., 2024a), which apply importance sampling at the token level. While these methods all leverage reweighting, on-policy correction. In contrast, HyPO gates the
their criteria typically noise, data quality, reference contribution by the reference model's 's own pessimism: when Aref 2 0 it behaves exactly like DPO, and when Aref it reduces an absolute margin update on those pairs.
Alternative Objective Formulations. Beyond manipulating the reference or loss weights, several methods alter the structure of the DPO objective. f-DPO generalizes DPO beyond reverse-Ke KL to broader f-divergences (Wang et al., 2024a). R-DPO diagnoses and mitigates length biases within the DPO objective (Park et al., 2024), and AIphaDPO adaptively sets a data-dependent reward margin (Wu et al., 2025). Techniques such as DPO-Shift (Yang et al., 2025) and BDPO (Cho et al., 2025) modify the loss function bound or rebalance the influence of the rejected response, while FocalPO (Liu et al., 2025) adds modulating factor to focus correctly ranked ones. Anchored methods like APO (D' O'OOsserlinckk et al.. 2024) retain the DPO form but add explicit constraints to control the absolute likelihood drift of chosen rejected responses. HyPO differs in where it acts: it preserves DPO's s gradient form (same logistic weighting and 3) but modifies the argument of the loss to Ag - Åref with Åref = max(0, ) thereby reallocating gradient mass on pessimistic pairs without introducing an extra multiplicative weight.
3

---

Published as a conference paper at ICLR 2026
3 MITIGATING MISMATCH IN DIRECT PREFERENCE ALIGNMENT
To fully appreciate the design trade-offs in modern alignment methods, we briefly recall the KL- regularized RLHF principle (Christiano al., 2017; Ziegler et al., 2019) that underpins them. We begin by considering a distribution of prompts Dp from which prompts x are sampled, and let т be the trainable policy initialized from a supervised policy TTFT. In KL-regularized policy optimization
(e.g., PPO; Schulman et al., 2017), a canonical KL-regularized objective used in practice is:
max Py (ro(s, T RR-D, Dit,,,,,KiL, DKc. (Ta(( a:) || Teer( [x)]], 28
(I)
where Tá is reward model trained from human preferences, Tee is fixed reference (typically TSFT), and 7 > O controls the regularization strength. For each x; Th denotes the optimal policy, the per-context optimum of equation has the Gibbs form (Peters al., 2010; Azar al., 2024):
Trerly x) exp(ro(z,w)/T) Ey Tre(y' x)) exp(rs(z,)'))
T x))
which exposes the role of the KL. term: anchors лH near Tiet making updates stable and discouraging distributional drift that could exploit reward misspecification (Schulman et al., 2015; 2017).
Direct Preference Optimization (DPO; Rafailov al., 2023) circumvents explicit reward modeling and reinforcement learning by reformulating the objective for an offline setting, directly optimizing pairwise preferences (x, y² y where y* > y under Bradley-Tene Terry likelihood (Bradley & Terry,
1952). We define the policy and reference margins as:
Ao(x, yt y ) = log ne (y** 1x) - log no ( |x),
(3)
(rt y,, )- = log Tret | log Tret | x))
DPO minimizes the logistic loss:
Dpo = E(z.yt -- log of (( - -ret) = E Bayyy ( log 1 + exp( 3(Ae - Aet))
where 3 = 1/T. Under the Bradley-Teryy model, the first-order optimality condition shares the Gibbs
structure of Kl-regullrized policy optimization (Rafailov al., 2023) and satisfies:
log ( x)) log ( x)) = B (x. v) log (xx).
(5)
This follows by taking logarithms of the KL-regularized optimum in Eq. 2:
(y x)
[zet(y) x) exp(r" (z. y)/
(6)
where r* is preference-consisteee reward and Z(x) the partition function. This makes explicit how DPO preserves proximity to ारef without explicit RL. Differentiating Eq. 4 with respect to the policy margin As yields the per-example muddeet-wweight form:
VolNPO = 3 of 3(As - Aret) VoAs.
(DPGE((.1)
Although the introduction of the reference policy anchors the policy Te near Nee via the relative margin (Ag - Aret) serving as the gradient weight (Pan et al., 2025) in Eq. 7, thereby providing stability and suppressing distributional drift, it simultaneously introduces training-infernnee -ifference metric mismatch (Chen et al., 2024; Meng et al., 2024). We refer to this side effect as premature satisfaction.
Premature Satisfaction. Let the scaled relative margin be:
1
z = 3(A Arer).
WDPO o(=))
I+ e= 2 + WDPO Se
Hence, from Eq. 7 we obtain: | = 30(-2) Vo < BE
When the reference is pessimistic on pair (Aref 0), even if the current policy is still wrong in absolute terms (Ag < 0), as soon as the relative margin becomes positive (Ag - Are > 0), the gradient shrinks exponentially, even though the policy remains wrong in absolute terms (A, < 0).
For example, take = 1, Arof = -3, and A = -1. Then Ag - Ard = 2, we have WDPO = a x 0.119, VoLDRO|| R 0.119 0. /oAlo/ As illustrated by the lower-left region of the heatmap
4

---

Published as a conference paper at ICLR 2026
in Figure 1(a) (i.e., Ag Aref > 0), the weight WDPO decays rapidly; when Aref 0, this low- weight band still covers many points with Ag < O. Premature satisfaction reveals the tension between stability (proximality) and mismatch: while DPO suppresses drift, it may also systematically attenuate the training signal of examples that disagree with Trof yet remain substantially wrong.
Reference-free Alignment. A natural reaction is remove the reference obtain unbiased signal based on the absolute correctness of Ag. aligned with inference- time likelihoods. Reference-fres objectives (e.g.. SimPO; Meng et al., 2024) optimize the absolute margin:
Cabs Ezz.ot . (aos(Ao) )
(ata(As) = log ( exp(_33 exp(--3) Ao)),
(9)
with gradient:
abs
= BB(- a
Volabs = -B0(-3A0) Vodo.
(10)
Compared to Eq. 7. attenuation now depends only the policy's 's own performance: pairs with
Ao < 0 inevitably receive large gradients until Ag crosses zero, directly aligning the training signal with the inference-time objective (increasing log (y+ 1²) and decreasing log no (y x)). However, Eq. discards the very mechanism that implemented proximality in Eq. 4 and Eq. 5. Although recent work suggests that RLHF (Chu et al., 2025; Mukherjee al., 2025) does not typically induce substantial parameter drift, removing this proximality nonetheless markedly reduces the robustness (Pan et al., 2025; Liu al., 2024b) of reference- free alignment.
Direct Preference Optimization with Better Reference. Another approach argues that we should improve reference model to enhance DPO framework. This idea motivates using a stronger reference policy within the DPO framework (Liu et al., 2024b; Azar et al., 2024; Pan et al., 2025). The premise that more accurate reference margin, Aref, makes the relative margin (Do ret) a better learning signal, which should improve optimization.
While often effective, these methods only modify the reference policy reef and leave the core DPO loss unchanged. Consequently. the phenomenon we term premature satisfuction persists when- ever the reference is pessimistic quantify this, we compute the sequence-level likelihood margin in Eq. equation 3 under three Llama-3-8B --adddd references (Base, SFT, and a SimPO-aligned model) on the Ultraeedback train- ing split (Cui et al., 2023), and report (i) the density of Ayer and (ii) summary statistics (full settings and results in Appendix A). Stronger references are indeed pessimistic less often and shift the distribution of Arel to the right; yet substantial fraction, approx- imately 45%, of pairs remains in the pessimistic region even for the SimPO-aligned reference, which is designed to mitigate such mismatch. This imposes a practical ceiling on the "better refer- ence" strategy: it improves stability and overall performance, but the training- training-inferrnce inference mismatch persists wherever Ayef < O.
-26.80
-22.05
14,15
p00
-7.02
11.83
pair (i.c., Arel < 0). To
0.005
0,004
SPT
SimPPO
0.003
0.002
0.00
0.000
400
-200
200
400
Refenence margin
Figure 2: Distribution of the ref- erence margin (Aret) across dif- ferent reference models. The ta- ble reports the mean and median (p50) of Aref for each model.
3.1 HYBRID-DPO
In this section, we introduce Hybrid-DPO (HyPO), an approach designed to mitigate the premature satisfaction problem while preserving the stability of original DPO (Rafailov et al., 2023). The stability benefit of DPO, inherited from KL-regularized RLHF (Eq. 6), relies on the assumption that the reference policy गref provides a constructive learning signal. We argue that this KL-induced proximality should be conditional: When the reference optimistic or neutral (Aref 2 0). it provides a valid anchor for stable, relative improvement, as the DPO objective effectively preserves proximity to the reference (Eq. 5). However, when the reference is pessimistic (Ayef < O), it acts as a misleading prior that causes premature satisfaction; analyzed in our discussion of the DPO gradient, the weight wpro in Eq. attenuates even when the absolute margin As remains negative. HyPO realizes this principle by clipping the reference margin: it preserves DPO's relative objective (relative margin, (A, Arer)) for reliable pairs but nullifies the reference's 's influence for pessimistic ones, compelling the model to optimize the absolute objective (absolute margin, An) instead.
5

---

Published as a conference paper at ICLR 2026
Objective. Formally, for given threshold 2 0 (with by default), modify the DPO
objective by replacing the reference margin Aref with its clipped, non-negative counterpart:
P max (or, 7) and LHYPO = E log [ + exp ( B(Aa - -
(11)
The gradient of the per-example loss retains the DPO structure, weighted by the modified margin:
VoLHyPO = - o 3 - -rer)) Vod0.
(12)
WHyPO
By construction, HyPO coincides with DPO on non-pessimistic samples (Aret 2 7). i.c., its per- example weight in Eq. 12 equals that of DPO in Eq. 7 (WHyrO wDPO), thereby preserving proximal stability where the reference is reliable. On pessimistic samples (Aref < <)), clipping nullifies the misleading anchor and restores an absolute absolute-margin dirginndriven update, pointwise dominating the reference-free weight Wabs := (--3Ae) induced by Eq. 10 (WHYPO 2 Wabs), thus preventing premature attenuation. An intuitive visualization is shown in Figure I(c).
For globally smooth objective, we can replace the hard maximum with a softplus function (Dugas et al., 2000). The smoothed reference margin is defined as:
A 7 + log(1 + exp(o(Are) - 7))),
0,
(13)
where controls the smoothness (a - oo recovers Eq. 11). This simple, smoothed formulation allows HyPO to be implemented as plug-in modification to the standard DPO loss.
Pairwise Win Rate Columns Rows
DPO
-80
0.58
HyPO
Rate (%)
HyPO (Ours)
-100
0.56
SFT
15.7
12.9
0.52
-14
DPQ HyyO
DPO
84.3
44.1
0.50
160
HyPO (Ours)
87.1
55.9
200
300
200
400
Tainiing Step
Training Slep
(a) Agreement Rate
(b) Absolute Margin
(c) Win-Rate Matrix
Figure 3: HyPO improves inference-aligned evaluation metrics and pairwise win rates. (a) Absolute agreement rate over training (higher is better). (b) Absolute margin on the pessimistic subset (Aref 0). (c) Pairwise win-rate. Each cell is the win rate (%) of the row model against the column model on AlpacaEval 2.0 (Li et al., 2023). All results use the same SFT checkpoint of Meta, 2024) trained UltraFeedback (Cui et al., 2023) with either DPO or our HyPO; the training/evalluttinn pipeline and optimization hyperparameters are identical and set to the DPO configuration from Zephyr (Tunstall et al., 2023). See Section 4. for more settings.
Llama-3-8B-BB
Effectiveness of HyPO. To quantify how effectively the HyPO objective mitigates the train- ing ing-inference mismatch compared DPO (Rafailov et al., 2023), we track two key metries on the evaluation set that are closely aligned with inference-time performance. The first, the Absolute Agree-
ment Rate, serves global measure of overall performance. Defined as Agree @t == Pr[A, 0] (with 2e the absolute policy margin as in Eq. 3), it measures the fraction of evaluation pairs whose absolute likelihood ordering matches the desired preference, directly mirroring the inference-time decision rule. Further, we use more targeted Absolute Margin on the Pessimistic Subset, which tracks the average absolute margin, E[A, Are! < 0). on the subset where the (fixed) reference policy is pessimistic. This metric isolates the challenging cases that trigger premature sutisfaction DPO and precisely measures HyPO's s ability improve learning on them. Further details on metric definitions provided in Appendix E. To directly compare final model outputs, report pairwise win rate matrix using AlpacaEval 2.0 (Li et al., 2023). For models i (row) and (column), model j] is the percentage of prompts
the entry W., 100 Pr Pmomole model's 's response is preferred to the column model's. s, under the same judge and prompts; diagonals undefined. Configuration and evaluation protocol are detailed in Section 4.1.
which the row
As demonstrated in Figure 3(a) and (b), HyPO achieves a significantly faster and higher increase in both the global agreement rate and the pessimistic subset margin, confirming its effectiveness

---

Published as a conference paper at ICLR 2026
in mitigating the vaining-inferene mismatch. In this run, the final agreement rate increases from 55.9% for DPO to 60.4% for HyPO. As shown in Figure 3(c), HyPO outperforms DPO head-to-head (Wis (WiyyO.DPO 55.9% vs. Hypo 44.1%). Moreover, against the common baseline SFT, both alignment methods win, with HyPO achieving the larger advantage.
A key advantage of HyPO is that it keeps DPO-like form and is therefore compatible with enhance- ments for the DPO. In our practical implementation, we leverage this compatibility in two ways. First, we adopt better reference (Liu al., 2024b; Pan et al., 2025) by using pre-aligned, same-family model instead of the SFT default. Second, we apply a uniform home advantage margin / to impose stronger learning pressure, leading to the final objective: De max [Arf. 7 h. We do not use length normalization (Park et al., 2024) due to its hyperparameter sensitivity.
4 EXPERIMENTS
We conduct a series of experiments to validate the efficacy of HyPO, We first establish its outperfor- mance against a set of offline preference alignment baselines on standard benchmarks. Subsequently, we perform targeted analyses and ablations to probe the robustness and scalability of HyPO.
4.1 EXPERIMENTAL SETUP AND RESULTS
To ensure comprehensive and fair comparison of our proposed HyPO against various direct pref- erence alignment baselines, closely follow the established experimental settings from existing works (Meng et al., 2024; Rashidinejad & Tian, 2025; Gorbatovski et al., 2025). More detailed experimental settings can be found in Appendix A.
Models and datasets settings. Our main experiments evaluate two open- model families, Mistral-7B (Jiang al., 2023) and Llama3-8B (AI@ Meta, 2024), using the UltraFeedback dataset (> 61k training samples; Cui et al., 2023 for preference alignment. We conduct these experiments in two distinct settings. In the Base setting, we first perform supervised fine-tuning (SFT) on the raw pretrained models (e.g., Mistral- -B--00, Meta-Llama-3-8B) using the UltraChat-200k dataset (z 200k training samples; Ding et al., 2023). The resulting SFT policy then serves as the initial policy for preference alignment on UltraFeedback. In the Instruct setting, we bypass the SFT step and directly apply our preference alignment methods to the official instruction-tur -unned models (e.g.. ---e---llaa------nttttet
Mistral-7B-IIntruecee
again using the UltraFeedback dataset.
Training settings. Our training methodology grounded in established practices from recent alignment literature to ensure robustness and fair comparison (Tunstall et al., 2023; 2025). For the initial SFT stage, we use a peak learning rate of 2 x 10-5, For each reproduced direct preference alignment method, we conduct small grid search over hyperparameters: the peak learning rate is selected from (5 x 10-7, 1 x 10 10-6), while the DPO temperature /3 is chosen from {0.01, 0.1} for standard objectives from {2, 10} for methods employing length normalization. For our proposed h
HyPO, we use a checkpoint of the same architecture pre-aligned with SimPO. The home advantage margin was
h 10, while the threshold was kept at its default of 7 = O. Detailed settings are listed in Appendix A. All models are trained for single epoch using the Adam W optimizer (Loshchilov & Hutter, 2017), and we employ a cosine learning rate schedule with a warm-up phase over the first 10% of training steps and a global batch size of 128.
Evaluation benchmarks. We primarily assess model performance on two popular instruction- following benchmarks: AlpacaEval 2.0 (Li et al., 2023; Dubois et al., 2024) and al., 2024a;b). Consistent with prior work (Rashidinejad & Tian, 2025; Gorbatovski al., 2025), we use GPT-4-Preview-111 106 as the automated judge to compute the primary metric: the win rate, which the percentage of times model's 's response preferred over a baseline's. The specific baseline model varies by benchmark: AlpacaEval reports win rates against GPT-44Preview-1I 106 itself, whereas Arena-Hard uses GPT-4-0314, Our evaluation protocols follow established practices from Tunstall et al. (2023); Meng et al. (2024); Tunstall et al. (2025), with the minor modification of using a more recent vLLM release (Kwon et al., 2023) for generation.
(Li
Direct preference alignment baselines. We compare our method against suite of direct preference alignment baselines. Our comparison starts with two classic off-policy methods: SLiC-HF (Zhao et al., 2023) and DPO (Rafailov al., 2023), the latter of which serves as our primary baseline.
7

---

Published as a conference paper at ICLR 2026
CPO (Xu et al., 2024) and KTO (Ethayarajh et al., 2024) DPO variants that reshape the loss from, respectively, a contrastive perspective and an asymmetric utility. SimPO (Meng et al., 2024) is tigh-performing reference-free representative that operates the absolute likelihood margin. FocalPO (Liu et al., 2025), TR-DPO (Gorbatovski et al., 2025), and RainbowPO (Zhao et al., 2025) recent DPO improvements: focal reweighting, trust-region constraints the relative margin for stability, and unified practical recipe combining length normalization and policy mixing.
Table 1: Main results reported from AlpacaEval 2.0 (Li et al., 2023; Dubois et al., 2024) and Arena- Hard-v0.1 (Li et al., 2024a;b). LC and WR denote length-controlled win rate and raw win rate, respectively. The best results are highlighted in bold, and the second-best are underlined.
Mistral-Base (7B)
Mistral-Instruet (7B)
Method
AlpacaEval
Arena-Hard
AlpacaEval
Arena-Hard
(C)
(%)
WR (%))%
(C(%)
WR (%)
WR WR(%) (%)
SLiC-HE (Zhao et al., 2023)
11.6
9.1
5.4
32.4
31.2
16.9
DPO (Rafailov al., 2023)
22.6
18.5
7.9
35.1
31.4
15.4
CPO (Xu et al., 2024)
13.1
11.6
6.4
34.9
39.9
21.0
KTO (Ethayarajh al., 2024)
12.9
9.3
6.6
35,0
31.3
17.5
SimPO (Meng et al., 2024)
27.3
25.4
11.2
38.4
40.0
20.5
FocalPO (Liu et al., 2025)
25.8
19.7
8.2
35.9
35.0
18.7
TR-DPO (Gorbatovski al., 2025)
24.9
21.4
9.5
36.5
33.7
RuinbowPO (Zhao et al., 202.5
28.4
26.7
9.2
35.7
33.9
18.2
HlyO (Ours)
32.8
29.6
13.9
38.9
47.9
25.2
Llama-3-Basy (8B)
Llama-3-Instree (8B)
Method
AlpacaEval
Arena-Hard
AlpacaEval
Arena-Hard
WR
LC(%)
SLiC-HF (Zhao al., 2023)
19.8
15.9
14.3
36.7
36.8
25.1
DPO (Rafailov et al., 2023)
24.3
21.9
23.0
40.9
41.3
31.5
CPO (Xu al., 2024)
22.3
24.6
12.2
38.1
40.4
30,0
KTO (Ethayarajh et al., 2024)
23.6
20.3
18.4
40.5
39.0
30.5
SimPO (Meng al., 2024)
30.7
26.2
30.1
46.0
43.1
32.1
FocalPO (Liu et al., 2025)
27.2
25.4
27.9
45.1
43.6
30.2
TR-DPO (Gorbatovski al., 2025)
31.8
30.2
31.0
46.7
42.7
32.5
RainbowPO (Zhao et al., 2025)
30.3
27.1
28.6
46.7
43.5
31.3
HyPO (Ours)
34.7
33.6
33.5
49.5
46.2
Main Results on Benchmarks. We present the main results of our comparative analysis in Table 1. Across all four experimental settings, including two model families (Mistral-7B and Llama-3-8B) and two initial states (Base and Instruct), our proposed method, HyPO, consistently and significantly outperforms all baselines on both AlpacaEval and Arena-Hard benchmarks. The results show that HyPO achieves average relative improvement of 41.2% over the original DPO. Furthermore, when compared to SimPO, strong reference- free competitor, HyPO delivers an average relative improvement of 15.1%. Taken together, these results highlight the practical value of our methodology. By mitigating the reference mismatch while preserving the core DPO framework, our method translates targeted theoretical improvement into superior performance benchmarks.
4.2 FURTHER ANALYSIS AND ABLATIONS
To further validate the effectiveness and robustness of HyPO, evaluate downstream transfer. ablations (including sensitivity to 7), measure training runtime, and study scaling. For comparability, benchmark against reference-based baseline (DPO; Rafailov et al., 2023) and reference- free baseline (SimPO; Meng et al., 2024) under largely the same settings as Section 4.1.
Downstream tasks. One of the challenges in alignment is that aligning model can reduce model's general capabilities. This degradation is often attributed to the over-optimization (Gao al., 2023; Rafailov et al., 2024) of the alignment objective. To assess whether approach preserves broad
8

---

Published as a conference paper at ICLR 2026
Table 2: Ablation study of HyPO's components.
Figure 4: Sensitivity the threshold 7, us- ing Meta-Llama-33-88 ----tuutttct...
Llama-3- 3-Models
Base
Instruct
50
LCX%)
WR(%)
LC()) WR(S)) Avp. A(9))
Alpacatival win
HyPO (Ours)
34.7
33.6
49.5
46.2
Home Advantage
33.2
32.5
47.4
45.1
1.5
44
Better Reference
29.3
27.1
45.3
45.9
-4.1
42
BR and HA
28.8
28.5
45.2
44,9
-4.2
40
30
Standard DPO
24.3
21.9
40.9
41.3
8899
-400
(0p maxx(eer.)) V)1
Y. ('0le (aex( V}1
Table 3: Further results reported from AlpacaEval 2.0 (Li et al., 2023; Dubois et al., 2024) and Arena-Hard-ve (Li et al., 2024a;b). LC and WR denote length-comtrolled win rate and raw win rate, respectively. The best results are highlighted in bold.
Helpsteer2
Mistral-Nemo-nnstruud (12B)
Qwen-2.5-7B
Method
AlpacaEval
Areua-Hard
AlpacaEval
Arena-Hard
AlpacaEval
Arena-Hard
WR (%)
WR (%)
WR (%)
(R)
LC (%)
WR
WR (%)
DPO (Rafailov et al., 2023)
16.2
14.3
4.5
50.4
49.2
35.5
27.9
22.4
28.8
SimPO (Meng et al., 2024)
18.6
16.0
7.0
52.1
46.4
33.9
33.7
22.2
33.8
HyPO (Ours)
22.3
19.5
9.3
55.7
54.9
38.9
38.0
30.7
36.2
utility, we evaluated the aligned models on a comprehensive suite of downstream tasks, using LM Evaluation Harness (Biderman et al., 2024). Our evaluation covers four key domains. We test knowledge and language understanding using MMLU (Hendrycks et al., 2020) and ARC-Challenge (Clark et al., 2018). For commonsense reasoning. we employ HellaSwag (Zellers al., 2019) and Winogrande (Sakaguchi et al., 2021). Factuality is assessed with TruthfulQA (Lin et al., 2021), and mathematical reasoning is measured by GSM8K (Cobbe et al., 2021). The results, presented in Table 5 (see Appendix D for full details), demonstrate that HyPO maintains or improves performance across all evaluated tasks relative to the corresponding baselines. This indicates that our method avoids additional performance degradation downstream tasks that can be induced by alignment.
Ablations and threshold y sensitivity. We ablate two enhancements in our HyPO implementation: the use of better reference model and home advantage margin. The results in Table 2 show that both components are beneficial. Notably, the ablated HyPO variants still significantly outperform the standard DPO baseline, finding consistent with our head-to-head comparisons in Figure confirming the effectiveness of our core mechanism. Furthermore, we analyze the model's 's sensitivity the threshold y from Eq. 11. As shown in Figure 4, performance remains stable across range of ? values around 0. Given this low sensitivity, we adopt the most principled and interpretable setting, y 0. These validate the contributions of proposed method and the stability of our configuration.
Running time. As a drop-in replacement for standard DPO, HyPO introduces negligible additional computation. In the experiment shown Figure 3, conducted with identical hardware and settings, the wall clock training time for HyPO differed from that of DPO by ~ 1% (7.1 hours vs. 7.2 hours).
Example Responses. To offer a qualitative view, Appendix compares sample model outputs. The responses suggest that HyPO can generate more considered answers that acknowledge subtleties in the prompt, distinguishing it from the more direct outputs of baseline methods.
Scalability. To further validate the scalability of our proposed method, we conduct extended experiments across varying scales, datasets, and architectures. We evaluate HyPO along three axes: (i) a dataset shift from UltraFeedback to HelpSteer2 (Wang et al., 2024b); (ii) model-size shift to Mistral-Nemo-111 12B (Mistral AI, 2024); and (iii) an architectural shift to Qwen-2.5-77 2.5-7B (Qwen, 2025). We follow the evaluation protocol and most hyperparameters from Section 4.1 (details in Appendix A). As summarized Table 3, HyPO consistently improves over baselines across all settings, indicating robustness distribution changes, parameter scaling, and diverse model families.

---

Published as a conference paper at ICLR 2026
5 CONCLUSION
In this paper, we introduced Hybrid-DPO (HyPO) to address premature satisfaction, a critical failure in Direct Preference Optimization (DPO) where pessimistic reference model down-weights learning signals. HyPO's s simple conditional mechanism, clipping the reference margin at zero, corrects this bias while retaining DPO's inherent stability. Empirically, HyPO consistently and significantly outperforms both various DPO and strong reference-free baselines across range of models and benchmarks, proving robust and scalable without an additional "alignment tax". Our work provides evidence that direct preference alignment could be enhanced by conditionally debiasing the reference signal, rather than discarding it, thereby achieving more effective and stable compromise.
LIMITATIONS AND FUTURE WORK
Our approach aims to mitigate rather than fully resolve the training-inferendd mismatch. By design, HyPÒ retains the reference- based behavior on optimistic pairs to preserve training stability and leverage better reference models, meaning the training-inferente Furthermore, our method assumes general label reliability; in cases of severe label noise where the reference model correctly disagrees with a wrong label, HlPOO enforces strong learning signal and may arguably amplify noise compared to DPO. Finally, while we prioritized minimalist, parameter- free clipping mechanism, more sophisticated functional forms, such as adaptive clipping thresholds or learned gating mechanisms, remain promising directions for future research.
mismatch persists for this subset.
ACKNOWLEDGMENTS
The authors would like to thank the anonymous reviewers for their insightful and constructive com- ments. Suqin Yuan thanks Muyang Li and Li He for their valuable advice. Tongliang Liu is partially supported by the following Australian Research Council projects: FT220100318, DP260102466, DP220102121, LP220100527, LP220200949. Suqin Yuan is supported in part by the OpenAl Researcher Access Program (projects 05950 and 19000). This research/project is supported by the National Research Foundation, Singapore under its National Large Language Models Funding Initiative (AISG Award No: AISG-NMLP 90GG-NMLP-202-0033 Any opinions, findings and conclusions or recommendations expressed in this material are those of the author(s) and do not reflect the views of National Research Foundation, Singapore.
ETHICS STATEMENT
Our research aims to improve the alignment of large language models with human preferences, a goal intended to enhance their safety, helpfulness, and reliability. We primarily use publicly available datasets, such as UltraFeedback (Cui et al., 2023) and HelpSteer2 (Wang et al., 2024b), which are aggregated from existing sources. While we did not collect new human data, we acknowledge that these datasets may contain inherent biases or sensitive information reflective of their origins. Our work does not aim create models from scratch but to refine existing ones (Mistral and Llama-3; Jiang et al., 2023; AI@ Meta, 2024). We recognize that any advancement in AI capabilities carries a dual-use risk (Hong et al., 2024b; Huang et al., 2024; Lin et al., 2025; Zhang et al.; Huang al., 2026). However, the explicit goal of our method, HyPO, is to make models' behavior more predictable and aligned with positive human values, thereby mitigating risks associated with misaligned AL We believe that research into robust alignment techniques is a crucial step toward the responsible development and deployment of AI systems.
REPRODUCIBILITY STATEMENT
Our method, Hybrid-DPO (HyPO), is designed as minimalist, plug-in modification to the standard Direct Preference Optimization (DPO) objective. This design ensures that our work can be easily reproduced by integrating our logic into popular alignment libraries and codebases that already support DPO, such as dlinment-handdoon in experiments
(Tunstall et al., 2025). All models and datasets used open-source and publicly accessible through the Hugging Face, ensuring full transparency. The base models include Mistral- (Jiang et al., 2023), Meta-Llama-3- (AI @ Meta, 2024), and Mistral-Nemo- 12B (Mistral AI, 2024). The datasets for supervised fine-tuning
10

---

Published as a conference paper at ICLR 2026
and preference alignment include UltraChat-225 200k (Ding al., 2023), UltraFeedback (Cui al., 2023), HelpSteer2 (Wang et al., 2024b), and OpenAssistan22 (Köpf et al., 2023). For evaluation on benchmarks like AlpacaEval 2.0 (Li et al., 2023; Dubois et al., 2024). Arena-Hard-ve (Li et al., 2024a;b), and LM Evaluation Harness (Biderman et al., 2024), we followed established protocols that utilize the GPP--4-Preview- 106 API commercial API is managed by third party, our evaluation methodology is public and can be adapted for use with other capable judge models. We have release our code f'cilitate further research: https: / /ggttub.hbb// www.commmmmmmoe.om http/mmllan Lalllaa//a 202
automated judge. While access this
THE USE LARGE LANGUAGE MODELS
During the preparation of this manuscript, the authors utilized large language models (LLMs) in several capacities. As writing and coding assistants, they were used to improve grammar, spelling, and sentence structure for clarity and to generate code for training and visualization. The subject of this research is the alignment of large language models themselves, by using LLMs such as Mistral and Llama-3 (Jiang et al., 2023; AI @ Meta, 2024). Furthermore, part of the evaluation methodology, the authors employed a closed-source LLM's API (GPT-4-Preview-- 106) to act as an automated judge for assessing the quality the trained LLMs. The authors maintained full intellectual control of this paper. LLMs were used as tools to augment the research process, and the final manuscript reflects the authors' own work and insights.
REFERENCES
AetMe Llama 3 model card. 2024.
Amanda Askell, Yuntao Bai, Anna Chen, Dawn Drain, Deep Ganguli, Tom Henighan, Andy Jones, Nicholas Joseph, Ben Mann, Nova DasSarma, et al. A general language assistant as a laboratory for alignment. arXiv preprint arXiv:21 2021.
Mohammad Gheshlaghi Azar, Zhaohan Daniel Guo, Bilal Piot, Remi Munos, Mark Rowland, Michal Valko, and Daniele Calandriello. A general theoretical paradigm to understand learning from human preferences. In International Conference Artificial Intelligence and Statistics, pp. 4447-4453 PMLR, 2024.
Stella Biderman, Hailey Schoelkopf, Lintang Sutawika, Leo Gao, Jonathan Tow, Baber Abbasi, Alham Fikri Aji, Pawan Sasanka Ammanamanchi, Sidney Black, Jordan Clive, et al. Lessons from the trenches on reproducible evaluation of language models. arXiv preprint arXiv:2405.14782 2024.
Ralph Allan Bradley and Milton E Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, (9(334))324-35, 1952.
Angelica Chen, Sadhika Malladi, Lily H Zhang, Xinyi Chen, Qiuyi Zhang, Rajesh Ranganath, and Kyunghyun Cho. Preference learning algorithms do not learn preference rankings. Advances in Neural Information Processing Systems, 17:109988-10196 2024.
Jay Hyeon Cho, JunHyeok Oh, Myunsoo Kim, and Byung-Jun Lee. Rethinking dpo: The role of rejected responses in preference misalignment. arXiv preprint arXiv:2506. /2725, 2025.
Sayak Ray Chowdhury, Anush Kini, and Nagarajan Natarajan. Provably robust dpo: Aligning language models with noisy feedback. arXiv preprint arXiv:2403.00409, 2024.
Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. Advances in neural information processing systems, 30, 2017.
Tianzhe Chu, Yuexiang Zhai, Jihan Yang, Shengbang Tong, Saining Xie, Dale Schuurmans, Quoc V Le, Sergey Levine, and Yi Ma. Sft memorizes, generalizes: A comparative study of foundation model post-training. arXiv preprint arXiv:2501.17 1766,, 2025.

---

Published as a conference paper at ICLR 2026
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint 0rXhv:1803005555 2018.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:21 10. 14168, 2021.
Ganqu Cui, Lifan Yuan, Ning Ding, Guanming Yao, Wei Zhu, Yuan Ni, Guotong Xie, Zhiyuan Liu, and Maosong Sun. Ultrafeedback: Boosting language models with high-quality feedback. 2023.
Tri Dao. Flashattention-2: Faster attention with better parallelism and work partitioning, arXiv preprint arXiv:2307.0869) 2023.
Ning Ding, Yulin Chen, Bokai Xu, Yujia Qin, Zhi Zheng, Shengding Hu, Zhiyuan Liu, Maosong Sun, and Bowen Zhou. Enhancing chat language models by scaling high-quality instructional conversations. arXiv preprint arXiv:2305. 14233, 2023.
Karel D'Oosterlinck, Winnie Xu, Chris Develder, Thomas Demeester, Amanpreet Singh, Christopher Potts, Douwe Kiela, and Shikib Mehri. Anchored preference optimization and contrastive revisions: Addressing underspecification in alignment. arXiv preprint Xxi::2408.06266 2024.
Yann Dubois, Balázs Galambosi, Percy Liang, and Tatsunori B Hashimoto. alpacaeval: A simple way debias automatic evaluators. arXiv preprint arXiv:2404. 04475, 2024.
Charles Dugas, Yoshua Bengio, François Bélisle, Claude Nadeau, and René Garcia. Incorporating second-order functional knowledge for better option pricing. Advances in neural information processing systems, 13, 2000.
Kawin Ethayarajh, Winnie Xu, Niklas Muennighoff, Dan Jurafsky, and Douwe Kiela. Kto: Model alignment as prospect theoretic optimization. arXiv preprint arXiv:2402.01306, 2024.
Leo Gao, John Schulman, and Jacob Hilton. Scaling laws for reward model overoptimization. In International Conference on Machine Learning, pp. 11835-10866. PMLR, 2023.
Alexey Gorbatovski, Boris Shaposhnikov, Alexey Malakhov, Nikita Surnachev, Yaroslav Aksenov, Ian Maksimov, Nikita Balagansky, and Daniil Gavrilov. Learn your reference model for real good alignment. arXiv preprint arXiv:2404.0966 09656, 2025.
Li He, Qiang Qu, He Zhao, Stephen Wan, Dadong Wang, Lina Yao, and Tongliang Liu. Unifying stable optimization and reference regularization in RLHF, In The Fourteenth International Confer- ence on Learning Representations, 2026. URL https:ttp///.cc.... OрqBaCTtW4.
htttps://operrevies opwnrevieeeeev.ee//eeee net/ / forum? id=
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and
Jacob Steinhardt. Measuring massive multitask language understanding. arXiv preprins arXiv:2009.0m 000:00009.0300, 2020.
Jiwoo Hong, Noah Lee, and James Thome. Orpo: Monolithic preference optimization without reference model. arXiv preprint arXiv:2403.07691, 2024
Ziming Hong, Zhenyi Wang, Li Shen, Yu Yao, Zhuo Huang, Shiming Chen, Chuanwu Yang, Mingming Gong, and Tongliang Liu. Improving con-transferable representation learning by harnessing content and style. In The twelfth international conference on learning representations, 2024b.
Zhuo Huang, Chang Liu, Yinpeng Dong, Hang Su, Shibao Zheng, and Tongliang Liu. Machine vision therapy: Multimodal large language models can enhance visual robustness via denoising in-context learning. In Forty-first International Conference on Machine Learning, 2024. URL https: /ppnnevveeeeeeeeeeeeeeeeeeeeeeee .nt// wtw.eeeeneeettoomiidiaaeaeem forum?id-LwwwWWee....
Zhuo Huang, Qizhou Wang, Ziming Hong, Shanshan Ye, Bo Han, and Tongliang Liu. Is gra- dient ascent really necessary? memorize to forget for machine unlearning. arXiv preprint 2Xi4:2602.064411, 2026.
12

---

Published as a conference paper at ICLR 2026
Dongsheng Jiang, Yuchen Liu, Songlin Liu, Jin'e Zhao, Hao Zhang, Zhen Gao, Xiaopeng Zhang, Jin Li, and Hongkai Xiong. From clip to dino: Visual encoders shout in multi-modal large language models. arXiv preprint arXinv2310.08825, 2023.
Andreas Köpf, Yannic Kilcher, Dimitri Von Rütte, Sotiris Anagnostidis, Zhi Rui Tam, Keith Stevens, Abdullah Barhoum, Duc Nguyen, Oliver Stanley, Richárd Nagyfi, et al. Openassis- tant converssatioossdddnorrriiinnnnn large language model alignment. Advances neural information processing systems, 16:47669-476666 2023.
Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles, 2023.
Tianle Li. Wei-Lin Chiang, Evan Frick, Lisa Dunlap, Tianhao Wu, Banghua Zhu, Joseph E Gonzalez, and Ion Stoica. From crowdsourced data to high-quality benchmarks: Arena-hard and benchbuilder pipeline. arXiv preprint arXiv:2406. /939, 2024a.
Tianle Li, Wei Chiang, Evan Frick, Lisa Dunlap, Banghua Zhu, Joseph E. Gonzalez, and Ion Stoica. From live data to high-quality benchmarks: The arena-hard pipeline, April 2024b.
Xuechen Li, Tianyi Zhang, Yann Dubois, Rohan Taori, Ishaan Gulrajani, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Alpacaeval: An automatic evaluator of instruction-follooing models, 2023.
Runqi Lin, Alasdair Paren, Suqin Yuan, Muyang Li, Philip Torr, Adel Bibi, and Tongliang Liu. Force: Transferable visual jailbreaking attacks via feature over-reliance correction. arXiv preprint arXiv:2509.21029 2025.
Stephanie Lin, Jacob Hilton, and Owain Evans. Truthfulqa: Measuring how models mimic human falsehoods. arXiv preprint arXiv:2109.07958, 2021.
Aiwei Liu, Haoping Bai, Zhiyun Lu, Yanchao Sun, Xiang Kong, Simon Wang. Jiulong Shan, Albin Madappally Jose, Xiaojiang Liu, Lijie Wen, al. Tis-dpo: Token-level importance sampling for direct preference optimization with estimated weights. arXiv preprint arXiv:2410.04350, 2024a.
Tong Liu, Xiao Yu, Wenxuan Zhou, Jindong Gu, and Volker Tresp. FocalPO: Enhancing preference optimizing by focusing correct preference rankings. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), July 2025.
Yixin Liu, Pengfei Liu, and Arman Cohan. Understanding reference policies in direct preference optimization. arXiv preprint arXiv:2407.13709, 2024b.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint 17X1v:1111.05101, 2017.
Yu Meng, Mengzhou Xia, and Danqi Chen. Simpo: Simple preference optimization with reference- free reward. Advances in Neural Information Processing Systems, 37:124198-124235 2024.
Mistral AI. distral-nemo--2b-inntt
2024. Model card; Apache-2.0; accessed 2025-09-21.
Sagnik Mukherjee, Lifan Yuan, Dilek Hakkani-Tur, and Hao Peng. Reinforcement learning finetunes small subnetworks in large language models. arXiv preprini arXiv:2505. /711,, 2025.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730- 27744, 2022.
Junshu Pan, Wei Shen, Shulin Huang, Qiji Zhou, and Yue Zhang. Pre-dpo: Improving data utilization in direct preference optimization using a guiding reference model. arXiv preprint arXiv:2504. 15843, 2025.
13

---

Published as a conference paper at ICLR 2026
Ryan Park, Rafael Rafailov, Stefano Ermon, and Chelsea Finn. Disentangling length from quality in direct preference optimization. arXiv preprint arXi::2403.1915,, 2024.
Jan Peters, Katharina Mulling, and Yasemin Altun. Relative entropy policy search. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 24, pp. 1607-1612, 2010.
Qwen. Qwen2.5 technical report. arXiv preprint urXiv:24/2./5115v4 2025.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances 2023.
in Neural Information Processing Systems,
Rafael Rafailov, Yaswanth Chittepu, Ryan Park, Harshit Sushil Sikchi, Joey Hejna, Brad Knox, Chelsea Finn, and Scott Niekum. Scaling laws for reward model overoptimization direct alignment algorithms. Advances in Neural Information Processing Systems, 37:126207-122242,
2024.
Paria Rashidinejad and Yuandong Tian. Sail into the headwind: Alignment via robust rewards and dynamic labels against reward hacking. In The Thirteenth International Conference on Learning Representations, 2025.
Jeff Rasley, Samyam Rajbhandari, Olatunji Ruwase, and Yuxiong He. Deepspeed: System opti- mizations enable training deep learning models with over 100 billion parameters. Proceedings of the 26th ACM SIGKDD international conference knowledge discovery & data mining, pp. 3505-3506, 2020.
Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge at scale. Communications of the ACM, 64(9)999-106, 2021.
John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. Francis Bach and David Blei (eds.), Proceedings of the 32nd International Conference Machine Learning, volume 37 of Proceedings of Machine Learning Research, pp. 1889-1897, Lille, France, 07-09 Jul 2015. PMLR.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347 2017.
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, et al. Deepseekmath: Pushing the limits of mathemat- ical reasoning open language models. arXiv preprint Xii::2442.0330000 2024.
Archit Sharma, Sedrick Scott Keh, Eric Mitchell, Chelsea Finn. Kushal Arora, and Thomas Kollar. A critical evaluation of ai feedback for aligning large language models. Advances in Neural Information Processing Systems, 37:29166-2911 29199099-9.,,,- 2024.
Shuaijie She, Wei Zou, Shujian Huang, Wenhao Zhu, Xiang Liu, Xiang Geng, and Jiajun Chen. Mapo: Advancing multilingual reasoning through multilingual gignment-assprefereee optimization. arXiv preprint arXiv:2401.0663 2024.
Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. Learning to summarize with human feedback. Advances in neural information processing systems, 33:3008-3021, 2020.
Lewis Tunstall, Edward Beeching. Nathan Lambert, Nazneen Rajani, Kashif Rasul, Younes Belkada, Shengyi Huang, Leandro Von Werra, Clémentine Fourrier, Nathan Habib, et al. Zephyr: Direct distillation of alignment. arXiv preprint arXiv:2310. 16944, 2023.
Lewis Tunstall, Edward Beeching, Nathan Lambert, Nazneen Rajani, Shengyi Huang, Kashif Rasul, Alvaro Bartolome, Carlos M. Patiño, Alexander M. Rush, and Thomas Wolf. The alignment handbook, 2025.
14

---

Published as a conference paper at ICLR 2026
Chaoqi Wang. Yibo Jiang, Chenghao Yang, Han Liu, and Yuxin Chen. Beyond reverse kl: Generaliz- ing direct preference optimization with diverse divergence constraints. In The Twelfth International Conference Learning Representations, 2024a.
Zhilin Wang, Yi Dong, Olivier Delalleau, Jiaqi Zeng, Gerald Shen, Daniel Egert, Jimmy Zhang. Makesh Narsimhan Sreedhar, and Oleksii Kuchaiev. Helpsteer 2: Open-source dataset for training top-perforning reward models. Advances Neural Information Processing Systems, 37: 14:4 1501, 2024b.
Junkang Wu, Yuexiang Xie, Zhengyi Yang, Jiancan Wu, Jiawei Chen, Jinyang Gao, Bolin Ding, Xiang Wang, and Xiangnan He. Towards robust alignment of language models: Distributionally robustifying direct preference optimization. arXiv preprint arXiv:2407.02 070000000...... 2024a.
Junkang Wu, Yuexiang Xie, Zhengyi Yang, Jiancan Wu, Jinyang Gao, Bolin Ding, Xiang Wang. and Xiangnan He. beta-dpo: Direct preference optimization with dynamic beta. Advances in Neural Information Processing Systems, 37: 1211129944299999 2024b.
Junkang Wu, Xue Wang, Zhengyi Yang, Jiancan Wu, Jinyang Gao, Bolin Ding, Xiang Wang, and Xiangnan He. Alphadpo: Adaptive reward margin for direct preference optimization. In International Conference Machine Learning, 2025.
Haoran Xu, Amr Sharaf, Yunmo Chen, Weiting Tan, Lingfeng Shen, Benjamin Van Durme, Kenton Murray, and Young Jin Kim. Contrastive preference optimization: Pushing the boundaries of Ilm performance in machine translation. arXiv preprint arXiv:2401.0848 08.11.......... 2024.
Zaiyan Xu, Sushil Vemuri, Kishan Panaganti, Dileep Kalathil, Rahul Jain, and Deepak Ramachandran. Robust IIm alignment via distributionally robust direct preference optimization. arXiv preprint arXiv:2502.0199: 2025.
Xiliang Yang, Feng Jiang, Qianen Zhang, Lei Zhao, and Xiao Li. Dpo-shift: Shifting the distribution of direct preference optimization. arXiv preprint arXiv:2502.07599 2025.
Suqin Yuan, Lei Feng, and Tongliang Liu. Late stopping: Avoiding confidently learning from mislabeled examples. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 16079-16088, 2023a.
Suqin Yuan, Lei Feng, and Tongliang Liu. Early stopping against label noise without validation data. In The Twelfth International Conference on Learning Representations, 2024.
Suqin Yuan, Lei Feng, Bo Han, and Tongliang Liu. Enhancing sample selection against label noise by cutting mislabeled easy examples. In The Thirty-ninth Annual Conference on Neural Information Processing Systems, 2025a.
Suqin Yuan, Runqi Lin, Lei Feng. Bo Han, and Tongliang Liu. Instance-dependent early stopping. In The Thirteenth International Conference on Learning Representations, 2025b.
Zheng Yuan, Hongyi Yuan, Chuanqi Tan, Wei Wang, Songfang Huang, and Fei Huang. Rrhf: Rank responses align language models with human feedback without tears. arXiv preprint arXiv:2304.05330. 2023b.
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can machine really finish your sentence? arXiv preprint
2019.
Yongcheng Zeng, Guoqing Liu, Weiyu Ma, Ning Yang, Haifeng Zhang, and Jun Wang. Token-level direct preference optimization. arXiv preprint arXiv:2404.1 119999....999,..... 1999, 2024.
Yichi Zhang, Siyuan Zhang, Yao Huang, Zeyu Xia, Zhengwei Fang, Xiao Yang, Ranjie Duan, Dong Yan, Yinpeng Dong, and Jun Zhu. Stair: Improving safety alignment with introspective reasoning. In Forty-secont International Conference on Machine Learning.
Hanyang Zhao, Genta Indra Winata, Anirban Das, Shi-Xiong Zhang, David Yao, Wenpin Tang, and Sambit Sahu. RainbowPO: A unified framework for combining improvements in preference optimization. In The Thirteenth International Conference Learning Representations, 2025.
15

---

Published as a conference paper at ICLR 2026
Yao Zhao, Rishabh Joshi, Tianqi Liu, Misha Khalman, Mohammad Saleh, and Peter J Liu. Slic-hf: Sequence likelihood calibration with human feedback. arXiv preprint arXiv:2305 10425, 2023.
Wenxuan Zhou, Ravi Agrawal, Shujian Zhang, Sathish Reddy Indurthi, Sanqiang Zhao, Kaiqiang Song, Silei Xu, and Chenguang Zhu. Wpo: Enhancing rihf with weighted preference optimization. arXiv preprint arXiv:2406. 1827, 2024.
Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences. arXiv preprint arXiv:1909.08593, 2019.
16

---

Published as a conference paper at ICLR 2026
A EXPERIMENTAL DETAILS
This section provides a comprehensive overview of the experimental settings used to validate our proposed method, Hybrid-DPi (HyPO). Our experimental design is grounded established practices from recent alignment literature to ensure fair, robust, and reproducible comparisons (Tunstall et al., 2023; Meng et al., 2024; Tunstall et al., 2025). We detail the models and datasets, the multi-stage training pipeline, hyperparameter tuning, and the evaluation protocols for all experiments.
A.1 MODELS AND DATASETS
Our experiments leverage suite of publicly available models and datasets to ensure full transparency and facilitate replication. We conduct experiments on two popular open-source model families: Mistral-7B (Jiang et al., 2023) and Meta-Llama-3-8FF (AI@Meta, 2024). We test each model in two settings: Base setting, starting from the raw pretrained weights, and an Instruct setting, starting from the official instruction-tuned checkpoints. For our scalability study, we also use Mistral-Nemo- Instruct- 12B (Mistral AI, 2024).
Supervised Fine- Tunnng (SFT) Datasets. For experiments in the Base setting, we first conduct supervised fine-tuning obtain a capable initial policy. We use UltraChat-200k (Ding al., 2023) for the main experiments and OpenAssistant2 (Köpf al., 2023) for the dataset transfer experiment.
Preference Datasets. For the core preference alignment stage, our main experiments use the UltraFeedback dataset (Cui et al., 2023), from which use the official training split for optimization and hold out fixed validation set. For the dataset transfer scalability experiment, we use the HelpSteer2 dataset (Wang et al., 2024b). In Instruct setting, UltraFeedback dataset released by Meng et al. (2024).
A.2 PIPELINE AND HYPERPARAMETERS
Our training process consists of two main stages for the Base setting (SFT followed by preference alignment) and one stage for the Instruct setting (preference alignment only).
Supervised Fine- Fine-Tuning (SFT) Stage.
For the Base models, we first create an SFT version by training on the relevant dataset for one epoch. We use the AdamW optimizer (Loshchilov & Hutter, 2017) with a global batch size of 128, a peak learning rate of 2 x 10-5, and cosine learning rate schedule with 10% warm-up phase. The maximum sequence length set to 2048, This SFT model serves as both the initial policy and the default reference model (Trer) for preference alignment.
Preference Alignment Stage.
We conduct a small targeted hyperparameter search for each method.
General Hyperparameters. To ensure consistent optimization dynamics, across all preference align- ment experiments, we train for single epoch with a global batch size of 1 128, maximum sequence length of 2048, and a maximum prompt length of 1800 tokens. We use the AdamW optimizer (Loshchilov & Hutter, 2017) with B1 0.9 and Bg 0.999, and cosine learning rate schedule with 10% warm-up phase. To maintain the fixed global batch size 128 despite varying memory requirements across stages (c.g., the additional reference model overhead in DPO) and hardware configurations (ranging from to 4 GPUs), we adjusted the per-device micro-batch size and gradient accumulation steps based the available GPU memory and model scale. For any specific set of comparative experiments, we ensured that these effective batch size settings were strictly identical across all baselines to guarantee fair comparison.
Seehodd-ppccfic Hyperparameters. We identified the peak learning rate and the DPO temperature / as the most sensitive hyperparameters. For each method we reproduced, we performed a small grid search and selected the best configuration based on the lowest validation loss on our held-out set. In our experiments, we used some of the checkpoints released by Meng et (2024).
Peak Learning Rate: Searched over (5 10-7, 7,, 10-9).
17

---

Published as a conference paper at ICLR 2026
Temperature 3: For standard objectives (DPO, CPO, KTO, FocalPO, TR-DPO, and HyPO.), we searched over {0.01, 0.13. For methods employing length normalization (e.g. RainbowPO, SimPO), we followed Meng et al. (2024) recommendations and searched over {2, 10).
Evaluation Stage.
Dnstruction-oollownn Benchmarks. We use GPT---Previee--1 106 as the automated judge. The primary metric is the win rate (%). For AlpacaEval 2.0 (Li et al., 2023; Dubois et al., 2024), we win rate (LC WR) against the GPT-4-
report both the raw win rate (WR) and the length-controlee Preview-1106 baseline model. Following Meng et al. (2024); Tunstall et al. (2023), we employed sampling-basd -ased decoding approach. The temperature was set to 0.7 for the Mistral-Base (Jiang et al., 2023) setting, 0.5 for the Mistral-Instrus models (7B; Jiang et al., 2023/12B; Mistral AI, 2024), and 0.9 across both Llama-3-8B (Al @ Meta, 2024). For Arena-Hard-vO. (Li et al., 2024a;b), we report the win rate against the GPT-4-0314 14 baseline model, and use greedy decoding.
Downstream Task Evaluation. To assess whether alignment methods introduce an aligument tax by degrading general capabilities, we use the LM Evaluation Harness (Biderman et al., 2024). We evaluate performance on MMLU (5-shot) (Hendrycks et al., 2020), ARC-Challenge (25-shot) (Clark al., 2018), HellaSwag (10-shot) (Zellers al., 2019), Winogrande (5-shot) (Sakaguchi et al., 2021), TruthfulQA (0-shot) (Lin et al., 2021), and GSM8K (5-shot) (Cobbe et al., 2021).
A.3 IMPLEMENTATION AND INFRASTRUCTURE DETAILS
Our implementation builds on the open-source Aignment-handbon codebase (Tunstall et al., 2025), which itself is built around Hugging Face Transformers and the tr1 library. HyPO is realized as a thin modification of trl's 's DPOTrainer (see Appendix G); all remaining components (data loading, logging, checkpointing, evaluation scripts) are reused from the Alignoent-handbook stack with minimal changes. See Appendix A.3 for low-level implementation details.
We train all models with DeepSpeed (Rasley al., 2020) ZeRO and bfloat 16 mixed precision. For the main runs, we use ZeRO stage-1 with CPU offloading of optimizer states, which substantially reduces GPU memory usage while keeping the implementation simple. On subset of configuration also ran ZeRO stage-3 without CPU offloading and observed that all reported metrics (loss, win-rate) match well. Based on this observation, we adopted the stage-1 CPU-offload configuration for all large-scale sweeps reported in the paper. All main results are obtained on compute nodes equipped with four NVIDIA H100 96GB GPUs We have verified that the same configuration can be run on two H100 GPUs, or on single H100 with CPU offloading, at the cost of longer wall-clock time but essentially identical performance Gradient checkpointing is enabled for all models to further reduce memory consumption. Unless otherwise stated, we use Flash Attention 2 (Dao, 2023) for faster attention computation, b mixed -reccssirn for accelerated throughput, and realized via data parallelism and gradient accumulation.
A.4 ANALYSIS OF REFERENCE MODEL PESSIMISM (FIGURE 2)
To motivate our work, the analysis in Figure quantifies the prevalence of pessimistic reference margins (Aref < 0). This experiment was conducted on the full training split of the UltraFeedback dataset (Cui et al., 2023). We computed the sequence-level log-likelihood margins for three distinct reference models derived from the Meta-Llama-3-8B family: (i) the raw base model, (ii) our SFT checkpoint trained on UltraChat-200k (use Zephyr settings; Tunstall et al., 2023), and (iii) model further aligned from the SFT checkpoint using SimPO (use SimPO settings: Meng et al., 2024), For each pair in the dataset, we calculated Ard without length normalization. We provide summary statistics for each distribution, including the mean and median (p50). The results confirmed that even for a strong, pre-aligned reference model, a substantial fraction of pairs (>x 45%) remained in the pessimistic region, underscoring the persistence of the training-inferent mismatch.
A.5 DIRECT DPO vs. HYPO COMPARISON (FIGURE 3)
Figure 3 presents controlled comparison between standard DPO and our proposed HyPO. The goal of this experiment was to isolate the impact of our conditional reference mechanism. Both methods started from the exact same SFT checkpoint of Llama-3-8B-Ban. Base and were trained UltraFeedback
18

---

Published as a conference paper at ICLR 2026
(Cui al., 2023). Critically, all hyperparameters were held identical and were based the DPO configuration from Zephyr (Tunstall et al., 2023), including a learning rate of 5 x 10 and [ = 0.01. For this specific comparison, HyPO was used in its most basic form: it used the SFT model as its reference (not better reference) and was configured with y = 0 and no home advantage margin (h = 0). This setup ensures fair, one-to-one comparison against the DPO baseline. The metrics in Figure 3(a) and 3(b) were computed periodically on our held-out validation set throughout the single training epoch. The win-rate matrix Figure 3(c) was generated by evaluating the final SFT, DPO, and HyPO checkpoints against each other on AlpacaEval (Li et al., 2023).
B ANALYSIS OF DPO WITH SFT LOSS
In this section, we explore natural baseline: adding supervised fine-tuning (SFT) loss on the chosen response the standard DPO objective. This acts a regularizer ensure the model maintains high likelihood for preferred responses. The DPO SFT-on-chosen objective is defined as:
LDOO+ Shosen-chosen = LDO + (yrr(
(14)
where (1F log ( x))
B.I HYPERPARAMETER SEARCH
To ensure fair comparison, we conducted hyperparameter sweep for the coefficient À over the range (0.01, 0.03, 0.1, 1, 0.3, 0.5). We established À = 0.5 as the upper bound for our sweep because we observed that, at this value, the magnitude of the SFT loss term was approximately equal the DPO loss term at the beginning of training. We posit that the auxiliary SFT loss should act regularizer rather than the dominant training signal: therefore, should not exceed the point where SFT dominates the preference optimization. Empirically, we found that the optimal performance on AlpacaEval 2.0 was achieved at X 0.03. We use this best- rest-performing configuration for the definitive comparison below.
B.2 COMPARISON AND ANALYSIS
We compared the optimized DPO + SFT-on-chosen (A = 0.03) against the standard DPO and our HyPO method. In addition to win rates, we measured the Agreement Rate (defined in Appendix E) to assess how well the model's likelihood ordering aligns with the ground truth preferences.
Table 4:
Comparison of DPO, DPO SFT-on-chosen (A = 0.03). and HyPO. All models are Llama- 3-Base-8B initialized from the same SFT checkpoint.
Method
A vaa I.C (%)
AlpacaEval WR (%)
Agreement Rate (%)
DPO (Baseline)
24.3
21.9
55,9
DPO SFT-on-chosen
26.1
21.9
56.3
HyPO (Ours)
28.8
28.5
60.4
As shown in Table 4, DPO SFT-on-chosen (A 0.03) yields only a marginal improvement in win rate (+1.8% LC) and Agreement Rate (+0.4%). HyPO significantly outperforms this baseline (+4.5% LC over baseline).
19

---

Published as a conference paper at ICLR 2026
C EXAMPLES RESPONSES
The following is an example question along with responses from various models from the Alpaca Eval 2.0 helpful subset base pipeline. HyPO response shows analytical thinking, a deeper reasoning process, and consideration of implicit factors.
1. gpt4-1106-preveew. Most comprehensive and beginner-safe; actionable progressions.
2. HyPO (ours): Well-structured with spss/warmup/cooldown,, few choices (Russian twists, Pallof press) may advanced or -eppmeentddpedeett.
3. SimPO: Solid variety and guidance; minor glitch (stray char) and several advanced core picks for true beginners.
4. DPO: Long single list with multiple advanced moves (hanging leg raises); fewer regressions and weaker structure.
Responses from Different Models
Question
What the best exercises for beginners?
GPT-4-Preview- 106 (Baseline):
For beginners, the best exercises are those that are easy to learn, safe, and effective at building strength, endurance, and flexibility. Here is list of exercises that are generally well-suited for beginners, covering different aspects of fitness:
1. Cardiovascular Exercises:
Walking: Start with brisk walk and gradually increase your pace and distance.
Cycling: Begin with flat route before tackling hills.
Swimming: Try few laps with rest in between, gradually increasing your en- durance.
Jumping Jacks: A simple full-body exercise to get your heart rate up.
2. Strength Training Exercises:
Bodyweight Squats: Focus on form and depth before adding weight. Push-ups: Start with wall knee push-ups if full push-up too difficult.
Planks: Begin by holding the position for short periods and gradually increase the time.
Lunges: Practice stationary lunges to master the form before attempting walking lunges.
Dumbbell Exercises: Use light weights to perform exercises like bicep curls,
shoulder presses, and tricep extensions.
3. Flexibility and Balance Exercises:
Stretching: Incorporate basic stretches for all major muscle groups, holding each stretch for 15-30 seconds.
Yoga: Basic poses like Child's Pose, Downward-Facing Dog, and Warrior can Bird-Dog: This exercise improves core stability and coordination.
improve flexibility and balance.
Tai Chi: This gentle martial art is great for balance and stress reduction.
4. Core Strengthening Exercises:
Dead Bug: Another great move for core strength that's easy on the back.
Seated Russian Twists: Begin without weight and focus on the twisting motion.
Functional Exercises:
Step-ups: Use low step first and foeus form.
Farmer's Walk: Carry a light weight in each hand and walk for a set distance,
maintaining good posture.
Glute Bridges: These help strengthen the glutes and stabilize the core.
It's essential for beginners to focus proper form prevent injuries and start with level of intensity that matches their current fitness level. It's also advisable to consult with fitness professional or a physician before starting any new exercise program, especially if you have pre-existing health conditions or concerns.
20

---

Published as a conference paper at ICLR 2026
Responses from Different Models
Question
What are the best exercises for beginners?
DPO: As beginner, it's essential to start with exercises that safe, effective, and enjoyable to help you build a strong foundation and momentum in your fitness journey. Here are some of the best exercises for beginners:
Squats: Works your legs, glutes, and core. Stand with feet shoulder-width apart, then bend knees and lower your body down, keeping back straight.
2. Push-ups: Targets chest, shoulders, and triceps. Start in a plank position with hands shoulder-width apart, lower your body until your chest almost touches the ground, then push back up.
Lunges: Works legs, glutes, and lower body. Stand with feet together, take large step forward with one foot, and lower your body down, keeping back knee almost touching the ground. Alternate legs.
4. Planks: Strengthens core and improves posture. Hold plank position for 30-60
seconds, engaging your abs and keeping your body straight from head heels.
5. Chest Press (using dumbbells or resistance bands): Works chest, shoulders, and triceps. Lie on a flat surface or sit with knees bent, holding weights or resistance bands, and press them upwards towards the ceiling.
6. Rows (using dumbbells or resistance bands): Targets back and biceps. Hold weights resistance bands and pull them towards your chest, keeping your elbows close to your body.
7. Bicycle Crunches:
Engages core and improves obliques. Lie on your back with hands behind your head, alternate bringing each elbow towards the opposite knee, as if pedaling a bicycle.
8. Leg Raises (hanging leg raises or leg lifts):: Works abs and lower back. Hang from pull-up bar or lie on your back with arms extended, lifting your legs straight up towards the ceiling.
9. Wall Sit: Strengthens legs and glutes. Stand with back against a wall, slide down into a seated position with knees bent at 90-degree angle, and hold for 30 seconds to minute.
10. Brisk Walking Jogging: Improves cardiovascular fitness and burns calories. Start with short sessions and gradually increase duration and intensity.
Remember to:
Start slow and focus on proper form and technique.
Begin with lighter weights and gradually increase the load as you build strength.
Warm up before exercising with 5-10 minutes of light cardio and stretching.
Listen to your body and rest when needed.
Consult with healthcare professional or fitness expert if you have any concerns or injuries.
As you progress, you incorporate more exercises and intensity to challenge yourself further. Happy exercising!
21

---

Published as a conference paper at ICLR 2026
Responses from Different Models
Question
What are the best exercises for beginners?
SimPO:
As beginner, it's essential to start with exercises that are safe, effective, and manage- able to build a strong foundation for your fitness journey. Here are some of the best exercises for beginners:
Bodyweight exercises:
Push-ups: Works chest, shoulders, and triceps.
Squats: Targets legs, glutes, and core.
Lunges: Works legs, glutes, and improves balance.
Planks: Engages core muscles for stability and endurance.
Pull-ups (or assisted pull-ups): Targets back, shoulders, and arms.
2. Resistance band exercises:
Banded squats, lunges, and chest presses: Add variety to bodyweight exercises with resistance bands.
Cardio exercises:
Easy to start and improves cardiovascular fitness. Jogging or running (begin with short intervals): Increases endurance and burns calories.
Brisk walking:
Swimming or cycling: Low-impact alternatives for those with joint concerns.
4. Upper body exercises with light weights or dumbbells:
Dumbbell chest press: Targets chest and shoulders.
Bicep curls: Works biceps.
Tricep dips (using chair or bench): Targets triceps.
Core exercises:
Russian twists: Engages obliques.
Leg raises: Targets lower abs. Bicycle crunches:
Works entire core.
Remember to:
1. Start slowly and gradually increase intensity and duration you become more fortable.
2. Focus proper form and technique to avoid injuries.
3. Begin with 2-3 sets of 8-12 repetitions for each exercise.
4. Listen your body and rest when needed.
5. Consult with a healthcare professional or fitness expert to create personalized workout plan.
These exercises will help you build solid foundation in strength, flexibility, and cardiovascular endurance. As you progress, you can always add more variety and intensity to your routine.
22

---

Published as a conference paper at ICLR 2026
Responses from Different Models
Question
What are the best exercises for beginners?
HyPO (ours): As beginner, it's essential to start with exercises that are gentle, easy to learn, and progressive in intensity to build a strong foundation and prevent injury. Here are some of the best exercises for beginners:
Bodyweight exercises:
Push-ups
Squats
Lunges
Planks
Bicycle crunches (modified or regular)
Wall sit
Resistance band exercises:
Banded squats and lunges
Banded chest press (using a resistance band or light dumbbells)
Banded rows (using resistance band)
3. Low-impact cardio:
Brisk walking
Jogging in place
Swimming (if accessible)
Cycling (stationary or outdoor)
4. Upper body strength exercises with light weights:
Dumbbell rows
Shoulder presses (using light weights or resistance bands)
Bicep curls
Tricep dips (using chair or bench)
5. Core exercises:
Leg raises (lying or hanging)
Russian twists (using medicine ball weight plate)
Pallof press (using a cable machine or resistance band)
Remember to:
Start slow and gentle, with 2-3 sets 8-12 repetitions for each exercise.
2. Focus on proper form and technique over the number of repetitions or weight used.
3. Listen to your body and rest when needed.
4. Gradually increase intensity and volume as you become more comfortable with the exercises.
Incorporate exercises that work multiple muscle groups at once (e.g., squats engage legs, glutes, and core).
Additionally, consider the following general tips:
Consult with healthcare professional or fitness expert to create personalized workout plan.
Warm up before exercising with 5-10 minutes of light cardio and dynamic stretching. Cool down afterwards with static stretches to help prevent soreness and improve flexibility.
By starting with these beginner-friendly exercises, you'll set yourself up for a strong foundation in fitness and enjoy a positive exercise experience.
23

---

Published as a conference paper at ICLR 2026
D DOWNSTREAM TASKS
One of the primary challenges in preference alignment the potential for model's general capabili- ties to degrade, phenomenon often attributed to over-optimization et al., 2023; Rafailov et al., 2024). To rigorously assess whether our approach introduces any addi- tional "alignment tax" and preserves broad utility,, we evaluated aligned models comprehensive suite of downstream tasks. For comparability, we benchmark against one reference- (DPO; Rafailov al., 2023) and reference-free baseline (SimPO; Meng et al., 2024). All three methods are evaluated starting from the same SFT checkpoint, with identical prompt formatting and decoding hyperparameters.
the alignment objective (Gao
baseline
The evaluation was conducted using the standard LM Evaluation Harness framework (Biderman et al., 2024), with inference performed via the Hugging Face Transformers library. We follow the default task implementations in LM Evaluation Harness and report accuracy (or exact-match for GSM8K) as provided by the framework. We measured performance across four key domains, using the following few-shot settings for each task:
Knowledge & Language Understanding: MMLU (Hendrycks et al., 2020) (5-shot) and ARC- Challenge (Clark et al., 2018) (25-shot).
Commonsense Reasoning: HellaSwag (Zellers et al., 2019) (10-shot) and Winogrande (Sakaguchi et al., 2021) (5-shot).
Factuality: TruthfulQA (Lin al., 2021) (0-shot).
Mathematical Reasoning: GSM8K (Cobbe al., 2021) (5-shot).
The results, presented in Table 5, demonstrate that HyPO consistently performs on par with or better than the initial model and other alignment methods. Notably, HyPO achieves the highest average score in three out of the four experimental settings. This indicates that method successfully aligns with human preferences without incurring a significant penalty on the model's 's core capabilities.
Table Downstream task evaluation results, using LM Evaluation Harness (Biderman et al., 2024). The best results are highlighted in bold, and the second are underlined.
MMLU ARC
HellaSwag TruthfulQA
Winogrande GSM8K Average
Mistral-Base (7B)
Initial Model
59.0
54.2
60.8
28.0
77.7
34.6
52.4
DPO (Rafailov et al., 2023)
57.6
58.7
64.4
40.3
77.4
31.5
55.0
SimPO (Meng et al., 2024)
58.4
58.6
63.6
33.1
78.1
35.0
54.5
HyPO (Ours)
58.5
60.2
63.5
39.1
78.9
31.8
55.3
Hiisral-IIntrute (7B)
Initial Model
59.2
58.7
66.1
52.6
78.1
44.1
59.8
DPO (Rafailov al., 2023)
58.8
60.9
67.6
51.2
78.3
41.6
59.7
SimPO (Meng et al., 2024)
58.1
62.5
67.5
55.0
77.4
39.8
60.1
HyPO (Ours)
59.6
66.8
67.4
54.0
77.6
40.6
61.0
Llama-3- B-ane- (8B)
Initial Model
63.7
56.9
61.4
31.5
77.4
50.3
56.9
DPO (Rafailov et al., 2023)
63.3
61.6
64.7
37.9
78.0
54.7
60.0
SimPO (Meng al., 2024)
62.1
63.4
64.5
38.0
77.2
50.4
59.3
HyPO (Ours)
63.6
65.4
64.3
42.8
78.9
49.7
60.8
Llama-3 1lama-3-Instruce (8B)
Initial Model
65.8
56.4
59.0
36.1
77.3
74.8
61.6
DPO (Rafailov et al., 2023)
65.6
61.3
60.6
42,4
76.4
73.5
63.3
SimPO (Meng et al., 2024)
64.7
59.8
56.0
43.8
72.5
71.3
61.4
HyPO (Ours)
65.0
64.4
58.7
45.3
73.3
70.3
62.8
24

---

Published as a conference paper at ICLR 2026
E EVALUATION METRIC DEFINITIONS
To provide a fine-grained analysis of model performance during training, as shown in Figure 3, we track two key anference-aligned metrics held-out evaluation set of preference pairs (z, Vee 31 where Vu > yt- These metrics are designed to directly quantify how effectively our proposed method, HyPO, mitigates the maining-infereme mismatch inherent in DPO, Both metries are computed using the policy and reference log-likeliine oggliikelhhodd margins, Ag and Art as formally defined in the main text in Equation 3. We denote the policy margin given training checkpoint
A
Absolute Agreement Rate. The Absolute Agreement Rate provides a global measure of how well the policy's log-likelihoods align with the ground-truth preferences. It defined as the probability that the policy assigns a higher log-likelihood to the winning response than the losing response y
Agree @f
Pr
A (Ir, You y1)
At inference time, model's performance depends on its absolute log-likelihoods, as the reference model is not used. This metric therefore directly reflects the desired inference-tine behavior. A higher agreement rate signifies that the policy is more accurately ranking responses in absolute terms, indicating a successful reduction of the training-inference mismatch.
Absolute Margin on the Pessimistic Subset. This metric is a targeted diagnostic designed to measure performance on the specific subset of data where DPO most prone to failure. We first define the pessimistic subset as all evaluation pairs where the reference model incorrectly prefers the losing response:
{(rr, Var. 31) Aref (3r, Vhe Mi < 0}. The metric is then the average policy margin, Ag, computed exclusively over this subset:
D pessimistic =
Margin pessimistic
pessiines Ql == B( M/) E2
A (1) (xc, Uw.
This metric directly probes the phenomenon premature satisfaction. For pairs in D pessimistics DPO's learning signal weakens soon the policy margin An surpasses the negative reference margin A even if Ag itself is still negative. A strong and increasing positive value for this metric demonstrates that the policy is successfully applying corrective pressure and overcoming the misleading signal from the pessimistic reference, a core goal of HyPO.
F ANALYSIS OF THE SMOOTHNESS PARAMETER a
In Eq. we introduced smooth variant of the clipped reference margin using a softplus transforma- tion, with smoothness parameter a::
A 7 log(1 exp(a - )))), 0,
where smaller values of a yield smoother transition, while - do recovers the hard max(Aref 7
To assess the sensitivity of HyPO this smoothness parameter,
small sweep over or aligned on UltraFeedback, evaluated on AlpacaEval 2.0. All other settings are identical to those in our main experiments.
{1, 10, 100} using -aaa--Llma------nmste
Table 6: Sensitivity of HyPO to the smoothness parameter in Eq. 13 Sitta-Llaaa3-8--Insttt
Method
a
AlpacaEvvl LC (%)
AlpacaEval WR (%)
HyPO
1 48.3
45.5
HyPO (default)
10
49.5
46.2
HyPO
100 49.6
46.2
The results in Table 6 show that HyPO is robust to the choice of a in a reasonable range: a = and 100 yield nearly identical performance, while much smoother transition (a 1) causes only modest degradation. This supports our default choice of a = 10 as a simple and effective setting, and suggests that the benefits of HyPO do not rely on fine-tuning this parameter.
25

---

Published as a conference paper at ICLR 2026
G MINIMAL CODE CHANGE FROM DPO TO HYPO
We implement HyPO by modifying only the computation of the DPO logits inside the original DPOTrainer .ww.nnerddpp..... pwt.ppppooooooooo..eee..oo in trl. Let
o log ( ax) log (b( 22),
net log (Be x)) log mee (Yr x),
be the policy and reference log-ratios. Standard DPO uses As Aer as the logit input to the sigmoid loss. HyPO simply replaces A with clipped (or smoothed) reference margin Are:
Dref - ))
A max Aeef ?)
A y 7
where 7 = 1/a, and uses Ag - Ere in the DPO loss.
The concrete code change in DPOTraines cecer.dpo_losse is:
Original DPO (simplificd)
pi lograaiaics policy_chosen poliey_rejecteddlogs
if relf.reeferenc__fiee
ref_logratios torch.zeros_In (pi_lliittipplllattiii..
else:
ret Legratoo = ceference_chesen
eeference_rejectenn
logits pi log_atits ref Logratics
replaced by the following HyPO version:
HyPO: Conditional Reference Clipping
pi pi_logratics policy_chosen__gge policy_rejected_
if relf.reee reeerenneeffe:: ref_logratios prrch.zerrs__ieee (pi_llgratto else:
ref logratios reference_chere chosen logps deference_rejectere logps
HyPO Modification Start gamma torch.tensor (self. (yyo_aaa dev.corref_logratioo......
if getattr (sclf. Smooth HyPO: Eq. 13 (using tau for temperature)
"hypo_tau" 0.0) > 0.0:
tau self.args wrw..argg.... hypo_typpaatt...c..p tau
ref ref_logratios gamma tau s.sofpplus (ref_logratios gamma)
else:
# Hard HyPO: Eq. 11 (standard max) ref_logratios torch. maximum (ref_log (ref_logratios, gamma) HyPo Modi ficaiion End
logits =p pi_lograaios ref logratics
In our experiments, we introduce hypo_gamma (scalar threshold y, default 0) and optionally hypo_tau (smoothing factor т = 1/a) as hyperparameters.
26


Reviews:
[{"review_id": "k7c3RoMgeU", "reviewer": "Reviewer_vgnX", "summary": "This paper introduces Hybrid-DPO (HyPO), a variant of DPO that aims to mitigate a key failure mode termed premature satisfaction, where DPO prematurely attenuates gradients for pessimistic pairs (when the reference model prefers the rejected response). The proposed modification clips the reference margin (replacing $\\Delta_{ref}$ with $\\max\\\\{0, \\Delta_{ref}\\\\}$) to conditionally remove the influence of pessimistic references. Empirical results across multiple benchmarks (AlpacaEval and Arena-Hard) show improvements over existing benchmarks.", "questions": "Please see the weakness part.\n\nOverall, I find the paper borderline but leaning toward acceptance. I'm open to reevaluating this work according to the further discussions.", "limitations": "", "rating": 6, "confidence": 2, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. This work is well-motivated. The paper identifies a concrete limitation of DPO and proposes a simple yet effective idea to fix it. \n\n2. Strong numerical behavior. The experiments on Mistral and Llama are reported with significant improvements over existing baseline methods (e.g., DPO and SimPO).", "weaknesses": "1. The Qwen series models are missing. It would be better to also include one of them to run the tests.\n\n2. The sample code is not provided. It would be better to share more details on experimental configurations.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "tv2QHNrhLI", "reviewer": "Reviewer_Af2K", "summary": "This paper identifies a key limitation in Direct Preference Optimization (DPO), termed **“premature satisfaction”**, where pessimistic reference models weaken learning signals too early. To resolve this, the authors propose **Hybrid-DPO (HyPO)** — a simple yet principled modification that conditionally removes the negative influence of the reference while maintaining DPO’s stability. Extensive experiments show that HyPO consistently improves alignment performance across diverse settings, confirming both its effectiveness and scalability.", "questions": "See weaknesses above.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper is well-organized, moving naturally from the identification of DPO’s weakness to the proposal of a concise and theoretically grounded fix, with clear motivation.\n\n2. HyPO establishes a thoughtful balance between reference-based and reference-free optimization, combining their respective advantages while eliminating their key drawbacks.\n\n3. The authors provide strong evidence that pessimistic samples are a real and significant issue, and this observation directly supports their design and problem formulation.\n\n4. The authors conduct systematic experiments under various configurations, demonstrating that the proposed method is both effective and robust without adding computational overhead.", "weaknesses": "1. The paper uses sequence-level log-likelihood difference to decide “pessimistic”, but explicitly “does not length-normalize”. Although evaluation uses length-controlled metrics (e.g., AlpacaEval LC), the training-time pessimism criterion remains length-sensitive.\n\n2. The paper fails to distinguish cases where a negative reference margin reflects genuine “pessimism” from cases where it arises due to noisy or incorrect preference labels, in which the reference may in fact be the more reliable anchor.\n\n3. HyPO’s hard threshold switches the objective at the sample level. Such hard switching can induce non-smooth optimization landscapes. Does the mixed objective’s fixed point correspond to an interpretable KL-regularized or f-divergence–constrained optimum? \n\n4. Lacking some recent related work, e.g., AlphaDPO [1].\n\n[1] Junkang Wu, et. al. AlphaDPO: Adaptive Reward Margin for Direct Preference Optimization. In ICML 2025.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4trzkx3nA3", "reviewer": "Reviewer_obfj", "summary": "This paper introduce a one-line change to DPO, by replacing the reference model margin $\\Delta_{ref}$ with a relu-regularized margin $\\max(0, \\Delta_{ref})$.\nThis change is motivated by the problem that: when  $\\Delta_{ref} < 0$, the margin $\\Delta_\\theta$ cannot deviate too far from the reference model, resulting in unintended optimization result: $0 > \\Delta_\\theta > \\Delta_{ref}$.\nBy applying the RELU function, the learned margin will be encouraged to be positive.\nEmpirical results on different models and benchmarks demonstrate the effectiveness of this simple modification.", "questions": "Does adding an SFT loss on the chosen response helps mitigate the premature satisfaction problem? It would be interesting to compare with this alternative solution.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. Well Motivated Solution: The paper clearly identifies a potential issue of reference model in DPO optimization and provides a simple yet effective solution ($\\max(0, \\Delta_{ref})$) to address it.\n2. Stable Empirical Improvement: The proposed method is empirically validated on different models and benchmarks to improve over several existing DPO variants. The downstream task evaluation further demonstrates the proposed method avoids performance degredation.", "weaknesses": "1. **Introduced Hyperparameter**: In practice, the objective is implemented with $\\Delta_\\theta - \\max(\\Delta_{ref}, \\gamma) - h$ which introduces two additional hyperparameters $\\gamma$ and $h$ to DPO. This complicates the hyperparameter tuning process in practice.\n\n2. **Concerns about Ablation Results**: The motivation of the proposed HyPO method is based on the \"premature satisfaction\" problem, which the authors solve by applying the relu function to the reference margin $\\max(\\Delta_{ref}, \\gamma)$.\nHowever, Figure 4 shows that the empirical result is not sensitive to the choice of $\\gamma$. A more reasonable explanation is needed to justify the effectiveness of the relu function.\\\nMoreover, the ablation study on each component in Table 2 misses the setting of DPO + relu on $\\Delta_{ref}$, without better reference and home advantage. It is unclear how much improvement comes from the relu function itself.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "QsqXZyyHiD", "reviewer": "Reviewer_MKDX", "summary": "This paper addresses a critical issue in DPO --- the standard method for aligning large language models.\n\nDPO relies on a reference policy that can introduce a \"training-inference mismatch.\" Specifically, when the reference model incorrectly prefers the rejected response (pessimistic pairs, $\\delta_{\\text{ref}} < 0$), DPO prematurely weakens the learning signal even when the policy is still wrong, a phenomenon the authors call \\textbf{\"premature satisfaction\"}.\n\nThe authors introduce \\textbf{Hybrid Preference Optimization (HyPO)}: Behaves like standard DPO when reference is helpful ($\\delta_{\\text{ref}} \\geq 0$). Nullifies the reference when pessimistic ($\\Delta_{\\text{ref}} < 0$), reverting to absolute margin optimization.\n\nEmpirically, HyPO consistently and significantly outperforms both various DPO and strong reference-free baselines across a range of models and benchmarks.", "questions": "See Weaknesses.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Core Innovation: First formalization and proof of the \"premature satisfaction\" phenomenon -- when the reference model is pessimistic ($\\Delta_{\\text{ref}} < 0$), DPO's gradient weight prematurely decays even when the policy is still wrong ($\\Delta_\\theta < 0$): $\\|\\nabla_\\theta \\ell_{\\text{DPO}}\\| \\leq \\beta e^{(-z)} \\|\\nabla_\\theta \\Delta_\\theta\\|$\n\n- Elegant Solution: Proposes a one-line code modification $\\Delta_\\theta - \\max\\{0, \\Delta_{\\text{ref}}\\}$ that mathematically guarantees two key properties: (1) when $\\Delta_{\\text{ref}} \\geq 0$, $w_{\\text{HyPO}} = w_{\\text{DPO}}$ maintains stability; (2) when $\\Delta_{\\text{ref}} < 0$, $w_{\\text{HyPO}} \\geq w_{\\text{DPO}}$ strengthens learning signal; also provides smooth version (Eq. 13) ensuring differentiability\n\n- Visualization Insights (Figure 1): Innovative heatmap visualization showing gradient weight distribution in $(\\Delta_\\theta, \\Delta_{\\text{ref}})$ space, intuitively revealing DPO's systematic problem in the pessimistic region\n\n- Empirical Support: Demonstrates that even SimPO-aligned strong reference still has $\\approx 45\\%$ pairs that are pessimistic, proving this is a pervasive issue that cannot be fully solved by \"better reference\" alone", "weaknesses": "1. Circular Dependency on Better Reference Model\n- **Page 7, Line 324-327 & Table 2**: The paper uses a SimPO-aligned model as the \"better reference,\" creating a fundamental circular dependency:\n  - To train HyPO effectively, you first need a strong aligned model (trained with SimPO or similar)\n  - This undermines the claim of HyPO being superior to reference-free methods like SimPO\n  - **Ablation shows severe degradation** (Table 2): Without better reference, performance drops by 4.1%, reducing the advantage over standard DPO from 8.9% to only 4.8%\n  - This suggests HyPO's gains may primarily come from using a better reference, not from the clipping mechanism itself\n- **Missing Critical Experiment**: No comparison of \"HyPO with SFT reference\" vs \"SimPO with no reference\" on equal footing\n\n2. **Training-Inference Mismatch Only Partially Addressed**\n- **Fundamental Issue** (Page 1, Line 50-53 & Page 4, Line 199-201): The paper claims to resolve training-inference mismatch, but:\n  - **At training**: HyPO still uses $Δθ - max{0, Δref}$, conditioning on reference model\n  - **At inference**: Uses only $Δθ$ with no reference whatsoever\n  - The mismatch is only **reduced** for pessimistic pairs, not eliminated\n- **On optimistic pairs** ($Δref ≥ 0$, which is ~55% of data per Figure 2): HyPO behaves identically to DPO, thus inheriting the exact same mismatch problem\n- **Contradiction with Main Claim**: \n  - Page 1, Abstract claims HyPO \"mitigates premature satisfaction\" and \"resolves tension\"\n  - But optimistic pairs still suffer from training-inference mismatch where reference is used in training but not inference\n  - This is inconsistent with the paper's critique of DPO\n- **Missing Analysis**: No comparison of how much mismatch remains (e.g., agreement rate between implicit reward ranking and likelihood ranking, as mentioned in Page 1, Line 52 for DPO's ~50%)\n\n3. **Theoretical Justification for Clipping Operation Lacks Rigor**\n- **Page 5, Eq. 11**: The choice of $\\max\\{0, Δref\\}$ appears ad-hoc without principled justification:\n  - Why clip at exactly 0? Why not clip at other values or use different functional forms?\n  - Why is $\\max\\{·\\}$ the optimal operation? What about soft-clipping, or learned thresholds?\n- **No Optimality Proof**: Unlike DPO which derives its objective from first principles (KL-regularized RL → Gibbs form → DPO loss), HyPO's clipping is presented as an intuitive fix without showing it's optimal\n- **Alternative Approaches Not Explored**:\n  - Adaptive clipping: $\\max\\{f(Δref), Δref\\}$ where f is learned\n  - Weighted combination: $αΔθ + (1-α)\\max\\{0, Δref\\}$ where $α$ depends on confidence\n  - Smooth transitions instead of hard clipping\n- **Missing Theoretical Analysis**: No proof that this specific form of clipping leads to convergence to the desired policy, or that it's better than alternatives\n- **Page 6, Eq. 13**: While softplus smoothing is provided, the paper doesn't analyze whether smoothness parameter $α$ matters (not in Figure 4's sensitivity study)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)