Uni-DPO: A Unified Paradigm for Dynamic Preference Optimization of LLMs
Authors: Shangpin Peng
Weinong Wang
Zhuotao Tian
Senqiao Yang
Xing W
Haotian Xu
Chengquan Zhang
Takashi Isobe
Baotian Hu
Min Zhang
Venue: ICLR 2026 Poster (2026)
Keywords: Deep Learning
Large Language Model
Preference Learning
Primary area: alignment, fairness, safety, privacy, and societal considerations

Abstract: Direct Preference Optimization (DPO) has emerged as a cornerstone of reinforcement learning from human feedback (RLHF) due to its simplicity and efficiency. However, existing DPO-based methods typically treat all preference pairs equally, overlooking substantial variations in data quality and learning difficulty, which leads to inefficient data utilization and suboptimal performance. To address this limitation, we propose **Uni-DPO**, a unified dynamic preference optimization framework that jointly considers (a) the inherent quality of preference pairs and (b) the model's evolving performance during training. By adaptively reweighting samples based on both factors, Uni-DPO enables more effective use of preference data and achieves superior performance. Extensive experiments across models and benchmarks demonstrate the effectiveness and generalization of Uni-DPO. On textual tasks, Gemma-2-9B-IT fine-tuned with Uni-DPO surpasses the leading LLM, Claude 3 Opus, by 6.7 points on Arena-Hard. On mathematical and multimodal tasks, Uni-DPO consistently outperforms baseline methods across all benchmarks, providing strong empirical evidence of its effectiveness and robustness.

Published as a conference paper at ICLR 2026
UNI-DPO: A UNIFIED PARADIGM FOR DYNAMIC PREFERENCE OPTIMIZATION OF LLMs
Shangpin Peng"!
Weinong Wang"
Zhuotao Tian
Senqiao Yang²
Xing Wu Haotian Xu
Chengquan Zhang
Takashi Isobe Baotian Hu' Min Zhang*
Harbin Institute of Technology. Shenzhen
X an Jiaotong University
The Chinese University Hong Kong
Univeesiity of Chinese Academy of Sciences
TTiinguua University
Huazhong University of Science and Technology
Code & Models:
? pttps://giituub..co///ppta
ABSTRACT
Direct Preference Optimization (DPO) has emerged as a cornerstone of reinforce- ment learning from human feedback (RLHF) due to its simplicity and efficiency. However, existing DPO-based methods typically treat all preference pairs equally, overlooking substantial variations in data quality and learning difficulty, which Icads to inefficient data utilization and suboptimal performance. To address this limitation, propose Uni-DPO, unified dynamic preference optimization frame- work that jointly considers (a) the inherent quality of preference pairs and (b) the model's evolving performance during training. By adaptively reweighting samples based on both factors, Uni-DPO enables more effective use of preference data and achieves superior performance. Extensive experiments across models and benchmarks demonstrate the effectiveness and generalization of Uni-DPO, On tex- tual tasks, Gemma-2-99B 2-9--TT fine-tuned with Uni-DPO surpasses the leading LLM, Claude 3 Opus, by 6.7 points on Arena-Hard On mathematical and multimodal tasks, Uni-DPO consistently outperforms baseline methods across all benchmarks, providing strong empirical evidence of its effectiveness and robustness.
Arena-Hard
AlpacaEval2
Minerua Math
WR(X)
10.7
24.1
WR(%)
Olympiad
MATH
29:3
18.5
54.5
13.3
18.8
62.6
Strict Ace
IFEval
42
40.1
20.9
15.5
AAlpacaEval2
LC(%)
AIME24
20.0
723-72.7
40.5
76.0
GSMBK
$3.0
41.9
11.81
52.5
55.1
96.6
IFEual
45.5
39.6
AMC23
57.5
$6.6
College Math
Loose Acc
SedarEval
GaoKao En
SinPO
Uni-PPO (Durs)
DPC
SinPO
Uni Uni-PPO (Durs)
(a) Results textual understanding tasks
(b) Results mathematical reasoning tasks
INTRODUCTION
Incorporating human feedback has become essential for developing large language models (LLMs), it aligns model behavior with human values and intentions, ensuring that generated outputs useful, reliable, and harmless (Leike et al. [2018) Askell et al. 2021). Reinforcement learning from human feedback (RLHF) is a widely adopted paradigm for fine-tuning LLMs (Wang al., 2024a). typically involves training a reward model to capture human preferences and using it to optimize policy via reinforcement learning algorithms such as Proximal Policy Optimization (PPO) (Schulman al. 2017). Despite its effectiveness, this multi-stage pipeline is often time-consuming and suffers from the difficulty of training a reliable reward model (Casper et al. 2023).
"Equal contribution, Project lead, 'Correspondigg author (tianzhuotao @hi..cdu.c))..

---

Published as a conference paper at ICLR 2026
Low
Human: How can keep my houseplants alive?
Quality
E: Water each plant once week in the morning, place
them in bright but indirect sunlight, and use well- draining potting mix to avoid root rot.
CHOGE
High Quality
High
E Water them randomly whenever you think of it, leave them in full shade, and use any soil you have.
Quality
I ref
ref
E You could water them now and then put them some- where with some light.
GHoGEN
Low
Human
E You could them the shade or water them when you remember.
Quality
Preference
Optimisation
Preference
Demonstrations of data quality and its effect. During preference optimization, high-quality sample pairs exhibit clear distinctions and effectively reflect human preferences. whereas low-quality pairs have ambiguous differences between positive and negative samples, failing to represent human preferences accurately. Therefore, it is crucial to consider data quality when optimizing LLMs with preference data.
Figure 2:
To streamline the training pipeline, Direct Preference Optimization (DPO) (Rafailov et al. 2023 reparameterizes the reward function so that the policy can be learned directly from the preference data without a reward model. Concretely, it uses the log-likelihood ratio between the policy and the reference model on the same input as an implicit reward signal, thereby rendering the training process simpler and efficient, while still maintaining performance (Jiang et al. 2024 Dubois et al. 2023).
Key observations. Although DPO and its recent variants (e.g., SimPO Meng et al. 2024 have achieved promising performance, they treat all preference pairs equally during training. neglecting the varying quality levels present in the data collected for preference learning (Deng et al. 2025 Amini al. 2024| Wu et al., 2024a: Wang et al.) 2024f, Pattnaik et al. 2024). However, as demonstrated in Fig. 2 the high-quality preference pairs can help the model better align with the human preference. In contrast, the low-quality pairs with ambiguity between positive and negative samples may instead cause difficulty for representation learning. Therefore, the indiscriminate treatment of different data pairs may prevent the model from fully leveraging the data, thereby limiting the final performance.
An intuitive solution to this issue is to assign higher weights to high-quality data pairs during training, where quality can be quantified by the score difference between positive and negative samples as assessed by external experts. However, shown in Fig. [cc] aggressively training on well-fitted high-quality pairs may lead to overfitting, ultimately degrading model performance (Wu et al. 2024b Azar al. 2024 Chen et al. 2024a). This naturally leads to a key question:
How can dynamically adjust the focus across training pairs by jointly considering both intrinsic data quality and the model's s learning dynamics during training?
Our solution. To address this challenge, we propose Uni-DPO, a dual-perse optimization paradigm that jointly addresses two critical aspects of dynamic preference learning: (1) the inherent quality of the preference data and (2) the evolving learning dynamics of the model. The overall objective of the proposed Uni-DPO and the comparison with vanilla DPO is illustrated in Fig. 3
Specifically, first, we incorporate quality-aware weight Vuual that adaptively prioritizes high- quality samples and down-weights low-quality ones. Then, we introduce a performance-based weight Upert that shifts the learning focus towards the sample pairs that are not well-fitted, thereby mitigating overfitting. Additionally, since DPO may decrease the probabilities of generating preferred samples (Rafailov et al., 2024, |al et al. 2024), we introduce a calibrated negative log- likelihodd loss Lo-NLL that targets difficult yet high-quality positive samples to further improve model's performance.
To this end, the overall preference learning objective obtained by combining the above components with the vanilla DPO loss, thereby explicitly modeling both the off-policy aspect of preference optimization, which accounts for intrinsic data quality, and the on-policy aspect that captures model learning dynamies during training. This persspetve paradigm facilitates adaptive corrections during preference learning, resulting in improved final performance. By holistically addressing both facets of preference optimization within a unified framework, introduce our method as Uni-DPO.
2

---

Published as a conference paper at ICLR 2026
To summarize, contributions are as follows:
In this study, we reveal the fact that DPO and its recent variants treat all pairs equally during preference learning, potentially limiting the models' learning capacity and impeding performance. To tackle this issue, we introduce Uni-DPO. This method facilitates unified dynamic preference optimization of LLMs by adjusting the learning focus to informative training samples based on the intrinsic data quality and the model's learning dynamics.
Uni-DPO is straightforward yet effective. The comprehensive experimental results on textual understanding tasks, math reasoning tasks, and multimodal tasks across multiple models and benchmarks demonstrate the superiority and generalization capabilities of our method.
BACKGROUND AND MOTIVATION
In this section, we briefly introduce the basic concepts and works relevant to this study Sec. 2.1 establishing the necessary background. Following this, in Sec. 2.2 we outline our key insights and elucidate the motivations behind our proposed designs.
2.1 PRELIMINARIES
DPO. Direct Preference Optimization (DPO) |Rafallov et al. 2023) has emerged as a cornerstone technique within RLHF due its simplicity and efficiency. DPO dispenses with an explicit reward model and instead learns a policy directly from preference data. By turning the traditional multi-stage RLHF pipeline into single preference learning step, its training objective can be expressed as:
DPO = - -a B log a ( (y x)) Tee (Yw x:)
Te y1
x))
-3 log
Tref (1l
Here, D represents the preference learning dataset, denotes the sigmoid function, Te indicates the policy model under training, and Net represents the unchanged reference model. Yo stands for the positive sample, and y represents the negative sample, both based on input æ. is parameter controlling the deviation from the reference policy Tze--
The variants of DPO. Recent works have provided theoretical insights and improvements to DPO. Some studies show that the DPO reduce the probabilities of preferred samples Yo (Feng et al.) 2024), causing LLMs to struggle with generating numan-preferee outputs. To counter this, Pal et al.
(2024) incorporates an auxiliary negative log-likelihood loss better balance learning. Others lind
that DPO induces length bias, where the policy model favors longer sequences (Meng et al. 2024
Park al. 2024 Lu al., (2024), which can be alleviated via length normalization (LN). Among
variants, SimPO Meng et al.] 2024)) stands out for removing the reference model, achieving better
training efficiency and performance. More related works are discussed in Sec.
Implicit reward margin. During optimization with preference pairs, the implicit reward signal derived from DPO objective (see Eq. ( proves valuable, serving both as effective metric for assessing alignment quality and as a useful tool for downstream analysis (Lu et al. [2224] Pan et al.. 2025). Accordingly, the closed-fome expression of DPO's implicit reward r(x, y) is formulated
r(I, v) = 3 log To x) x))
3 log Z(x),
where (zx) denotes the partition function, independent of ky- The implicit reward defined in Eq. 2 measures the difference in log-likelihood between the policy and the reference policy the same sample. Based on this, we further define the implicit reward margin A, as the difference between the implicit rewards of the preferred (yw) and inferior (in) samples, which can be expressed as:
A, r(x, yw) r(x, 91).
A recent study (Houliston al. 2024) shows that the implicit reward margin A, of a pair (Nre U1) reflects both the policy model's performance and its learning difficulty for that instance. In this study, the reward margin A, will be utilized to dynamically adapt the focus towards various samples.
3

---

Published as a conference paper at ICLR 2026
Cara
Bin Kolat
DPO
Preferenice Optimization
lazer (8 (oke
Blog
Nolsi Korrst
Cussavo
π Tref
T
Miqual (tin 26) apper(t)) logo/A))) AGoou
Local
Weight
Human Optima
Mi
where:
w) ofo-(S. Sill,
Uni-DPO
Weight
Preference
Npper
[1 [1-0( log loguolaula)) molomooloo
bail
π ref
Weight
T
molste s)) A. mil s)
Weight
Humaa
b- mullo- ii)
Preference
1 ( logmottall (S6
Low quality
High quality
Well fitted
Under fitted
dats
data
data
data
Figure 3: Comparison of DPO and Uni-DPO objectives. The Uni-DPO objective introduces dual seeealepespeetivee
weighting mechanism, including quality-aware weight Vquas a performance-basee weight Wgeer, and a calibrated negative log- ligliikl term LASUA. that emphasizes challenging and high-quality positive samples. Left: schematic illustration of the two objectives. Right: compared with DPO, Uni-DPO dynamically reweights data pairs during training, guiding the optimization trajectory toward regions that hetter reflect human preference.
2.2
KEY OBSERVATIONS
Although DPO simplifies RLHF into single preference learning step. it implicitly assumes that every preference pair contributes equally to optimization. However, human-annotalee
or model-
preference data vary widely in quality (Wu et al. 2024b Amini al. 2024 Wu al. 2024a). Treating all sample pairs uniformly during optimization gives rise to the critical issues of under- utilization high-quality pairs that could more effectively facilitate the policy learning compared to low-quality counterparts (Wu et al., 2024b; Penedo et al. (023). A straightforward solution to this issue is allocating distinet training weights to various data pairs based on their quality.
However, we observe potential mismatch between the inherent quality of a preference pair, measured by the score margin (S... Si) assigned by external experts, and the model's dynamic performance on that pair, captured by its reward margin A, (see Eq. (3). As shown in Fig. 4 high-quality pairs can already be well-fitted by the model. While such data aids alignment, overly emphasizing them with large training weights could amplify overfitting risks (Wu et al.) 2024b), shown in Fig. 5c thus may undermine the generalization performance. More details about this study are in Sec. A.2
Consequently, this motivates us also consider the model's learning dynamics when adjusting the training focus based on data quality, i.e., specifically, by prioritizing underperforming pairs while downweighting well-fitted ones to mitigate overfitting. To this end, the contributions of data quality and model performance are balanced for weight assignment during the optimization process.
3 METHOD
3.1 OVERVIEW
To address the issues discussed in Sec. in this section, propose adaptive weighting mechanism that dynamically adjusts the importance of each pair based on two key factors: (1) Data quality (Wqual), which reflects the inherent value of the preference pair (Nw, 31); (2) Model performance (Wpert), which measures how well the current policy मæ aligns with the pair. To address all four cases mentioned in Fig. 4al These two factors combined multiplicatively (Miual Wpert) in method.
Additionally, recognizing the importance of difficult yet high-quality positive responses (yw), we incorporate a calibrated negative likelihoooo (c-NLL) loss LONNL that selectively amplifies the policy's confidence in such examples. To this end, the overall training objective is formulated as:
Luni DPO = (r.. D Mauuat U1) Upert ( (T)) log of (A,)) + XLe
(4)
where A, denotes the implicit reward margin defined Eq. The comparison between Uni-DPO and the vanilla DPO is presented in Fig.
In the following sections, the detailed formulations of perrppeeree adaptive weights Wqual and
Wpert presented in Secs. 3.2 and 3.3 and the design of the c-NLL loss is elaborated in Sec. 3.4
4

---

Published as a conference paper at ICLR 2026
reward margin
Low quality margin
W perf +1
High qualitty margin Easy to distinguish
Easy to distinguish
The probability that Lee has soccer
the courdi rutes paint symmatric origin (1, 6) with nepect the line 2212200 are
today (y/rs]5509) what the peebutiitty thit Lee does not nave socoer game today?
Ye. To find the ocor di of the point symmetric..Tue Thr.............. the fiaal ancwer aillooeddi (ifroi4155) WiatcE The coondinones the point
Mus The complement of pnocobilly FLA)
(b) Distribution of reward margin (y- uxass) versus score margin (x-axiis) and their Spear cocrelation coefficient
Pial, So, the probabiiltt that Lee does not have a sosoer game today is...
%t The protusility that lee does not have soover gome today 1- (1reet51999
symmetric to the or gii () with respect in the line x 2
score
Own liboseijn 2
margin
score margin: 2.10 reward marginst.3
score margin: 9.76 reward margind..3
Wigual
low
high quality
quality
Uqual
0
@! Solve for (h 2) 9*(X))
Q: The y-intercap the iin So- 20m0
W quai
Yai So, the solution i
on the y uais
set's serity this solution by. Since boon
sloes are caual the solution is conrect.
ya: To find the intercape the line given by the couation Sx 2y 10 O, we need
data quality
The flaal sooune loboennt 21 yi Therefoor the solution Sa -25.
to.. 1s (iboccel:
yi cor reot intercepti is WhbeediSi
Wporr
score margin: 2.05 reward margin-d.4t
scone margin: 9.00 reword marginod..aa
parfiem pooely
peffom
will
Law quality margin
High quality margin
Hard distinguish
Wperf" +
Hard distinguish
model performance
(a) Example of reward margin versus score margin
(c) Dual weight demo
Figure 4: Analysis of reward margin versus score margin. (a) Illustrative reward margin versus score margin examples across training data. The four quadrants reflect the combinations of high/low data quality and easy/hard learning difficulty. (b) Distribution of reward margin (y-axis) versus score margin (x-axis), along with their Spearman correlation coefficient p. indicating data quality is not necessarily aligned with learning difficulty, suggesting the need to account for both factors during training. (c) Demonstration of dual-perspectiv weighting mechanism: the quality weight Viqual (top) rises with data quality, while the performance weight ppet (bottom) decreases model performance improves, ensuring updates target both high-quality yet under-fitted samples.
3.2 QUALITY WEIGHTING FACTOR
In this section, propose measure to quantify the quality of each preference pair (yur B1) based on the distinction between the preferred (yw) and inferior (yi) samples (Wang et al. 2024a). Specifically, introduce quality- -based weighting factor yuull for the preference pair (the, y1) that is defined as:
Wuual (yw: 31) = (1 (S.. - S))
(5)
where Sw and St are the scalar quality scores for the preferred and inferior responses, derived from prior knowledge that reflects human preferences, either through human annotations, strong proprietary models (such GPT-4 (Achiam al. 2023)), or domain-specific reward models. Score sources in this work are detailed in Sec. 4 where we observe that better data quality yields better results. denotes the sigmoid function to ensure Vqual [0, 1]. and the scalar factor 7) is used to adjusts the score margin (S... Si) to an appropriate scale. In this work, 7) is consistent across models and datasets.
large score margin (Sw Si) implies confident preference, making Vqual close and amplifying the training signal of this preference pair. A small margin indicates ambiguity, leading Vqual to minimize, filtering noisy or uncertain pairs, and enhancing the robustness of training.
3.3 PERFORMANCE WEIGHTING FACTOR
Although prioritizing high-quality samples may be beneficial for policy learning, as shown in Fig. 5c
overemphasizing well-fitted examples may cause overfitting (Wu et al.) 2024b Azar et al.) 2024 Chen
ct al. 2024a). Therefore, the model's current performance is critical for dynamically rebalancing learning focus across training samples.
Revisit the focal loss. To better adapt to the model's learning dynamics during training, focal loss (FL) (Lin et al. 2017) has been introduced to address the imbalance learning issue in object detection tasks (Zu al. 2023). It adds modulating factor (1 px)7 to the standard cross-entropy loss (CE(P) = log(p)) where P. is the predicted probability for the target. This factor weights easy, well-fitted samples (with high p.) while preserving gradients for harder ones, thus rebalancing the training focus. The focal loss is defined as:
(p)(p) -(1 -1--))" -.) log(p))
(6)
5

---

Published as a conference paper at ICLR 2026
(a) Gradient comparison DPO and DPO with focal weight at different factors ?
(b) Gradient comparison the original focal- weighted
(c) Train loss and eval loss curves without weight "iperr durinng train-
(d) Train loss and eval loss with weight wyyr during training effectively mitigates overfitting
DPO stabilized variant
ing shows issues of overfitting
Figure 5: Illustration of model performance weighting, (a) Focal-weighted DPO gradients are larger for hard examples and decrease as the model improves, encouraging the focus on harder examples. (b) Direct focal weighting (Eq. (7) can lead to unstable training. whereas stabilized form (Eq. (TO) applies uniform constraint on each sample, resulting in more stable training. (c)-(d) Incorporating Uper can reduce overfitting.
where 7 2 0 is tunable parameter. When 7= the focal loss becomes equivalent to the standard cross-entropy loss. As y increases, the modulating factor's influence strengthens, causing the loss function to increasingly discount well-classifedd examples and prioritize hard, informative ones.
The direct integration with DPO, Given the concept of focal loss, we first directly incorporate it into our framework as an adaptive weighting factor Whecal based on the model's performance:
To (Be x)) B log Trer (x)
To (T(()(a))
log
Wfocal
Teee (ur (r)
1 a (Blog To 1z)) B log ( (3 log [ve((V B log mee m (z)) [=))]
Here, 7 controls the sharpness of weight decay. This formulation ensures that when the implicit reward margin A, (see Eq. (3) is significantly large, the weight approaches zero, effectively reducing emphasis on already well-learned samples. The resulting weightwe DPO objective becomes:
LDPO = - ()) [Tiee((Fa) loog o (A,)]]
(8)
and the corresponding gradient is given by (the derivation process is presented in Sec. B.2
Ve CoCo W/ = -EED://HHEE(E....... (1c -vo)
As illustrated in Fig. 5a the gradient of the focgl-weighted DPO objective peaks when the model
performs poorly and diminishes as performance improves. This adaptive weighting prioritizes
challenging examples by reducing the influence of well-fitted samples during training.
Issues of the direct integration. However, as shown in Fig. 5b) directly applying the focal weighting factor Wiscal into preference learning leads to unstable training dynamics. We attribute this to the focal factor imposing rigid, per-sample constraint. Concretely, for every preference pair (yws y1). Wlocal requires the policy's 's log-probability gap (logma (yw (x) log (x)) to surpass that of the reference model. While helpful for disambiguating hard examples, this constraint also forces the policy to further enlarge the margins gap between Yw and yt is already large, risking overfitting to noise or spurious samples. This suggests that extending the focal weighting mechanism to preference learning requires relaxed formulation that reduces emphasis on well-fitted samples.
already well-handled cases, the reference model's log-probability
Calibrated weighting factor. To mitigate the above issue, we first apply length normalization (LN) following prior work (Meng et al. 2024, Park et al.) 2024 Lu et al. 2024), ensuring the learning objective reflects preference differences rather than sequence length. Then, we relax the per-sample constraint by introducing fixed, uniform performance threshold Tref across all training pairs. Since this threshold is fixed, the margin required from the policy model no longer depends on how well the reference model performs pressure on easy samples and lowers the risk of overfitting to already well-fitted cases. The resulting performance factor Upert is formulated as:
given pair. This decoupling reduces unnecessary optimization

---

Published as a conference paper at ICLR 2026
WLN focal = -
(
B
log (Uha (x)
B
Logro(Ur(ar)
B yeu
(ew
3 lyal
log ( (z))
Merl
18vl
unify margin
B 13h
B
- Tref ]
Wperf
log ne (ywlzz)
log no (1)
(10)
Here, Trer is hyperparameter controlling the required performance margin. Empirically, increasing strengthens the down- nown-weighting of well-learned samples during training, which makes the optimal Tee rise, suggesting stronger margin constraints are needed for optimal performance (see Sec. 4.4)
More details about the rationale for relaxing the original focal weight are discussed in Sec. E.4
3.4 CALIBRATED NEGATIVE LOG-LIKELIHOOD LOsS
While the DPO objective enforces the likeliilloooo of the preferred response (yw) exceeds that of the inferior one (y)). it does not explicitly promote a higher absolute probability for you (Rafailov) et al. 2024 Pal al. 2024). Empirically, this "relative" optimization mechanism probabilities of preferred responses (yw) thereby hindering the ability of LLMS trained with DPO to generate outputs that align with human preferences. To remedy this, we introduce calibrated negative log-likelihood (c-NLL) loss targeted exclusively at difficult yet high-quality positive samples:
suppress the
log To (Uw x))
LeNLL = - 1 (log (Vw 2² > log To (B x))
1 (Sw 2 Tgood)
(11)
only active when refierence model's likelihood exceeds the policy's
only active high-quality positive samples
The first indicator 1(.) ensures LoNLL is only applied when the reference model's log- bigelikeddood exceeds that of the policy's, s, and the second 1(-) restricts the penalty only to high-quality positive samples. This calibration strengthens the policy's 's confidence in challenging, high-quality responses without disturbing well-fitter or low-quality ones. Further discussions about Le-NLL are in Sec. E.3
4 EXPERIMENTS
In this section, we present a comprehensive suite of experiments to demonstrate the superior perfor- mance of Uni-DPO across multiple dimensions. In Sec. [.1] evaluate Uni-DPO open-ended cntruuctionfollooiing tasks to verify its effectiveness in textual understanding: in Sec. 4.2 we inves- tigate its ability enhance mathematical reasoning; and in Sec. 4.3 we conduct ablation studies to assess the contribution of each component within Uni-DPO. In the appendix, we further present
evaluations on multimodal tasks in Sec. D.2 along with additional ablation analyses in Sec. D.3
4.1 TEXTUAL UNDERSTANDING EVALUATION
Experimental settings. We apply Ini-DPO to Llama 8B ( @ Meta 2024b) under two setups: Base and Instruct, and to 1-eema----9B... (Tea et al., 2024b). All training configurations are aligned with those used in SimPO to ensure a fair comparison] Additionally. we extend our evaluation to Qwen2.5| (Yang et al., 2024a), more recent and stronger model. Following the Base setup in SimPO, we perform off- ofl-policy preference training the UtraFeedhacci (CahlaaaakCCCiiiiaaa et al. 2023) dataset. We primarily evaluate our method on four widely adopted open-ended
cnrrruction-folllwwwi benchmarks: AlpacaEval 2.0 (Li et al. 2023b 2 Arena-Hard v0.1 (Li et al. 2024a). IFEval (Zhou et al. 2023).
and SedarEval (Fan et al., 2025). These benchmarks evaluate the models' conversational abilities across diverse set of queries. We compare Uni-DPO against DPO and its improved variant, SimPO. Supervised fine-tuning (SFT) is also included as a baseline. All methods share identical training corpora guarantee rigorously fair evaluation. Details of evaluation settings are in Sec. D.
Wee also compare with SimPO v0.2 setting, which utilizes preference labels from a stronger reward model
ArmoRM WWan al. 2024c) for Instruct setup. For base setup. we adopt GPT-4o (Hurst et al. 2024) to obtain
higher-quality preference data. The details of the data generation process in Sec. C.2
2 As noted by the original authors, the AlpacaEval results for SimPO were obtained using an older version of the evaluation toolkit, which is no longer compatible with the current setup. We re-evaluated SimPO using the publicly available checkpoint under the updated evaluation framework for fair comparison.
7

---

Published as conference paper at ICLR 2026
Table Main evaluation result of textual understanding. WR denotes the Win Rate, LC denotes the Length- Controlled win rate, Acc. denotes the Accuracy. The results highlighted in bold. The results show that Uni-DPO consistently outperforms the SimPO and DPO methods across models and benchmarks.
AlpacaEval2
Arena-Hard Hard
Arena
IFEval
SedarEval
Model
Method
LC(%)
WR(
WR(S)
Strict (cc
Loose Acc.
Overall(%)
SFT
5.7
3.7
3.3
33.5
35,3
20.85
DPO
15.5
13.3
15.9
40.5
31.80
Llama3-8B
SimPO
19.4
18.1
23.4
40.5
45.7
32,43
Buse
Uni-DPO
23.8
20.5
23.9
38.1
47.9
38.49
SimPO
15.5
18.5
29.3
40.1
45.3
36.23
Uni-DPO va.
22.8
24.1
30.7
42.1
48.6
39.62
SFT
28.5
28.4
25.7
59.4
62.2
40.93
DPO
43.7
41.7
32.6
Llama3-8B
SimPO
44.7
40.4
33.8
61.2
65.8
44.81
Instruct
Uni-DPO
47.2
44.2
37.3
61.2
67.8
46.32
SimPO
41.2
37.1
36.5
60.8
68.6
42.37
Uni-DPO
46.8
42.6
40.6
66.9
73.2
46.50
SFT
52.5
40.5
40.8
Gcmma-2-9B-IT
SimPO
53.2
48.0
60.4
67.7
57.7
Uni-DPO
54.7
52.7
67.1
71.2
72.8
57.5
Qwen2.5-7B
SimPO
20.9
18.4
39.5
43.6
48.2
48.2
Uni-DPO
25.8
24.7
43.5
43.3
49.9
Table 2: Main evaluation result of mathematical reasoning. All benchmarks evaluated with zero-shot chain-of- oh-iouht prompting greedy decoding. The best results are highlighted bold. The Uni-DPO method delivers significant improvements across all eight benchmarks for both 1.5B and 7B model scales.
Model
Method
GSMSK
MATH
Minerva Math
Olympiad Bench
AIME
2024
AMC
2023
GaoKao 2023 En
College Math
Average
Baseline
72.5
62.6
16.2
29.3
40.5
41.20
Qwen2.5. Math
DPO
72.7
64.4
18.8
31.4
3.1
47.5
56.6
41.9
42.08
1.5B
SimPO
68.4
32,9
45.68
Uni-DPO
80.1
70.2
27.9
34,5
20.0
57.5
58.2
47.9
49.54
Baseline
64.3
65.8
10.7
24.3
23.3
47.5
44.7
32.3
39.11
Qwen2.5-Math
DPO
83.2
75.8
20.2
37.9
26.7
$7.5
61.6
49.5
$1.55
7B
SimPO
85.7
26.4
32.0
39.3
26.7
57.5
50.1
53.73
Uni-DPO
88.9
78.2
34.6
41.5
26.7
67.5
65.7
51.3
56.80
Experimental results. As in Tab. T. Uni-DPO consistently and significantly outperforms existing preference learning techniques across all settings. While both DPO and SimPO improve upon the SFT baseline, Uni-DPO achieves the highest overall scores, underscoring its robustness and effectiveness. Notably, Uni-DPO surpasses DPO and SimPO on both AlpacaEval2 and the Arena-Hard datasets across all models. Specifically, on the Arena-Hare dataset, Gemma-22 2-9B- 9B-ITT finetuned with Uni-DPO attains score of 67.1, outperforming the leading models Claude 3 Opus (2024-02-29) (Anthropic and otherss 2024 (60.4) and Llama-3.1 1.11---..-70B-... 7-0B-Inntructt (AI@ Meta 2024a (55.7
Details are in Sec. D.2
Scalability analysis. As indicated by the v0.2 entry in Tab. our method's performance steadily increases as the quality of training data improves, validating the effectiveness of our dynamic weighting mechanism in leveraging high-quality samples. Besides, as illustrated in Figs. [6dland 6e Uni-DPO consistently outperforms SimPO across a spectrum model sizes, revealing its scalability.
4.2
MATHEMATICAL REASONING EVALUATION
Experimental settings. To validate the effectiveness Uni-DPO in mathematical reasoning, we employ the Qwen2.5 Math base model Yang et al.) (2024b) with 1.5B and 7B parameters. For training data, we select 20K math problems from the NuminaMath (Li et al. 2024) dataset. We use the same training data across all methods to compare fairly. The performance assessed eight widely used math reasoning benchmarks. All evaluations employ greedy decoding and zero-shot chain-of tf-ohhttht prompting for consistency. More details provided in Sec. D.I
Results from the official Chatbot Arena Leaderboard
8

---

Published as a conference paper at ICLR 2026
Table Main ablation results. Removing either the quality weight Wquat. the performance weight ugers length normalization (LN), or the calibrated NLL L.NU. consistently degrades performance compared Uni-DPO, confirming that each component of Uni DPO contributes meaningfully to its overall effectiveness.
AlpacaEval2
Arena-Hard
IFEval
SedarEval
Model
Method
LC(%)
WR(%)
WR(%)
Striet Acc.(%)
Loose Acc.(%)
Overall(%)
SFT Ours
5.7
3.7
3.3
33.5
35.3
20.85
23.8
20.5
23,9
38.1
47,9
38.49
Llama3-8B Base
w/o Mqqul
19.7
15.9
22.8
39.4
44.5
37.43
w/o pert
22.1
18.5
21.4
37.5
44.5
40.46
w/o LN
5.2
3.8
2.7
36.8
39.6
28.18
w/o NLL
22.4
19.4
23.3
40.7
47.5
37.73
SFT Ours
28.5
28.4
25.7
59.4
62.2
40.93
47.2
44.2
37.3
61.2
67.8
46.32
Llama3-8B Instruct
w/o Mquul
44.7
41.0
36.1
61.9
66.7
44.94
w/o Werert
44.2
40.5
37.0
57.5
66.0
43.76
w/o LN
47.0
43.7
38.4
59.5
66.5
46.60
w/o NLL
46.2
43.2
37.6
58.6
65.4
44.48
(a) Arena Hard WR Arena Hard WR (c) Relationship between (d) Model size scaling Model scaling when Tie 0.8, varying y when 7 m 3.0, varying Tel Mier parameter and Tat sistency in AlpacaFval
sistency in Arem--Hard
Figure 6: Parameter sensitivity and scaling consistency. (a) (c) Sensitivity of the performance weight Wpert (Eq. (10)) hyperparameters y and Tier' (a).(b) show practical ranges of [1.0, 5.0] for and [0.5, 2.0] for (c) plots optimal Tief and corresponding Arena-Hard WR scores across different ? values, revealing that higher y indicates sharper gradient decrease on casy sample, requires stronger margin constraints Ter- (d). Uni-DPO consistently outperforms SimPO on Qwen2.5 base model across different sizes, revealing its scalability.
Experimental results. As shown in Tab. 2 our proposed Uni-DPO method yields significant improvements over both SimPO and DPO across model scales of 1.SB and 7B parameters. Specifically, Uni-DPO-trainet models consistently outperform the basclines on all eight mathematical reasoning benchmarks, providing strong empirical evidence for the effectiveness and robustness of our approach. Moreover, the improvement margin of Uni-DPO grows with model scale. Uni-DPO achieves average gain of 8.3 points on the 1.5B model, which further increases to 17.7 points on the 7B model. This observation indicates that our approach scales effectively with model capacity, exhibiting stronger advantages as the model size increases.
4.3 ABLATION STUDY
In this section, we conduct an ablation study to evaluate the individual contributions each compo- nent in our Uni-DPO framework. As shown in Tab. 3 models without Uqual or Wperf exhibit significant performance drops, indicating the critical roles of these weighting factors in the optimization process. The base model without length normalization shows notable decline, suggesting that the base model is more unstable and sensitive to data length. Finally, models without Le.NLI. perform poorly across all benchmarks, highlighting the essential role of c-NLL loss during training. Further ablation about the components of Lo-NLL and the selection of the expert model are discussed in Sec. D.3
4.4 PARAMETER SENSITIVITY
In our textual experiments, most hyperparameters are fixed across all models and benchmarks,
Wqual = 0.7, c-NLL coefficient À 0.001, and Tgood 3.2, yielding stable and strong performance, which demonstrates the robustness of Uni-DPO. Only the sefformmannewwiihhte Wpert parameters vary: sensitivity analysis in Figs. 6alto 6o shows the practical ranges of these parameters are y E [1.0. 5.0] and Tier E [0.5. 2.0], leading us to use -= 3.0 throughout all experiments. Detailed training settings and environments are provided in Sec.

---

Published as a conference paper at ICLR 2026
5 CONCLUDING REMARKS
Further discussions. Within the Unified Paradigm proposed by Shao et al. (2024b). all methods, such as SFT, DPO, PPO, and GRPO, can be conceptualized as RL methods. Therefore, we re-examine the underlying principles of Uni-DPO from the RL perspective. In algorithms such as PPO and GRPO, the gradient coefficient derived from the advantage function, which is computed based on a reward model and (where applicable) value model, enabling them to dynamically reinforce or penalize responses with varying intensity. By contrast, DPO relies solely on its intrinsic reward margin (Eq, ( as the weighting signal, without incorporating a fine-grained external reward signal. As a result, cannot dynamically weight data samples according to their quality. Additionally, during training, DPO is limited in its ability to effectively distinguish between samples of varying difficulty. Uni-DPO closes this gap by incorporating both a data quality weight Wqual and a performance weight Wperfs thus explicitly enabling the model to adaptively prioritize samples based on both their intrinsic quality and training difficulty in a manner analogous to advantage-bass based reweighting. This dual- tual-perspeetive weighting mechanism allows Uni-DPO achieve more fine- fine-grained credit assignment during policy optimization, addressing the limitations of DPO. More discussions are provided in Sec.
Summary. this work, we introduce Uni-DPO, novel unified optimization framework for dynamic preference learning of LLMs. Our approach incorporates a dual- perspeetveee weighting mechanism that dynamically adjusts each sample's s contribution during training based on the intrinsic data quality and the model's learning dynamics, enabling the model to focus on high-quality yet challenging examples and thereby improving training efficiency and overall performance.
Limitation and future work. While our method demonstrates significant performance improve- ments compared to existing methods, the quality of the training data remains a critical factor. Future work could explore more sophisticated methods for estimating data quality for preference alignment.
Reproducibility statement. To facilitate a clearer understanding of our contributions and to enable broader use and adoption of our work, we provide extensive materials to support reproducibility. the main text, we describe the key components of our method in Sec.Bland present the main experimental results in Sec. 4 In the supplementary materials, we provide further detailed information, including Motivation Details (Sec. A), Method Details (Sec. B Training Details (Sec. C). and Evaluation Details (Sec. D). which we hope will help reproduce our results. Furthermore, we release our code, data, training scripts, and model checkpoints to facilitate future research and applications.
This work was supported by the Guangdong Basic and Applied Basic Re- search Foundation (2025A1515011 and by the Shenzhen Science and Technology Program (CC11202448 2DCY22000000 11000000100
Acknowledgement.
10

---

Published as a conference paper at ICLR 2026
REFERENCES
Grok-1.5 1.5 vision preview. ptcpps//s..aa//ew//groooooe
2024.
Josh Achiam, Steven Adler. Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida. Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprini (xXiv:2303.08774 2023.
AI@Meta. Introducing Ilama 3.1: Our most capable models to date. https://ai.meta / meta 11ama 3-1 2024a.
Al@Meta. Llama 3 model card. MODEL CARD. md 2024b.
Afra Amini, Tim Vieira, and Ryan Cotterell. Direct preference optimization with an offset. arXiv preprint arXiv:2402. 1057), 2024.
Thomas Authony, Zheng Tian, and David Barber. Thinking fast and slow with deep learning and tree search. Advances in neural information processing systems, 2017.
Anthropic. Claude 2, 2023a. URL
pdf
Authooiic. Introducing Claude, 2023b.
URL http:://w..atttrggic..co////
int Introciing--caae
Anthropic. The Claude 3 model family: Opus, Sonnet, Haiku, /ttp/////tt//////ccccccouu octa0//6666aa33377776666666aaaad
/oo//de 0001
Anthropic and others. The claude 3 model family: Opus, sommet, haiku, http://ttppppp///cco.poopo. ccom/ Sba hrt.aa/cco.ctaaaaaaaaaaaaaaeeae0
2024.
Amanda Askell, Yuntao Bai, Anna Chen, Dawn Drain, Deep Ganguli, Tom Henighan, Andy Jones, Nicholas Joseph, Ben Mann, Nova DasSarma, et al. A general language assistant as a laboratory for alignment. arXiv preprint arXiv:21 00.01 2021.
Mohammad Gheshlaghi Azar, Zhaohan Daniel Guo, Bilal Piot, Remi Munos, Mark Rowland, Michal Valko, and Daniele Calandriello. A general theoretical paradigm to understand learning from human preferences. In International Conference on Artificial Intelligence and Statistics, 2024.
Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A versatile vision-language model for understanding, localization. Text Reading, and Beyond, 2023.
Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, et al. Qwen2. 5-vl technical report. arXiv preprint arXiv:2502. 13923, 2025.
Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Decp Ganguli, Tom Hemighan, et al. Training helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint 224::22204008622 2022.
Edward Beeching, Clémentine Fourrier, Nathan Habib, Sheon Han, Nathan Lambert, Nazneen Rajani, Omar Sanseviero, Lewis Tunstall, and Thomas Wolf. Open LLM leaderboard, hettp:://hugiiggaae. /o/ spaces / Huggi ngl aagaacees ceH4 /open/ 11 Teaderboar rd 2023.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 2020.
Stephen Casper, Xander Davies, Claudia Shi, Thomas Krendl Gilbert, Jérémy Scheurer, Javier Rando, Rachel Freedman, Tomasz Korbak, David Lindner, Pedro Freire, al. Open problems and fundamental limitations of reinforcement learning from human feedback. arXiv preprint 2rXiv:2307.15211 2023.
Shreyas Chaudhari, Pranjal Aggarwal, Vishvak Murahari, Tanmay Rajpurohit, Ashwin Kalyan. Karthik Narasimhan, Ameet Deshpande, and Bruno Castro da Silva. Rlhf deciphered: critical analysis of re- inforcement learning from human feedback for 1lms, arXiv preprint arXiv:2404.088. 40555555, 2024.
Sapana Chaudhary, Ujwal Dinesha, Dileep Kalathil, and Srinivas Shakkottai. Risk-averse fine-tuning of large language models. Advances in Neural Information Processing Systems, 2024.
11

---

Published as a conference paper at ICLR 2026
Angelica Chen, Sadhika Malladi, Lily Zhang, Xinyi Chen, Qiuyi Richard Zhang, Rajesh Ranganath, and Kyunghyun Cho. Preference learning algorithms do not learn preference rankings. Advances in Neural Information Processing Systems, 2024a.
Lichang Chen. Chen Zhu, Davit Soselia. Jiuhai Chen, Tianyi Zhou, Tom Goldstein, Heng Huang, Mohammad Shoeyhi, and Bryan Catanzaro. Odin: Disentangled reward mitigates hacking rlhf. arXiv preprint arXiv:2402.073/9, 2024b.
Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Zehui Chen, Handong Duan, Jiaqi Wang, Yu Qiao, Dahua Lin, et al. Are we on the right way for evaluating large vision-language models? arXiv preprint arXiv:2403.228 20333003, 2024c.
Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. Advances in neural information processing systems, 2017.
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? Try ARC, the A12 reasoning challenge. ArXiv, 2018.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Retichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprins arXiv:21 14168, 2021.
Ganqu Cui, Lifan Yuan. Ning Ding, Guanming Yao, Bingxiang He, Wei Zhu, Yuan Ni, Guotong Xie, Ruobing Xie, Yankai Lin, et al. Ultraecdback: Boosting language models with scaled ai feedback. arXiv preprint arXiv:2310.0137, 2023.
Ganqu Cui, Lifan Yuan. Zefan Wang, Hanbin Wang, Wendi Li, Bingxiang He, Yuchen Fan, Tianyu Yu, Qixin Xu, Weize Chen, et al. Process reinforcement through implicit rewards. arXiv preprint xxXi:2502.001456 2025.
Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong. Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi, Instructblip: Towards general- purpose rision-languee models with instruction tuning, 2023.
Xun Deng, Han Zhong, Rui Ai, Fuli Feng, Zheng Wang. and Xiangnan He. Less is more: Improving llm alignment via preference data selection. arXiv preprint Xix:22502..1466 2025.
Hanze Dong, Wei Xiong, Bo Pang. Haoxiang Wang, Han Zhao, Yingho Zhou, Nan Jiang, Doyen Sahoo, Caiming Xiong, and Tong Zhang. RL.FF workflow: From reward modeling to online RLHF, arXiv preprint arXiv:2405.0 07863, 2024.
Haodong Duan, Junming Yang, Yuxuan Qiao, Xinyu Fang, Lin Chen, Yuan Liu, Xiaoyi Dong, Yuhang Zang, Pan Zhang, Jiaqi Wang, et al. Vimevalkit: An open-source toolkit for evaluating large multi-modality models. In Proceedings of the 32nd ACM international conference on multimedia, 2024.
Yann Dubois, Chen Xuechen Li, Rohan Taori, Tianyi Zhang, Ishaan Gulrajani, Jimmy Ba, Carlos Guestrin, Percy S Liang, and Tatsunori B Hashimoto. Alpacafarm: A simulation framework for methods that learn from human feedback. Advances in Neural Information Processing Systems, 2023.
Yann Dubois, Balázs Galambosi, Percy Liang, and Tatsunori B Hashimoto. simple way debias automatic evaluators. arXiv preprint arXiv:2404.04475, 2024.
alpacaeval: A
Esin Durmus, Faisal Ladhak, and Tatsunori Hashimoto. Spurious correlations in reference freccncee-feeeeeeeee evaluation of text generation. arXiv preprint #XXi:2204.0989/ 2022.
Hugging Face. Open rl: A fully open reproduction of deepseek-rl. wuuggingfacc//penn. r_ 2025.
https: com/
Zhiyuan Fan, Weinong Wang, Xing Wu, and Debing Zhang. Sedareval: Automated evaluation using self self-adaptive rubries. arXiv preprint arXiv:2501.15595 2025.
Duanyu Feng, Bowen Qin, Chen Huang, Zheng Zhang, and Wenqiang Lei. Towards analyzing and understanding the limitations of dpo: A theoretical perspective. arXiv preprint arXiv:2404.044: 046666 2024.
Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Jinrui Yang, Xiawu Zheng, Ke Li, Xing Sun, et al. Mme: A comprehensive evaluation benehmark for multimodal large language models. arXiv preprint arXiv:2306. 13394, 2023.
Leo Gao, John Schulman, and Jacob Hilton. Scaling laws for reward model overoptimization. In International Conference on Machine Learning, 2023a.
12

---

Published as a conference paper at ICLR 2026
Leo Gao. John Schulman, and Jacob Hilton. Scaling laws for reward model overoptimization. In International Conference on Machine Learning, 2023b.
Leo Gao. Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Ie Noac"h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou, The language model evaluation harness, 2024. URL hittps:/zzencds wttpreecora//120050000000
Alexey Gorbatovski, Boris Shaposhnikov, Alexey Malakhov, Nikita Surnachev, Yaroslav Aksenov, Ian Maksimov, Nikita Balagansky, and Daniil Gavrilov. Learn your reference model for real good alignment. arXiv preprint arXiv:2404.09633 2024.
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, al. Deepseck-rl: Incentivizing reasoning capability in Ilms via reinforeement learning. arXiv preprint arXiv:2501. 12948, 2025.
Alex Havrilla, Yuging Du, Sharath Chandra Raparthy, Christoforos Nalmpantis, Jane Dwivedi- Yu, Maksym Zhuravinskyi, Eric Hambro, Sainbayar Sukhbaatar, and Roberta Raileanu. Teaching large language models to reason with reinforcement learning. arXiv preprint arXiv:2403.04660 2024a.
Alex Havrilla, Sharath Raparthy. Christoforus Nalmpantis, Jane Dwivedi- Yu, Maksym Zhuravinskyi, Eric Hambro, and Roberta Railneau. Glore: When, where, and how improve Ilm reasoning via global and local refinements. arXiv preprint 2xXiv:2402.10933 2024b.
Chaoqun He, Renjie Luo, Yuzhuo Bai. Shengding Hu, Zhen Leng Thai, Junhao Shen, Jinyi Hu, Xu Han, Yujie Huang. Yuxiang Zhang. et al. Olympiadbench: A challenging benchmark for promoting agi with olympiad-level bilingual multimodal scientific problems. arXiv preprint arXiv:2402. 14008, 2024a.
Lehan He, Zeren Chen, Zhelun Shi, Tianyu Yu, Jing Shao, and Lu Sheng. topic-level self-correctionl approach to mitigate hallucinations in milms. arXiv preprint 2rXiv:2411.17266 2024b.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Stein- hardt. Measuring massive multitask language understanding. In International Conference on Learning Representations, 2020.
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang. Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. NeurIPS, 2021.
Sam Houliston, Alizéc Pace, Alexander Immer, and Gunnar Rätsch. Uncertainty-pentn penalized direct preference optimization. arXiv preprint arXiv:2410.204 2024.
Jingeheng Hu, Yinmin Zhang, Qi Han, Daxin Jiang, Xiangyu Zhang, and Heung- Srung-Yeung..a Shum. Open-rcasoner- zero: An open source approach to scaling reinforcement learning on the base model. arXiv preprint D.Xiv:25033242900 2025.
Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-40 system card. arXiv preprint arXiv:2410.21276, 2024.
Albert Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, et al. Mixtral of experts. arXiv preprint arXiv:2401.0400 2024.
Samia Kabir, David N Udo-Imeh, Bonan Kou, and Tianyi Zhang. Is stack overflow obsolcte? an empirical study of the characteristies of chatgpt answers to stack overflow questions. In Proceedings of the 2024 CHI Conference on Human Factors in Computing Systems, 2024.
Timo Kaufmann, Paul Weng, Viktor Bengs, and Eyke Hüllermeier. A survey of reinforcement learning from human feedback. arXiv preprint arXiv:2312. 14925, 2023.
Dahyun Kim, Yungi Kim, Wonho Song, Hyeonwoo Kim, Yunsu Kim, Sanghoon Kim, and Chanjun Park. sDPO: Don't use your data all at once. ArXiv, 2024.
Keyi Kong, Xilie Xu. Di Wang. Jingfeng Zhang, and Mohan Kankanhalli. Perplexity- aware correction for robust alignment with noisy preferences. In Advances in Neural Information Processing Systems, 2024.
Tomasz Korbak, Kejian Shi, Angelica Chen, Rasika Vinayak Bhalerao, Christopher Buckley, Jason Phang. Samuel R Bowman, and Ethan Perez. Pretraining language models with human preferences. In International Conference on Machine Learning 2023.
13

---

Published as a conference paper at ICLR 2026
Xin Lai, Zhuotao Tian, Yukang Chen, Yanwei Li. Yuhui Yuan, Shu Liu, and Jiaya Jia. Lisa: Reasoning segmentation via large language model. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024a.
Xin Lai, Zhuotao Tian, Yukang Chen, Senqiao Yang. Xiangru Peng, and Jiaya Jia. Step-dpo: Step deie preference optimization for long-chain reasoning llms. arXiv preprint arXiv:2406. 18629. 2024b.
Nathan Lambert, Valentina Pyatkin, Jacob Daniel Morrison, Lester James Validad Miranda, Bill Yuchen Lin, Khyathi Raghavi Chandu, Nouha Dziri, Sachin Kumar, Tom Zick, Yejin Choi, Noah A. Smith, and Hanna Hajishirzi. Rewardhench: Evaluating reward models for language modeling. ArXiv, 2024.
Jan Leike, David Krueger, Tom Everitt, Miljan Martie, Vishal Maini, and Shane Legg. Scalable agent alignment reward modeling: a research direction. arXiv preprint 1xXiv:811107888 2018.
Hector Levesque, Ernest Davis, and Leora Morgenstem. The Winograd schema challenge. In Thirteenth international conference confereec on the principlex of knowledge represeniation and reasoning, 2012.
Aitor Lewkowycz. Anders Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay Ramasesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Soloo, et al. Solving quantitative reasoning problems with language models. Advances in Neural Information Processing Systems, 2022.
Bohao Li, Rui Wang, Guangzhi Wang, Yuying Ge, Yixiao Ge, and Ying Shan. Seed-hench: Benchmarking multimodal IIms with generative comprehension. arXiv preprint arXiv:2307.16122 2023a.
Jia LI, Edward Beeching, Lewis Tunstall, Ben Lipkin, Roman Solctskyi, Shengyi Costa Huang, Kashif Rasul, Longhui Yu, Albert Jiang, Ziju Shen, Zihan Qin, Bin Dong, Li Zhou, Yann Fleureau, Guil- laume Lample, and Stanislas Polu. Numinamath. lbttps://g pectt
aimo- poono-progres--ee wetssprrieebbppe wwt.b/wwwbbmmaaaaaeaaeeaaaaaeaaa http://man//eept//mmiinnoo
dataset..dF 2024.
Jingyao Li, Senqiao Yang, Sitong Wu, Han Shi, Chuanyang Zheng, Hong Xu, and Jiaya Jia, Logits-based finetuming. arXiv preprint arXiv:2505. 24461, 2025.
Tianle Li. Wei-Lin Chiang, Evan Frick, Lisa Dunlap, Banghua Zhu, Joseph E. Gonzalez, and lon Stoica. From live data to high-quality benchmarks: The Arena-Hard pipeline. https://Lmsss.oret
2024-04-2 11-aarena---a---- hard/ 2024a.
Xianchen Li, Zheng-Xin Yong, and Stephen H Bach. Preference tuning for toxicity mitigation generalizes across languages. arXiv preprint arXiv:2406. /6235, 2024b.
Xuechen Li, Tianyi Zhang, Yann Dubois, Rohan Taori, Ishaan Guirajani, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. AlpacaBval: An automatic evaluator of instruction-followee following models. https //github. www/ tat su- lab/ wwwpapaaaaaaaaaaeeeaaaaaaaaaaaee 2023b.
Yifan Li. Yifan Du Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji- Ji-Rong Wen. Evaluating object hallucina- tion in large vision-language models. arXiv preprint Ttt::2305.10335. 2023c.
Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let's verify step by step. arXiv preprint 2305:2.300.00000 2023.
Stephanie Lin, Jacob Hilton, and Owain Evans. TruthfulQA: Measuring how models mimic human falsehoods. In ACL, 2022.
Tsung- Yi Lin, Priya Goyal, Ross Girshick, Kaiming Hc, and Piotr Dollár. Focal loss for dense object detection. In Proceedings of the IEEE international conference on computer vision, 2017.
Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang. Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng. Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 technical report. arXiv preprint arXiv:2412.1943) 2024a.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural
information processing systems, 2023.
Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jac Lec. Improved basclines with visual instruction tuning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024b.
Haotian Liu, Chunyuan Li, Yuheng Li. Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. Llava-next: Improved reasoning, ocr, and world knowledge. 2024c.
22024-01330 1100--laaaa-aeeee
Yijun Liu, Jiequan Cui, Zhuotao Tian, Senqiao Yang. Qingdong He, Xiaoling Wang, and Jingyong Su. Typicalness-aware learning for failure detection. arXiv preprint arX1v22411.01981, 2024d.
14

---

Published as a conference paper at ICLR 2026
Yixin Liu, Pengfei Liu, and Arman Cohan. Understanding reference policies in direct preference optimization. arXiv preprint arXiv:2407. 13709, 2024c.
Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangho Zhao, Yike Yuan, Jiaqi Wang. Conghui He, Ziwei Liu, et al. Mmbench: Is your multi-modal model an all-around player? In European conference on computer vision, 2024f.
Yuliang Liu, Zhang Li. Mingxin Huang, Biao Yang. Wenwen Yu, Chunyuan Li, Xu- Xu-Cheng Yin, Cheng Lin Liu, Lianwen Jin, and Xiang Bai. Ocrhench: the hidden mystery of oer in large multimodal models. Science China Information Sciences, 2024g.
Zichen Liu, Changyu Chen, Wenjun Li, Penghui Qi. Tianyu Pang. Chao Du, Wee Sun Lee, and Min Lin. Understanding rl-zero-like training: critical perspective. arXiv preprint arXiv:2503.20783, 2025.
Blya Loshchilov and Frank Hutter. Decoupled weight decay regularization, arXiv preprint arXiv: 11Xvv:1111.050011 2017.
Junru Lu. Jiazheng Li. Siyu An, Meng Zhao, Yulan He, Di Yin, and Xing Sun. Eliminating biased length reliance of direct preference optimization via down-sampled kl divergence. arXiv preprint arXiv:2406. 10957, 2024.
Pan Lu, Swaroop Mishra, Tony Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn to explain: Multimodal reasoning via thought chains for science question answering. In The 36th Conference on Neural Information Processing Systems (NeurIPS), 2022.
Haipeng Luo, Qingfeng Sun, Can Xu, Pu Zhao, Jianguang Lou, Chongyang Tao, Xiubo Geng, Qingwei Lin, Shifeng Chen, and Dongmei Zhang. Wizardmath: Empowering mathematical reasoning for large language models via reinforced evol-instruct. arXiv preprint 23Xxx:2308.0983,, 2023.
Sadegh Mahdavi, Muchen Li, Kaiwen Liu, Christos Thrampoulidis, Leonid Sigal, and Renjie Liao. Leveraging online olympiad-leve level math problems for Ilms training and contaminatiion-resistaa evaluation. arXiv preprint arXiv:2501. /4275, 2025.
Ahmed Masry, Do Xuan Long, Jia Qing Tan, Shafiq Joty, and Enamul Hoque. Chartqa: A benchmark for question answering about charts with visual and logical reasoning, arXiv preprint arXiv:2203. 22044422003..244, 2022.
Minesh Mathew, Viraj Bagal, Ruben Tito, Dimosthenis Karatzas, Emest Valveny, and CV Jawahar. Info- graphievqa. In Proceedings of the IEEF/CVF Winter Conference on Applications of Computer Vision, 2022.
Yu Meng, Mengzhou Xia, and Danqi Chen. Simpo: Simple preference optimization with reference-free reward. Advances in Neural Information Processing Systems, 2024.
OpenAL. Gis--V(ssion) system card, 2023.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedhack. Advances in neural information processing systems, 2022.
Arka Pal, Deep Karkhanis, Samuel Dooley, Manley Roberts, Siddartha Naidu, and Colin White. Smaug: Fixing failure modes of preference optimisation with dpo-positive. arXiv preprint arXiv:2402. 13228, 2024.
Junshu Pan, Wei Shen, Shulin Huang, Qiji Zhou, and Yue Zhang. Pre-dpo: Improving data utilization in direct preference optimization using a guiding reference model. arXiv preprint arXiv:2504. 15843, 2025.
Richard Yuanzhe Pang, Weizhe Yuan, He He, Kyunghyun Cho, Sainbayar Sukhbaatar, and Jason Weston. Iterative reasoning preference optimization. Advances in Neural Information Processing Systems. 2024.
Ryan Park, Rafael Rafailov, Stefano Ermon, and Chelsea Finn. Disentangling length from quality in direct preference optimization. arXiv preprint arXiv:2403. 19159, 2024.
Pulkit Pattnaik, Rishabh Maheshwary, Kelechi Ogueji. Vikas Yadav, and Sathwik Tejaswi Madhusudhan. Enhancing alignment using curriculum learning & ranked preferences. In Findings of the Association for Computational Linguistics: EMNLP 2024, 2024.
Guilherme Penedo, Quentin Malartic, Daniel Hesslow, Ruxandra Cojocaru, Alessandro Cappelli, Hamza Alobeidli, Baptiste Pannier, Ebtesam Almazrouei, and Julien Launay. The refinedweb dataset for falcon Ilm: outperforming curated corpora with web data, and web data only. arXiv preprint arXiv:2306.01. 2306.01116.. 2023.
Shangpin Peng, Senqiao Yang, Li Jiang, and Zhuotao Tian. Mitigating object hallucinations via sentence-level carly intervention. arXiv preprint 250: /2455, 2025.
15

---

Published as a conference paper at ICLR 2026
Tianyuan Qu, Longxiang Tang. Bohao Peng, Senqiao Yang. Bei Yu, and Jiaya Jia. Does your vision-language model get lost in the long vidco sampling dilemma? arXiv preprint arXiv:2503. 12496, 2025.
Alec Radford. Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry. Amanda Askell, Pamela Mishkin, Jack Clark, et al. learning transferable visual models from natural language supervision. In International conference on machine learning. 2021.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning. Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 2023.
Rafael Rafailov, Joey Hejna, Ryan Park, and Chelsea Finn. From r to q": Your language model secretly a q-function. arXiv preprint arXiv:2404.12533 2024.
Rajkumar Ramamurthy, Prithviraj Ammanabrolu. Kianté Brandly, Jack Hessel, Rafet Sifa, Christian Bauck- hage, Hamnanch Hajishirzi, and Yejin Choi, Is reinforcement learning (not) for natural language process- ing: Benchmarks, baselines, and building blocks for natural language policy optimization. arXiv preprint arXiv:2210.0124), 2022.
Corby Rosset, Ching-An Cheng, Arindam Mitra, Michael Santacroce, Ahmed Awadallah, and Tengyang Xie. Direct nash optimization: Teaching language models to self-improve with general preferences. ArXiv, 2024.
Michael Santacroce, Yadong Lu, Han Yu, Yuanzhi Li, and Yelong Shen. Efficient RLHF: Reducing the memory usage of PPO. arXiv preprint arXiv:2309.008 007554 2023.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint urXiv:1707.000 2017.
Ruichen Shao, Bei Li, Gangao Liu, Yang Chen, Xiang Zhou, Jingang Wang, Xunliang Cai, and Peng Li. Earlier tokens contribute more: Learning direct preference optimization from temporal decay perspective. arXiv preprint 2r02::2502. 14340, 2025.
Tong Shao, Zhuotao Tian, Hang Zhao, and Jingyong Su. Explore the potential of clip for training-free open vocabulary semantic segmentation. In European Conference on Computer Vision, 2024a.
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li. Y Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.033: 03300000 2024b.
Amanpreet Singh, Vivek Natarajan. Meet Shah, Yu Jiang. Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. Towards yqa models that can read. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2019.
Prasann Singhal, Tanya Goyal, Jiacheng Xu, and Greg Durrett. A long way to go: Investigating length correlations in rlhf. arXiv preprint arX1::2310.03716, 2023.
Nisan Stiemnon, Long Ouyang, Jeffrey Wu, Daniel Ziegler. Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. Learning to summarize with human feedback. Advances in neural information procexsing systems, 2020.
Alon Talmor. Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. Commonsenseqa: A question answering challenge targeting commonsense knowledge. Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume I (Long and Short Papers), 2019.
Yingshui Tan, Yilei Jiang, Yanshi Li, Jiaheng Liu, Xingyuan Bu, Wenbo Su, Xiangyu Yue, Xiaoyong Zhu, and Bo Zheng. Equilibrate rlhf: Towards balancing helpfulness-safety trade-off in large language models. arXiv preprint 2502v:2502. 1555, 2025.
Zhengyang Tang, Xingxing Zhang, Benyou Wang, and Furu Wei. Mathscale: Scaling instruction tuning for mathematical reasoning. arXiv preprint 2403 02884, 2024.
Gemini Team, Petko Georgiev, Ving Ian Lei. Ryan Burnell, Libin Bai, Anmol Gulati, Garrett Tanzer, Damien Vincent. Zhufeng Pan, Shibo Wang, et al. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv preprint ar arX3v:40030053300 2024a.
Gemma Team, Morgane Riviere Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, I.énnard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, et al. Gemma 2: Improving open language models at practical size, arXiv preprint arXiv:2408.0m 00/ 18, 2024b.
16

---

Published as a conference paper at ICLR 2026
Hunyuan Vision Team, Pengyuan Lyu, Xingyu Wan, Gengluo Li, Shangpin Peng. Weinong Wang, Liang Wu, Huawen Shen, Yu Zhou, Canhui Tang. Qi Yang, Qiming Peng, Bin Luo, Hower Yang. Xinsong Zhang, Jinnian Zhang, Houwen Peng, Hongming Yang, Senhao Xie, Longsha Zhou, Ge Pei, Binghong Wu, Kan Wu, Jieneng Yang, Bochao Wang, Kai Liu, Jianchen Zhu, Jie Jiang, Linus, Han Hu, and Chengquan Zhang. Hunyuanocr technical report, 2025.
Katherine Tian, Frie Mitchell, Huaxiu Yao, Christopher D Manning, and Chelsea Finn. Fine-tuning language models for factuality. In The Twelfth International Conference on Learning Representations, 2024.
Zhuotao Tian, Michelle Shu, Pengyuan Lyu, Ruiyu Li, Chao Zhou, Xiaoyong Shen, and Jiaya Jia. Learning shape-aware aware embedding for scene text detection. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2019.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaci, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:230..09288 2023.
Binghai Wang, Rui Zheng, Lu Chen, Yan Liu, Shihan Dou, Caishuang Huang, Wei Shen, Senjie Jin, Enyu Zhou, Chenyu Shi, al. Secrets of rlhf large language models part i:: Reward modeling. arXiv preprint axXiv:2401.06088 2024a.
Haoxiang Wang, Yong Lin, Wei Xiong, Rui Yang, Shizhe Diao, Shuang Qiu, Han Zhao, and Tong Zhang. Arithmetic control of Ilms for diverse user preferences: Directional preference alignment with multi-objective rewards. In ACL, 2024b.
Haoxiang Wang. Wei Xiong, Tengyang Xie, Han Zhao, and Tong Zhang. Interpretable preferences via multi- objective reward modeling and mixture-of-experts arXiv preprint arXiv:2406. 12845, 2024c.
Junjie Wang, Bin Chen, Yulin Li, Bin Kang, Yichi Chen, and Zhuotao Tian. Declip: Decoupled learning for open-vocabulary dense perception. In Proceedings of the Computer Vision and Pattern Recognition Conference, 2025.
Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, al. Qwen2-vl: Enhancing vision-language model's perception of the world at any resolution. arXiv preprint arXiv:2409. 12191, 2024d.
Shiqi Wang, Zhengze Zhang. Rui Zhao, Fei Tan, and Nguyen Cam-Tu. Reward difference optimization for sample reweighting in offline rlhf. In Findings of the Association for Computational Linguisticx: EMNLP 2024, 2024e.
Shiqi Wang. Zhengze Zhang, Rui Zhao, Fei Tan, and Cam Tu Nguyen. Reward difference optimization for sample reweighting in offline rlhf. arXiv preprint arXiv:2408. 09385, 2024f.
Yizhong Wang, Hamish Ivison, Pradeep Dasigi. Jack Hessel, Tushar Khot, Khyathi Chandu, David Wadden, Kelsey MacMillan, Noah A Smith, Iz Beltagy, et al. How far can camels go? exploring the state instruction tuning on open resources. In Teirth-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2023.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoe V Le, Denny Zhou, et al. of-huuh prompting elicits reasoning in large language models. Advances in neural information processing systems, 2022.
Junkang Wu, Yuexiang Xie, Zhengyi Yang. Jiancan Wu, Jiawei Chen, Jinyang Gao, Bolin Ding. Xiang Wang, and Xiangnan Hc. Towards robust alignment of language models: Distributionally robustifying direct preference optimization. arXiv preprint (tt::2407.07880 2024a.
Junkang Wu, Yuexiang Xie, Zhengyi Yang, Jiancan Wu, Jinyang Gao, Bolin Ding, Xiang Wang, and Xiangnan He. Vbeta-dpo: Direct preference optimization with dynamic Vbeta. In Advances in Neural Information Processing Systems, 2024h.
Violet Xiang, Charlic Snell, Kanishk Gandhi, Alon Albalak, Anikait Singh, Chase Blagden, Duy Phung, Rafael Rafailov, Nathan Lile, Dakota Mahan, et al. Towards system reasoning in lmss: Learning how to think with meta chain of though. arXiv preprint arXiv:2501 04682, 2025.
Wenyi Xiao, Ziwei Huang, Leilei Gan, Wanggui He, Haoyuan Li, Zhelun Yu. Fangxun Shu, Hao Jiang, and Linchao 7hu. Detecting and mitigating hallucination in large vision language models via fine-grained ai feedback. arXiv preprint 2404 14233, 2024.
17

---

Published as a conference paper at ICLR 2026
Wei Xiong, Hanze Dong, Chenlu Ye, Ziqi Wang, Han Zhong, Heng Ji. Nan Jiang, and Tong Zhang. Iterative preference learning from human feedback: Bridging theory and practice for RLHF under Ko-constaiin.. In Forty-first International Conference on Machine Learning, 2024.
Haotian Xu, Xing Wu, Weinong Wang, Zhongzhi Li, Da Zheng, Boyuan Chen, Yi Hu, Shijia Kang, Jiaming Ji, Yingying Zhang, et al. Redstar: Does scaling long-cot data unlock better slow-reasoning systems? arXiv preprint 2rXiv:2501.11284 2025a.
Shiyue Xu, Fu Zhang, Jingwei Cheng, and Linfeng Zhou, MWPO: Enhancing LLMs performance through melth- preference strength and length optimization. In Findings of the Association for Computational Linguistics: ACL, 2025, 2025b.
An Yang, Baosong Yang. Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li. Dayiheng Liu, Huang, Haoran Wei, al. Qwen2.5 technical report. arXiv preprint arXiv2241..15115. 2024a.
An Yang, Beichen Zhang, Binyuan Hui, Bofei Gao, Bowen Yu, Chengpeng Li, Dayiheng Liu, Jianhong Tu, Jingren Zhou, Junyang Lin, et al. Qwen2.5-math technical report: Toward mathematical expert model via self-improvement. arXiv preprint 2rX 2024b.
An Yang. Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng. Bowen Yu, Chang Gao, Chengen 2025a.
Huang, Chenxu Lv, et al. Qwen3 technical report. arXiv preprint 2505,
Senqiao Yang, Jiaming Liu, Ray Zhang, Mingjie Pan, Zoey Guo, Xiaoqi Li, Zehui Chen, Peng Gao, Yandong Guo, and Shanghang Zhang. Lidar-llm: Exploring the potential of large language models for 3d lidar understanding. arXiv preprint arXiv:2312.14074, 2023a.
Senqiao Yang, Tianyuan Qu, Xin Lai, Zhuotao Tian, Bohao Peng, Shu Liu, and Jiaya Jia. An improved baseline for reasoning segmentation with large language model. arXiv preprint arxiv:231..17240, 2023b.
Senqiao Yang, Yukang Chen, Zhuotao Tian, Chengyao Wang. Jingyao Li, Bei Yu, and Jiaya Jia. Visionzip: Longer is better but not necessary vision language models. arXiv preprint arXiv:2412.04467 2024c.
Senqiao Yang. Zhuotao Tian, Li Jiang, and Jiaya Jia. Unified language-driver zero-shot domain adaptation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024d.
Senqiao Yang, Junyi Li, Xin Lai, Bei Yu, Hengshuang Zhao, and Jiaya Jía. Visionthink: Smart and efficient vision language model via reinforcement learning. arXiv preprint arXiv:2507. 2025b.
Yueqin Yin, Zhendong Wang, Yi Gu, Hai Huang, Weizhu Chen, and Mingyuan Zhou. Relative preference optimization: Enhancing Ilm alignment through contrasting responses across identical and diverse prompts. arXiv preprint 240.:22002.19958 2024.
Tianyu Yu, Haoye Zhang, Yuan Yao, Yunkai Dang, Da Chen. Xiaoman Lu, Ganqu Cui, Taiwen He. Zhiyuan Liu, Tat-Seng Chua, al. Rlaif-v: Aligning milms through open-source ai feedback for super gpt-4v trustworthiness. arXiv preprint arXiv:2405. 2024.
Weizhe Yuan, Richard Yuanzhe Pang, Kyunghyun Cho, Sainbayar Sukhbaatar, Jing Xu, and Jason Weston. Self-rewarding language models. arXiv preprint 2rXiv:2401. 10020, 2024.
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. HellaSwag: Can a machine really finish your sentence? In Anna Korhonen, David Traum, and Lluís Mårquez (eds.), Proceedings of the 57th Anmual Meeting of the Association for Computational Linguistics, 2019, URL
Drq/P19-144 orq/p19-1412
Weihao Zeng, Yuzhen Huang, Qian Liu, Wei Liu, Keqing He, Zejun Ma, and Junxian He. Simplerl-zoo: Investigating and taming zero reinforcement learning for open base models in the wild. arXiv preprint arXiv:2503. 2025.
Hanning Zhang, Jiarui Yao, Chenlu Ye, Wei Xiong, and Tong Zhang Online-dpo-rl: Unlocking effective reasoning without the ppo overhead. /tttps://gitttttta
com/ Wwt.AALLLO////One/eeea
2025a.
Yi-Fan Zhang, Tao Yu, Haochen Tian, Chaoyou Fu, Peiyan Li, Jianshu Zeng Wulin Xie, Yang Shi. Huanyu Zhang, Junkang Wu, et al. Mm-rlhf: The next step forward in multimodal Ilm alignment. arXiv preprint arXiv:2502.1 10399) 2025b.
Zhenru Zhang, Chujie Zheng, Yangzhen Wu, Beichen Zhang, Runji Lin, Bowen Yu, Dayiheng Liu, Jingren 7hou, and Junyang Lin. The lessons of developing pracess reward models in mathematical reasoning. arXiv preprint arXiv:2501.0730/ 2025c.
18

---

Published as a conference paper at ICLR 2026
Zhiyuan Zhao, Bin Wang, Linke Ouyang. Xiaoyi Dong. Jiaqi Wang. and Conghui He. Beyond hallucinations: En- hancing Ivims through hallucination-awar direct preference optimization. arXiv preprint arXiv:3311.16839, 2023.
Chujie Zheng, Pei Ke, Zheng Zhang, and Minlie Huang. Click: Controllable text generation with sequence likelihood contrastive learning. In Findings of ACI, 2023a.
Chujie Zheng, Zhenru Zhang, Beichen Zhang, Runji Lin, Keming Lu, Bowen Yu, Dayiheng Liu, Jingren Zhou, and Junyang Lin. Processbench: Identifying process errors in mathematical reasoning. arXiv preprint arXiv:2412. 2024.
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang. Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-as-a-judge with mt- mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 2023b.
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng. Siyuan Zhuang. Zhanghao Wu, Yonghao Zhuang. Zi Lin, Zhuohan Li, Dacheng Li, Brie. P Xing. Hao Zhang, Joseph E. Gonzalez, and lon Stoica. Judging Im-aa-a--udge with mt-bench and chatbot arena, 2023c.
Rui Zheng, Shihan Dou, Songyang Gao, Yuan Hua, Wei Shen, Binghai Wang. Yan Liu, Senjie Jin, Qin Liu, Yuhao Zhou, et al. Secrets of RLHF in large language models part PPO. arXiv preprint arXiv:2307.049: 2394444944 2023d.
Zhisheng Zhong, Chengyao Wang, Yuqi Liu, Senqiao Yang, Longxiang Tang, Yuechen Zhang, Jingyao Li, Tianyuan Qu, Yanwei Li, Yukang Chen, et al. Lyra: An efficient and speech-centric framework for omni- cognition. arXiv preprint arXiv.2412.09501, 2024.
Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou, and Le Hou. Instruction-following evaluation for large language models. arXiv preprint 2rXiv2231.009//1 2023.
Wenxuan Zhou, Ravi Agrawal, Shujian Zhang, Sathish Reddy Indurthi, Sanqiang Zhao, Kaiqiang Song. Silci Xu, and Chenguang Zhu. Wpo: Enhancing rlhf with weighted preference optimization. arXiv preprint arXiv:2406. 1827, 2024a.
Yiyang Zhou, Chenhang Cui, Rafael Rafailov, Chelsca Finn, and Huaxiu Yao. Aligning modalities in vision large language models via preference fine-tuning, arXiv preprint 2rXiv:2402.1141) 2024b.
Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: Enhancing vision- language understanding with advanced large language models. arXiv preprint arXiv:2304. 10592, 2023.
Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences. arXiv preprint arXiv: (xxiv:190900859) 2019.
Zhengxia Zou, Keyan Chen, Zhenwei Shi, Yuhong Guo, and Jieping Ye. Object detection in 20 years: A survey. Proceedings of the IEEE, 2023.
19

---

Published as a conference paper at ICLR 2026
Uni-DPO: A Unified Paradigm for Dynamic Preference Optimization of LLMs Supplementary Material
OVERVIEW
This material provides supplementary details to the main paper, including the following sections:
Motivation Details
A.11
Analysis of Reward vs. Score Margins
A.2
Overfitting Behavior of Quality Weight
A.3
Stability of Performance Weighting
B
Method Details
B.T
DPO with Length Normalization
B.2
Performance Weighting Factor
B.3
Overall Optimization Objective
C Training Details
C.1
Training Setup
Training Data
Training Pseudocode
Training Complexity
Training Dynamics
D
Evaluation Details
D.1
Evaluation Setup
D.2
Evaluation Results
D.3
Further Ablation Study
D.4g
Downstream Task Evaluation
Discussions
E.
Revisiting Uni-DPO through the RL Paradigm
Impact of Length Normalization on Alama3--B-Basee
E.
Further Discussions on c-NLL Loss
Rationale for Relaxing the Original Focal Weight
Related Work
Case Studies
Broader Impact
LLM Usage Statement
Math
TextvQA
OCRBench
MATH 78,2
73.2
Olympiad
77.6
POPE
86.0
50.6
677
InfoVQA
Bench
41.5
30.4
GSMBK
MME
1905 1811
11.7 13.8
ChartQA
60.0
54.4
$4.8
AMC23
$1.3
College
MMBench
RealWorldOA
GaoKao En
Math
70.8
68.1
SEEDBench
ScienceQA
Base
DPO
SimPO
Uni-OPO (Duns)
(a) Results on mathematical reasoning tasks (7B)
(b) Results multimodal tasks

---

Published as a conference paper at ICLR 2026
A MOTIVATION DETAILS
In this section, we deepen the discussion supporting the key observations from the main paper Sec. 2.2 We focus on three critical phenomena: (1) the empirical relationship between reward margins and expert-assigned score margins of training pairs, as illustrated in the main paper Fig. (2) the overfitting dynamics that emerge during preference learning if only consider the data quality. shown in the main paper Fig. [cl] and (3) the stability of performance weighting factors, shown in the main paper Fig. [5b These analyses help to shed light on the challenges inherent in preference learning and suggest the need for a more flexible approach, motivating our design of the Uni-DPO framework.
A.1 ANALYSIS OF REWARD VS. SCORE MARGINS
To investigate how the data quality of preference samples relates to the model's learning dynamics during training, we fine-tune Qwen2.5-Math base model on a 50/50 split of the preference data, using half of the dataset for training and reserving the remainder for validation, to emulate exposure to unseen pairs during training. As illustrated in main paper Fig. 4 the results reveal that a pair's intrinsic quality (measured by its expert-assigned score margin (S... Si)) does not necessarily correlate to the model's learning difficulty, as quantified by its reward margin A, (defined the main paper Eq. (3). For instance, a pair with high quality (i.e., large score margin) may be already well-fitte by the model. This observation prompts us to consider whether relying solely on intrinsic data quality in preference training is sufficient and to examine the potential negative consequences further.
In addition, we conducted a parallel analysis of the textual understanding. We fine-tune Qwen2.5-7D on the prai__prefs split of the UltraFeedback (Cui et al., 2023) dataset and then evaluate it on the held-out test preffs split. The resulting Spearman correlation between expert score margin (S S and model reward margin A, was only P 0.08 (using the same metrics in the main paper Fig. 4 indicating almost no correlation between intrinsic data quality and learning difficulty. This result further reinforces the need for weighting mechanisms that go beyond only considering data quality.
A.2 OVERFITIIG BEHAVIOR OF QUALITY WEIGHT
DPO with Uqual only. We perform an investigation of overfitting in preference learning. Specifically, during fine-tuning of Qwen2.5-7B (Yang et al., 2024a) on the train_prefss split of the UltraFeed- back (Cui et al., 2023) dataset, we extend the standard DPO objective (the main paper Eq. by incorporating a data quality weighting factor Wqual (the main paper Eq. (5)) We tracked both the training loss and the evaluation loss measured on the held-out test foeff_presseeeeeeeeerer split. The main paper Fig. [cc] shows that as training progresses, the training loss steadily decreases, whereas the evaluation loss, after initially falling, reaches minimum (marked by the dashed vertical line) and then begins to rise. This divergence signals that the model starts to overfit to the training data beyond that point. This suggests that effective training strategies must also adapt to the model's learning dynamics in addition to the data quality.
DPO with iqual and Wperf- After we also incorporate the performance weighting factor Wperf (the main paper Eq. (TO) into the DPO objective, we observe that both the training and evaluation losses exhibit a consistent downward trend throughout the training process, as shown in the main paper Fig. [dd] This suggests that our proposed Wper effectively mitigates overfitting, allowing the model to better leverage the data pairs during training and better align with human preferences.
A.3 STABILITY OF PERFORMANCE WEIGHTING
Finally, we compare the gradient norms of the original weigh-weighted DPO (the main paper Eq. 77) and our stabilized variant (the main paper Eq. (T0)) during fine-tuning Llama3-88 (amma3-BB-Bil -B--esse (Al @ Meta 2024b) on the UltraFeeddback (Cui et al. 2023) dataset. The results show that the original focal- weighted DPO exhibits frequent, large spikes in gradient norm throughout training, whereas our stabilized variant maintains a much smoother profile. This finding suggests that naively applying focal loss (Lin et al. (2017) weights in preference learning can induce unstable updates, while our stabilized form, which utilized length normalization (LN) and a relaxed, unified constraint Tie yields consistently more stable training dynamics.
2

---

Published as a conference paper at ICLR 2026
B METHOD DETAILS
In this section, we provide a detailed exposition of the key components of our proposed Uni-DPO framework, including its formulations and gradient derivations. We begin with review of the fundamental principles of DPO, then introduce the length normalization (LN) in Sec. B.I. next present the performance weighting strategies in Sec. B.2, and finally conclude with the derivation of the overall objective function in Sec. B.3
B.1 DPO WITH LENGTII NORMALIZATION
DPO objective Rafailov et al. 2023 augmented with length normalization (LN) demonstrated significantly more stable training dynamics in experiments. In the following, briefly introduce DPO with length normalization to prepare for the subsequent sections. The gradient the DPO loss (the main paper Eq.
①T) with respect to the policy model wy parameters 0 can be written
V e DPO
-B( ((((,, 91) Bw))
V log (Vwl
Ve log Te (1) )]
(12)
higher when reward estimate wrong increase likelihood of Vw decrease likelihood ofy!
where the implicit reward function (excluding the partition function (xx) for simplicity) is: To x))
r(x, v) = log
(13)
Fe (y x))
The gradient of DPO with length normalization (DPO w/ LN) is:
V VoLDrowLa L/
log (o
log To xr)
(14)
- (FIx(1, sn) (a, (.)),,))))))))))))))( VV
yw
130
where the reward function with length normalization (fIN) is defined as follows:
B
To
x))
(L, v)
log
(15)
y Tref
y
Recent studies (Meng et al. 2024 Park 2024
al. 2024) have observed that DPO training
allows the policy to exploit length bias in the data, assigning disproportionately high probabilities longer sequences. As a result, DPO-trained models tend produce unduly long responses, even
though the quality of response should remain independent of the token length of it (Dubois et al.
2024; Singhal et al., 2023] Durmus et al. 2022 Kabir et al.) 2024). Thus, length normalization (LN)
has been introduced in the framework of DPO by recent studies (Meng al. 2024
Lu et al. 2024
B.2 PERFORMANCE WEIGHTING FACTOR
Building on our previous discussion of length normalization, in this section, we further discuss the performance- sooommane--eightinng factor. We first derive the gradient for the original DPO objective augmented with vanilla focal loss (Lin et al.) 2017) weighting factor Witcaal (the main paper Eq. (8) ). and then
transition to the gradient derivation for our more stable variant Wpen (the main paper Eq. 110 This analysis prepares the groundwork for deriving the gradient of our overall Uni-DPO optimization
objective in Sec. B.3
The direct integration with DPO.
We first derive the gradient of the focal-weights DPO loss LDPO FL. arriving at the expression in the main paper Eq. (9) Recall that C wT. is defined as:
LDPO w/FO. FL. = -B Bez )~ Wrocaa (T)) log a (Ar)],
(16)
where the implicit reward margin A, and the weighting factor Wiocaa are defined as:
A, = r(x, yw) r(x, y1) = -lggg (o x) Bog Tee /
No (yl x)
Thet x))
(17)
(Blog (T
To (30 (u)(2) )"²
Wfocca -
3 log Tea (vr
= 11 (A.))
Teet (Ven x.)
3

---

Published as a conference paper at ICLR 2026
The gradient of this loss concerning the policy model no parameters proceeds as follows:
V DPO w/ Ft. - =<o
B
(1 -o0
(1-o(A)) Vo0(4,)
+ ( ((())
-B
(4,)
(18)
-B
((4)) Vo(A,)
(1 (1- (1- 7) (1 (1- (A,) 7o -.,) Vo(A))
where Vo(A,) - 3 Vo log To (y [x) Ve To
As shown in the main paper Fig. 5a the gradient the weightee DPO objective peaks when the model performs poorly and diminishes as performance improves. This dynamic naturally prioritizes harder examples during training by downweighting the contribution of already well-fitted samples.
Calibrated weighting factor. Since directly incorporating focal-loss weights into preference learning destabilize training (as described in the main paper Sec. B.3), incorporate length normalization (LN) together with a unified performance margin Tiet, thus arrive at our more stable performance weight Wperr defined in Eq. TT)) of the main paper. In the sequel, derive the DabilizedDDOOO gradient of this stabilized form. The stabilized DPO loss with performance weighting is defined as:
LDoO w/ perf (.. [marr(Ta) llg (Ar).
(19)
where the implicit reward margin A, and the performance weight Wperr defined
3
To ( x)
B
To (3)1 xr)
A
VI,N (2, Bw)) - -IN(2 (1) =
log
log
[yaol
Tee (hh
(x)
Wee
(1I
x))
(20)
3 [Bhhll
B
]
Uprr
log No (vwl)) Tevl
log No To (z:)
Trer
First, we introduce the adjusted reward margins Andj for simplicity, which is defined as:
3
B
Adjj
log To (y
x))
log To( xr)
(21)
ywel
[30|
Thus, we have:
3
VODni VoA,
Vo log To(w [x)
Vo log colu zz)
(22)
yuvl
lyal
For brevity, denote the performance weight:
Uperf = (1-o))
(23)
We also require the following intermediate gradients:
1 V (og) o(A,) - -1-
VoL, - (1- VoAr,
(24)
(oo = o (ayy ) 10ll--(Ann o(An))) VODa..
follows that:
V Vowper Vo (1
( |
-1 )) 7- ((Am)) (1 o(Aui)) VoAulj
(25)
-1 7 ((aa)) VoDad).
Consequently, the overall gradient of the loss be expressed
4

---

Published as a conference paper at ICLR 2026
Ve DDCD w/ perf
= E
Vo logr
= E
Wpert Vo log oo
(A.)) +V (Vor))))0))
(26)
= E
= E
-- (1 o(Ana) (1- (0
Both the gradient of the original focal-weighted DPO LDO WL (Eq. 16 and its stabilized
performance-weightee counterpart LDPO were (Eq. (19) share common structural backbone: they modulate the base DPO gradient VgA, , by a multiplicative "focusing" term of the form (1 (1-o(-))7 [(1 o(.)) - (-) In (A,)].
This factor down- weights examples for which the model is already confident (i.e.o((A,) is high) and up-weights those where underperforms (i.e..o(Ar) is low). both the original weighte seeermannne-eighter variant LDOO w/ perf, the exponent y
DPO loss LDo FL and stabilized determines how sharply the model concentrates on harder samples. Both losses share the same - - focal multiplier, preserving targeted emphasis difficult cases.
With this formula established, we now incorporate it into the overall objective function of Uni-DPO.
B.3
OVERALL OPTIMIZATION ORJECTIVE
Recall that the overall objective of Uni-DPO is defined as:
LUi--DO (E(..... Mquua (Vies y1) (Tpe log or (A,)) ACUNLLL
(27)
Based the derivation in Eq. (26). the gradient of Uni-DPO can be expressed as:
V VeLuni-PO
--E(g.ve Uquul (1 (A ) (11 (1-o(A,) - (Any)) log (4,) VOAr
-
log To (ywr x))
+ A. (log (Ber x²) > log TO / x)) 1Sw 2 Tgaor) Vo
(28)
= --B { [1 ((1-0(A,) - ~ (A (ogggg/(())) (AA,) +
log To (Bw Mull (x)
V
1yve
log (
+ [1- (110(4))))))))))) log 0 (A.)) V
where 1 denotes the indicator function / 1 (log (yw x) > log ( 1)) > Tgood Wquat is the quality waighting factor, and S.p is the score of the preferred completion Yw- The A, is the reward margin in Eq. (20) and Awij is the adjusted reward margins in Eq. (.
Upon deriving the gradient of the overall objective LUiiDPO we proceed with the discussion from the main paper (in main paper Sec. (5) by analyzing Uni-DPO's gradient coefficient GCA within the Unified Paradigm proposed by [Shao et al. (2024h) and comparing it to that of DPO in Sec. E.
C TRAINING DETAILS
In this section, we present comprehensive details related to training, including the training setup (Sec. C.II. the training datasets (Sec. C.2), the training pseudocode (Sec. C.3) the training complexity (Sec. C.4). and the training dynamics (Sec. C.5). By making these details available, we aim to enhance reproducibility and further validate the effectiveness of Uni-DPO.
5

---

Published as a conference paper at ICLR 2026
Table C.I: Training setup for textual understanding evaluation. We maintain largely consistent hyperpa- rameter configurations across model variants and datasets to demonstrate robustness and scalability of Uni-DPO. In this table, the placeholder x in Qwen2.5-rB denotes the model size in billions of parameters, with a E {0.5, 1.5, 3, 7, 14). We set the threshold Fgood 3.2 since this value corresponds to the median score of the chosen response Yu in the training datasct, thereby reflecting the central tendency of the score distribution.
Setting
Model
Laama3-0B-Bas
Llama3-8B-- -Bt-Bace Saama--SU-Instrur
Cemma-2-08-I7
Qwen2.5-cB
Leuming
1.0c-6
Open puram Til
1.0
Wlyal param ?
0.7
"ted puram
c.NLL loss lambda A
0.001
Data threshold
3.2
Mix precision training
B6foa116
Leaniing scheduler
Cosinee
128
Warmup
stops)
Opsimizer
AdaoWW Loshchilov
Hutter 2017
Epoch
Seed
Table C.2: Prompt template for math reasoning ability training and evaluation. All evaluations employ
greedy decoding and widely used zero-shot chain-of-thought (CoT) (Wii et al. 2022) prompting for consistency.
<lim_ startbsssttmm
Please reason step by step, and put your final answer within \hoxed(] _<cli_eenll>>>>>>>> _nddl>
<lim__tartte startbuser
[Question] } Let's think step by step and output the final answer within \boxed| ><lim_endl>>
<lim stattaassisttn
C.I TRAINING SETUP
General training byperparameters. All general training settings, including the batch size, num- ber of epochs. learning rate schedule, optimizer choice, and on, are identical to those used in SimPO (Meng et al.) 2024), ensuring a fair and controlled comparison.
Method-specific training hyperparameters. For the training hyperparameters of DPO (Rafailoy al. 2023) and SimPO (Meng et al. 2024), we follow the officially recommended best practices and perform grid search to select configurations that achieve the best performance. For Uni-DPO, we observed that the choice of hyperparameters can vary between models and affects the benchmark outcomes. To address this, we run preliminary experiments to identify appropriate values for the performance weight Wpert hyperparameters Y and Tref as illustrated the main paper Figs. 6a and 6b) the practical range of is [1.0, 5.0] and Tref is [0.5, 2.0]. Once these ranges were established, we maintained largely consistent hyperparameter configurations across all models, datasets, and benchmarks to demonstrate the robustness and scalability of Uni-DPO, Detailed training configurations for textual understanding tasks are given in Tab. C.I and for mathematical reasoning tasks and multimodal tasks are provided in Tab. C33
Computation environment. Our experiments were mainly conducted based the code from the SimPO repository to ensure a fair comparison. All computations are on a suite of 8xH800 GPUs.
C.2 TRAINING DATA
Textual training data. All training configurations, including the training data and training pipeline, are kept strictly aligned with those used in SimPO (Meng al.) 2024) to ensure a fair comparison. For the v0.l version of the data, the quality score of cach preference pair is directly provided by the publicly available UltraFeedback dataset. Unlike DPO and SimPO, which can only exploit paired preference signals, Uni-DPO leverages these supervision signals in more fine-grained manner and performs adaptive correction throughout the preference learning process.
For the v0.2 version of the training data, which utilizes preference labels from more powerful reward model, we obtain the quality scores of data using two models, GPT-4o (Hurst et al.] (2024) and ArmoRM ( (Wang al. 2024c). For GPT-4o, we adopt the widely used 'single-vi" judge prompt from
judde

---

Published as a conference paper at ICLR 2026
FastChat (Zheng al. 2023c to call the model, which first asks the model explain its judgment in tenns of the helpfulness, relevance, accuracy, depth, creativity, and level of detail of the response, and then produce overall quality score. The range of the quality score is from 1 to 10. For ArmoRM, the reward model evaluates responses along dimensions such as helpfulness, correctness, coherence, complexity, and verbosity, and we use the offfiiiaall scoring released with ArmoRM to obtain the final scores Except for the way quality scores are obtained, the experimental settings for v0.2 are identical those of v0.1 in all other aspects.
Math training data. For mathematical reasoning ability training data, following recent stud- ies [Mahdavi et al. 2025; Face 2025 Xiang et al., 2025 Zhang et al. 2025a), we randomly sampled 20K math problems from the widely used NutinaMath [I et al. 2024) dataset. We generate prefer- ence training data in an on-policy manner: using the base model To under training. we sample N = candidate solutions y ~ (Y((x) for each problem prompt æ using the zero-shot chain of thought (CoT) prompting template shown in Tab. C.2 with sampling temperature of 0.8 and top-p of 0.8. We then apply a rule-based classifier to label each sample correct incorrect using the provided ground truth answer Furthermore, we score each response using a open source domain-specifte reward model. Qwen2.5- -Math--R-77B (Zhang et al., 2025c). M7 To maintain the training data quality, we exclude problem r for which all sampled responses y To (Y (x) are labeled as incorrect, as these cases prove excessively challenging for the model learn. From the remaining pool, we select only those positive samples whose solutions are correct and whose reward scores S., exceed those of corresponding negative samples Sw. > Si), thereby ensuring a dual level of data quality control. This process yields final set of 20K high-quality math reasoning training examples, which we use uniformly across DPO, SimPO, and Uni-DPO training objectives to ensure a fair comparison.
Multimodal training data. For multimodal training, we employ the recently publicly available MM-RLHF (Zhang et al., (0255) dataset, which is a high-quality human-annotated preference dataset with reliable ranking rationales. We use the Image subset, which includes three question types: Long (long-text), Short (short-text), and MCQ (multileecchoice), totaling approximately 50K preference learning pairs. We then utilize the critique-base MLLM reward model, MM-RLHF--Rewald Ilava-ov-qwen (Zhang et al.) 20255)), to score the sampled responses using the human- human-provided provided critiques, to ensure both quality and efficiency We apply this same set of around 50K multimodal examples across DPO, SimPO, and Uni-DPO objectives to maintain fair comparison.
C.3 TRAINING PSEUDOCODE
The pseudocode for computing the training loss in Uni-DPO is presented in Algorithm [ In brief, the procedure (1) performs forward passes under both the policy To and reference models गeर to obtain log- bog-probability ratios; (2) computes dual adaptive weights; (3) applies these weights to the DPO loss; (4) introduces the calibrated NLL loss for high-quality, hard positive samples; and (5) returns the batch-mean of all loss components.
C.4 TRAINING COMPLEXITY
In this section, we provide a comparison of the training complexity between DPO and our Uni-DPO. Both DPO and Uni-DPO share the same computational bottleneck: each training sample requires four LLM forward passes (two from the policy model and two from the reference model), typically in the range of trillions of FLOPs per sample.
Any extra operations in Uni-DPO are negligible. The only additional costs come from computing the dual weights and the c-NLL term, which are simple element-wise arithmetic (differences, sigmoid,
"Since ArmoRM is a regression model, its output scores are not constrained to a strict range. Therefore, after scoring all entries in the datasot, we apply Z-score normalization to standardize the scores, and then use the CDF of the standard normal distribution map them into the range 0-10, match the v0.1 setting.
$ *Our evaluation code for math answer correction is adapted from simploRL--rcasn (Zeng et al. 2025).
'We use the solution reformatting method suggested by Zheng et al. (2024) to split the multi-step responses
into individual steps for PRM to process.
"The prompt we use to call the math PRM is suggested by the official implementations
*The prompt use call the multimodal reward model suggested by the official implementations
7

---

Published as a conference paper at ICLR 2026
Table C.3: Training setup for math reasoning and multimodal tasks. We maintained largely consistent
hyperparameter configurations across all models demonstrate the robustness scalability of Uni-DPO.
Setting
Model
Owen2.5-Math-1155 Qmen2.5-Math-78
Qwen2-VIL-28
c-NIJ. lamidda Faee
0,0011
Ujer param fut
Mixa param ?
"ter parim
3.0
Mis precision training
Bllcai16
Leaming rate schedaler
Cosine
Global batch size
128
DPO label smoothing
D.D
Warmup rano
D.D
Seel
42
Algorithm Uni-DPO loss calculation
Input: Training samples (æ, Mw: yr) D
Hyperparameters: eta(T)) beta((3), gamma(?), lambda(A), tao ref((ne), tao_aaood ggod() Output: Uni-DPO loss
1: import torch
2:
import torch.nn. www..cioonnoiaalaommmmmmmmmmmmaa as
3:
4:
def' get Uni DPO loss(self, (æt, Wu. 31)) -> torch.eensor:
5:
# policy model forward
6:
policy_chosee logpp_oppspppppp = model.dpo fitpo_forwrrdrrrr. yw)) Note that the 'logps' are length-normalized
7:
wwwpereeeeeeeeeeeee policy_logratioe logps model.dpo fororfooowrd[
model.dpo_fforward (æ, 3/))
8:
9:
# reference model forward with torch.no grad():
11:
12:
ref chosen logps ref model.dpo forward((æ; yy.)))
rejected logps = ref model.dpo forward((2;, 3)))
ref_logratios ref cof_chosen_logny ref_rejected_logee
15:
16:
logits mmllicyllogratinn ref_ lograaios
17:
18:
# calculate dual adaptive weights
19:
w_quality compute seciteteeeehh__e quality(score, margin, self.eta)
20:
w performance = (1 (sif policy logratios self.tao__eff) self.gamma
21:
# calculate loss losses w __quality * ___peffrrmacce * Flogssigmoid/sel..bei * logits)
24:
25:
# calculate e-NLL loss
26:
positive positive_coeffi self lambda sign(reference__hhg cchosen_lognet policy_chosen_L
27:
positive_cceli coeffi = torch. where(score chosen >= tau_good, positive_coeffi coeffi, 0)
28:
losses = positive_cnefff policy_chosenn_lgg
29:
30:
return loss meann(
exponentiation, masking) on vector outputs after the LLM forward pass. These operations contribute only constant-factor overhead C' of around few hundred FLOPS. Formally, have:
FUn--PO RS FDRO - C, with C FDPO-
(29)
In practice, this means that training complexity is effectively identical between DPO and Uni-DPO. Our experiments also verify that the total FLOPs difference between DPO (3.870951c+17) and Uni-DPO (3.870986e+17) 17) is negligible.
In terms memory, Uni-DPO requires storing additional intermediate tensors such as the weight (Wqual and Wpaat) and binary masks. However, these also contribute only small constant overhead relative to the total activation memory required by LLM inference, which dominates overall GPU memory consumption.
8

---

Published as a conference paper at ICLR 2026
DPO
SimPO
Uni-DPO
0.703
0.65
0.55
0.522
0.500
0.375
1.23
0.525
0.43
700
120
500
32
Gradlenn Nores
300
22
300
90
200
100
-00.0
62.5
Pollcy LogPs (Chosen)
M AM M MMM
67.5
-0.4
-0.6
10.0
0.5
-0.5
0.6
-1.
5..1
-0.7
-77.3
230
300
-10
-75
0.2
-32
-85
-50
-55
300
0.8
0.8
-330
20
105
15:0
250
30
120
130
200
50
100
120
200
Step
Figure C.1: Training dynamics mathematical reasoning tasks. From top to bottom: training loss, gradient norm, policy log probabilities for chosen responses, and those for rejected responses across training steps.
C.5 TRAINING DYNAMICS
To more intuitively illustrate the behavior of our Uni-DPO compared with DPO and SimPO during
training, we provide training curve comparisons in this section, as shown in Fig. C.T The training configuration for Uni-DPO is detailed in Tab. C.3 For DPO and SimPO, we perform grid search over the hyperparameters and report the best-performing configuration. The analysis is as follows.
Uni-DPO provides more stable training dynamics. From the first two columns of the figure, observe that Uni-DPO exhibits more stable training behavior, with smoother and more consistent loss curves and fewer spikes gradient norms. This suggests that Uni-DPO is less prone to gradient explosion or vanishing issues, enabling more effective learning and convergence. This observation consistent with the results in Fig. [b We attribute this stability to our improved design of the performance-weightee factor Wpert in the optimization objective.
Uni-DPO better preserves the generation probability of positive samples. As shown in the third column of the figure, Uni-DPO better preserve the generation probability of positive samples compared with SimPO. This benefit comes from the c-NLL loss Le-NL introduced in Sec. 3.4 (main
"hhe corresponding evolution of test performance with respect training steps is shown in Fig. D.1

---

Published as a conference paper at ICLR 2026
paper), which explicitly encourages the model increase the likelihood of high-quality and hard positive samples, thereby helping it better capture the distribution of good responses.
Uni-DPO better separates positive and negative samples. From the last two columns of the figure, that Uni-DPO enables the model to better distinguish between positive and negative samples compared with DPO. This indicates that Uni-DPO leverages preference information more effectively during training, leading to stronger discriminative capability. This also validates the effectiveness of the performance-weije weighting factor uport proposed in Sec. 3.3 (main paper).
D EVALUATION DETAILS
We present detailed descriptions of the evaluation setup (Sec. D.I], D. the evaluation results (Sec. D.2]. additional ablation studies (Sec. D.3). and downstream task evaluation (Sec. (.4).. By providing these, we aim further validate the effectiveness and scalability of Uni-DPO and to enhance reproducibility.
D.1 EVALUATION SETUP
We present the evaluation setup, including the evaluation benchmarks and decoding strategy.
Evaluation benchmarks. We provide the details about the evaluation benchmarks used in this work. The evaluation benchmarks for textual understanding ability are summarized Tab. D.1
The descriptions of the benchmarks of textual understanding evaluation are as follows:
AlpacaEval 2.0 (Li al. 2023b) comprises 805 questions sourced from five diverse datasets. We report both the raw win rate (WR) and the length-controlled win rate (LC) (Dubois et al. 2024). where LC mitigates verbosity bias using a regression- based causal inference approach adjust for and control potential response length bias, thereby enhancing the accuracy and robustness of automated evaluation metries. The prompt template shown in Tab. D.2]
Arena-Hard v0.1 (Li al. 2024a) builds on MT--eecc (Zheng et al. 2023b) with 500 technically challenging problem-solving queries. We measure performance via WR against the designated
bascline model GPT-4-0314.
The prompt template is shown in Tab, D.3
IFEval (hou et al. 2023) contains approximately 500 prompts based on 25 "verifiable instructions" (e.g., "write more than 400 words," "mention the keyword 'Al' at least three times"). We report both strict accuracy (Strict Acc.), which demands exact compliance down to formatting details, and loose accuracy (Loose Acc.), which allows minor variations such differences case or markup.
SedarEval (Fan et al. 2025 Xu et al. 2025a) a novel evaluation paradigm that introduces a self-adaptive rubric for scoring 1,000 questions spanning domains such as long-tail knowledge, mathematics, coding, and logical reasoning. Unlike traditional scoring methods, SedarEval's s approach provides more accurate reflection of the problem-solving process. Using evaluator LM trained on the same dataset, SedarEval achieves higher concordance with human judgments than GPT-4 (Achiam et al. 2023). offering a more faithful assessment of problem-solving quality. The template from the official repository
For math reasoning ability, we evaluate model performance on standard mathematical reasoning benchmarks, including GSM8K (Cobbe al. (02T). MATH 500 (Hendrycks et al., (02T)) Minerval
Math (Lewkowycz et al. (022), GaoKao 2023 En CollegeMath (Tang et al. 22024) 4). and Olympiad competition-levee benchmarks AIME2024 and AMC2023
Bench (He al. 2024a), well
For multimodal evaluation, we assess model performance across a diverse set of benchmarks:
Chart and document understanding: ChartQA Masryy et al. 2022). InfoVQA (Mathew al. 2022
Optical character recognition: OCRBench (Liu et al 2024g),
TextVQA (Singh et al. 2019
Hallucination detection: POPE (Li et al. 2023c)
Generaa--Knoleedee reasoning: MME (Fu et al. 2023). MMBench LLu et al. 2024f SEED
Bench (Li et al. 2023a), ScienceQA (Lu al. (022))
High-resolution and real-world utility: RealWorldQA Rea 2024
10

---

Decoding strategies. For both textual understanding and mathematical reasoning tasks, we adopt greedy decoding, selecting the highest-probabillty token at each step. All math evaluations use the same prompt template as training (see Tab. C.2). and we report single-run pass@ results for consistency. For multimodal tasks, we strictly follow the recommendation of the widely used (Chen Published as a conference paper at ICLR 2026
Table D.I: Evaluation benchmarks for textual understanding. The baseline model refers to the model being
compared against. A unified judge model GPT-4o 2024-05-13 employed across benchmarks to ensure fairness.
# Exs.
Baseline Model
Judge Model
Scoring Type
Metric
AlpacaEval eL 2023b
805
GPT-4 Turbo
Pairwise comparison
LC raw win rate Win
GPT-4o 2024-05-13
Arena-Hard vo. (Li et 20245
500
GPT-4-0314
Pairwise comparison
SedarEval |an et ct 2025
1,000
Single-answer grading
Rating of 0w Strict/Lonse Acc.
1FEval (Zhou 2025
541
Rule buse- checking
Table D.2: Prompts for AlpacaEval 2.0 (Li et al. 2023b This template lets the assistant compare two model responses for given instruction and return preaan-refeernce rank. This template is from official repository
<lim__stane stttssttbbssstemm
You are a helpful assistant that ranks models by the quality of their answers.
<lim_endl>>
<lim startiuuser
want you to create leaderboard of different large-language models. To do so, I will give you the instructions (prompts) given to the models and the responses of two models. Please rank the models based on which responses would be preferred by humans. All inputs and outputs should be Python dictionaries.
Here is the prompt:
"instruction": ""instructioo""";
Here are the outputs of the models:
"model": "model_ 1",
"answer": ""* "output_
"model": "model_2" "answer": """ [outpui_2)"
Now, please rank the models by the quality of their answers, that the model with rank has the best oulput. Then return list of the model names and ranks, iee, produce the following output: [
"modell: mmodel-namez. 'rank': <model-rank>), modell: <model-nanpel name>, 'rank": <model-rank>>
Your response must be a valid Python dictionary and should contain nothing else because we will directly execute it in Python. Please provide the ranking that the majority of humans would give.
<lim_endli endl>
al. 2024c) evaluation framework VLMEvalKit (Duan et al. 2024) and perform all the evaluations.
D.2 EVALUATION RESULTS
In this section, we provide additional details on the evaluation results to supplement the main paper.
Dynamics of mathematical reasoning performance over training. To gain a deeper understanding of the training dynamics of Uni-DPO on mathematical reasoning tasks, we periodically evaluate the model on multiple benchmarks throughout training. The overall training dynamics are shown in Fig. C while Fig. [.I the test scores of Uni-DPO, DPO, and SimPO on eight mathematical benchmarks at different training steps. From the figure, Uni-DPO exhibits two clear advantages over DPO and SimPO. First, under exactly the same training data, Uni-DPO typically achieves higher average scores and also reaches higher peak performance. Second, Uni-DPO maintains its performance more stably in the later stages of training, whereas the scores of DPO and SimPO tend to fluctuate or degrade. These observations further validate the effectiveness and robustness of Uni-DPO for mathematical reasoning tasks.
11

---

Published as a conference paper at ICLR 2026
Table D.3: Prompts for Arena-Hard benchmark. This template instructs the system generate its answer, then compare and judge two assistants' responses against that answer, outputting one of five verdict labels to indicate which assistant is preferred. This template is from the official repository
system_prompe prompt:
"Please act impartial judge and evaluate the quality of the responses provided by Al assistants to user prompt displayed below. You given assistant A's answer and assistant B's answer. Your job is to evaluate which assistant's 's answer is better. An
Begin your evaluation by generating your own answer to the prompt. You must provide your answers before judging any answers. \n \n
When evaluating the assistants' answers, compare both assistants' answers with your answer. You must identify and correct any mistakes inaccurate information. \n'n
Then consider if assistant's answers are helpful, relevant, and concise. Helpful means the answer correctly responds to the prompt or follows the instructions. Note when user prompt has any ambiguity or more than one interpretation, is more helpful and appropriate to ask for clarifications or more information from the user than providing an answer based on assumptions. Relevant means all parts of the response closely connect are appropriate to what is being asked. Concise means the response is clear and not verbose excessive. An
Then consider the crcativity and novelty of the assistant's answers when needed. Finally, identify any missing important information in the assistants' answers that would be beneficial to include when responding to the uset prompt. An
After providing your explanation, you must output only one of the following choices as your final verdict with a n \n
1. Assistant A is significantly better: [[AC>B]]\\n
2. Assistant A is slightly better: [[A>BB]]\n
3. Tie, relatively the same: [[A=B]]\n
4. Assistant B is slightly better: [[B>A]]\v
5. Assistant B is significantly better: [[>>A]>''n
Examplc output: "My final verdict is tic: [[A=B]]"."
coompt_template.
"<|User Prompt>> rqqesstin
nrmiippiinnnnnnooonn..mommm 1aaoo______a I)n),n
</The Start Assistant A's Answerl>\n//anseen 1) An</The End Assistant Answerlz
<|The Start Assistant Answerba)n (answer__] An<TThe End Assistant B's s Answerl>"
Table D.4: Main result of multimodal tasks. We evaluate the performance of Qwen2-VL.s (Wang et al.
2024d) base model various multimodal benchmarks. The best results are highlighted in bold and the second best in underline. The results demonstrate that Uni-DPO consistently outperforms the baseline, DPO, and SimPO across all benchmarks, achieving significant improvements overall performance.
ChartQA
InfoVQA
OCRBench
TexiVqa
POPE
Model
Method
Overall(%)
(6)
Final Score
Overall(%)
Overall(%)
Bascline
5.04
37.77
669
50.21
79.55
Qwen2-VL-2B
6.40
39.73
677
52.73
SimPO
11.68
41.51
635
60.63
78.71
Uni-DPO
13.84
45.28
709
73.23
86.03
MME
MMBencha..
SEEDBenchaage
ScienceQA...
RealWordQA
Model
Method
Perception Ressoning
Overall(%)
Overall(%)
Overall(%)
Overall(%)
Buseline
I 174.77
50.09
53.98
65.57
53.86
Qwen2-VL.sd VL-2B
1799,9
43.73
50.65
63.00
$4.77
SimPO
1813.2
54.38
59.57
66.81
53.99
Uni-DPO
1905.1
63.75
70.83
68.10
55.29
Results on multimodal tasks. To further demonstrate the effectiveness of Uni-DPO, we extend our evaluation to the multimodal domain, shown in Tab. T.b. Uni-DPO outperforms the baseline model
and preference optimization methods DPO (Rafailov et al. (2023) and SimPO (Meng et al.) 2024) by
significant margin in all benchmarks, showing the effectiveness of Uni-DPO multimodal tasks [0
Due to computational resource limitations, training on multimodal tasks was restricted to a 2B 2B-22rapme parameter model. Nevertheless, the encouraging results suggest that scaling to larger models is likely yield further nerformance improvements, which defer future work.
12

---

Published as a conference paper at ICLR 2026
Table D.5: Mathematical reasoning evaluation and comparison with additional methods. We compare SFT methods, alignment methods, and RL-based methods on the Qwen2.5-Matte Matthh-B model. Results with from Zhang et al. (2025a), and those with * are from Cui et (005). The remaining are from our evaluations. Our Uni-DPO consistently and significantly outperforms the baseline model, SFT-style methods, and other alignment approaches. In particular, Iterative DPO applies seven DPO iterations and thus uses roughly seven times more training data than Uni-DPO, whereas Uni-DPO uses only one iteration. Uni-DPO surpasses Iterative DPO on almost all benchmarks as well as on the average score, demonstrating its effectiveness. Moreover, compared with RL-based methods (Eurus2-PRIMe) OpenReasoner, SimpleRL, Dr. GRPO, and PPO), Uni-DPO achieves noticeably better performance while maintaining much simpler and more efficient training procedure.
Model
Methed
Methodl
GSM8K
MATII
Math
Olympiad Bunch
2824
2023
GaoKan 2823 En
Callege Math
Average
Baseline
64.3
65.8
63.4
10.7
9.9
24.3
23.4
23.3
23.3
47.5
47.5
44.7
32.3
38.85
39.11
SET
91.3
73.2
73.2
30.5
30.5
35.6
35.5
20.0
20.0
62.5
62.5
63.6
51.6
53.54
53.54
Sepervised Fine-turing
Earus2 SIT
89.5
64.0
65.1
52.2
RAFT
77.6
30.5
55.0
52.5
83.2
20.2
57.5
$1.55
Preference Alignment
DPO
29.0
SimPO Uni-DPO
85.7
57.5
53.33
35.9
78.2
34.6
41.5
26.7
67.5
65.7
31.3
56.80
EanIN--RRIME
87.7
75.4
34.6
57.3
23.3
57.5
61.3
49.7
53.35
93.0
79.2
31.6
44.0
13.3
54.2
68.8
48.7
54.10
Reisfonecmui Leariing
SiupleRtt-Zae-e
89.2
7644
27.6
40.3
26.7
57.5
65.5
50.3
54.19
Dr. GRPD (Oon-Zero-11))
89.5
79.0
33.1
42.4
31.3
200
64.9
30.4
36.58
H00-MAIH15500
33.8
31.1
33.5
O-eenI-s-ath-
GPT-4o
Table D.6: Result of textual understanding in Qwen2.5 model. WR denotes the Win Rate, LC denotes the Length-Corn Controlled rate, and Acc. denotes the Accuracy. The best results are highlighted in bold. The results show that our Uni-DPO method consistently outperforms SimPO across various model sizes and benchmarks.
AlpacaEval2
Arena-Hard
IFEval
SedarEval
Model
Model Size
Method
LC(%)
WR(%)
WR(%)
Strict Aoc.(%)
Loose Acc.
Overall (%)
SimPO
0.92
1.37
12.0
0.5B
Uni-DPO
3.37
4.10
4.4
16.6
18.9
1.5B
Uni-DPO
SimPO
0.44
6,49
7.46
0.87
12.4
2.9
23.1
16.1
26.2
17.7
Qwen2.5
3B
UniDPP
SimPO
10.81
5,81
11.32
6.02
18.5
32.5
32.3
34.8
35,3
SimPO
20.9
18.4
39.5
43.6
48.2
48.2
7B
Uni-DPO
25.8
24.7
43.5
43.3
48.1
49.9
SimPO
30.07
26.52
48.7
46.0
53.6
14B
Uni-DPO
31.99
30.75
54.5
47.1
52.5
DPC
SimiPo
Avengg Seate Atrows Math Tasks
Uni-DPO
$3.48
52.10
Epoch
Epoch
Epoch
80
120
150
200
220
Taining Stups
Figure D.I: Dynamics of performance mathematical reasoning benchmarks. We periodically evaluate Uni-DPO, DPO, and SimPO eight mathematical benchmarks throughout training.
Compare to more advanced methods. We also compare Uni-DPO with more advanced methods, as summarized in Tab. D.5 We first consider the Iterative DPO method proposed by Zhang et al. (2025a) on mathematical reasoning tasks, which adopts the same training setups and training datasets Uni-DPO. Although Iterative DPO performs seven DPO iterations, which about seven times the training data as Uni-DPO, it is still outperformed by Uni-DPO on almost all benchmarks as well in terms of the final average score, highlighting the overall effectiveness of Uni-DPO.
13

---

Published as a conference paper at ICLR 2026
Table D.7: Calibrated negative log-likelihood loss LENLA. ablation results. The full c-NLL loss delivers substantial performance gains, and each of two indicator functions makes meaningful contribution to overall improvement, which demonstrates the effectiveness of our proposed loss function.
AlpacaEval2
Arena-Hard
IFÉval
SedarEval
Model
Method
LC(%)
WR(S)
WR(%)
Strict (cc.(%)
Loose (cc.(%)
Overall (%)
SFT Uni-DPO
3.7
3.3
33.5
35.3
20.85
23.8
20.5
23.9
38.1
47.9
38.49
wfol
22.8
20.0
25.7
37.2
45.5
38.
Llama3-8B-Base
w/o II
24.7
20.9
20.2
36.8
47.9
36.82
I-11
24.3
20.9
21.9
38.8
47.1
36.82
w/o CeNTL.
22.4
19.4
23.3
40.7
47.5
37.73
SFT
28.5
28.4
25.7
Uni-DPO
47.2
44,2
37.3
61.2
67.8
46.32
Llama3-8B-Instre
42.7
38.6
67.8
45.85
w/o II
46.0
42.9
37.0
61.6
67.5
44.50
w I-II
46.4
43.3
38.0
60.4
67.5
44.80
LESLLL
46.2
43.2
37.6
58.6
65.4
44.48
In addition, compare Uni-DPO with RL-based approaches such as Eurus2-PRIME Cuii al. 2025), OpenReasoner (Hu et al. (025), SimpleRL (Zeng et al.) (2025). Dr. GRPO -eat-Zer--7B LLu al. 2025), and PPO (Schulman et al. 2017). Uni-DPO outperforms Eurus2-PRIME, OpenReasoner, and SimpleRL on nearly all benchmarks and achieves an overall score that is on par with the model trained with PPO, while enjoying a much simpler and more efficient training procedure
Model size scalability. As shown in the main paper Figs. and 6e Uni-DPO consistently outperforms SimPO (Meng et al. 2024) across range of model sizes, demonstrating its effectiveness scale. Tab. provides detailed breakdown of these results, comparing SimPO and Uni-DPO performance metrics for each model variant
Detailed statistical results. In addition to the primary metrics, Tab. D.8 reports further statistics, including standard errors (Std Error), confidence intervals (CT), average response lengths (Avg. #tokens), to illustrate the distribution and the reliability of the evaluation results.
Radar plot details. To better visualize Uni-DPO's performance across multiple domains, we generate four radar charts comparing our method against strong baselines highlight key differences:
Textual understanding (8B, the main paper Fig. La Results for Slama3-BB-Barl (AI@ Metal 2024b). SimPO and Uni-DPO results correspond to the v0.2 setup; see the main paper Tab. for exact numbers.
Mathematical reasoning (1.5B, the main paper Fig. lb]: Performance of 10wen2.5-Math-1155 (Yng et al. 20245) base model after finetuning. Detailed scores are in the main paper Tab. 2
Mathematical reasoning (7B, the appendix Fig. Results for Qwen2.5-Mathe
Yang et al.
2024b) base model; Tab. [D.5 for the underlying data.
Multimodal tasks (2B, the appendix Fig. .7b): Evaluation of Qwen2-VL-22 (Wang et al. 2024d
base model on our multimodal benchmark suite. Exact metrics appear Tab. D.4
D.3 FURTHER ABLATION STUDY
which evaluate the individual contributions of each component in the Uni-DPO framework, further conducted studies focusing on the role and strength of the calibrated negative log-likelihood loss (Lo-NLL). This loss specifically targets challenging yet high-quality positive samples. We also examined the effectiveness of different selections of the expert model, which demonstrates that the Uni-DPO's performance gains stem from its ability dynamically allocate focus across training examples.
In addition to the ablations presented in the main paper Sec. 4.3
"l Details of the Iterative DPO method and its implementation be found in the official blog The SFT. RAFT, DPO, Iterative DPO, PPO-MATH7500, and PPO models reported Tab. [D55 all from the Iterative DPO work, while the Fur and Furus2-PRIME: models are taken from Cui et al. (2025).
12Dee the high costs associated with the extensive GPT-4o (Hurst et al. 2024 API calls required by the report only partial results this benchmark for reference.
SedarEval benchmark (Fan 2025
14

---

Published as a conference paper at ICLR 2026
Ablation of LO-NLL components. As defined in Eq. [I]. c-NLL loss includes two indicator functions. To assess the contribution of each component, we conduct ablation experiments by removing it individually. The results, presented in Tab. D.7 provide further insights into the effectiveness of different parts of the loss design.
Without the first indicator (VV | x)) > log (y =)) (w/o I): This corresponds to not restricting samples where To performs worse than Tee. The performance drops slightly. indicating that this term somewhat important. However, this term may not be optimally configured, potentially requiring margin parameter Tin to achieve the best results. In other words, this indicator
term may ideally be formulated as: 1
log (ar()ull [o
We leave this for future exploration.
Without the second indicator 1 (S 2 Tgood) (w/o II): This means we apply NLL loss compensation positive samples of all quality. The performance drops significantly, showing the importance of controlling the quality of positive samples when performing NLL loss, co..compensttion should only be applied to nigh-quality ones.
Without both indicators (w/o I-II): In this variant, the NLL loss is applied uniformly to all positive samples you without any filtering. Performance degrades substantially, confirming the necessity of selectively augmenting the hard yet high-quality positive examples. Nevertheless, this configuration still slightly outperforms the complete removal of the NLL loss ("w/o LeNNI." " entry in the table), indicating that the NLL component contributes positively to overall model performance.
Ablation of LENLL strength X. We further evaluate the effect of the strength À on the Le-NLL term across different domains. While NL. is necessary for boosting model capabilities (as discussed above), its optimal strength may vary by task. We sweep À over wide range of values:
f0.0001, 0.0005, 0.001, 0.005, 0.01, 0.05, 0.1, 0.2, 0.3, 0.5, 1},
and report the final settings in Tabs. and C.3 For mathematical reasoning tasks, a larger weight (better at À 0.2) is required to achieve peak performance. In contrast, for text understanding and multimodal tasks, much smaller weight (e.g..) 0.001) suffices. These observations are consistent with the settings of prior work (Pang et al. 2024). We hope these findings will inspire further research into the optimal strength of Lo-NLL-
Ablation of expert model. In our main experiments, we strictly follow related works (such as SimPO) in using the same publicly available dataset, UltraFeedbcck, with preference samples and quality scores given by the dataset our training signal. Compared with SimPO, Uni-DPO exploits these resources at a finer granularity, enabling adaptive corrections during preference learning. As (main paper) and Tab. D.4 (Appendix), this design consistently improves performance across modalities and benchmarks, demonstrating its effectiveness.
shown in Tabs. D and 2
To further verify that the improvement does not stem from the use of strong expert model, but rather from Uni-DPO's ' fine- fine-graing grained utilization of training data, we conduct ablation studies using open-source models to provide the quality scores. Specifically, as shown in Tab. D.9 we replace GPT-4o with Qwen2.5-72B and ArmoRM-7.55B as weaker expert models for data annotation. Except for replacing the expert model, all other settings (e.g., the scoring prompts) are kept unchanged.
On the left, we apply the Uni-DPO framework using data quality scores derived from a weaker and more accessible open-source model, Qwen2.5-72B. The results indicate that even with weaker expert model, Uni-DPO still achieves competitive performance. On the right, we use the 7.5B ArmoRM model, which is weaker and more accessible than GPT-4o, for scoring. The results also outperformed SimPO, demonstrating the stability and scalability of Uni-DPO. These results highlight that the effectiveness of Uni- Uni does not depend on the availability of particularly strong expert model, but instead arises from its ability to dynamically assign focus to each training example.
D.4 DOWNSTREAM TASK EVALUATION
To investigate how preference optimization methods impact downstream task performance, we evalu- ate models on the benchmarks included in the Huggingface Open LLM Leaderboard (Beeching et al. 2023). The SFT, DPO, and SimPO checkpoints used for comparison are provided by SimPO (Meng et al. 2024h, which is consistent with those in the main paper. Specifically, we report results on MML (Hendrycks et al. 2020), ARC (Clark et al. 2018 HellaSwag (Zellers et al. 2019). Truth-
15

---

Published as a conference paper at ICLR 2026
Table D.8: Detailed statisties for textual understanding experiments. This table presents standard errors, 95%
confidence intervals (CI), and average response token lengths for each model variant across the benchmarks.
AlpacaEval2
Arena-Hard
Model
Method
LC(%)
WR(S)
Std Error
Avg. #tokens
WR(S)
95% CI
Avg. #tokens
SFT
5.7
3.7
0.66
1000
3.3
-0G 0.8)
389
DPO
13.3
1.20
1708
1.6,
542
Llama3-88
SimPO
19.4
18.1
1.36
1883
23.4
-199.1.7)
704
Base
Uni-DPO
20.5
1.43
1757
23.9
1.7.1.6)
SimPO,aa
15.5
18.5
1.37
2349
29.3
2.2.2.4)
751
Uni-DPO,a:
22.8
24.1
1.51
2139
30.7
1.9.
668
SFT
28.5
28.4
1.59
1955
25.7
1.8)
585
DPO
43.7
41.7
1.74
1890
32.6
2.3.222)
528
Llama3-8B
SimPO
44.7
40.4
1.73
1779
33.8
2.8.
504
Instruct
Uni-DPO
47.2
44.2
1.75
1862
37.3
2.1
524
SimPO,a1
41.2
37.1
1,70
1787
36.5
2.0.
530
Uni-DPO,e.1
42.6
1.74
1811
2.5.2.1)
Gemma-2-9B-IT
Uni-DPO
SimPO
54.7
53.2
52.7
1.76
1.76
1759
1968
59.1
67.1
2.4,2.2)
222,,
693
751
Qwen2.5-005B
SimPO
0.92
1.37
0.41
4585
1.3
0.44.0.)
1089
Uni-DPO
3.37
4.10
0.70
2763
4.4
0.9,0.8)
678
Qwen2.5-1.5B
SimPO
0.44
0.87
0.33
6626
2,9
-..5
2199
Uni-DPO
6.49
7.46
0.93
3591
12.4
1.6,
938
Qwen2.5-3B
SimPO
5.81
6.02
0.84
2556
-13,,3
1816
Uni-DPO
10.81
11,32
1.12
2551
18.5
1.7.
1050
Qwen2,5-78
SimPO
20.9
18.4
1.37
1809
39.5
1.6.2.9)
585
Uni-DPO
25.8
24.7
1.52
2005
43.5
224,2222
567
Qwen2.5-14B
SimPO
30.07
26.52
1,56
1629
48.7
2277 1.9)
471
Uni-DPO
31.99
30.75
1.63
2126
54.5
221,
Table D.9: Ablation study on the expert model used for scoring. Uni-DPO maintains competitive or superior performance even when using weaker and more accessible open source models such as Qwen2.5-72B and ArmoRM-7.5B, for scoring the preference training data.
Llama3-S0 -BB-Basee
Lhama3-88-Insreee
Method
Arvna-Hard
IFEval
Method
Arvna-Hard
E'Eal
WRS
Striet Acc.(Sk)
Ace.(%)
Striee Acc.(Sk)
Acc.(%)
SFT DPO (GPT-4)
33.5
35.3
SFT
25.7
15.9
40.5
45.5
DPO
32.6
SimPO (GPT-4)
23.4
40.5
45.7
SimPOvaz
36.5
60.8
68.6
Uni-DPO (GPT-4)
Uni-DPO (GPT-4o)
73.2
Uni-DPO (Qwen)
24.2
30.2
46.0
Uni-DPOsas (AmoRM)
37.5
66.2
73.8
fulQA [in et al. 2022), Winogrande (Levesque et al., 2012) and CommonsenseQA (Tamorr et al. We strictly follow the standard evaluation protocols, and show the results in Tab. D.10
2019
Substantial gains in knowledge-intensive understanding. Compared with the SFT checkpoint, DPO, SimPO, and Uni-DPO all yield improvements on MMLU, For the Llama3-8B family, Uni-DPO achieves gains that are comparable to those of DPO and SimPO. contrast, for Gemma-2-9B- and Qwen2.5-7B, Uni-DPO delivers noticeably larger improvements, suggesting that our method can be particularly beneficial for knowledge-intent -iiteeevieeeiiveeeeeeee tasks.
Reading comprehension and commonsense reasoning generally improve. On ARC, Winogrande, and CommonsenseeA, preference optimization methods consistently improve performance over the SFT checkpoint. We hypothesize that preference training helps the model better exploit context and strengthens its reading comprehension and commonsense reasoning abilities. An exception is HellaSwag, where most preference optimization methods lead to a moderate performance drop. Nevertheless, Uni-DPO exhibits the smallest degradation among DPO, SimPO, and Uni-DPO, thus best preserving the original capability on this benchmark.
Truthfulness consistently benefits from preference optimization. Interestingly, we observe that preference optimization methods consistently improve TruthfulQA performance relative to the SFT checkpoint, with gains exceeding 16% in some settings. In particular, Uni-DPO achieves substantial improvements over the SFT baseline and outperforms both DPO and SimPO on TruthfulQA. We conjecture that the preference data contain non-trivial amount of supervision that implicitly empha-
16

---

Published as a conference paper at ICLR 2026
Table D.10: Downstream task evaluation. We strictly follow the standard evaluation protocols provided by Language Model Evaluation Harness et al. 2024 for each The best and second-best results are highlighted in bold and underline, respectively.
Model
Method
MMLU
ARC
HellaSwag
TruthfalQA
Winogrande
CommonseenseeA
Average
SFT
77.44
75.62
46.66
70.72
66.24
Llama3-88
Base
SimPO
Uni-DPO
DPQ
60.41
61.95
61.83
81.65
80.81
74.79
72,64
71.98
67.00
59.23
63.48
65.82
60.85
57.74
70.35
67.24
63.99
69.18
Uni-DPO (Qwen)
79.08
64.57
Baseline
58.08
77.10
64.80
52.81
64.17
45.70
60.44
DPO
62,95
68.86
48.17
57.72
57.30
73.22
61.37
Lluma3-8B
SimPO
61.91
74.16
42.13
62.24
57.38
75.84
62.23
Instruct
SimPOxa2
63.77
79.59
52.19
65.33
61.33
76.17
66.39
Uni-DPO
63.19
76.35
47.95
61.84
58.96
75.18
63.9]
Uni-DPO.a:: (GPT-4o)
DPO-e2 (AmoRRM
63.86
79.84
60.60
64.51
65.19
74.77
68.13
Baseline
33,46
79.12
67.11
61.41
70.40
37.59
58.18
Gemmo-2.9B.
DPQ
36.84
61.15
57.17
44,48
67.51
Uni-DPO
63.82
79.04
61.59
62.05
69.22
59.46
66.19
Qwen2.5-77
Uni-DPO
Baseline
53,16
65.87
75.02
51.65
67.55
69.22
78.05
65.49
sizes factuality and honesty, which encourages the model to produce more truthful responses. The construction process of open-source preference datasets such as UltraFeedback likely incorporates considerations of response truthfulness, further contributing to the observed gains on TruthfulQA.
Overall, Uni-DPO not only outperforms DPO and SimPO on textual understanding benchmarks but also delivers great improvements on a broad range of downstream tasks, with especially pronounced gains knowledge-intent -edgee-nteeniive understanding and truthfulness. We believe that a more systematic study of how different preference optimization strategies affect downstream performance would be valuable and call for rigorous, large-scale analyses future work.
E DISCUSSIONS
E. REVISITING UNI-DPO THROUGH THE RL PARADIGM
Unified RL paradigm. While the main paper (Sec. outlines the RL perspective underlying Uni-DPO, here we provide a more detailed examination of its foundational principles. The Unified Paradigm provided by (hao et al. (2024b) shows that different training methods, such as SFT, DPO Rafailov et al. 2023). PPO (Schulman al., 2017), and GRPO (Shao et al., (024b) can be conceptualized as either direct or simplified RI. methods. The paradigm can be expressed as:
VoL. (((
CC Vo log (oula,
(30)
Grudient Coeflicient
There exist three key components: a) Data source D, which specifies the set of training examples used learn preferences: b) Reward function Ti. which produces the scalar reward signals that guide model updates; c) Algorithm AA, which consumes both the training data and the reward outputs compute gradient coefficient GCA,, thereby modulating the magnitude of the reinforcement or penalty applied to each example.
We begin by presenting the gradient coefficient of DPO as derived by Shao et al. (2024b), which serves as the foundation for our subsequent analysis. For notational convenience, we denote the positive sample yo in each preference pair by y and the negative sample yI by y. The objective of DPO is:
Gradient coefficient of DPO.
(0) =B(r ~ (an(Q), (y" )
log (8
1yt1 maly (1, mar v2e)
1 1v
Tolye HCe)
(31)
log
log
Tur U2e)
The gradient of (0) is:
17

---

Published as a conference paper at ICLR 2026
V (0) Blz ParlQ), ~
(32)
GCoro v, Voloog To(ue lx, met))
( 1) Volog to lr, vee)
Data Source:
question in the SFT dataset with outputs sampled from the SFT model.
Reward Function: human preference in the general domain.
Gradient Coefficient:
GCreo v, = (olo [aVe [z, Mere)
To(ue" (x,, wtee [eer (v, (2, v
log
(33)
(ye [z Mee)
Building on this formulation, we can compare the gradient behavior of DPO and Uni-DPO.
Gradient coefficient of Uni-DPO. The objective of Uni-DPO is:
Dusi E[z Pa(Q)) (v' [~))(((a))
3 log (M. Ia, wice
Mestl
1
lv
(M (m.
Feel
(34)
(o lr. vZ.) mna v2e)
IM
mo la, MZ2)
A.1 1u
log ( 1z wixi }.
log
log
log
19
anr v2,)
1y+1
The gradient of (Um) is:
Vo (elomeno(0==Ble-- (a) (vt wtt"" )
tu (0n u. mat) Voloo mo Ja, wtx)
GCy Umi (it Mi m.) Volog mo (1l (e.)))))
Data Source:
questions are drawn from the SFT dataset, with outputs sampled from the SFT model. Reward Function: integrating human preference data from the general domain with predictions from reward model.
Gradient Coefficient:
GO (I,, y, L, ma) = 3 Wigual (1 (Ans))) ((-))))7((11)))))) - (o (ow) + A-I,
(36)
GC (x, y, t, mi) = B Wigul - (Aut)) (1 - (Amt) log (ogo(A))).
where is the indicator function 1 (log (Be x) >> log (B the score of the preferred completion Ywu, the A, the reward margin in Eq. (20) and Andj is adjusted reward margins in Eq. 21
x) x)) (S.. 2 Tgood) and S. is
Discussions on the gradient coefficient. Here compare the gradient coefficients in Eqs. 33 and (36) and reveal several key advantages of our Uni-DPO over the standard DPO.
snfficulty-agnostte weighting in DPO. The gradient coefficient GCA in DPO depends solely on the implicit reward margin A,. As result, it cannot fully account for the varying difficulty of the on- on-policy samples. Even when the reference model wraf exhibits a sufficiently large log-probi pog-bobbyiylity gap. DPO will continue apply strong updates if A, not large enough, potentially risking overfitting. In contrast, Uni-DPO addresses this limitation by introducing a performance-weighhigg term uper into the objective and the focal term (1 o-))) into the gradient coefficient GC. This modification automatically downweights well-fitted examples and shifts the emphasis to more challenging cases.
Lack of quality-adaptive scaling in DPO. The Gradient Coefficient GCA in DPO is largely determined by the policy margin between positive and negative samples. The model's ability to distinguish preferences inherently limited by its policy margin between positive and negative samples. Consequently, DPO struggles effectively differentiate between high-quality (large
18

---

Published as a conference paper at ICLR 2026
score margin in this work) and low-quality (small score margin) preference pairs, contradicting the principle that qual-quality examples should exert a greater influence on learning, while low cow-quality pairs should induce more conservative policy updates. Uni-DPO resolves this by introducing an external quality judge Wqqual via a scaling function over expert-assigned margins (S.. Si), which adaptively adjusts each pair's s weight. This mechanism enables the model modulate the learning signal based on sample quality, analogous to the use of proxy reward signals in PPO and GRPO, thus ensuring that higher quality examples are more likely to receive greater emphasis during policy optimization.
Uniform sample-level gradient magnitude. In DPO, positive and negative samples in pair share the same update strength Lie. = (Po)) which overlooks the fact that different examples may need different levels of adjustment. PPO and GRPO solve this by assigning each sample its coefficient GCAA through advantage estimation. In Uni-DPO, we take similar approach by adding calibrated negative ligllike loss L. NLI. applied selectively to high-quality, difficult positive samples. Thus, the incorporation of the c-NLL loss naturally enables the model to extract more information from high-quality positive samples and thereby enhance overall learning performance. Although this mechanism does not achieve the sample-level granularity of GRPO, it allows positive and negative samples to be assigned different weights based on their differing qualities, enabling the model to effectively leverage information from high-quality positive samples and thereby enhance overall learning performance.
E.2 IMPACT OF LENGTH NORMALIZATION ON LLAMA3-8B-BAL
we observe that, without length normalization, Llama3-8B-Bas exhibits minimal post-training gains on AlpacaEval2 and Arena-Hard compared its instruct counterpart. In contrast, applying length normalization substantially reduces reward oscillations during training, resulting in significant performance improvements that surpass those achieved by DPO and SimPO. Specifically, we attribute this effect to two factors:
As demonstrated in Sec. 4.3
Weak initial preference aligmment in the base model. We observe that during training, Llama3- 8B-Bast assigns lower and more unstable probabilities to preferred samples (yw,) compared the instruct-tune models. This suggests that it may not be well-calibrated to human preferences-an unsurprising result for a purely pre-trained model that was not trained with human preference data. As a consequence, the reward signal it receives is both weaker and noisier and may make it harder for the model to learn reliable preference signals during training.
Off-policy data training, We strictly follow the experiment setups of SimPO, where the base model learns from off-policy samples (publicly available dataset UltraFeedback,, whereas instruct- tuned models learn on-policy. We hypothesize that off-policy training makes learning more cefficult-especian for a base model with weak initial preference alignment. Thus, without length normalization (LN), the model may struggle to learn true preference signals during training, leading higher reward variance and weaker overall performance.
SimPO shows (in Sec. 4.2) that, without length normalization, models exploit length bias producinn ever-longer, but not necessarily higher-quality, outputs-whereas wwereas with length normalization, this bias is suppressed and models focus on true preference signals. This finding is consistent with our observations.
E.3 FURTHER DISCUSSIONS ON C-NLL LOSS
In this section, we provide further insights into the design and effectiveness of the calibrated negative log- likelihood loss (Le-NLL) introduced in Sec. 3.4] We show that incorporating Le-NLL does not lead to overfitting and also compare its behavior with the standard SFT loss as used in SimPO.
Combining c-NLL won't cause overfit. The c-NLL loss is a corrective measure for the preference learning algorithm DPO's inherent tendency to shrink generation probabilities. It will not raise concerns about overfitting. Empirically, we strictly follow recent work, such as SimPO, to train the model exclusively on UltraFeedback (2023) and evaluate it on a broad suite of benchmarks, including SedarEval (2025) and -ard (2024), both released after the training data. Uni- DPO consistently outperforms previous methods on these unseen benchmarks, demonstrating robust generalization. Besides, we have evaluated Uni-DPO on both mathematical reasoning Tab. and
19

---

Published as a conference paper at ICLR 2026
multimodal tasks Tab. D.4 which demonstrate that Uni-DPO extends effectively to wide range of domains and modalities. Furthermore, our ablation study in Tab. shows that adding the c-NLL term improves performance across almost all benchmarks, confirming that c-NLL addresses limitation of DPO without causing overfitting.
Comparison of c-NLL and SFT loss introduced SimPO. SimPO reports their Table 14 (Appendix G) that adding generic SFT loss degrades performance on benchmark AlpacaEval2, whereas in our framework, incorporating the well-designed calibrated NLL (c-NLL) term improves results across modalities and benchmarks. We attribute this difference two key factors:
Addressing the DPO's 's inherent limitation. We observed that, after standard DPO training, models tend to decrease the absolute generation probabilities of both positive (ym)) and negative samples (wi), which can hinder their ability to learn and internalize human preferences. The proposed c-NLL can effectively address this issue by directly encouraging the model to generate human- human-preferred outputs. Our ablation study Tab. 3 clearly shows that adding c-NLL boosts model performance, confirming that it remedies the deficiency of vanilla DPO.
Being selective of high-quality, hard positive samples. Unlike uniform SFT loss used in SimPO, our well-designed c-NLL applied only those positive examples that are both high-quality and difficult for the model during training. As detailed in the ablation study (Appendix Table D.7), this selective strategy enables the model to extract more information from the most informative preference samples, thereby enhancing overall learning performance without over-emphasizing easier or lower-quality training data.
We note that the impact of SFT loss in preference optimization may vary depending on the training setup and evaluation task. Our results demonstrate that the proposed calibrated NLL (c-NLL) objective can better complement DPO.
E.4 RATIONALE FOR RELAXING THE ORIGINAL FOCAL WEIGHT
We introduce focal weighting adaptively emphasize under-fitted training samples based on the policy model's 's learning dynamics. However, the original focal factor imposes rigid, per-sample constraint that can destabilize preference learning. We provide more detailed explanation below.
Limitations of the original focal weight. As defined in the first line of Eq. ( the focal weight dynamically increases when the policy model my underperforms on a poorly-perforned sample (i.e. when its generation probability for the positive sample Yu lower than the reference model's, ss or its generation probability for the negative sample yı is higher than the reference model's ss), encouraging the model to focus on such under-fitted examples.
However, when rewritten as the second line of Eq. (7) it becomes clear that this formulation may over- amplify the weight for samples on which the reference model already exhibits extreme log- bog-prbbability differences. For such samples, regardless of how the policy model performs relative to the reference model (i.e., whether these examples are poorly-performed for the model or not), this weighting will further widen the policy model's probability gap between positive and negative samples, potentially leading to overfitting on those cases, which may cause unstable training dynamics.
Relaxed formulation with performance weighting. To address this issue, we replace the reference- dependent margin with a fixed threshold, yielding Wpert in Eq. (IO). This modification regulates the margin subtracted from the policy's log-probability disparity, preventing extreme reference scores from dominating the update. As shown in Fig. 55, this relaxation improves training stability.
Preserving hard-easy balancing. Although the margin is fixed via Tief, the weight Wperf still adapts to the policy model's per-sample log-probability margin between yoe and yr. Moreover, the multiplier (1 (1- (1-o(-))" is retained (shown in Eqs. [18) and (26)). ensuring that the relaxed formulation preserves the focal mechanism's 's ability target difficult examples. The effect is thus smoother but not weaker. maintaining adaptive balancing between well-fitted and under-fitted cases.
Effectiveness of performance weighting. Performance weighting (Wper) yields consistent gains across tasks. It stabilizes optimization (Fig. 5b), mitigates overfitting (Figs. 5c and 5d). and improves
20

---

Published as a conference paper at ICLR 2026
both overall comparisons (Tab. 1 and ablation results (Tab. 3). demonstrating its necessity and effectiveness in preference learning. The unified RL analysis in Sec. E.1 further demonstrates that the performance weighting remains a theoretically sound addition preference learning.
F RELATED WORK
F.1 LARGE LANGUAGE MODELS
Recent rapid progress in large language models (LLMs) [ouvrrn et al.,
2023
Achiam et al. 2023
A Metal 20246
Hurst al., 2024 Yang al. 2024a
AI@ Meta
2024a Yang et al. 2025a
Brown et al.J 2020 Team et al. 2024a Anthropic
2023b 22trr
2024 Liu et al. 2024a Guo ct al. 2025
Li al. 2025) has accelerated the path toward artificial general intelligence (AGI). In particular,
combining extensive self- self-super pre-training on massive text corpora with subsequent high-quality supervised tieettnnng (SFT) has endowed LLMs with surprising emergent properties in real-world understanding. Building these milestones, current efforts focus refining the alignment between model outputs and human preferences, thereby extending the utility LLMs across a wider array of real-world applications (Ouyang al. 2022
In recent years, vision-langueel language models (VLMs) have achieved remarkable progress (Radford et al.
2021 (Shoo /0aoo et al., 2024a Wang et al., 2025; Tian et al. 2019 Liu al. 2024d Yang et al. [024d
Team et al. 2025). ). With the advancement of LLMs, multimodal large language models MM..MMs)
have achieved remarkable alignment of visual and textual representations through cross-modal feature
inlegration, marking a crucial milestone toward truly general- purpose Al systems (Wang et al. 2024d
Bai et al. 2023
2025
Liu et al. 2023 2024b Dai al. 2023 OpenAl| 2023 Zhu
2023
Qu et al. 2025 Yang et al. [023b; Zhong et al. 2024] Yang et al. 2023a 2024c Lai et al. 2024a
Peng et al. 2025 However, the challenge of building models that are useful, reliable, and harmless
remains central concern. In this paper, we propose a new algorithm, Uni-DPO, to advance the reliability of MI.LMs.
F.2 REINFORCEMENT LEARNING FROM HUMAN FEEDBACK
RLHF is powerful paradigm ga to align LLMs with human judgments and values (Christiano et al.
2017 Ziegler et al., 2019 Kaufmann et al. 2023 Stiennon al. 2020 Touvron et al.) 2023 Wang
et al.
2024a| Ouyang et al. [022 Bai et al. 2022). In the classical RLHF workflow, reward model
is first trained on human preference data (Gao et al. 2023b Luo et al. 2023 Chen et al.) 2024b Lightman et al. 2023 Havrilla et al.) 2024b Lambert et al. 2024). and this learned reward is then used
to optimize the policy, typically via reinforcement learning algorithms such as PPO (Schulman et al.
2017: Ouyang al. [022] Ramamurthy et al. [202 Anthony al. (017). RLHF has demonstrated
its utility in wide range of settings, from toxicity mitigation (Amini et al.
2024 Korbak et al. 2023 2024) to enhancing helpfulness ('Tian et al.
Zheng et al. (023aa] Li et al. 2024b Chaudhary al.
2024 Wang et al. [024bb an et af. (025). reduce hallucination phenomena
(Peng et al. 2025 ZZhoo
al.
2023 Xiao et 2024 Zhou et 2024b Yu al. 2024 He et al.
2024b), and bolstering
reasoning skills (Zhang et al. 2025a Pang et al. 2024 Havrilla et al. 2024a
Lai cl al.
2024b Yang
al.
2025b).
F.3 DIRECT PREFERENCE OPTIMIZATION
DPO. Although RLHF has been successfully applied across many domains, online preference optimization algorithms are often complex, hard to optimize efficiently, and may compromise the stability and reliability of the learned reward model (Casper et al.) 2023 Gao et al. 2023bb Chaudhari al. 2024 Zheng et al., 2023d, Santacroce et al. 2023). This has motivated the development of simpler offline alternatives such as Direct Preference Optimization (DPO) (Rafailov et al.) (023). DPO dispenses with an explicit reward model and instead learns a policy directly from the annotated preference data. However, without an explicit reward model, DPO cannot fully utilized preference pairs by the improved policy after several steps of training (Pan et al. 2025; Gorbatovski et al. 2024; et al. 2004e, Gao et al.) 2023a Kong et al.] (024)). To address this limitation, several works adopt an iterative scheme in which the reference policy is periodically replaced with the latest policy parameters or fresh preference pairs sampled at each iteration (Dong al. 2024
21

---

Published as a conference paper at ICLR 2026
Table F.1: Comparison of Uni-DPO with 3-DPO and DOPO Reported results for B-DPO and DPPO are taken from their original papers. Uni-DPO consistently achieves superior performance across benchmarks, demonstrating its effectiveness.
AlpacaEval2
Arena-Hard
IFEval
SedarEval
Model
Method
LC(%)
WR(%)
WR(%)
Strict Acc.(%
Loose Acc.(%)
Overall(%)
SFT
28.5
28.4
25.7
59.4
62.2
40.93
DPO
43.7
41.7
32.6
Llama3-8B SimPO
44.7
40.4
33.8
60.8
68.6
44.81
Instruct
3-DPO
43.4
38.2
DºPO
43.0
43.5
65.6
Uni-DPO
47.2
44.2
37.3
66.9
73.2
46.50
Kim et al. 2024, Rosset et al. 2024| Xiong et al. 2024, Yuan et al. 2024). In contrast, Uni-DPO
incorporates a -aul-tdee weighting mechanism that dynamically accounts for both the model's learning dynamics and the effectiveness of training data. Consequently, it runs entirely in a single pass without any iterative retraining and achieves superior performance (see Sec. D.2
Limitations of standard DPO. DPO has emerged as a cornerstone technique within RLHF due to its simplicity and efficiency. However, several limitations of DPO have been identified. For example, DPO can reduce the probabilities of preferred samples you (Feng et al. 2024 Rafailov et al., 2024), leading LLMs struggle with generating human- human-prefeeee preferred outputs. To mitigate this issue, Pal et al. (2024); | Pang et al. 2024) propose incorporating a negative
ligllikel loss to balance learning. Building on this idea, Uni-DPO applies a calibrated NILL term selectively to high-quality and challenging positive samples, enabling more focused updates and yielding superior performance.
Length bias and normalization. Another line of research highlights that DPO could induce a length
bias, where the policy model tends to favor longer sequences (Meng et al.) 2024 Park et al.) 2024 Lu
et al. 2024 Dubois et al., 2024; Singhal et al. 2023 Wang et al. 2023). Length normalization (LN) has been shown alleviate this problem. Uni-DPO integrates L.N into the training objective, and our ablation experiments confirm its effectiveness in improving training stability.
Adaptive weighting strategies. Beyond these refinements, other DPO variants explore different weighting strategies. For instance, 3-DPO (Wu et al., (0024b introduces a dynamic weighting factor to rebalance training samples by applying idea further, Uni-DPO unifies the off-policy aspect (intrinsic data quality) and the on-policy aspect (model learning dynamics) within a single adaptive weighting scheme. Similarly, D2PO (Shao et al.) 2025) argues that uniformly weighting all tokens underestimates the fact that earlier tokens contribute more to sequence-level preference optimization. Uni-DPO generalizes this principle to the sequence level, allowing each example's 's weight to adapt jointly to its judge-assigned quality and its difficulty for the current policy. The comparison results in Tab. F.T demonstraat that Uni-DPO consistently outperforms these variants across different models and benchmarks.
both sample filtering and loss scaling. Extending this
Besides, several recent works propose improvements to DPO on sample-wise weighting. RDO (Wang et al. 2024e) observes that current offline RLHF mainly captures the ordinal relationship between responses, while overlooking how much one response is preferred over another. Thus, RDO directly predicts a differential score between two responses and uses this margin to more precisely guide the alignment. WPO (Zhou et al. (024a) reweights preference pairs according to their likelihood under the policy, so that off-policy data are adjusted better match the on-policy distribution. MWPO (Xu et al. (0255) proposes DPO-
multi-weighting scheme that jointly considers implicit reward margins and response length margins. These two factors are geometrically combined into synthetic weights, which are then used to rescale preference pairs for optimization. RPO (Yin et al.) 2024) introduces contrastive weighting mechanism that allows the model to train on broader range of preference data, including both paired and unpaired examples.
Comparison with SimPO. Among recent variants, SimPO MMeg et al. 2024) stands out for elimi- nating the reference model, thereby achieving improved training efficiency and strong performance. Our Uni-DPO combines fine- greingr data utilization, length normalization, and unified weighting into a coherent framework, leading further gains across diverse tasks.
22

---

Published as a conference paper at ICLR 2026
G CASE STUDIES
This section presents two types of studies illustrating the effectiveness of Uni-DPO: one textual understanding, which demonstrates the model's ability to generate coherent, foolooiion-following responses, as illustrated in Figs. G. to G.4, and one in mathematical reasoning tasks, which showcases its proficiency in solving complex problems, as shown Figs. G.5 to G.7
H BROADER IMPACT
This work contributes to the advancement preference learning in large language models. While the proposed methods may have broader societal implications, we believe the primary impact lies in improving the alignment between model outputs and human preferences. We have carefully considered potential concerns and have not identified specific issues that require immediate attention beyond standard considerations for language model deployment.
I LLM USAGE STATEMENT
In this work, large language models (LLMs) were used as general- cenpral-purpose writing assistants. Specifi- cally, we employed LLMs to improve the grammar and readability of the manuscript through language polishing and minor rephrasing.
23

---

Published as a conference paper at ICLR 2026
Figure G.I: An example Arena-Hard (Li et al. 2024a) response generated by the Qwen2.5-7B (Yang
2024a) model after preference learning. The outputs are compared against those the SimPO-trained model and the proprietary GPT-4 model.
Input
Write function in Scheme that reverses 1ist of strings?
SimPO
write funstioo in Scheme reverse 1sst strings recuision. flere' an example:
schame (dafine (reverse-1ist 1st]
(if (mull? 1st)
(I) (append (reverse-list (odr 1st)) (1ist 1st111))
This function takes
list 'lat' input and recursively appenda the sar (firat element) the 1ist to reversed rest of the 1ist. When the 1185 is empty, it returns empty 1ist.
can
Function this:
seschame
(raverse-1ist (hello world)) Curput: worll hello)
Uni-DPO llere's s Scheme function that zeverses lis: of strings:
schhame (datine (reverse-list 1st]
(if (mull? 1st) ()
(append
1ist 1st)) (List (oar 1e())1))
However, this function uses 'append', which is not efficient for reversing lists, creates new liats and copies elements. à more efficient approach
would be to recursion with accumilator:
schhame
(define (reverse-list 1st)
(define (reverse-helper acc)
(it Inu.1? 1st)
acc
(reverse-helper (edr 1st) (cons (car 1st) acc)
(rovarse-helper lst (]))
This function uses inner halper eriginal 1ist and accumulator 1iat. It recursively processes the input list, adding each element to the front of the accumulator list. Finelly, it returns the reversed list when the ingut list is empty.
"raverse-helpe"
takes the
Here's 5 function schame that reverses
strings:
"echene (define (zeverse-list lst)
(if (mul1? 1st)
'(1 (append (reverse-list (adr 1st)) (list (car 1st))
(deSine (reverse trevere--atring-le string string-List)
(zeverse- list string- list))
can 'reverae-atring-11e function zeverse strings. The 'reverse-list function is general function that reversen any list, and Trevereestrinng1lee
specific tunction for reversing 11st of strings.
Example usage:
***scheme
string 1ist ("hello* "world" Dutputt (*scheme* "vorld" thello")
24

---

Published as a conference paper at ICLR 2026
Figure G.2: An example Arena-Hard response generated by the Qwen2.5-7B model after preference learning. The outputs are compared against those of the SimPO-trained model and the proprictary GPT-4 model.
Input
Q: Write python function to find the nth occurrence of specific character in string.
Python function Find the character in string using the following approach:
specific
""pythen
fand_atthooccureene (string, char, nl:
count =
range (itrinn))
if string(1] =s char: count if count =s return return -1 Return -1 if the character doesn't oceur times
Example string "Hello World!" chor *o"
n
result find find_nth_oocurreeee oocurranee (string, print (1*The (ajth occurrence of [char]' 18 at index (result]*)
This function iterates through string, counting eccurrences specified character. When it reaches the ath occuzrence, it returns the index. the choracter doas not ocaur times, returns -1.
Uni-DPO
Here's a Python function that finds the ath occusrence of specific character in string:
.. "pythos
def find_ trn__nth_occureene (string, char, nl:
for i, in (sumerate(stzing)
count +-
return
return
not n
Example usage: string "Hello, Korld!"
char "6"
index Eind_ath_occurzznee (striing char, if index != -1:
print The (n)th else:
of (charr) is index (index).")
The character (charr' does not
times in the string.")
This function 1terates through the string, counting the ocoucrences of the specified charactes. If the character is not found a times, it returns -1.
occurcence found,
GRT-4
llere's Python function that finds the ath occurrence of specific character in string:
""pyhhor _tcouunnt_ceeeeeeeeeee (charactez, string,
count = t index,
enumerate (
if char =- character:
if count ==
return -1
Example usage:
string "hello wozld, how are you?"
charscter = 'o'
3
result Eihh_nntho (cheracter, string,
print (I*The (alrd occurcence of "cheracceer" La at index (resuit).")
This function takes threa argumenta:... 10mitted for brevity-)
25

---

Published as a conference paper at ICLR 2026
Figure G.3: An example Arena-Hard (Li al. 2024a) response generated by Llama3-8B Maima--B-BBase Base (Al@ Metal 2024b) after preference learning. The outputs are compared against those of the SimPO-trained model and proprietary GPT-4 model.
Input
Q: How do i do tuple comprehension in pythen?
SimPO
Tuple comprehensions are not available An Python, they are specific to Python 3.8 and later versions.
tuples using list comprehensions and then them to tuples using the "tuple!)" function.
Python, you
Here's s example:
""pythen Exemple: Creating a tuple of squares from a list of numbers
numbers - [1, 2, 3, 4, 5]
Using list comprehension
squares_ [x.42 for numbers]
converting the comprehension result
aquares_tuple (squatesssss
print (squarea_tuple) Output: (1, 4, 9, 16, 25)
In this example, we first create a Hst comprehenston that ca culates the 'numbera" list. Then, convert
squares of comprehension result to tuple using the 'tuple fuaction.
Note that tuple comprehensions are not available in Python versions prior to 3.8, but you can achieve similar functionality using a list comprehensions and converting them tuples shown above.
Uni-DPO
Python does not have tuple comprohensions, but you can use generator expression to create tuple. liere's s an example:
"pythor
Example:
tuple aquares range
range (L,
aquare_tuple txpla for x in numbers)
print (square_tuple) Output: 4, 9, 16, 25)
In this example, we use generator expression Mxxx2 for in numbers) to number in the 'numbers' iterable, then
calculate the squares convert the resulting generator cbject to cuple using the 'tuple() function.
Note that tuple comprahensions expressions can be used to achieve similar results. If you want to create tuple simple transfozmation, generator expressions are readable way do it.
feature in Python, but generator
GPT-4
tuple comprahension using generator expression inside a tuple constructor. Here'a an example:
Python,
pyyhhon
Senerator expression inside tuple constructor sople__ompreenntton tuple(x for in range (5))
print (tuple comprehansionn Output: (0,
in this example, we create tuple comprehension that contains the double of
each number the range from to 4. genarasor expression "x range used to generate the doubled values, and the 'tuple() () ructor is used to coavert the generator expression inte tuple.
26

---

Published as a conference paper at ICLR 2026
Figure G.4: An example Arena-Hard et al. 2024a) response generated by the Llama3-8B-Instie AT Meta 2024b) model after preference learning. The outputs compared against those of the baseline model.
Input
Q:I have pazt of Javascript function that I want to rewrite. Currently
searches every property Matches find the minimum, and makes Flayer2 always be the first member. Instead, want Flayerl to the lowest result sorting by Matches, and Player2 to be random each time the code is run.
function cdaada
// the two players with fewest matchas.
let minMacches Number.MAX_GAPE MAX_SAPP_INTTEEEN
let Playerl aull;
let Player2 mull;
for (let playar of datal
(player.Xatchee minNasches)
minMatches
Playerl player;
Flayer2 data. find(p !-- Player11;
Baseline
Hlere's revised version of Function that sozta the playera by the number of matches and then assigns Playerl as the player with the fewest matches and Flayer2 randon player from the
" "avvsserrppt
function (data)
Sort the players by the rumber matchea
deta.sorl b) a.Matches b.Matches);
// Assign Playeri the player with the fewest matches let Playar1 dd]a
// Assign Player2 as randon player from the rest
1et Player2 data (Math. floor (Math. randon (data. 1)1 11z
// Make suze Player2 is not the same as Pleyerl while (Player2 Playarll Player2 = data [Math. floor (Math. rancom!1 (data. 11) 1];
This function first serts the players by the number natches ascending order. Then it assigns first player in sorted array ayer1. asslgns raadom playez from the rest of the array as Playe:2, making aure Player2
the same Playeri.
Uni-DPO + Here's s the rewritten function:
Javascript
function clo(data)
// Find the player with the fewest matchea (Player1)
let minMsrches
(lachr.Xatches)
let Playerl (playe Nitchs
minMatches)
// Randemly salect Playar2 from the remaining players
playaroptions data, dita.filter filteri p Player11;
let Player2mplayerdption (Math. floor raachh (eaaaalpppaaayeeeiions.ee.ee
Here's what's changed:
minimun 'Katches' value, we use the 'Math, 'Math.min() function with the spread operator ("...) to find the concise way.
Instead iterating through
2. We 'data. find()) (inda.find)) to retrieve the player with the minimum 'Matches"
value, which becomes 'Playeri',
filter 'Playes!' from the original data arrsy using 'data. filter() ', and then use Math random() " to select an integer
3. solect 'Flayer2' rendamly,
randon index using 'Math. floor and use it to access the corcesponding player from the array.
index from the remaining players.
Note that this imglementation
that 'dats" is arrey of objeste with a 'Mathess' prcpesty, and there are at least two players in the data. if these assumptions don't hold, may nead add additional handling or checks.
27

---

Published as a conference paper at ICLR 2026
Figure G.S: An example AIME2024 response generated by the Qwen2.5-Math-1.58 (Yang al. 2024b model after preference learning. The outputs are compared against those the SimPO-trained model.
Input
@:
Jen enters Lottery by pieking 4 distinct rumbers Irom S 21.2,... .., 9. 10). numbers are rendomly chosen from S. She wins a prize if at least two of her numbers,
wins the grand prize if all four of her numbers are the randemly chosen numbers. The probabillty of her winning the grand prize given that she woz prize is m where m relativaly prime positive intagers. Find +n.
numbers were 2 of the randonly
are
SimPO
lo solve this problem, need to calculate the probability of cen winning
the grand priza given that sha ven a prize, We can break this down into the lowiining steps:
1. Calculate the Lotal number of ways to choose 4 distinct numbers from the
set (1,,,3,..,,11 ,00..
2. caleulate the number of ways to win prize (i.e., at least 2 of her numbers are among the randonly chosen numbers).
3, Calcilate the number of ways to win the grand prize ti.e,, all 4 of her numbers aze among the 4 randonly chosen aumbers).
4. Use the conditionsl probability fermula find the probability of winning the grand prize given that sha wen a prize.
Let's start by eslculating the tetal number ways choose 4 distinst
numbers from the set S. This is giver by the combinstion formuls (4))
Next, we need to calculate the number ways to win a prize. by considering the cases where exactly 2, 3, or 4 of het numbors are among the randonly chosen numbers.
done
Finelly, we need to calculate the number of ways to win the grand price, which is the case where all 4 of her numbers are among the 4 zandomly chosen numbers. This is given by the combination. formuls (1).
... (Onitted for brevity.1
The probability of winning the grand prize given that she prize is I.. Therefore, the velues of m and n are and 11, respectively. The sum m + 1+11=
Thus, the final answer is 12
Uni-DPO
To solve this problam, need to calculate the probability of Jen winning the grand prize given that ahe won prize, Me can break this down into the following steps:
1. Caleulate the totel number of ways to choose 4 distinct numbers from the
set == (1,2,3,...,,1f
2. calculate the number of ways to win prize (1.0,, at least 2 of her numbers match the 1 randomly chosen numbers).
3. Caleulate the number of ways win the grand prize (i.e., all 4 of her
numbers match the 4 randomly chosen numbers).
4, Dae the conéitions
probability formula find the probabil ttt of winning the grand price given that she wen price.
Let's start with the calculations:
1. The total number of ways to choose 4 distinct numbers Irom 10 Le given by combination fornula (1)):
10!
4 41(10 4)
210
To calculate the number of ways win prize, need to consides the cases where exsctly 2, 3, or of her numbers natch the 4 randemly chosen numbers:
2.
Exactly 2 matches: Choose from her numbers and 2 from the rems iniig 6 numbers.
6 15 90
Exactly 3 metchen: Choose 3 from her numbora and from the remaining 6 numbers.
4.66 6 24
Exact ly matches: Choose all from her numoers.
=1-1=1
So, the total number ways to win a prize is:
+ 24
The number of ways to win the grand prize 1a the number of ways to choose
all 4 of her aumbers to natch the 4 zandomly chosen numbers, which is:
-1.11
4. The probability of winning the grand prise given that she won prize is the ratio the number ways to win grand pzize the number ways to P(grand prize prizz)= 115
win a prize:
So, the probability is d.. the values of m are and 115,
respectively. Therefore, m m 1 115 = 116.
the final answer is 116
28

---

Published as a conference paper at ICLR 2026
Figure G.6: An GCample/AMM200 response generated by the Qwen2.5-Math-1.33 (Yang et al. 2024b model after preference learning. The outputs compared against those of the SimPO-trained model.
Input
0: Mra. Jones is pouring orange juice into four identical glasses for her four sons. She fills
first three glasses cemplately but runs out juice when the fourth glass is only full. What fraction of glaas must Mra. Jonas poured from each of the firat three glaases into the fourth qlasa so that be
all four glasses will have the same amount of juice? The final written in the form m, where m and n are relatively prime pasitive integers. What +
SimPO
To solve this problem, we need to determine how much juice Mra. Jones must from eaca of the first threee glasses into the fourth şlase that all four glasses have the same amount of juice.
1. Let's assume each glass
capacity of unit of juice.
2. The first three glasses are completaly filled, so each of them hea 1 unft
of juice.
The Fourth glass is only $ full,
has unit of juice.
4. The total amount of juice is 1+1+ 11l++ +++++ = 3 3 = 10 units.
5. since
are glasses, each glass
10 +
units
of juice.
6. The fourth glass currently has 1 unit of juice, so it needs -
2 unit of juice to reach 3 unita.
completely filled, lose 1 unit of juice te the fourth glass.
7. Since the first three glasses
then
3o, Mrs. Jones must pour unit of juice from each the first three glasses into the Tourth glass. The fraction ef glass that must be poured Irom each of the first three glasses into the feurth glass is 1-
answer is i, m + ===2++=3.
form m
Therefore,
The final answer 1s
Uni-DPO
determine how much juice Mrs. Jones must. Four from each of the first three glasses into the fourth glass that all four glasses have the sane amount of juice,
To solve problem, we
1. Let's assune each glass has capacity of unit of juice.
2. The first three glasses are completely filled, so each of them hes 1 unit
of juice.
3. The Sourths glass is only 1 full, it has unit of juice.
4. The total amount of juice 1+1+ +1++ = 3 z = 18 units.
there of juice,
glasses, each glass should
3 +4
10 = unita
6. The fourth glass currently has 1 unit juice, s0 it needs 3 3-3-0-3
unit of juice reach å unita.
4. Since the juice is being poured Iron the firet three glasses into the fourth glass, each of the first three glasses must lose 1+3= g unit of juice. first three glasses
must pour & unit of juice into the feurth glass. The fraction of glass that must be poured from each of the firat three glasses into the fourth glass is #
The final answer is 4r and m 1
answer is
29

---

Published as a conference paper at ICLR 2026
Figure G.7: An example A1ME202 generated by the Qwen2.5 1rren.5--Mth--7B (Yang et al. 2024b) model after preference learning. The outputs are compared against those of the SimPO-trained model.
Input
Q: list positive integers
the following properties:
The aun of the itema in the 1ist is 30.
The unique mode of the 1is: is 9,
The median of the list is positive intager that does not appear in the list itself.
Find sum
aquares
the items in 1ist.
SimPO + Given these constraints, let's start by considering the possible lengths of the list. Since 9 is the unique mode, it has on add number of clements, the median will be the middle alement, and it Tf the 11st has even number elements, the mediar will be the average of the two middle elements, and again, it cannot be 9,
appear least twice. if the list
Let's start by considering the smallest possible list that satisfies these conditions, The 1ist must have at least 5 elements (since 9 nust appear least twice and the median cannot be 9), Let's try a 1ist with 5 elements: of these elements mast be 20,
19, 9, b, el, The o = 30, which simplifies to a + +
9 9
12, The median of this 1ist is the must one of a, b, or G, and cannot be
third element when sorted,
9. Let's tay 1, b - 2, and - 9. This gives us the list [1, 2, 9, 9, 91, the medien is 9, which is not allowed. so, let's try
is the mode, another combination.
... (Omitted for brevity.)
Let's Lry a with 7 elements: 19, a, b, o, d, el. sum elements must be 30, so 9 + 9 + a + b + e + d e- 30, which simplifies to this list is
fourth element when sorted, so it must be one of the numbers in the list, and it cannet be 9. Let's try a = L, b 2, c = 3, d= 4, and 9, 9], and the medion is which is
+ 12. The median
2. This gives us the list [1, 2, 2, 3, 4,
9 and appears only once. This Hist
aatisties all the conditions. Comment: 9 is not the unique mode.
Now, let's calculate the sum of the squares of all the items in the list: 1 + 2 +22 + + 4 + + == =1+4+ +444+++00 ++ 16 + 81 81 196.
The sum of the aquazes of all the items in the 1ist is 196
Uni-DPO + Let's denote the 1ist of integers as 01, 02.42..... Anr where as are positive integers and n the number of elements in the list. Since the mode is 9 and it is unique, the number 9 must appear more frequently than any other number in List, bet'a since the median is positive integer that doas not appear in the list itsalf, the 11st must have an even number of elementa lotherwi the mediar an would be the middle element, which would appear in the list). Let's assume the list has 2m elements. The median would then be the average of the m-th and (m + 1)-th elements when the list is sorted ascending ordes. Since the median does not appear in the 1ist, it must be a difrertnt number.
that appears k times in the 11at, where & 2.
Let's stazt by considering the simplest case where the 1ist has elements. The median would be the average of the 2nd and 3rd elements. since the median doea not appear in the 1iat, it muat be different number, let's s denote the list d, in ascending order. The median would be T and it must be different number from c,d. c,d.
Civen that the aun of the alenents is 30 and the mode is 9, lat's assume the Has is 0,9, The aue of theae elementa is + 9 = 30, so a + = 12. median would be $49 and mus: a different number from b, let's try different values for and b.
If = 3 and b= 9, the 1ist would be 3,9,9,9, but the node would not be unique. If and b= 8, the 1ist would be 4,8,9,9 and the median would be o1 = 8.5, which is not an inzeger. If a = 5 and b = 7, the list would be 5, 7,7, 9, and the median would T49 list.
which 15
fffrentt number and does
appear in the
So, the 1ist 5,7,9.9 satiefies all conditions. The sun the squares the elements in the 1ist is 5² 4 9" + 9² = 25 49 +811 + = 236.
The sum of the squares of ell the itens in the 1ist is
236
30


Reviews:
[{"review_id": "rgMSsbFxMO", "reviewer": "Reviewer_uYa8", "summary": "The paper proposes Uni-DPO, a “dual-perspective” extension of Direct Preference Optimization that dynamically reweights each preference pair by (i) a quality-aware weight $w_{\\text{qual}}$ estimated from external expert scores and (ii) a performance-based weight $w_{\\text{perf}}$ that emphasizes pairs the model currently fails to fit; it also adds a calibrated NLL term for difficult, high-quality positives. The stated goal is to correct DPO’s uniform treatment of pairs and better utilize data during preference learning for text, math, and multimodal LLMs. Empirically, the authors report consistent gains over SFT, DPO, and SimPO on instruction-following (AlpacaEval2, Arena-Hard, IFEval, SedarEval), math (GSM8K, MATH, MinervaMath, etc.), and multimodal suites, and claim Gemma-2-9B-it+Uni-DPO surpasses Claude 3 Opus on Arena-Hard.", "questions": "- Described in the Weakness section.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 4, "contribution": 3, "strengths": "- Clear motivation and formulation: The paper explicitly identifies DPO's uniform-pair limiattions and proposes a dual weighting + c-NLL objective.\n- Empirical gains across domains: Reported improvements over SFT/DPO/SimPO on text, meth and multimodal data depicts the strength of Uni-DPO\n- Ablations: The paper shows that removing each of the component degrades performance, empirically supoprting the necessity of each component.", "weaknesses": "- Lack of transparency and reproducibility of the scalar quality score $w_{\\text{qual}}$.\n    - The core of Uni-DPO's contribution, $w_{\\text{qual}}$, depends on external scalar scores produced by \"expert evaluators\" (e.g., GPT-4). However, the exact procedure is not disclosed. Since $w_{\\text{qual}}$ is an important component of Uni-DPO, the procedure of obtaining this score should be thoroughly described in the paper.\n- Insufficient engagement with existing sample-wise weighting preference optimization (PO) literature.\n    - There are no related work section regarding the prior methods that apply per-sample or instance-wise weighting in PO[a,b,c,d]. This weakens the novelty and makes it unclear whether if Uni-DPO has any overlaps between the related works.\n- Potential unfairness from applying Length Normalization (LN) only to Uni-DPO.\n    - The paper notes that LN is used when computing rewards or scores for Uni-DPO, but not for baseline method such as DPO in Table 1. As LN can boost the performance regardless of the underlying loss, the fairness of the baseline is questionable.\n- External scorer dependence and fairness\n    - The $w_{\\text{qual}}$ depends on strong proprietary LLM scorers, which hinders the fairness against baselines trained with weaker/cheaper labels. How is using these external strong models justified?\n- If these weaknesses are taken care of, I am willing to raise my score.\n\n[a] Reward Difference Optimization For Sample Reweighting In Offline RLHF, ACL, 2024\n\n[b] Enhancing RLHF with Weighted Preference Optimization, EMNLP, 2024\n\n[c] MWPO: Enhancing LLMs Performance through\nMulti-Weight Preference Strength and Length Optimization, ACL, 2025\n\n[d] Relative Preference Optimization: Enhancing LLM Alignment through Contrasting Responses across Identical and Diverse Prompts, ArXiv, 2025", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "8iJAlkQjA6", "reviewer": "Reviewer_5qbs", "summary": "The paper proposes Uni-DPO for fine-tuning large language models with human feedback that dynamically reweights preference training pairs based on each pair’s quality and the model’s evolving performance. This addresses the limitation of standard DPO, which treats all feedback equally. By explicitly emphasizing high-quality, challenging preference pairs during training, Uni-DPO enables more effective use of data, which in turn yields better alignment performance across diverse benchmarks.", "questions": "1. Could Uni-DPO be combined with reward model–based RLHF methods (e.g., PPO) to further improve alignment? How would the dual perspective weighting interact with explicit reward signals?\n2. How sensitive is Uni-DPO to the source and accuracy of the quality scores $w_{qual}$? For example, if one uses a weaker judge model or noisy human preference ratings to derive the score margin, does Uni-DPO still outperform vanilla DPO by a large margin?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1.\tThe paper clearly identifies that standard DPO treats all preference pairs uniformly, which underutilizes high-quality feedback and fails to adapt to varying task difficulty. Uni-DPO tackles this by adaptive sample weighting, allowing the model to focus on informative training examples and thus improving learning efficiency. This insight targets an important problem in RLHF.\n2.\tThe quality aware weight prioritizes pairs with larger expert score margins, while the performance aware weight, inspired by focal loss, shifts focus toward underfitted examples. This addresses observed mismatches between external score margins and the model’s reward margin.\n3.\tUni-DPO shows consistent improvements on diverse tasks. Experiments show that Uni-DPO consistently outperforms vanilla DPO and SimPO baselines on multiple language understanding benchmarks, mathematical reasoning datasets, and multimodal tasks.", "weaknesses": "1.\tA notable concern is that Uni-DPO requires a “quality score” for each preference pair, often obtained via human annotation or a powerful proxy model like GPT-4. This introduces an extra dependency on external evaluators (in effect, a form of reward signal), partially undermining the simplicity of the reward model-free DPO paradigm. If these quality scores are noisy, biased, or unavailable, it’s unclear how well the method would perform. The authors themselves acknowledge that training data quality is critical and better methods for estimating it are needed. In scenarios without reliable prior scores, the applicability of Uni-DPO could be limited or require additional effort (e.g. training a proxy reward model), which diminishes its practicality.\n\n2.\tThe contribution of Uni-DPO, though useful, can be seen as a combination of existing ideas in preference optimization rather than an entirely novel invention. The method builds on known techniques: using an auxiliary NLL loss to counter DPO’s bias (as in Pal et al., 2024), applying length normalization to remove sequence-length bias, and adapting focal loss from classification to reweight easy vs hard examples. The quality-based data filtering is conceptually akin to curriculum learning or prioritizing high-confidence preferences, which have precedents in the literature, although not in the exact form used here. While the unification of these components into one framework is valuable, the paper does not derive fundamentally new theory beyond this integration, nor introduce a wholly new training paradigm. It primarily refines DPO with well-motivated heuristics. \n\n3.\tBy focusing training on “hard” and high-quality examples, there is a risk, which is not deeply explored in the paper, of overfitting or bias if these weights are mis-calibrated. It would be better to introduce a calibrated performance weight to avoid instability, indicating the approach required careful handling to train reliably. It would be good to know how consistently training converges under Uni-DPO versus standard DPO. For instance, do some runs diverge or get stuck if the thresholds are set poorly?\n\n4.\tWhile c-NLL is proposed to counteract DPO’s preference suppression bias, the paper does not explore its impact on diversity or other alignment metrics. It also does not evaluate fairness or bias trade offs introduced by weighting high quality data more heavily.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jApo14ObVN", "reviewer": "Reviewer_Azer", "summary": "This paper introduces Uni-DPO, which injects two jointly learned adaptive weights into single-pass offline training: 1) a quality weight derived from external score margins that up-weights high-confidence preference pairs, and 2) a performance weight computed from the current policy margin that down-weights already well-fitted samples while emphasizing those that remain hard. Additionally, to counteract DPO’s tendency to suppress the absolute likelihood of preferred responses, the authors devise a calibrated negative log-likelihood term that selectively reinforces difficult yet high-quality positive examples. The resulting objective preserves the simplicity of DPO but enables gradient allocation at a fine-grained level that accounts for both sample difficulty and data quality.", "questions": "Some suggestions here.\n\n- The right-hand side of Figure 3 does not clearly or intuitively convey the core idea of the proposed method. Its meaning only becomes apparent after reading the entire paper and integrating multiple sections. It is recommended to redesign or annotate the figure to make the conceptual distinction between DPO and Uni-DPO more self-explanatory and visually interpretable.\n\n- It would be helpful to include training curve comparisons (e.g., loss or evaluation score versus training steps) to further illustrate the differences in learning efficiency among Uni-DPO, DPO, and SimPO.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- This paper derives the gradient coefficient of Uni-DPO in closed form, explicitly integrating two modulation factors, including quality weight and performance weight, to provide a principled explanation for online sample re-weighting.\n- Uni-DPO preserves the simplicity of single-stage offline training without introducing additional reward models or iterative sampling overhead. With only two learnable weights and a calibrated loss, it can achieve consistent performance improvements across diverse benchmarks.", "weaknesses": "- Although the paper proposes a \"unified dynamic weighting paradigm\", its core idea essentially combines a quality-aware weight with a performance-aware weight. Such sample reweighting concepts have already been extensively studied in machine learning and RLHF literature, including focal loss, curriculum learning, and advantage reweighting. Therefore, the conceptual novelty of Uni-DPO appears somewhat incremental rather than fundamentally new.\n\n- A major concern lies in the lack of essential baselines. While the paper focuses on DPO-based alignment, it would be important to include comparisons with more recent RL-based alignment methods beyond PPO, since several advanced RL algorithms have recently demonstrated improved stability and performance in preference optimization. Moreover, to my knowledge, there already exist studies on data selection and adaptive weighting for DPO that explicitly aim to exploit heterogeneous data quality during training. Including these approaches as baselines would provide a fairer and more convincing empirical validation.\n\n- Although Uni-DPO claims to \"outperform SimPO\", its improvement may come from the weighting strategy rather than a new optimization formulation. In contrast, SimPO already removes the reference model and introduces length normalization, while Uni-DPO still relies on the log-probability ratio between the policy and the reference model. This raises questions about whether the claimed superiority originates from a fundamentally different principle or just an additional heuristic weighting scheme.\n\n- Uni-DPO is only evaluated on textual understanding and mathematical reasoning tasks, while multimodal experiments are mentioned in the abstract but not presented in the results section. The lack of evaluations on other important aspects such as dialogue safety, factual consistency, or instruction robustness weakens the \"Unified\" claim. Additionally, the models used for evaluation (e.g., LLaMA-3-8B, Gemma-2-9B) appear to be relatively early versions; including results on more recent model releases would strengthen the credibility and contemporaneity of the paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "zlNejAFO58", "reviewer": "Reviewer_HSvf", "summary": "The paper develops what’s called a “unified dynamic preference approach”, UNI-DPO, aiming to improve extend the basic DPO. Some key elements of UNI-DPO include: two weighting factors, w_qual and w_perf, which apply, respectively, to the preference pair (y_win, y_loss) and to the current policy \\pi_\\theta; the former takes into account data quality, and the latter measures how the policy aligns with the pair. Furthermore, a “calibrated negative log-likelihood loss” (c-NLL) term is added to the training objective, to amplify the policy’s confidence on y_win. These are elaborated in \\S3.2, 3.3. and 3.4.", "questions": "Can the author(s) comment on how does the dual weighting scheme in UNI-DPO compare with other approaches to preference alignment and reward optimization, and specifically the weighting factors motivated by importance sampling?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The main strength of the approach is to dynamically adjust each sample’s contribution based on both data quality and the model’s learning performance. This effectively incorporates ideas like advantage-based schemes in RL and thereby improves training efficiency and overall performance.", "weaknesses": "The weight w_qual defined in (5) is a bit simplistic in view of the complexity in defining (or capturing) what is data quality. This is also partially acknowledged by the paper (at the end).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)