$\alpha$-DPO: Robust Preference Alignment for Diffusion Models via $\alpha$ Divergence
Authors: Yang Li
Songlin Yang
Wei Wang
Xiaoxuan Han
Jing Dong
Venue: ICLR 2026 Poster (2026)
Keywords: diffusion model; preference alignment; noise robustness
Primary area: reinforcement learning

Abstract: Diffusion models have demonstrated remarkable success in high-fidelity image generation, yet aligning them with human preferences remains challenging. Direct Preference Optimization (DPO) offers a promising framework, but its effectiveness is critically hindered by noisy data arising from mislabeled preference pairs and individual preference pairs. We theoretically show that existing DPO objectives are equivalent to minimizing the Forward Kullback–Leibler (KL) divergence, whose mass-covering nature makes it intrinsically sensitive to such noise. To address this limitation, we propose $\alpha$-DPO, which reformulates preference alignment through the lens of $\alpha$-divergence. This formulation promotes mode-seeking behavior and bounds the influence of outliers, thereby enhancing robustness. Furthermore, we introduce a dynamic scheduling mechanism that adaptively adjusts $\alpha$ according to the observed preference distribution, providing data-aware noise tolerance during training. Extensive experiments on synthetic and real-world datasets validate that $\alpha$-DPO consistently outperforms existing baselines, achieving superior robustness and preference alignment.

Published as a conference paper at ICLR 2026
a-DPO: ROBUST PREFERENCE ALIGNMENT FOR DIF- FUSION MODEL VIA a DIVERGENCE
Yang Li Songlin Yang" Wei Wang'* Xiaoxuan Han 1.2 & Dong Jing' NNew Laboratory of Pattern Recognition, MAIS, CASIA 2 School of Artificial Intelligence, UCAS The Hong Kong University of Science and Technology
ABSTRACT
Diffusion models have demonstrated remarkable success in high-fidelity image generation, yet aligning them with human preferences remains challenging. Di- rect Preference Optimization (DPO) offers a promising framework, but its effec- tiveness critically hindered by noisy data arising from mislabeled preference pairs and individual preference pairs. We theoretically show that existing DPO objectives are equivalent to minimizing the Forward Kullback-Leibler (KL) diver- gence, whose covsrrn nature makes it intrinsically sensitive to such noise. To address this limitation, we propose a-DPO, which reformulates preference alignment through the lens of a-divergence. This formulation promotes mode- seeking behavior and bounds the influence of outliers, thereby enhancing robust- ness. Furthermore, we introduce a dynamic scheduling mechanism that adaptively adjusts according to the observed preference distribution, providing data-aware noise tolerance during training. Extensive experiments on synthetic and real- world datasets validate that a- a-DPO consistently outperforms existing baselines, achieving superior robustness and preference alignment. The code and project page are available at httpt:ts://// //g thub. com/yang i 1aab Di fuusicn in a phppaaai DD0- TCLR20262
INTRODUCTION
Diffusion models Labs 2024 Esser et al. 2024 Podell et al. 2023) have recently emerged as pow- erful generative frameworks capable of producing high-fidelity and photorealistic images. However, beyond visual quality, aligning these models with human preferences is crucial for practical deploy- ment, since users often care about semantic relevance, stylistic choices, and aesthetic qualities that not fully captured by lkelihood-basee training. A common paradigm to address this alignment problem is reinforcement learning from human feedback (RLHF) (Black et al.) 2023 Fan et al, 2023 Zhu et al. 2025 Yang al., 2024), which leverages preference annotations train reward model and subsequently optimize the generator via reinforcement learning. While effective, RI.F introduces additional complexity and instability due to the need for explicit reward modeling and policy optimization. Direct Preference Optimization (DPO) (Wallace et al. 2024) offers a more streamlined alternative: instead of constructing a separate reward function, DPO directly optimizes the generative model to increase the likelihood of human- human-preferred samples over less preferred ones, thus achieving preference alignment in a simpler and more stable manner.
However, the effectiveness of DPO paradigms critically hinges on the quality of their preference data. In practice, such datasets inevitably contain noisy preference pairs from two primary sources: (i) mislabeled preference pairs, where annotation errors introduce signals that contradict real pref- erences; and (ii) individual preference pairs, arising from annotator subjectivity, where either as- signment of winner and loser can be considered valid ("Also OK" labels). Although these noisy pairs originate differently, they both alter the intended preference label. From this perspective, we unify them tabel-flippin noise, which allows us explicitly control the ratio of noisy pairs in the dataset and quantify how different noise levels affect DPO optimization. As shown in Fig. T block (a) provides examples of noisy preference pairs sampled from the Pick-a-Pic v2 dataset (Kirstain
"orrreponndngg Author.

---

Published as a conference paper at ICLR 2026
Noisy Data Pair Type
(b) Dataset Distribution
Impact Noisy Preference Pairs
Mislabeled Preference Pair
DPO
-P0 (Oun)
Low Noise ratio
"A cuts cat"
Gien Labe:
Real Label
Loser
Winner
True
High Noise ratio
Individual Preference
Tiny clear plastic
Noise
degi mode of crysticl, internal mechaniama"
"D digital illuttration Burger with wheels speeding on the rooe track supershorged, detaileed hypereslistic, 48"
Bacn Labe:
"Min labe:
Figure I: Impact of noisy preference pairs on DPO. Block (a) shows two types of noisy preference
pairs from the Pick-a-Pic v2: mislabeled and individual pairs, which can be unified as lahel-fipping noise. Block (b) shows our estimated data distribution of the Pick-a-Pic V2 dataset. Block (c) illustrates that DPO is highly sensitive to such label-flipping noise, leading to degraded performance.
et al. 2023), and block (c) compares optimization under low and high noise ratios, illustrating how a larger proportion of noisy pairs substantially degrades the training efficacy of DPO.
To address this critical vulnerability of DPO to noisy data, previous research has focused on de- veloping noise-robust DPO variants. They (Chowdhury et al. 2024, Fujisawa et al., 2025 Liang al. 2024) typically rely on modifying loss functions applying heuristics under simplified noise assumptions (e.g.. L.I.. flip assumptions, known noise rates). These approaches often lack the fun- damental ability to distinguish the true human preferences from spurious modes introduced by noisy data in real-world preference datasets. As shown in Fig. block (b), the noisy data used in pref- erence alignment shows different distribution from the clean data. Moreover, these methods primarily developed for Large Language Models (LLMss..
To better understand this limitation, revisit DPO from distributional perspective. In essence, DPO optimizes the model to align with the distribution of preferred (winner) samples while di- verging from that of dispreferred (loser) ones. The presence of mislabeled preference pairs and individual preference pairs significantly compromises this process. Our theoretical analysis reveals that DPO's s optimization objective is equivalent to minimizing the Forward Kullback-Leibler (FKL) divergence between the model distribution and the target preference distribution. However, FKI. is inherently noise-sensitive due to mass-covering behavior: it heavily penalizes underestimation even in regions where the target distribution assigns negligible probability. This property amplifies the influence of noisy or mislabeled samples, degrading alignment.
To achieve robust preference alignment in the presence of noise, two key criteria must be satisfied: (i) Mode-Seeking: Optimization should actively favor high-density regions over full support coverage. prioritizing the learning of salient preferences; (ii) Bounded Influence of Outliers: The loss function should be resilient to individual erroneous or outliers in the preference dataset. Motivated by these criteria, we introduce a-DPO, which replaces the FKL with the promoting mode-seeking behavior and enhancing tolerance to noisy preferences. Crucially, since the optimal a depends on the underlying structure of these noisy data, we introduce dynamic a scheduling mechanism that adaptively adjusts a based on the observed preference probability of each sample. This makes our approach inherently adaptive preference deviations, leading to more reliable and robust alignment in practice. Extensive experiments showing that a- a-DPO consistently surpasses existing moise-robuss baselines across diverse benchmarks and noise regimes, affirming its effectiveness and generality. Comprehensive ablation studies further confirm the individual impact of each component in our framework.
general a-divergence, naturally
Our contributions are summarized as follows: (i) We are the first to explore a noise-robust DPO method for image generation and theoretically prove Diffusion-DPO's inherent vulnerability to noisy preferences by demonstrating its optimization objective's equivalence to FKL divergence. (ii) We propose a-DPO, which applies a-Divergence to DPO to improve noise tolerance, and design dynamic noise assessment method that adaptively adjusts its noise robustness during training for enhanced preference alignment. (iii) Extensive evaluations across synthetic and real-world datasets reveal that our aDDPO exhibits robust noise tolerance, achieves superior preference alignment, and
2

---

Published as a conference paper at ICLR 2026
consistently surpasses existing noise robust preference learning baselines, all without incurring ad- ditional computational cost.
2 RELATED WORK
Preference Alignment of Diffusion Model. Preference learning for human preference alignment
has recently achieved remarkable advancement diffusion image generation (Fan et al. [023]
Black et al., 2023 Yang et al., 2024 Zhu et al., 2025 Lu et al. 2025). Early explorations, such as DDPO (Black et al. 2023) and DPOK (Fan et al., 2023). utilize Reinforcement Learning (RL)
algorithms to steer backbone models towards alignment with high-reward images (Xu et al., 2023
Kirstain et al, 2023 Wu et al. 2023). However, numerous studies indicate that these RL-based methods often suffer from reward hacking (Jena et al. 2025) and tend to yield suboptimal re- sults. Another line of work leverages the DDIM sampling process and trains a step-wise reward model (Liang al., 2025) for better preference alignment. However, these methods explicitly re- quire reward model training. Alternatively, D3PO (Yang et al.] 2024) directly uses binary human feedback to fine-tune the diffusion model. Similarly, Diffusion-DRR DPO (Wallace al. 2024) adopts
the DPO algorithm (Rafailov et al. 2023) from Large Language Models (LLMs) (Han et al. 2024
Go al. 2023), implicitly reparameterizing the reward model via the pre-trained model and the
optimized model. Another large range of methods (Lu et al. 2025 Wang et al., 2025 Li et al.,
2024 Hong et al. 2024) (Shen et al.) 2025] Yuan et al., 2024) builds on DPO to further improve the performance of preference alignment in Text-to-lmage generation. However, these direct fine-tuning methods remain vulnerable to data noise, which can lead to a significant performance degradation.
Divergence- Based Preference Optimization. Several recent works explore divergence-based pref- erence optimization (Lee et al.) 2025) Gupta et al. 2025 Wu et al. 2024b) Santurkar et al. [023] Song et al. 2023). AlphaPO (Gupta et al.) 2025) leverages an a-transform to reshape the reward model to stabilize preference optimization, while preserving the original KL-based DPO objec- tive structure. FKPD (Shan et al. 2025) introduces forward-KL regularization into Diffusion-DPO to improve mode coverage and mitigate out-of-distribution generations in diffusion models. a- DPO (Wu et al., 2024a) proposes a dynamic margin control version of IPO (Azar al. 2024), aiming at enhancing the preference alignment when the reference model is not well-calibrated. In addition, a-divergence has been applied as a policy regularizer in RLHF-style objectives, such as AlphaPO (Gupta et al. 2025) and f-DPO (Wang et al. 2003). by replacing the standard KL regularization term between the learned policy and the reference policy. In contrast to these ap- minimiiaaiinniiiiioonnn objective between
proaches, our method directly reformulates DPO as a the learned preference distribution and the target preference distribution. Furthermore, a mode- seeking a-divergence is adopted to improve robustness against noisy preference labels.
Robust Direct Preference Optimization. Following advances learning with noisy labels in image classification (Song et al. 2022). A broad range of DPO variants have been developed to improve preference alignment under noisy feedback. These approaches can be broadly grouped into three categories: (i) Sample Selection methods (Kong al. 2024; Deng et al. 2025 Cheng et al.) 2024 Singh et al. 2024) filter out noisy samples by leveraging models trained on clean datasets. However, they often limited by the requirement of clean data and the need to train multiple noise-validation models. (ii) Regularization-bass
methods, such as ROPO (Liang et al.) (0024). assign regularized gradient weights according label uncertainty
mitigate noise, while Con- servative DPO (Mitchell] 2023) applies label smoothing. (iii) Loss modification methods improve robustness by optimizing loss. Robust-DPO (Chowdhury et al. (0024) introduces theoretically grounded noisc-robust loss design, but its impracticality demands prior knowledge of the noise ratio 8 is unfeasible in real-world scenarios. NAPO (Zhang et al., 2025) combines the noise robust Mean Absolute Error (MAE) (Ghosh et al. 2015) with Binary Cross-Entropy (BCE) to alleviate modality bias in MLLMs. More recently, Hölder-DPO (Fujisawa et al., 2025) evaluates the robustness using the statistics concept of redescending properties, revealing that no existing DPO variant satisfies this criterion. Its robustness, however, is derived under an I.L.D. flip assumption, which limits predictive reliability under structured noise, such annotator bias or divergent group preferences. Critically, all of these methods are designed for autoregressive language models, whereas our approach focuses on diffusion models, offering fundamentally distinct inspiration for aligning the Markov chains.
3

---

Published as a conference paper at ICLR 2026
3 METHOD
Our goal is to achieve robust preference alignment for diffusion models under noisy human feed- back. In Sec. [.I,, starting from Reinforcement Learning from Human Feedback (RLHF) and its simplified variant Direct Preference Optimization (DPO), we extend DPO to diffusion models and show that its objective reduces minimizing Forward Kullback-Leibler (FKL) divergence, which is intrinsically prise-sensitive In Sec. 3.2 to overcome this limitation, we introduce a-DPO, re- formulation based on a-divergence that enhances robustness by balancing mass-cooering and mode- seeking. Furthermore, we design a dynamic scheduling strategy that adapts to data quality, ensur- ing consistent alignment under varying noise levels.
3.1 REFORMULATING DIFFUSION PREFERENCE ALIGNMENT AS FKL DIVERGENCE
We trace the evolution for preference alignment, from RLHF to DPO and its extension to diffusion models (i.e., Diffusion-DPO), and show that the resulting Diffusion- DPO objective can be reformu- lated FKI. divergence minimization problem.
Reinforcement Learning from Human Feedback. RLHF has emerged as a powerful paradigm for aligning generative models with human preferences. At its core, the generative model's policy (0 e) is updated using reinforcement learning (e.g., PPO (Schulman et al. 2017)) to maximize the reward predicted by reward function ro(c, 2o), with the optimization objective of:
max Re Re (æle) (c, ar)] 3Drr (p)l
Here, Prer( (e) is fixed reference policy and is hyperparameter controlling the FKL.
Direct Preference Optimization. The solution to the above objective Eq. can written as:
p = Z(c) Peer Prerexp(p rraee
(2)
where Z(c)= = En Prer(aule) exp (8 (c,, xo)) is the partition function. However, directly opti- mizing this objective challenging, as necessitates a separate reward model ro xo) and relies on computationally expensive, often unstable, algorithms for policy updates. To circumvent these complexities, DPO reparameterizes the reward function directly in terms of the policy, allowing for end-to-end optimization without explicit reward model:
r(c, mn) = 3 log (o(ol))) Pre((zolc)
+ 3 log ((c),
(3)
DPO grounds its formulation in the Bradley-Tere (BT) model. Given a preference pair (xl 2/p c), where ag and wg is the "winner" and "loser" samples, the probability of ax > as f is:
PBr(# > =//lc) a (r(c, mx) r(c,,)))
where a is the sigmoid function. Substituting Eq. into Eq. that trains the policy po to satisfy these preferences:
DPO derives a direct loss function
[oog
Pe (xx (c) (Blg
B log po (a Peetl (ar (elle) )
(o - Eang ont e)-D
(5)
Peet (x (c)
Diffusion-DPO. To apply DPO to diffusion models, the parameterized distribution po (c) is not tractable, therefore, Diffusion-DPOO (Wallace al. (024) utilize the evidence lower bound (ELBO) and define R(cc, Co.r) as the reward on the whole diffusion trajectory (#),..., ær))
r(c, vp) Epo
[s).e) [R(c, (o.T))
The FKL regularization term in Eq. I is replaced with its upper bound EELL (xo. (e)oorr/e))rreeeeeee (2o.T)))))) Through algebraic manipulation, the objective of Diffusion- DPO is formulated as:
LDPO Diffission (0) Ex at DDD log σ [E By pr (a log po(2 log po (10b.))
Prerl (x Peet (
4

---

Published as a conference paper at ICLR 2026
SP OP Date (3)
OPO (Plp 1953
OFD 20N1
Backhone
Flip rate
CLIPT
HPSv21
PS
IR1
Aest
Pretrained
0.3142
25.06
20.66
0.09535
5,450
SD1.5
10%
0.3152
25.92
20.92
0.3031
5.586
20%
0,3146
25.86
20.89
0,3015
5.606
30%
0.3102
25.11
20.75
0.2205
5.468
Rats 47%
40%
0.3143
24.92
20.62
0.1289
5.421
Pretrained
0.3240
28.20
21.99
0.7234
5,932
10%
0.3312
29.43
22.38
0.9424
5,960
SDXL
20%
0,3310
29.12
22.27
0,9102
5,940
30%
0.3260
28.50
22.05
0.8649
5.926
0.3256
28.77
22.00
0.7308
5.921
Figure 2: Qualitative DPO results SDXL with different label flipping.
Table DPO results with different label flipping.
by approximating the reverse process with the forward process, this loss can finally be simplified to per-step alignment loss across the diffusion trajectory. This enables the diffusion model to be optimized directly on pairwise preference data.
DPO Loss as FKL Divergence. As shown before, the objective of preference optimization is to minimize the distances between pa and p". Therefore, we can explicitly define it as a distribution matching problem: Let po (e) pa (m(oorle)) "Pter(to:r (c)=-P and
p" (xorTle) x Pret exp(r(xe:T, c)). We have the Diffusion-DPO objective Eq. 7 as:
CDPO-Diffusion EE=D [DKL [0" (xoor|c)) (xo:r|c)]]]
This objective aims to optimize po (3o:rle) towards p" (aprrlc) which in terms is equivalent with optimizing Po towards p". The full proof can be found in the Appendix. A.2
3.2 a- RPO: ROBUST DIFFUSION PREFERENCE ALIGNMENT VIA DIVERGENCE
We first analyze why the FKI. divergence underlying Diffusion-DPO intrinsically sensitive to noisy preferences. Motivated by this limitation, we then introduce the a-divergence, which pro- vides a principled trade-off between mass-covering and mode-seeking behaviors. Building on this foundation, reformulate Diffusion-DPO under a-divergence obtain more noise-robust ob- jective, termed a-DPO. Finally, we propose a dynamic a scheduling strategy that adaptively ad- justs the divergence parameter during training, further enhancing robustness to varying noise levels. Noise Sensitivity of FKL Divergence. As il- lustrated in Fig. 3 the FKL divergene's'' "mass- covering" property is susceptible to noise. This vulnerability stems from its severe penalty for instances where ("(xorr() assigns negligible probability regions where po (o (c) has a small mass. Experiments results can be found Fig. 2 and Tab. [ To address this, introduce the a-divergence (Eq. 9). which balances mass-covering and mode-secking via a, improving robustness to noisy data and en- abling more targeted alignment (Fig. 3
Figure 3: Distributions learned from noisy data (blue) under FKL divergence (left) and a- divergence (right). The FKL tends to fit the noisy distribution closely, while a-divergence yields distribution closer to the noise-free case.
o-Divergence. The a-divergence offers continuous spectrum of divergences that measures the difference between two distributions, parameterized by a & 0, 1. For two probability distributions P and Q, the a-divergence defined
{
((())
1-a
P(x)
(P(l))) = a(a 1)a Ben
Q(x) - -a1-a) Q(x)
as a + 1, a-divergence simplifies to D. - (PII),,) while a - 0, D. + (QL(Q) The hyperparameter fundamentally controls the relative weighting of errors when Q(x) is small versus when P(x) is small, thereby influencing the divergence's 's mode-seeking or mass-covering behavior, which is crucial for tailoring divergence behavior in the presence of noise. Please Fig.
a-DPO: Preference Alignment via a-Divergence. Given these observations, we propose our noise-aware learning objective, a-DPO. This optimization objective replaces the FKI. with more
5

---

Published as a conference paper at ICLR 2026
noise-robust a-divergence to align with general human preferences. According Eq. 2 we can rewrite the reward model as R(c, Tv.r) = log (poor Pog (orrlc)) By defining the rela- tive log ratio of log Po (Zo:T(c) (c) - log Pier Prer(Zo:T as go (c, Zn:r), the target can be simplified as:
1 )1-aaa - (1 a)
p pou
- )
Da IF" || po] End a1(x-1) 1) pdo
(10)
Zao (0,2o.T) exp (e To.r). However, the partition functions are intractable. By using the Monte Carlo estimation, we assume there are K images for every prompt c, and we can approximate the distribution of p" and yoo as:
where p"
Z, (2orT) exp r(c, T..T) and poo(e.arr)
p' ((](( c)) = exp (r(c, {x)=)):))
(11)
Σラ= exp (r(c, ( (
exp (go(e, (x):r
nd (il(foo: c))
(12)
Σラ
K exp go(c, (
In our setting, the preference dataset is composed of pair-wise data (a a c), i..,, K 2. Thus, the final objective La-Dro can be written as:
LO-DPO = BOoDDEE trppie aa (w-11- (1-a)u" - a)
(13)
where u = (go(c, ) - (c, )) Following Diffusion-DPO (Wallace et al. 2024 recall
that the pa c) is evaluated over the full diffusion trajectory To.T. apply Jensen 's inequality to evaluate the upper bound of pa (To.T, c) and approximating the reverse process pe p2 2t (2)) using the forward distribution ((F1:T (o). This yields:
a(aa 1) u.(1 (0) (1-a)u -0) -0)
D.DDPO = TE (a anf (..,,-,(0 a prgq (z)"), 2i~g(ai|=0)) (m))
(14)
the u0) is defined as:
u0( ( |1E" - ==(""""",)) t))1 -|E"" (xor t)ll
(15)
- -(1l' col { - I|e' - ener(ad, 0113) 1
xi'), at q(>/(=b). and X is the signal to noise ratio. w()e) is the fixed weighting function. T is the maximum training timesteps. The full derivation can be found in the Appendix A.2
with e NN(O, I). U(O,T), 2t"r
Dynamic Scheduling. The a-divergence, parameterized by a, encompasses a broad spec- trum of divergence measures. A crucial char- acteristic this family its heightened sensi- tivity to discrepancies in the distributional tails as the absolute value of a increases. This in- herent property also implies that the divergence exhibits varying sensitivities to noise, depend- ing on the chosen a. Consequently, relying on fixed a value may not consistently yield optimal performance, particularly when con- fronting diverse noise levels. To address this limitation, we propose a novel mechanism for dynamically adjusting the value. Our ap- proach integrates information about the current sample's 's potential noise level to guide this adjustment. To achieve this, introduce the func- tion f(x"". x', c) = Stop Grad u, (f)), which does not participate the gradient backpropagation process, as auxiliary indicator quantify the noise level computed from each preference sample.
HHP99 (pairwiis))
681
6005
Pairwise Marsin O0g den-deysyy
Figure 4: Comparison the value of f(a", x', against the AHPSv2 score. f(x", x e) exhibits a strong monotonic correlation with AHPSv2, and could act effective internal confidence signal for guiding the dynamic adjustment of a.

---

Published as a conference paper at ICLR 2026
f(=", x², c) as implicit preference classifier. Given target minimizing the Eq. 14 analyze the behaviour of the gradient of La-DPO with respect to uy uf))). The gradient is calculated as:
Vu, = (a (a- [
= (1 (1-a)
U1 1-a
(16)
Given that the constrains are 0 < < and 0 < u < 1, we have La. < 0. This in- dicates that Vu, CPD decreases monotonically, Consequently, to minimize the loss La-DPO. the model is guided to maximize the u. (0) (f""",,', c)), thereby effectively increasing the likelihood of the preferred sample (x²") over the rejected sample (x'). This indicates that the auxiliary indi- implicit preference classifier, where its value is analogous to
cator f(x",, x², c) could act confidence score, with a higher score indicating a superior preference alignment. Furthermore, we validate this implicit classification role by comparing the value of f(x" xx' .)) against the reference A APPSv2 score in each sample pair (Singh et al. 2024). As shown in Fig. [xx" xt, c) exhibits strong monotonic correlation with AHPSv2, confirming that the f(x x c) is suitable for act as an effective internal confidence signal for guiding the dynamic adjustment of
Therefore, we leverage the prediction of the fine-tuned model simultaneously specify a as: a uf("", xx' c) where is dynamically adapted: higher a is assigned when samples achieve a larger confidence score, thereby indicating less noisy sample. This adaptive strategy allows our model to maintain robust performance across varying data quality.
controlling the scale of a. This process ensures that
EXPERIMENTS
Table 2: Quantitative comparison with other backbones on Label Flipping dataset. The Flip rate is 20%. For more results, please refer to the Appendix. A.4.2 The best results are in highlighted bold and the second-best ones are underlined (same in the following tables).
SD1.5
SDNL
1390
CLIP?
0,3142
0.3146
0.3139
0.3155
0.3112
0.3183
0.3240
0,3310
0.3247
0.3278
0.3304
0,3312
Pick Tal
HPSv21
PST
20.66
25.06
30,89
25.86
20.59
25.15
35.51
20.00
20.51
23.62
21.00
26.69
21.59
28.20
22.27
20.12
22.14
28,53
22.17
28.37
22.30
29.12
30,35
22,50
IKh
0,08355
0.3015
0,2393
0,1916
0.7629
0.4812
0.7334
0.910
03568
0.8119
0,9211
1.001
5.589
5.604
S961
CLIP?
0,3086
0.3122
0.3100
0,3089
0.3073
0.3132
0.3197
0,3249
0.3228
0.3236
0.3251
0,3270
PickowPic Pic
Valldation
HPS-21
PST
IR
0,1635
24.99
20.60
02642
2054
25.88
0.2127
25.66
2080
0.1861
2061
25.45
0.2617
25.57
20.64
0.3281
20.96
26.54
0.6770
21.89
28.02
0.0013
22.24
20.55
0,8095
22.06
28.57
0.279
22.09
28.54
0.9157
29.45
22.28
0,9450
38.18
22,89
Aze
5.478
5.567
5.509
5.563
5.573
5.682
5.938
5.963
5.970
5.945
5,921
5.971
0.3192
0.3419
HPSx21
24.68
25.49
25.11
25.56
25:21
26,38
28:26
29.75
29.01
29.01
29.85
36.64
HPSv2
PST
20.91
20.43
20.12
20.99
20.32
21.24
22.53
22.81
22.70
22.72
22.64
22.98
IR1
0.1583
0,2913
0.2912
0,2720
0.3104
0.4492
0.868]
1.002
0.7544
0.9900
1,012
1.123
Aza"
5.634
5.647
3.522
5.631
5,604
5,698
6.082
6.083
6.062
6.073
6.064
6.098
Table 3: Comparison with other baselines with models trained the Pick-a-Pic V2 dataset.
Donaset
Metric
Frstrained
DPO
-DFO
SD1.5
1500
Ouas
Freuaincd
D20
SOXL
1UPO
D.D
Ouu
(1IPP
0.3143
0.3158
0.3150
0.3823
0.3931
0.3122
HPSx21
26.35
29.77
Test
PS1
IRT
0.99335
20.66
0.3805
0.3637
21.18
0.3369
20/33
0.9725
1,030
Ars"
5,490
5.578
5,663
5.559
5,000
5,962
5,972
CI IPt
0.3886
0.1131
0.3111
0.3135
0.3082
0.3152
0.3197
0.3285
0.3235
0.3271
0.3078
0.328)
Pick
Valicesion
PS1
24.99
26.24
21.0
26.20
25.61
21.05
27.29
29.73
22.36
22.35
30.10
22.41
36.64
5.475
5,565
3,650
5.533
5.965
5,954
CLIPT
0,3252
0.3292
0.3248
0.3246
0.3208
0.3302
0.3386
0.3442
0.3439
0.3436
0,3483
0,3451
HT
24.68
25.50
26.65
26.55
25.75
27.68
28.26
30,05
30.53
30.68
30.22
31.42
HPS
PS1
20.91
21.33
21.30
21.29
21.44
21.72
22.53
22.94
23,000
22.23
23.04
0.1585
0.3638
0.4480
0.4299
0.4204
0U6991
0.8681
1.887
1.138
E.133
1.116
1.17%
6.084
6.149
4.1 EXPERIMENT SETUP
Dataset and Baselines. Following Diffusion-DPO Wallace al. 2024), select Stable Diffusion v1.5 (SD1.5) and Stable Diffusion XL Base 1.0 (SDXL) as the base models. We adopt Diffusion- DPO (Wallace al. 2024) and robust alignment methods: conservative DPO (cDPO) (Mitchell)
7

---

Published as a conference paper at ICLR 2026
robust DPO (rDPO) (Chowdhury al., 2024) and Hölder-DPO (H-DPO) (Fujisawa al. 2025 for comparison. We train all the methods on the Pick-a-Pic v2 (Kirstain et al. (023) Dataset. After excluding the 12% of pairs with ties, the final training data is composed of 851293 pairs of preference data, with 58960 unique prompts.
2023
Implementation Details. Training is distributed across 8 NVIDIA H100 80GB GPUs, with each GPU having a local batch size of 4, and we set the gradient accumulation over 64 iterations to achieve the desired global batch size 2048 (same Diffusion-DPOO As models are trained with square resolutions (512 for SDI.5 and 1024 for SDXL) with set to 0.9999. A linear warmup of 200 steps is used for all methods. Finally. for a-DPO trained on SD1.5, we set 3 2000 with a learning rate of 4.2- 10- For a-DPO trained on SDXL, we set 5 = 4000 with a learning rate of 1.7 10-8 We train all the baseline methods following their official guidelines. For more details of the baselines, please refer to the Appendix. A.3
Evaluation Details. We evaluate the performance of our a-DPO from two perspectives: (1) Label Flipping: We conduct experiments on a dataset where of the labels are flipped, is the flip- ping probability. These validate the effectiveness of our method compared with baselines in noise- awareness. (2) Real World Benchmark: Despite rigorous multi-round cleansing, current real-world datasets inevitably retain substantial amounts of noisy data. Therefore, we evaluate our proposed method and existing baselines on the real-world dataset, i.c., Pick-a-Pic V2, to demonstrate our approach's 's superior ability to effectively mitigate the influence of noise, highlighting its significant practical relevance. To ensure fair comparison, we fix all the inference settings, where the guidance scale is 7.5 for SDI.5, 5 for SDXL, and the sampling step is 50. We include five automatic evalu- ation metrics evaluate the performance of a-DPO text-image alignment, general preference, and aesthetic preference: (1) CLIP (Radford et al.) (02T), (2) HPS V2 (Wu et al. 2023). (3) Pick Score (Kirstain et al., 2023). (4) ImageReward (Xu et al., 2023). (5) and Aesthetic Score (Schuh- mann et al.) 2022). For final testing, we generate images using captions from the HPSv2 (3200 captions) benchmark, PartiPrompt (Yu et al., 2022). along with 500 unique test captions and 500
unique validation captions sourced from the Pick-a-Pic V2 dataset.
4.2 EVALUATION ON SYNTHETIC DATASET
We started by testing our method's effectiveness synthetic label-flipped samples. To do this, we randomly selected a specific proportion of training samples and inverted their preference labels. This synthetic dataset was then used to fine-tune SD1.5 and SDXL models with our proposed a- DPO and other baseline methods. This preliminary experiment allows us to verify our method's robustness towards non- non-prevagling preferences. We rigorously evaluated our method and existing approaches across varying label flipping rates of 0.1, 0.2, 0.3. and 0.4. As shown in Tab. [ we show quantitative results when setting the flipping rate to be 0.2 (More results please refer to the Appendix. A.4). The results demonstrated that our a-DPO consistently yields significantly stronger performance compared to models fine-tuned with standard DPO at the same label-flipped level. Furthermore, method outperforms cDPO and H-DPO across variety of metrics. While rDPO achieves excellent performance in the noisy domain, our method surpasses it by a substantial margin. This suggests that the characteristics of non-prevailing preferences differ from the simplified noise models often assumed, highlighting the unique advantages of our approach.
Winning Rate with different E. As shown in Tab. 4 our a-DPO consistently outperforms all other methods across the CLIP, HPS V2, Pick Score, and ImageReward metrics. With a flipping rate of 0.2, our method maintains high winning rate of 82.6% on HPS V2 and 76.6% on Pick Score. In contrast, methods such as H-DPO and rDPO experience significant performance degradation in most test cases. Even at a higher flipping rate of 0.4, a-DPO continues to outperform SDXL, achieving a 70.6% winning rate on HPS V2 and 62.0% on ImageReward. In comparison, most other methods fail to exceed 50% winning rate against SDXL.
4.3 EVALUATION ON REAL- WORLD DATASET
While experiments with synthetic flipped labels offer initial insights, they do not fully capture the real-world value of our method due fundamental differences between the distribution of synthet- ically generated noise and that of label flipping data arising from actual annotation processes. To address this, we directly fine-tune pretrained models on the Pick-a-Pic V2 dataset using our proposed
8

---

Published as a conference paper at ICLR 2026
Own
Ql: Oversl Preference
QEI Vauel Aftroctneness
QJ Text Aigrmenn
Preteiined
bPg
<OPO
10FO
IADRO
lege
SD1.5
weering lede hakding glazz
onine with oyen halr powerfll
tecntiful white eutfit. extrenaly danalled
SDXL
Shoppoth, helding onlley
Figure 5: Top, human evaluations, our method shows superior performance over DPO on SDXL. Bottom, qualitative comparison with other baselines with models trained Pick-a-Pic V2 Dataset.
11112
1111/
1/11
1410
1/11
1111
6-0
--0.1 0.1
5- 0.2
E- D.3
6-0.4
Figure 6: Winning rate (%) comparison for all methods versus SDXL on Pick-a-Pic V2 Test Dataset. From left to right, the winning rate of models trained on Pick-a-Pic V2 Dataset with label flipping rate E 0, 0.1, 0.2.0.. 0.3, 0.4.
a-DPO and various comparison methods. The results are presented in Tab. 3 Our method achieves better text-image alignment, general preference alignment, and aesthetic preference alignment.
For clearer comparison, in Fig. we present a qualitative comparison of generated images from various methods and backbones. Our approach consistently yields superior image quality compared to Diffusion-DPe
finding corroborated by our quantitative results. Specifically, when utilizing the SD1.5 backbone, our fine-tuned model generates correct semantics with significantly richer de- tails in both human subjects and backgrounds. Furthermore, for the SDXL backbone, our method demonstrates enhanced performance over other techniques, showcasing improved text alignment alongside superior aesthetic qualities. This qualitative assessment unequivocally highlights the ad- vanced capabilities and robustness of our proposed method.
4.4 ABLATION STUDY
We conduct ablation studies to evaluate the effect of the μr, a, and the proposed Dynamic Schedul- ing on SDXL. Experimental results are shown in Tab, 4
Ablation on Parameter The value of μ decides the robustness of our a-DPO against noise. We choose E 0.9999, 0.8, 0.5, 0.3, 0.1 for the ablation study. We observe that as μ decreases, the alignment objective would overly emphasize the alignment of main modes, neglecting some other important information, which weakens the alignment precision.
Ablation of the Dynamic Scheduling, Our Dynamic Scheduling adaptively adjusts the noise- aware parameter a. As evidenced Tab. in the 2nd row, disabling this strategy results a substantial performance degradation.

---

Published as a conference paper at ICLR 2026
Table Ablation on hyperparameters jr and & and the Dynamic a Strategy, evaluated on the Pick-a- Pic Test dataset with the fine-tuned SDXL model. (1) Effect of μc: As μ grows, model performance first increases then decreases. (2) Effeet of the Dynamic a Strategy (Fixed-a): Without dynamic allocation of the a, the model performance degrades significantly. (3) Effect of dynamic start step: the increasing of starting step slightly decrease the performance. (4) Effect of 3: As / in- creases, model performance first increases and then decreases.
f
0.9999
0.8
0.5
0.3
0.1
0.9999
0.8
0.5
0.3
0.1
22.51
22.45
22.42
22.36
22.31
IR
1.054
1.032
1.025
1.021
1.013
Fixed-cx
0.9999
0.8
0.5
0.3
0.1
0.9999
0.8
0.5
0,3
0.1
PS
22.45
22.41
22.42
22.31
22.29
IR
1.019
1.018
1.014
1.002
0,9933
dynamic
0
50
100
150
200
50
100
150
200
a starts
PS
22.51
22.46
22.45
22.47
22.44
IR
1,054
1.047
1.048
1.041
1.031
8 2000
3000
4000
5000
6000
2000
3000
4000
5000
6000
PS
22.28
22.38
22.51
22.43
22.31
IR
1.022
1.031
1.054
1.042
1.002
Ablation on the starting step of Dynamic a Scheduling. To investigate the impact of when to activate the proposed Dynamic a Scheduling, we conduct an ablation study by varying the starting step at which the scheduling replaces the fixed a. Specifically, we evaluate the model performance when Dynamic a Scheduling is introduced at the 0-th, 50-th, 100-th, 150-th, and 200-th training steps. This allows to analyze how the timing of the transition from fixed dynamic influences the final results. The results are shown in Tab. 4 As the dynamic scheduling start step increases, the model performance exhibits a slight decline.
Ablation on Parameter B. As the quantitative results demonstrated in Tab, starting from 4000, increasing 3 makes the KL penalty become overly restrictive, limiting the model's optimization. On the other side, as B decreases, the model learn to overly emphasize the reward, leading to a performance drop.
Table 5: Combining our a-DPO with other DPO variants. SPO" denotes our reimplementation results. Our a-DPO acts com- plementary component, which successfully integrates with SPO and significantly boosts its efficacy.
DPO Variant Integration. Fundamentally, our a-DPO is formulated robust en- hancement to the core DPO optimization mechanism, de- signed specifically to improve resilience against data noise. This architectural compatibility allows a-DPO serve as "plug-and-play" module that can be seamlessly integrated into other DPO-based frameworks. To demonstrate this versatility, we examine methods such as SPO (Liang et al.) 2025). which enhance alignment by incorporating an auxiliary Reward Model (RM) prior into standard DPO training (conceptualized as RM + DPO). By substituting the conventional DPO component in these frameworks with our method, we construct an enhanced training paradigm: RM + a-DPO. We present numerical comparisons on SDXL in Tab. [.] which confirm that this integration yields distinct performance gains over the original baselines. These results validate that a-DPO is not only effective as a standalone method but also capable of generalizing and significantly boosting the effectiveness of other DPO variants.
Metries SPO
CLIPt
HPSv21
PSt
IR1
Aest
0.3178
0.3227
22.89
1.107
6.366
SPO*
0.3119
0.3208
22.86
1.116
6.417
SPO + Ours
0.3201
0.3231
22.89
1.134
6.501
CONCLUSION
In this work, we investigated the vulnerability of existing DPO formulations to noisy human feed- back, showing that their reliance on FKL divergence makes them intrinsically sensitive to noise. To address this challenge, we introduced a-DPO, which reformulates preference alignment through a-divergence and thereby achieves a principled balance between mass-covering and mode-seeking behaviors. Furthermore, we proposed a dynamic a scheduling mechanism that adapts to the esti- mated noise level of each sample, providing stronger robustness during training. a-DPO achieves superior reliability in preference alignment for diffusion models across both synthetic and real-world datasets over previous baselines.
10

---

Published as a conference paper at ICLR 2026
6 ETHICS STATEMENT
We acknowledge the ethical considerations associated with the development and application text- to-image models, especially in the context of preference alignment. Our work prioritizes the align- ment of these models to ensure that generated images reflect socially beneficial, fair, and inclusive preferences. We are mindful of the potential for bias in generative models and the harm that could arise from their misuse. To mitigate these risks, we have focused on approaches that promote posi- tive, non-harmful applications the technology.
REPRODUCIBILITY STATEMENT
We have made substantial efforts to ensure the reproducibility of our work. novel algorithms and theoretical results are thoroughly explained in the main text, with detailed proofs provided in the Appendix. A.2 For experiments, we include comprehensive descriptions of the computational resources, parameter settings, and experimental protocols in both the main text and Appendix, A..3 Additionally, the source code and datasets used this study, along with any relevant data processing steps, will be made available at Appendix. to support reproducibility. We encourage readers to refer to these resources to replicate our results.
8 ACKNOWLEDGMENTS
This work is supported by the National Key Research and Development Program of China un- der Grant No. 2024YFF0907202 and the National Natural Science Foundation of China (NSFC) under Grant 62372452, Jiangsu Provincial Science and Technology Major Project under Grant BG2024042, and the Joint Research Project on Integration of Culture and Science and Technology between Chinese Academy of Sciences and Hunan Provinee(
REFERENCES
Mohammad Gheshlaghi Azar, Zhaohan Daniel Guo, Bilal Piot, Remi Munos, Mark Rowland, Michal Valko, and Daniele Calandriello. A general theoretical paradigm to understand learn- ing from human preferences. In International Conference on Artificial Intelligence and Statistics, pp. 4447 4455. PMLR, 2024.
Kevin Black, Michael Janner, Yilun Du, Ilya Kostrikov, and Sergey Levine. Training diffusion models with reinforcement learning. arXiv preprint 33X1 2023.
Jie Cheng, Gang Xiong, Xingyuan Dai, Qinghai Miao, Yisheng Lv, and Fei- Yue Wang- Rime: Robust preference-based reinforcement learning with noisy preferences. arXiv preprint arXiv:2402.17257, 2024.
Sayak Ray Chowdhury, Anush Kini, and Nagarajan Natarajan. Provably robust dpo: Aligning language models with noisy feedback. arXiv preprint 2rXiv:2433.00409, 2024.
Xun Deng, Han Zhong, Rui Ai, Fuli Feng, Zheng Wang, and Xiangnan He. Less more: Improving Ilm alignment via preference data selection. arXiv preprint 2r02v:2502. 14560, 2025.
Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, et al. Scaling rectified flow transformers for high-resolution image synthesis. In Forty-first international conference machine learning, 2024.
Ying Fan, Olivia Watkins, Yuqing Du, Hao Liu, Moonkyung Ryu, Craig Boutilier, Pieter Abbeel, Mohammad Ghavamzadeh, Kangwook Lee, and Kimin Lee. Dpok: Reinforcement learning for fine-tuning text-to-image diffusion models. Advances in Neural Information P Processing Systems, 86.79858-79883 2023.
Masahiro Fujisawa, Masaki Adachi, and Michael A Osbore. Scalable valuation of human feedback through provably robust model alignment. arXiv preprint arXiv:2505. /7859, 2025.
11

---

Published as a conference paper at ICLR 2026
Aritra Ghosh, Naresh Manwani, and PS Sastry. Making risk minimization tolerant to label noise. Neurocomputiing 160:93-1011 107, 2015.
Dongyoung Go, Tomasz Korbak, Germán Kruszewski, Jos Rozen, Nahyeon Ryu, and Marc Dymet- man. Aligning language models with preferences through f-divergence minimization. arXiv preprint arXiv:2302.0822/5 2023.
Aman Gupta, Shao Tang. Qingquan Song, Sirou Zhu, Jiwoo Hong, Ankan Saha, Viral Gupta, Noah Lee, Eunki Kim, Siyu Zhu, et al. Alphapo: Reward shape matters for I1m alignment. arXiv preprint arXiv:2501.00 03884, 2025.
Jiaqi Han, Mingjian Jiang, Yuxuan Song, Stefano Ermon, and Minkai Xu. f-po: Generalizing preference optimization with f -diieergence minimization. arXiv preprint arXiv:2410. 2024.
Jiwoo Hong, Sayak Paul, Noah Lee, Kashif Rasul, James Thome, and Jongheon Jeong. Margin- aware preference optimization for aligning diffusion models without reference. In First Workshop on Scalable Optimization for Efficient and Adaptive Foundation Models, 2024.
Rohit Jena, Ali Taghibakhshi, Sahil Jain, Gerald Shen, Nima Tajbakhsh, and Arash Vahdat. Eluci- dating optimal reward-diversity tradeoffs in text-to-image diffusion models. In 2025 IEEE/CVF Winter Conference on Applications of Computer Vision ( (WACV), pp. 232-242. IEEE, 2025.
Yuval Kirstain, Adam Polyak, Uriel Singer, Shahbuland Matiana, Joe Penna, and Omer Levy. Pick- a-pic: An open dataset of user preferences for text-to-image generation. Advances in neural information processing systems, 36:36652-36663, 2023.
Keyi Kong, Xilie Xu, Di Wang, Jingfeng Zhang, and Mohan S Kankanhalli. Perplexity-aware cor- rection for robust alignment with noisy preferences, Advances in Neural Information Processing Systems, 37:28296-28321, 2024.
Black Forest
Labs.
Inference repository. flux 2024. Accessed: 2024-10-25.
Flux:
https://githut: com/
black-
Sangkyu Lee, Janghoon Han, Hosung Song, Stanley Jungkyu Choi, Honglak Lee, and Young- jae Yu. KI penalty control via perturbation for direct preference optimization. arXiv preprint vrxiv:2502.13177 2025.
Shufan Li, Konstantinos Kallidromitis, Akash Gokul, Yusuke Kato, and Kazuki Kozuka. Align- ing diffusion models by optimizing human utility. Advances in Neural Information Processing Systems, 37:24897-24922. 2024.
Xize Liang, Chao Chen, Jie Wang, Yue Wu, Zhihang Fu, Zhihao Shi, Feng Wu, and Jieping Ye. Robust preference optimization with provable noise tolerance for 1lms. CoRR, 2024.
Zhanhao Liang, Yuhui Yuan, Shuyang Gu, Bohan Chen, Tiankai Hang, Mingxi Cheng, Ji Li, and Liang Zheng. Aesthetic post-training diffusion models from generic preferences with step-by- step preference optimization. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 13199-13208, 2025.
Yunhong Lu, Qichao Wang, Hengyuan Cao, Xierui Wang, Xiaoyin Xu, and Min Zhang, Inpo: Inver- sion preference optimization with reparametrized ddim for efficient diffusion model alignment. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 28629-28639, 2025.
Eric Mitchell. note on dpo with noisy preferences & relationship to ipo, 2023.
Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion models for high-resolution image synthesis. arXiv preprint arXIv:2307.01995, 2023.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp- 8748-8763. PmLR, 2021.
12

---

Published as a conference paper at ICLR 2026
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in neural information processing systems, 36:53728-5333 53741, 2023.
Shibani Santurkar, Esin Durmus, Faisal Ladhak, Cinoo Lee, Percy Liang, and Tatsunori Hashimoto. Whose opinions do language models reflect? In International Conference on Machine Learning, pp. 29971-30004, PMLR, 2023.
Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, al. Laion-5b: An open large-scale dataset for training next generation image-text models. Advances in neural in- formation processing systems, 35:25278-252999 2022.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv: 1707.00777 2017.
Zhao Shan, Chenyou Fan, Shuang Qiu, Jiyuan Shi, and Chenjia Bai. Forward kl regularized pref- erence optimization for aligning diffusion policies. Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pp. 14386-14395, 2025.
Xiangwei Shen, Zhimin Li, Zhantao Yang, Shiyi Zhang, Yingfang Zhang. Donghao Li, Chunyu Wang, Qinglin Lu, and Yansong Tang. Directly aligning the full diffusion trajectory with fine- grained human preference. arXiv preprint arXiv:2509.0 06942, 2025.
Kunal Singh, Mukund Khanna, and Pradeep Moturi. Effective text-to-image alignment with quality aware pair ranking. In NeurIPS 2024 Workshop on Fine-Tuning in Modern Machine Learning: Principles and Scalability, 2024.
Hwanjun Song, Minseok Kim, Dongmin Park, Yooju Shin, and Jae-Gil Lee. Learning from noisy labels with deep neural networks: A survey. IEEE transactions on neural networks and learning systems, 34(1 1):8131-81553 2022.
Ziang Song, Tianle Cai, Jason D Lee, and Weijie Su. Reward collapse in aligning large language models. arXiv preprint arXiv:2305.17600 2023.
Bram Wallace, Meihua Dang. Rafael Rafailov, Linqi Zhou, Aaron Lou, Senthil Purushwalkam, Stefano Ermon, Caiming Xiong, Shafiq Joty, and Nikhil Naik. Diffusion model alignment using direct preference optimization. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 0228-8238, 2024.
Chaoqi Wang, Yibo Jiang, Chenghao Yang, Han Liu, and Yuxin Chen. Beyond reverse kl: Gen- eralizing direct preference optimization with diverse divergence constraints. arXiv preprint arXiv:2309.16240, 2023.
Fu- Yun Wang, Yunhao Shui, Jingtan Piao, Keqiang Sun, and Hongsheng Li. Diffusion- npo: Nega- tive preference optimization for better preference aligned generation of diffusion models. arXiv preprint arXiv:2505. /245, 2025.
Junkang Wu, Xue Wang. Zhengyi Yang. Jiancan Wu, Jinyang Gao. Bolin Ding, Xiang Wang, and Xiangnan Hc. a-dpo: Adaptive reward margin is what direct preference optimization needs. arXiv preprint arXiv:2410. 10148 2024a.
Junkang Wu, Yuexiang Xie, Zhengyi Yang. Jiancan Wu, Jinyang Gao, Bolin Ding, Xiang Wang, and Xiangnan He. 8-dpo: Direct preference optimization with dynamic B. Advances in Neural Information Processing Systems, 37: 12994441299966, 2024b.
Xiaoshi Wu, Yiming Hao, Keqiang Sun, Yixiong Chen, Feng Zhu, Rui Zhao, and Hongsheng Li. Human preference score v2: A solid benchmark for evaluating human preferences of text-to- image synthesis. arXiv preprint arXiv:2300.00000 09000, 2023.
Jiazheng Xu, Xiao Liu, Yuchen WL, Yuxuan Tong, Qinkai Li. Ming Ding, Jie Tang, and Yuxiao Dong. Imagereward: Learning and evaluating human preferences for text-to-image generation. Advances in Neural Information Processing Systems, 36: 1595555 2023.
13

---

Published as a conference paper at ICLR 2026
Kai Yang. Jian Tao, Jiafei Lyu, Chunjiang Ge, Jiaxin Chen, Weihan Shen, Xiaolong Zhu, and Xiu Li. Using human feedback to fine-tune diffusion models without any reward model. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 8941-89511 2024.
Jiahui Yu, Yuanzhong Xu, Jing Yu Koh, Thang Luong, Gunjan Baid, Zirui Wang, Vijay Vasudevan, Alexander Ku, Yinfei Yang, Burcu Karagol Ayan, et al. Scaling autoregressive models for content- rich eoxttt--imaee generation. arXiv preprint arXiv:2206.10789, 2(3):5, 2022.
Huizhuo Yuan, Zixiang Chen, Kaixuan Ji, and Quanquan Gu. Self-play fine-tuning of diffusion models for text-to-image generation. Advances in Neural Information Processing Systems, 37: 73366-73398, 2024.
Zefeng Zhang. Hengzhu Tang, Jiawei Sheng, Zhenyu Zhang, Yiming Ren, Zhenyang Li, Dawei Yin, Duohe Ma, and Tingwen Liu. Debiasing multimodal large language models via noise- pref- erence optimization. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 9423-9433, 2025.
Huaisheng Zhu, Teng Xiao, and Vasant G Honavar. Dspo: Direct score preference optimization for diffusion model alignment. In The Thirteenth International Conference on Learning Representa- tions, 2025.
A
APPENDIX
This supplementary material provides additional details as follows.
Use of Large Language Models.
Proofs and Derivations. We present a theoretical analysis of all the theorems and our method.
Experimental Details. We provide experimental details for the baselines.
More Analysis & Future Work. We provide more quantitative and qualitative results for o-DPO fine-tuned on different base models and discuss the future work of a-DPO.
14

---

Published as a conference paper at ICLR 2026
A.1 USE OF LARGE LANGUAGE MODELS
Large Language Models (LLMs) were used solely for minor grammar correction and stylistic pol- ishing of the manuscript text. They were not involved in the design of the methodology, execution of experiments, analysis of results, or any other aspect of the scientific contribution.
A.2 PROOFS AND DERIVATIONS
A.2.1 PROOFS OF EQ.
In this section, we provide detailed proof of Eq. showing that DPO's optimization objective is equivalent to minimizing the forward KL Divergence. Following f-PO Han et al. (2024), starting from the definition of the DPO loss and combining with Eq. 3, we have:
exp (rle, (2).:):))
LDro BewDBrac(EooTT
K 25=1 exp (r(c, foor bj))
(enor exp (3 log (a(( Jule)
(17)
ned (
log
K Eラm exp B log
pul (u(f rilei parl (2oer ())))
Here, using the Monte Carlo estimation of Eq. 11 and Eq. 12, the distribution of the target and optimized distribution are:
K
exp (r(c, (xoT):))
p (i) (1o.T) (c)
(18)
دز= exp (r(e, (xoTT)
cxp
Blog pulfwrir do tole) pail (2000 hile)
(0 (KoT)ix (c) log
(19)
Σ exp (3 log
po 1 lslo) Par P (sle)
r(0) is the optimized distribution parameterized by 0. With the definition of Po (a x pe(zorlle) (re((Zo:)l)) 1-8, we can extend the dpo loss to:
exp exp(r(e,(0o.r1) (xor)))
(D) Ee- BeenDBpartt Bpar((
r Fi:K)
== exp (r(c, (zorT))))
(20)
(o kile
log
Pial ( llle)
Pul [wrrr kylc)
2==1 1=
Peil( (mn c)(
when K + co, we have Σx= exp go (2orr, Ic)/K KK = Efei exp (xp.o(xarrlc))
Note that
(o(c, Vo.r) = log po (x log Pre (No.T c)), then we have:
K po (o(oTT)
K,
(21)
Peerl (Zo:r |
K
exp (r({Zu:r)i [e) = Epa((sarle) lexp (r(#ur)) c))]
(22)
KK(c),
15

---

Published as a conference paper at ICLR 2026
here, Z(c) is the partition function. Putting the above to the Eq. 20
obtain:
exp(r(20rr c))
(Dro()) BEND
Prer(aor: (c)
((c)
Bo.TED
Po Forr
c)
log
p (O.T
c)
(23)
Po (xo.
BeND
Σ
p² p' c) log
D' (oorr
BewD [DL (Ze(
c) || p² (* c))l
which finishes the proof.
A.2.2 DIVERGENCE IS NOT EQUIVALENT TO A WEIGHTED FKL DIVERGENCE
We illustrate the inequality by comparing the generator function f(u) of these two divergences. The generator function of weighted FKL divergence is fFKL(u) x W loog u (w is the reweighting (u* (-1---())))-a)) (Wang et al. (a(a-1))
factor), while for a divergence, its generator function is fu( x 2023). Unlike ad-hoc FKL reweighting. the a divergence structurally reshapes the nonlmearity of the underlying generator function, fundamentally altering how deviations from the target density penalised. For example, for two samples 21, I2 with corresponding ú2 = u/. here in and U2 are the density of and 22 respectively. And we have:
(u) (u (1 a)uu aa)
w log 11
(24)
(a(a - 1))
we can obtain
- log 22 log [u] 1-a - (1 11-0mm - a)
(25)
(a(a 1))
On the other hand,
(u² a- (1 - 01)u2 at) (ng (1 - ox)ut - a)
(26)
(a(a 1))
(a(a 1))
When wo and a are fixed, the equality:
(u) (1-a)u - a) (u - (1 -1-a aa) (a(a (1)
(27)
(a(a 1))
does not hold for wide range of This illustrates that a-divergence introduces qualitatively different regularisation.
Conceptually, conventional DPO remains oversensitive to label noise due to the inherently mass- covering behaviour of the forward KL divergence. Even when applying reweighted KL formu- lations, this tendency persists, leaving the model vulnerable to interference from noisy data. In contrast, the a-divergence inherently promotes mode-seeking behaviour, effectively reducing the impact of noisy labels. This fundamental distinction motivates replacing the divergence formulation itself, rather than merely adjusting the weight of the forward KL term.
A.3 EXPERIMENTAL DETAILS
A.3.1 DETAILS OF BASELINES
1. Diffusion-DPO Walllac al. 2024). We use their released official checkpoint.
2. Conservative DPO Mitchell 2023). We train the cDPO with learning rate of /3 2.002.048
10 8 with a linear warmup of 200 steps. We set 3 = 2000 and the contamination ratio as
0.1 for SD1.5. For SDXL,, set 3 5000 and the contamination ratio 0.1.
16

---

Published as a conference paper at ICLR 2026
3. Robust DPO (Chowdhury al. 2024). We train the rDPO with learning rate of 2000 2.048 10 with a linear warmup 200 steps. We set 3 = 2000 and the con- tamination ratio as 0.1 for SD1.5. For SDXL, we set B = 5000 and the contamination ratio 0.1.
8
4. Holder-DPO Fujjiaaw et al., 2025). We train the H-DPO with a learning rate of 2000 2.048 10 with linear warmup of 200 steps. We set 8 2000 and the =
8
for SD1.5. For SDXL., we set / = 5000 and y=1
5. Diflusion-KIT (Li et al. 2024). We their released official checkpoint.
6. Diffusion-SPIN Yuan et al. 2024). We their released official checkpoint.
7. SPO (Liang et al. 2025). We use their released official checkpoint and the released of- ficial code to retrain the model. For training the enhanced SPO + Our model on SDXL, we adhered to the standard official hyperparameters provided by SPO, with two specific exceptions: B = 3 and our a-DPO introduced hyperparameter μ = 0.9999.
A.3.2 DETAILS OF DATASETS
1. Pick-a-Pic V2 (Kirstain et al. 2023) Dataset. Pick-a-Pic is comprehensive dataset of over 500,000 real user preferences, derived from more than 35,000 unique prompts submit- ted via web platform. Each entry includes a text prompt. two Al-generated images, and user feedback on their preferred image or cases with no clear preference. The collection features outputs from various generative models, such as Stable Diffusion 2.1, Dreamlike Photoreal 2.05, and Stable Diffusion XL.
2. HPSv2 /Wu et al. 2023) Dataset. Human Preference Score v2 (HPS v2) is large-scale and comprehensive benchmark for text-to-image models. The dataset comprises 798,090 human preference choices 430,060 pairs of images. This makes it one of the largest datasets of its kind. Its test set is made up of 3200 unique prompts, which are used to generate controlled set of images for evaluation. The prompts in the rized into four distinct styles to ensure a comprehensive evaluation of model's generative abilities: Animation, Concept-art, Painting, and Photo.
are catego-
3. PartiPrompt YYu al. 2022) Dataset. PartiPrompts is a rich set of over 1600 prompts that can be used to measure model capabilities across various categories and challenge aspects.
A.3.3 DETAILS OF USER STUDY
For the human evaluation, we randomly sampled set of prompts and used each method to generate 40 groups of images. We then recruited 40 participants, who were asked evaluate the generated images along three dimensions:
(I) Which image is your overall preferred choice? ? (2) Which image more visually attractive? (3) Which image better matches the text description?. In the human evaluation, we utilize diverse and comprehensive set of prompts. The complete listing of prompts used generate the images is:
1. Anime art featuring Hatsune Miku with symmetrical shoulders.
2. A lifelike anime girl in steel plate armor takes a selfie in a castle courtyard.
3. A Rika Furude plush toy.
4. An anime girl, masterpiece, good line art, trending in pixiv,
5. a dog riding bicycle.
6. two girls-sisters in dark heavy techno armor with red-shine eyes, long black hair, futuristic helmet on head, full body view, gorgeous face, Cute face, cantasy-style, highly detailed, VFX, 4k, ultrarealistic photo, realistic-style photo.
7. two horses running in field in the foggy day time.
8. Eldrich Old God Shoggoth, holding smiley face.
9. nice lake in austria mountains.
17

---

Published as a conference paper at ICLR 2026
10. dark elf armor of world of warcraft, mistique background, ultra detail, full body shot, full body portrait, detailed, professional photo, Cinematic lighting, Cinematic, Color Grading.
11. floral textile A letter.
12. diamond ring on girls hand.
13. Two fake looking giraffes are on display at an exhibit.
14. Spongebob wearing Rick Owens clothing in an avant garde fashion look on r/Streetwear.
15. Man at park.
16. Photorealistic beautiful African girl age 25.
Koa muscular adventurer with a rugged tan complexion, piereing blue eyes, and weathered face. He sports full beard and wide-brimmed hat. For his latest adventure, rides atop the back of magnificent flying turtle with a sleek shell and majestic wings. The turtle soars through stumning sky filled with fluffy white clouds and vibrant colors. Koa grips tightly to the turtle's shell.
18. car that is made of wood.
19. full body, slender anime girl with long cyan hair, powerful arcane wizard, beautiful white outfit, extremely detailed, realistic shading.
20. Walter white lego.
21. Danny DeVito appears in Jojo's Bizarre Adventur.
22. A hyper-realistic 3D render of the Doge meme featuring a shiba inu, portrayed in cinematic lighting.
23. Harry Potter book cover with Van Gogh-inspired design and visible book title.
24. Anthropomorphic orange crocodi.
25. Two cats playing chess on a tree branch.
26. She has heart-shaped face, large expressive eyes, and soft, lustrous hair. Her figure is slender and graceful, and she moves with quiet confidence and inner strength that com- mands attention. But what truly sets her apart is her inner beauty, which radiates from within and lights up any room she enters.
27. stop sign with a large tree behind it.
28. Batman wearing metal gear armor with a cinematic, dramatic background.
29. A sign that says "PICK A PIC".
30. 3D digital illustration, Burger with wheels speeding on the race track, supercharged, de- tailed, hyperrealistic, 4K.
31. Spinger Spaniel liver and white.
32. a building sized muscular bear.
33. dinossaur rpg character, planet with the volcanic landscape, concept art, smooth fog, high quality, symmetry in objects.
34. A cat jumping for toy.
35. A Sign that says: Free Candy!
36. cyberpunk mutant cute blond girl.
37. flying crocodile.
38. beautiful portrait of young woman made of glossy glass skin surrounded with glowing birds.
39. pikachu in a pinstripe suit.
40. logo of a blue elephant, flat modern vector icon.
18

---

Published as a conference paper at ICLR 2026
Table 6: Human Evaluation Robustness Analysis. This result strongly demonstrates the statistical robustness the proposed method in hu- man evaluation.
In Fig. leverage hu- man evaluation to compare the generated results of our method and baselines. The results depiet that method shows superior per- formance on "Text Align- ment", "Visual Attractive- ness". and "Overall Preference". To ensure that our human evaluation statistically robust, we additionally calculate pairwise comparisons of the human evaluation results among SDXL, DPO, and a-DPO, and compute the corresponding p-values. The results in Tab. reveal that our a-DPO significantly outperforms both DPO (p < 0.001) and SDXL. (p < 0.001) in terms of human pref- erence. The small p-value (p < 0,001) indicates that the observed preference difference has less than 0.1% probability of occurring by random chance. This outcome strongly demonstrates the statistical robustness of the proposed method in human evaluation.
SDXL vs DPO SDXL vs a-DPO DPO a-DPO
P 1.000c+00
5.002c-09
2.961e-9
SDXL
DPO
Ours
rDPO
Hölder DPO
Ours
Overall Preference
Q1: Overall Preference
Q2: Visual Attractiveness
Q2: Visual Attractiveness
Q3: Text Alignment
Q3: Text Alignment
Figure 7: User Study. We conduct user studies to evaluate the performance of method and the baselines. On the left block, we compare our a-DPO with SDXI. and DPO. On the right block, compare a-DPO with rDPO and Hólder-DPO. Our method shows superior performance "Text Alignment", "Visual Attractiveness", and "Overall Preference".
A.3.4 COMPUTATION CoST
We compare the computational cost of our method with the other baselines. Our method needs the same time and GPU consumption as the DPO and the other baselines. In Fig. & we show the training curve of them. Our method does not need extra training cost while showing better performance.
A.3.5 EXPLAINATION OF PICKSCORE
As shown Tab. I as the label-fliping rate in- creases from 10% to 40%. PickScore exhibits con- sistent downward trend. Although its absolute change appears small (e.g. from 22.38 22.00), this is a property of the metric's scale rather than an indi- cation of robustness. The qualitative degradation in the generated images, such as reduced text alignment and weaker visual appeal, is clearly visible in Fig. which confirms that PickScore is correctly reflecting a decline in preference alignment, albeit within a narrow numeric range. To show that this limited PickScore change in amplitude does not signify robust performance, but rather intrinsic failure of its scoring mechanism to be sensitive to underlying quality signals. Here, we leverage the Coefficient of Variation (CV) analyze the property of PickScore. This analysis stems from
Table 7: The Insufficient Amplitude Change in PickScore a Symp- tom of its Low Intrinsic Sensitivity. The PickScore's low CV (23.12% demonstrate that the metric operates with narrow scor- ing range. In contrast, ImageReward exhibits an extremely high CV (188.24%), which indicates sparse and highly selective scoring mechanism.
Metries
CV(%)-2000
CV(%)-50000f
PickScore
23.12
22.65
ImageReward
188.24
189.05
19

---

Published as a conference paper at ICLR 2026
Plot of Second vs Third Element
<DPO
0.70
DPO
6:DPO
0.85
Third Elament (Valua)
0.60
0.55
0.50
300
400
500
800
Second Element (Index)
Figure Training curve comparison of the used methods.
Figure 9: show the PickScore and the ImageReward of pairs of samples. Although PickScore exhibits smaller amplitude fluctuations, it follows the same trend as ImageReward.
experiment conducted 2, 000 and 50000 randomly selected image pairs from the Pick-a-Pie Test Dataset, where scores were normalized to compute the CV for comparative analysis. The results depicted in Tab. 7 We have also provided visualizations of both PickScore and ImageReward outputs in Fig. g which offer immediate intuitive support for the following statistical observations: (1) The PickScore's low CV (23.12%) demonstrates that PickScore operates with narrow scoring range relative to its mean. This results in an inherent insensitivity (low amplitude) to changes in sample quality (the observed small amplitude to the model's fundational scoring philosophy). (2) In contrast, ImageReward exhibits significantly sharper score change under varied sample quality.
A.4 ADDITIONAL EXPERIMENTAL RESULTS
To further demonstrate effectiveness of our method, we provide additional qualitative and quan- titative evaluations with other methods.
A.4.1 QUALITATIVE RESULTS
provide more visualization results. Com- pared with other baselines. our tine-tuned model generates significantly richer details in both human subjects and backgrounds, showcasing improved text alignment alongside superior aesthetic quali- ties. This qualitative assessment unequivocally highlights the advanced capabilities and robustness of our proposed method.
In Fig. Fig Fig 12 Fig. and Fig. 15
20

---

Published as a conference paper at ICLR 2026
A.4.2 QUANTITATIVE RESULTS
Other label flipping ratio. We provide more experimental results of our method and the other baselines under label flipping rate 0.1, 0.3, and = 0.4 on SDXL. The results are demonstrated in Tab. TO Tab. [T] and Tab. 12 respectively, our method consistently outperforms the other methods across a wide range of flipping rates. We also notice that, trained on dataset with & 0.1, our method even outperforms the Diffusion-DPO trained on
Comparison with other SOTA alignment methods. To further validate the effectiveness of our method, we conducted comprehensive comparison with additional state-of off-th--rt approaches for preference alignment on real-world datascts, reported in Tab. 8 and Tab, These baselines include Diffusion-KTO (Li et al., 2024), Diffusion-SPIIN (Yuan et al. 2024). and MAPO (Hong al. 2024). Our method consistently achieves superior preference alignment across metrics. In particular, on the Stable Diffusion 1.5 model, our approach significantly outperforms or matches all competing methods. This demonstrates that method is not only highly robust to noise but also achieves superior alignment performance on real-world datasets compared to other dedicated alignment methods.
A.4.3 FUTURE WORK & LIMITATIONS
(I) While a-DPO demonstrates clear advantages noise robustness and consistently surpasses ex- isting baselines, it introduces additional hyperparameter μ that controls the strength of dynamic a-scheduling. Designing an automated or more adaptive mechanism for tuning μi represents a valu- able direction for future work. (2) We have preliminarily explored combining a-DPO with SPO, and we believe that developing more principled integration strategies in this direction offers substantial potential for future improvements.
Table 8: Quantitative comparison with other baselines with SD15 trained on the Pick-a-Pic V2 dataset.
SD1.5
Dataset
Metric
Pretrained
DPO
KTO
SPIN
Ours
CLIPt
0.3142
0.3176
0.3193
0.3095
0.3184
HPSv21
25.05
26.11
27.09
28.05
27.59
Pick-a-Pic Test
PSt
20.66
21.02
21.13
21.18
21.27
IRt
0.09535
0.3050
0.6123
0.4901
0.5831
Aest
5.596
5.678
5.700
5.937
5.702
CLIP
0.3086
0,3131
0.3160
0.3057
0.3152
HPSv2t
24.99
26.24
27.02
27.33
27.29
Pick-a-Pic
PSt
20.60
21.02
21.09
21.36
21.15
Validation
IRT
0.1035
0.3191
0.6041
0.4632
0.5337
Aest
5,478
5.587
5,709
5,823
5.692
CLIP
0.3252
0.3292
0.3322
0.3205
0.3302
HPSv2t
24.68
25.89
28.71
28.21
27.68
HPSv2
PSt
20.91
21.33
21.44
21.32
21.44
IRT
0.1585
0.3628
0.5321
0.5474
0.6391
Aest
5,634
5.749
5.712
5,702
5.782
21

---

Published as a conference paper at ICLR 2026
Table 9: Quantitative comparison with other baselines with SDXL trained on the Pick-a-Pic V2
dataset.
Dataset
Method
CLIP
HPSv2t
PSt
IRt
Aest
Pick-a-Pic
MaPO
0.3251
29.18
22.07
0.853
5.932
Test
Ours
0.3326
30.86
22.51
1.054
5,972
Pick-a-Pic Validation
MaPO
0.3225
29.11
22.00
0.8596
5,874
Ours
0.3289
30.64
22.41
1.041
5.983
MaPO 0.3408
29.41
22.64
0.9844
6.072
HPSv2
Ours
0.3451
31.42
23.04
1.178
6.140
Table 10: Quantitative comparison with other backbones on Label Flipping dataset with SDXL. The
Flip rate is 10%. The best results are in highlighted bold and the second-best ones are underlined.
SDXL
Dataset
Metric
Pretrained
DPO
cDPO
IDPO
Н-DPO
Ours
CLIP
0.3240
0.3312
0.3302
0.3307
0.3315
0.3320
HPSv21
28.20
29.43
29.51
29.85
29.69
30.57
Pick-a-Pic Test
PSt
21.99
22.38
22.35
22.41
22.37
22.50
IR
0.7234
0.9424
0.9466
0.9641
0.9324
1.021
Aest
5.932
5.960
5.933
5.966
5.926
5.971
CLIP
0.3197
0.3272
0.3263
0.3269
0.3260
0.3273
HPSv21
28.02
29.53
29.41
29.67
29.64
30.33
Pick-a-Pic
PSt
21.89
22.32
22.27
22.35
22.33
22.40
Validation
IRt
0.6770
0.8973
0.8775
0.9137
0.9312
0.9710
Aest
5.933
5.963
5.924
5,961
5,915
5.962
CLIPt
0.3223
0.3280
0.3271
0.3277
0.3279
0.3281
HPSv2t
27.24
28.69
28.73
28.93
29.12
29.87
PartiPrompt
PSt
22.39
22.64
22.69
22.72
22.71
22.80
IRt
0.7489
1.010
1.017
1.047
1.052
1.129
Aest
5,705
5.765
5.723
5,764
5.747
5.819
CLIP
0.3386
0.3437
0.3432
0.3440
0.3448
0.3444
HPSv2t
28.26
29.95
29.82
30.07
30.02
30.99
HPSv2
PSt
22.53
22.85
22.90
22.96
22.73
23.05
IRT
0.8681
1.041
1.047
1.081
1.084
1.133
Aest
6.082
6.082
6.042
6.085
6.050
6.123
22

---

Published as a conference paper at ICLR 2026
Table 11: Quantitative comparison with other backbones on Label Flipping dataset with SDXL. The Flip rate is 30%. The best results in highlighted bold and the second-best ones underlined.
SDXL
Dataset
Metric
Pretrained
DPO
cDPO
rDP
H-DРO
Ours
CLIP
0.3240
0.3260
0.3253
0.3221
0.3259
0.3271
HPSv21
28.20
28.50
27.15
27.19
28.51
28.70
Pick-a-Pic
PSt
21.99
22.05
21.97
21.90
21.99
22.12
Test
IRt 0.7234
0.8649
0.7862
0.7163
0.8871
0.9213
Aest
5.932
5.926
5.924
5.925
5.862
5.965
CLIPt
0.3197
0.3222
0.3200
0.3192
0.3217
0.3247
HPSv21
28.02
28.35
27.91
27.83
28.38
28.62
Pick-a-Pic Validation
PSt
21.89
22.02
21.91
21.85
21.93
22.10
IR 0.6770
0.8037
0.7057
0.7006
0.8352
0.8762
Aest
5.933
5.921
5.932
5,957
5,894
5.973
CLIP
0.3223
0.3245
0,3245
0.3238
0.3260
0.3272
HPSv2t
27.24
27.64
27.24
27.15
27.54
27.83
PartiPrompt
PSt
22.39
22.47
22.42
22.37
22.37
22.52
IR
0,7489
0.8459
0.7941
0.7642
0.8853
0.9046
Aest
5.705
5.716
5.681
5.681
5.662
5.789
CLIP
0.3386
0.3407
0.3392
0.3389
0.3422
0.3433
HPSv21
28.26
28.59
28.14
28.11
28.51
30.15
HPSv2
PSt
22.53
22.62
22.52
22.48
22.54
22.71
IR
0.8681
0.9453
0.8911
0.8773
0.9723
1.005
Aest
6.082
6.024
6.038
6.040
6.002
6.078
Table 12: Quantitative comparison with other backbones on Label Flipping dataset with SDXL. The Flip rate is 40%. The best results in highlighted bold and the second-best ones underlined.
SDXL
Dataset
Metric
Pretrained
DPO
cDPO
rDPO
H-DPO
Ours
CLIPt
0.3240
0.3256
0.3245
0.3247
0.3231
0.3254
HPSv21
28.20
28.77
27.75
28.28
28.73
29.55
Pick-a-Pic Test
PSt
21.99
22.00
21.81
21.93
21.89
22.11
IR
0.7234
0.7308
0.7034
0.7592
0.7489
0.8530
Aest
5.932
5.921
5.941
5.930
5.941
5.963
CLIPt
0.3197
0.3202
0.3185
0.3201
0.3207
0.3221
HPSv2t
28.02
28.64
27.47
27.99
28.24
29.37
Pick-a-Pic
PSt
21.89
21.93
21.78
21.83
21.87
22.04
Validation
IRt
0.6770
0.7343
0.6462
0.6827
0.7240
0,8080
Aest
5.933
5.913
5.955
5.962
5,902
5.962
CLIP
0.3223
0.3236
0.3244
0.3236
0.3234
0.3258
HPSv2
27.24
27.72
27.91
27.26
27.93
28.47
PartiPrompt
PSt
22.39
22.43
22.41
22.36
22.38
22.45
IRt
0.7489
0.8326
0.8482
0.7709
0.8007
0.8703
Aest
5.705
5,746
5.806
5.686
5.770
5.762
CLIP
0,3386
0.3383
0,3383
0.3384
0.3399
0.3405
HPSv2t
28.26
28.32
27.95
28.46
28.43
29.89
HPSv2
PSt
22.53
22.58
22.40
22.50
22.48
22.60
IR
0.8681
0.8912
0.8410
0.8785
0.9195
0.9627
Aest
6.082
6.046
6.056
6.067
6.042
6.072
23

---

Published as a conference paper at ICLR 2026
Pretrained
DPO
Halry Potter
cDPO
rDP
Hats Poller
H-DPO
HARRY POTTER
Ours
Figure 10: Images generated by different models (based on SDXL) for various prompts. From left right, the prompts are follows: "Danny DeVito appears Jojo's Bizarre Adventure.", "A hyper-realistic 3D render of the Doge meme featuring a shiba inu, portrayed in cinematic lighting.". "A photorealistic Bob Odenkirk sitting under with smiling anime girl with black hair and hime cut in digital art anime key visual.". 4. "Harry Potter book cover with Van Gogh-inspired design and visible book title.".
24

---

Published as a conference paper at ICLR 2026
Pretrained
DPO
cDPO
rDPO
H-DPO
Ours
Figure 11: Images generated by different models (based on SDXL) for various prompts. From left right, the prompts are follows: 1. "A lifelike anime girl in steel plate armor takes selfie in a castle courtyar''. 2. "Two fake looking giraffes are on display at an exhibit", 3. "An anime girl, masterpiece, good line art, trending in pixiv,,", 4. "A diamond ring on girls hand".
25

---

Published as a conference paper at ICLR 2026
Pretrained
A
A
DPO
cDPO
rDPO
H-DPO
A
Ours
Figure 12: Images generated by different models (based on SDXL) for various prompts. From left to right, the prompts are as follows: 1. "full body, slender anime girl with long cyan hair, powerful arcane wizard, beautiful white outfit, extremely detailed, realistic shading". 2. "two horses running field in the foggy day time". 3. "Floral textile A letter", 4. "dark elf armor of world of warcraft, mistique background, ultra detail, full body shot, full body portrait, detailed, professional photo, Cinematic lighting, Cinematic, Color Grading".
26

---

Published as a conference paper at ICLR 2026
Pretrained
DPO
cDPO
rDPO
H-DPO
Ours
Images generated by different models (based on SDXL) for various prompts. From left right, the prompts are follows: 1. "Man park", 2. "Anthropomorphic orange crocodile". 3. "Koa is a muscular adventurer with rugged tan complexion, piercing blue eyes, and weathered face. He sports full beard and a wide-brimmed hat. For his latest adventure, he rides atop the back of a magnificent flying turtle with sleek shell and majestic wings. The turtle soars through a stunning sky filled with fluffy white clouds and vibrant colors. Koa grips tightly to the turtle's shell.", " 4. A car that is made of wood.".
Figure 13:
27

---

Published as a conference paper at ICLR 2026
Pretrained
DPO
cDPO
rDPO
H-DPO
Ours
Figure 14: Images generated by different models (based SD15) for various prompts. From left
right, the prompts are as follows: 1. "A pencil draw portrait of a grey haired 30 years old vampire with an extended goatee beard, trending on artstation.". 2. "Two cats playing chess on a branch.", ", 3. "She has a heart-shaped face, large expressive eyes, and soft, lustrous hair. Her figure is slender and graceful, and she moves with quiet confidence and inner strength that commands attention. But what truly sets her apart is her inner beauty, which radiates from within and lights up any room she enters.". 4. "Cute guy, Asian popular pop hair style, Anime character.".
28

---

Published as a conference paper at ICLR 2026
Pretrained
STOP
DPO
STOP
cDPO
TT
STOP
rDPO
TO
STOP
H-DPO
TP TO019
Ours
STOP
Images generated by different models (based on SD15) for various prompts. From left to right, the prompts cyberpunk setting, with intricate and highly detailed elements crafted in digital painting.". 2. "a stop sign with a large tree behind it.", "A highly detailed digital portrait of the mythological figure Hephaestus, featuring a fantasy interpretation inspired by D&DD and well-known French rugby player Sebastien Chabal, as illustrated by artists Artgerm, Greg Rutkowski, and Magali Villeneuve and gaining popularity on ArtStation.", 4. "Batman wearing metal gear armor with cinematic, dramatic background."
Figure 15:
follows: 1. "The image depicts anime woman in futuristic
29


Reviews:
[{"review_id": "wiUeGYZiVO", "reviewer": "Reviewer_AiHs", "summary": "This paper introduces α-DPO, a noise-robust variant of Direct Preference Optimization (DPO) for aligning diffusion models with human preferences. The authors identify the sensitivity of standard Diffusion-DPO to label-flipping noise and propose replacing the Forward KL divergence with α-divergence, along with a dynamic α-scheduling mechanism. The method is evaluated on both synthetic and real-world datasets and shows consistent improvements over existing baselines.", "questions": "1. Have you experimented with fixed α values across a wider range? \n2. How does dynamic scheduling compare to the best fixed α?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This work is to systematically address noise robustness in preference alignment for diffusion models, with a solid theoretical grounding in α-divergence.\n2. The connection between DPO and FKL divergence is clearly established, and the motivation for using α-divergence is well-justified.\n3. Extensive experiments on SD1.5 and SDXL with varying noise levels demonstrate the effectiveness of α-DPO. The method consistently outperforms strong baselines across multiple metrics.", "weaknesses": "1. The paper lacks a thorough exploration of α values beyond the dynamic scheduling setup. A fixed-α baseline with a sweep could better contextualize the contribution of the scheduling mechanism.\n2. The human evaluation is based on only 20 participants and 40 prompts, which may not be statistically robust.\n3. There is a lack of comparison with many baselines.  like [1]SPO(SD1.5 SDXL), [2]SmPO(SD1.5 SDXL),  [3] MaPO(SD1.5 SDXL). These baselines (checkpoints) are open-sourced.\n [1]Aesthetic Post-Training Diffusion Models from Generic Preferences with Step-by-step Preference Optimization. CVPR2025\n [2] Smoothed Preference Optimization via ReNoise Inversion for Aligning Diffusion Models with Varied Human Preferences. ICML 2025\n [3] Margin-aware Preference Optimization for Aligning Diffusion Models without Reference", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "B6mJMrLxdB", "reviewer": "Reviewer_8b67", "summary": "This paper addresses the sensitivity of diffusion model to noisy labels in T2I. The paper first shows that the DPO objectives are equivalent to minimizing the Forward KL divergence, thus explaining the sensitivity of DPO to noise. The paper then proposed an alpha-DPO method aimed to alleviate the above problem. The alpha in the proposed method can be adjusted in a data-aware way, giving the training process more control over noise tolerance. The method was tested on both synthetic and real world datasets.", "questions": "Please see weaknesses.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "The main strength is to use alpha-DPO to bypass the over-sensitivity in conventional DPO, as revealed by the FKL analysis. Figure 3 illustrates the difference between the FKL and alpha-divergence in covering noisy distribution, which may account for a small amount of data but cause big penalty in regions with small masses. Addressing this point is a strength of the paper. Results are good.", "weaknesses": "A major weakness is that to address the over-sensitivity to noise in conventional DPO, the proposed alpha-divergence incurs hyperparameter such as the mu on line 304 to tune, this increases the complexity of the method. \nThe essence of the alpha-divergence is a weighted control of the KL measurement, which is good but novelty is not particularly high. \nIt is not clear why Monte Carlo approximation, starting at line 257, would be a good approximation to p^r and p^g(theta)? Can authors give some reasoning behind this statement.\nWhat is the meaning of Figure 4? Can the authors elaborate?\nFrom line 301 to 303, how is f(x^w, x^l, c) maximized?\nFor the ablation study of Table 4, it seems there was no meaningful difference in the results for different mu, alpha, beta, and the starting point of dynamic alpha scheduling. Does it mean there was little impact on how to choose these parameters. \nWriting can be better as some abbreviations were defined more than once.\nWhile line 250 refers to equation 2, there is no equation 2.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "da1nkInNPn", "reviewer": "Reviewer_kScQ", "summary": "This paper proposed the modified DPO algorithm with $\\alpha$-divergence for diffusion model alignment based on the learning dynamics analysis on the plain DPO's implicit reward. With simple modification, the authors test their method, $\\alpha$-DPO, on varying axes of datasets, models, and metrics, consistently outperforming previous methods.", "questions": "1. Is it still within the range of “noise robustness” when the downstream evaluation results remain unchanged across 10–40% label flipping?\n   * In Tables 2 and 5–8, the evaluation scores exhibit little to no variation as the label-flipping ratio increases. Intuitively, if the flipping ratio rises to 40% (nearly half of the 1M Pick-a-Pic V2 dataset), one would expect a gradual decline in human alignment metrics. For instance, PickScore, which originates directly from the Pick-a-Pic V2 authors, should display a noticeable drop. Moreover, if the proposed $\\alpha$-divergence formulation demonstrated the least degradation under increasing noise, that would provide strong empirical evidence that $\\alpha$-divergence genuinely mitigates label noise. It is therefore questionable whether maintaining nearly constant evaluation scores across 0–40% flipping can be interpreted as true robustness, since the underlying preference distribution has substantially changed by flipping the labels while the reported outcomes remain unaffected.\n\n2. Typos in Tables 5-8?\n   * The explanations and captions in Appendix A.4.2 state that the results correspond to SDXL, whereas the tables themselves label the backbone as SD 1.5. Could these be typographical errors, or do the results indeed correspond to SD 1.5?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. The empirical scope of the paper is extensive, with different models (SD 1.5 and SDXL), a lot of baselines (Diffusion-DPO, SPIN, KTO, …), and metrics/datasets.\n2. The evaluation results demonstrate that $\\alpha$-divergence is superior in nearly every case mentioned in the paper, shown through the direct win rate and average evaluation scores.\n3. Although it is not a novel approach, he motivation for using $\\alpha$-divergence is straightforward (see weakness for the details).", "weaknesses": "The main weakness of the paper lies in the **novelty and completeness of the divergence analysis on DPO and $\\alpha$-divergence as the solution, which serves as the conceptual foundation of the work**. The paper’s discussion of DPO through the lens of divergence minimization and the proposed adoption of $\\alpha$-divergence as a solution overlooks several important prior works that have already analyzed these aspects in detail:\n\n1. Theoretical analyses of DPO as divergence minimization (Section 3.1) [2, 4].\n\n2. Adoption of $\\alpha$-divergence in DPO-style formulations (Section 3.2) [1–3].\n\nA few previous studies, highly relevant to this paper’s core idea, have already investigated how different divergences articulate the implicit reward under the DPO framework, including the explicit use of $\\alpha$-divergence [1–3]. While [1] and [2] directly address $\\alpha$-divergence and its policy-based variants, [3] generalizes DPO to a broader family of $f$-divergences that naturally subsume $\\alpha$-divergence. None of these works is cited or discussed in the current manuscript.\n\nGiven that this paper also emphasizes the “nice properties” derived from adjusting $\\alpha$ within the $\\alpha$-divergence family, these prior studies are non-concurrent and directly relevant for establishing the paper’s originality and theoretical positioning. Even considering the domain difference (language model and diffusion model), these references are essential for clarifying what is newly contributed here versus what is conceptually inherited.\n\n(Additional potential weaknesses are further elaborated in the Questions section.)\n\n&nbsp;\n\n**References**\n\n[1] Wu et al., 2024, “α-DPO: Adaptive Reward Margin is What Direct Preference Optimization Needs.” (Preprint)\n\n[2] Gupta et al., 2025, “AlphaPO: Reward Shape Matters for LLM Alignment.” (ICML 2025)\n\n[3] Wang et al., 2024, “Beyond Reverse KL: Generalizing Direct Preference Optimization with Diverse Divergence Constraints.” (ICLR 2024)\n\n[4] Shan et al., 2025, “Forward KL Regularized Preference Optimization for Aligning Diffusion Policies.” (AAAI 2025)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)