Diffusion Negative Preference Optimization Made Simple
Authors: Joshua Tian Jin Tee
Hee Suk Yoon
Sunjae Yoon
Tri Ton
Chang D. Yoo
Venue: ICLR 2026 Poster (2026)
Keywords: Diffusion Models
Text-to-image generation
Preference Alignment
Primary area: generative models

Abstract: Classifier-Free Guidance (CFG) improves diffusion sampling by encouraging conditional generations while discouraging unconditional ones. Existing preference alignment methods, however, focus only on positive preference pairs, limiting their ability to actively suppress undesirable outputs. Diffusion Negative Preference Optimization (Diff-NPO) approaches this limitation by introducing a separate negative model trained with inverted labels, allowing it to capture signals for suppressing undesirable generations. However, this design comes with two key drawbacks. First, maintaining two distinct models throughout training and inference substantially increases computational cost, making the approach less practical. Second, at inference time, Diff-NPO relies on weight merging between the positive and negative models, a process that dilutes the learned negative alignment and undermines its effectiveness. To overcome these issues, we introduce Diff-SNPO, a single-network framework that jointly learns from both positive and negative preferences. Our method employs a bounded preference objective to prevent winner-likelihood collapse, ensuring stable optimization. Diff-SNPO delivers strong alignment performance with significantly lower computational overhead, showing that explicit negative preference modeling can be simple, stable, and efficient within a unified diffusion framework. Code will be released.

Published as a conference paper at ICLR 2026
DIFFUSION NEGATIVE PREFERENCE OPTIMIZATION MADE SIMPLE
Joshua Tian Jin Tee!
Hee Suk Yoon
Sunjae Yoon²
Tri Ton!
Chang D. Yoo
Korea Advanced Institute of Science and Technology (KAIST)
=Chung-Ang University
{joshuateeti, hskyoon, tritth, yo .c.kr
jaunjaeyoonecau. .cckr
ABSTRACT
Classifier-Free Guidance (CFG) improves diffusion sampling by encouraging con- ditional generations while discouraging unconditional ones. Existing preference alignment methods, however, focus only on positive preference pairs, limiting their ability to actively suppress undesirable outputs. Diffusion Negative Prefer- ence Optimization (Diff-NPO) approaches this limitation by introducing sep- arate negative model trained with inverted labels, allowing it to capture signals for suppressing undesirable generations. However, this design comes with two key drawbacks. First, maintaining two distinet models throughout training and inference substantially increases computational cost, making the approach less practical. Second, inference time, Diff-NPO relies weight merging between the positive and negative models, a process that dilutes the learned negative align- ment and undermines its effectiveness. To overcome these issues, we introduce Diff-SNPO, single-network framework that jointly learns from both positive and negative preferences. Our method employs a bounded preference objective prevent winner-likelihood collapse, ensuring stable optimization. Diff-SNPO de- livers strong alignment performance with significantly lower computational over- head, showing that explicit negative preference modeling can be simple, stable, and efficient within unified diffusion framework. Code and models are available at whtttss::/// //qi thub. com/ T/
INTRODUCTION
Diffusion models (Ho et al. 2020) have become the backbone of modern visual content generation, achieving remarkable fidelity in synthesizing images, videos, and multimodal content (Rombach
et al. 2022 Ho al. 2022 Ruan et al. 2023 Ton et al. 2025 Hong et al. 2025a Yoon et al.
2025b). However, models trained on vast, uncurated web-scale datasets often inherit biases and fail to align with human notions of quality, aesthetics, safety. Consequently, fine-tuning models with human feedback through preference alignment has become a critical step for bridging the gap between model's raw capabilities and user intent (Black et al. 2024 Wallace et al. 2024 Li al.
2024 Tee et al.) 2025).
At the heart of high-quality diffusion sampling lies Classifier-Fre Guidance (CFG) (Ho & Sali- mans 2021). a technique that enhances sample quality by amplifying the contrast between condi- tional and unconditional (or negatively conditioned) likelihoods. This mechanism guides generation toward prompt-aligned outputs by explicitly pushing samples away from the lower-quality uncon- ditional distribution. However, challenge arises with many preference optimization methods, such as Diffusion Direct Preference Optimization (Diff-DPO). These methods often apply the same opti- mization objective to both the conditional and unconditional branches. This uniform reinforcement of preferred attributes, while shifting the overall distribution in the desired direction, fails to heighten the critical contrast that CFG relies on. Consequently, the model's ability to generate outputs aligned with user intent remains limited.
"Corresooniing Author

---

Published as a conference paper at ICLR 2026
Table 1: Comparison of methods by alignment type, model setup, and use of merging strategy.
Method
Negative Alignment
Dual Model
Merging Strategy
DPO
NPO
CHATS
SNPO
X
X
To further strengthen the contrastive effect of CFG, recent methods (Fu et al., 2025 Wang al. 2025) have incorporated negative preference alignment into the training process. In particular, CHATS (Fu al., 2025) and Diff-NPO (Wang et al. 2025) train two separate models: "posi- tive" model on standard preference data and "negative" model on inverted preferences. This setup allows the system explicitly learn from dispreferred samples and steer generation away from un- desirable attributes, resulting in improved human preference alignment. Despite their effectiveness, a key limitation of both approaches lies in their reliance on distinct sets of parameter for the two models, which often leads to misaligned outputs and complicates guided sampling (Wang et al.) 2025]. Diff-NPO addresses this by merging the weights of the two models, whereas CHATS per- turbs conditional embeddings to blend their signals. These differences summarized in Table I In this work, we focus on Diff-NPO, since its sampling procedure more closely follows standard CFG. Building this foundation, highlight two central limitations in its design. First, training and sampling from two independent models inherently doubles both computational and memory costs, creating significant scalability challenge. Second, merging weights that were optimized separately introduces a distribution mismatch that weakens the contribution of the negative model. This re- duces the method's ability suppress dispreferred outputs and ultimately limits the gains expected from negative alignment.
In this work, we ask: Can we achieve the benefits of explicit negative modeling without the costs and compromises of a dual-model architecture? We introduce Diffusion Simple Negative Prefer- ence Optimization (Diffusion-SNPOL
a framework that integrates positive and negative preference signals into a single, unified network. Our approach leverages the inherent dual-branch structure of CFG-enabled models, training the conditional branch on positive preferences and the unconditional branch negative (inverted) preferences. This design eliminates the need for separate models and weight merging, preserving a strong, explicit contrast between preferred and dispreferred distribu- tions within one efficient architecture.
However, we find that naively applying this strategy results in an unintended side effect: the gener- ated images become progressively blurrier training continues. We attribute this to the likelihood decrease observed in the DPO algorithm (Rafailov et al. 2024b Pal et al. 2024, Cho et al., 2025), which, when coupled with conflicting gradients from flipped preferences between the conditional and unconditional branches, causes instability. As a result, the model converges to a suboptimal so- lution, with blurring becoming more pronounced as the likelihood of winning samples decreases. To address this, we adapt Bounded DPO (Cho et al. (2025), preference optimization method designed increase the likelihood of winning samples during training, to Diffusion Models. This adaptation stabilizes training by preventing the loss from being dominated by low-likelihood "losing" samples, thereby boosting the likelihood of winning samples throughout preference optimization.
In detail, our contributions can be summarized as follows:
We identify the challenges of dual-model negative preference optimization (NPO), namely its high computational cost and the performance trade-off's inherent in its two separate model design.
We demonstrate that simply applying opposing preferences on different branches within a single model leads to poor interactions with DPO-based algorithms, resulting in blurry images as training progresses.
We introduce Diff-SNPO, single-model negative preference optimization method for Diffusion Models. By adapting the BDPO algorithm and deriving an upper bound, our approach effectively eliminates the blurring effeet caused by DPO in negative preference optimization setting.
We conduct extensive evaluations the Pick-a-Pic v2 benchmark with both SD1.5 and SDXL, showing that Dif delivers strong performance across multiple alignment metrics, while being more efficient in both training and inference.
2

---

Published as a conference paper at ICLR 2026
2 RELATED WORK
2.1 PREFERENCE OPTIMIZATION
Preference optimization aims to align generative models with human judgments. One common strat- egy is to train a reward model that scores prompt-inage pairs based on semantic or aesthetic quality,
and then fine-tune the diffusion model to maximize these scores (Xu et al. 2023; Clark al. 2024). Another direction draws on policy optimization, which frames denoising
sequential decision process: DDPO Black et al. 2024) applies reinforcement learning across the sampling steps, while DPOK (an et al. (2033) introduces KL-regularized reward objective. More recently, methods that bypass explicit reward modeling by learning directly from curated positives or pairwise preferences
have attracted growing interest (Wallace al. 2024 Lu et al. 2025 HHon al., 2025h) Tee al.) 2025 Yoon et al., 2025a). While these approaches simplify training, they typically handle nega- live feedback only implicitly. by training models to favor preferred outputs in relative comparisons. This indirect treatment leaves little control over explicitly suppressing undesirable generations. In contrast, explicitly modeling dispreferred outcomes offers a more direct form of control: it steers generation away from undesirable regions of the distribution, reducing artifacts and improving align- ment.
2.2 NEGATIVE PREFERENCE OPTIMIZATION
In response the limitations of standard preference optimization algorithms, recent work has plored dual-model strategies that explicitly separate positive and negative preferences. Diff-NPO (Wang al., 2025) trains negative model on inverted preferences and substitutes it for the un- conditional branch during inference, which improves alignment but introduces a mismatch between training and inference due to weight interpolation. CHATS (Fu et al. (025) addresses this issue by jointly training positive and negative models within a contrastive objective, achieving stronger alignment but at the cost of increased computation and memory from maintaining two networks. These challenges highlight the need for approaches that unify positive and negative preference mod- eling within single framework. Our method, DIFF-SNPO, takes this direction by integrating both signals into shared training objective, reducing overhead while better balancing alignment across positive and negative preferences.
3 PRELIMINARIES
3.1 DIFFUSION MODELS
Denoising Diffusion Probabilistic Models (DDPMs) (Ho al. 2020) consist of a forward noising process and a learned reverse denoising process. The forward process gradually perturbs clean data To with Gaussian noise according to a variance schedule (3
T
((11 2o) = II a(x 1z-1), g(x, 21-1) = -N 1 22-111111
(1)
t=1
To invert this process, a neural network es parameterizes the reverse transitions by predicting the injected noise:
p0 xx)) No (1 1 you Viー co( 1))
where a; =1 Br, āt IT's 1 as,, and of denotes the variance.
Model training minimizes a variational bound, which simplifies to a weighted noise-prediction loss:
CDOPM Exo.e,t (w()) ||e t)(3),
(3)
with e ~ A((0,) I) and u(1...., TT.. The weighting function w(t) controls the relative con- tribution of different timesteps, reflecting the varying difficulty of denoising across the diffusion trajectory. This objective provides a simple and stable training criterion that underpins most modern diffusion-based generative models.
3

---

Published as a conference paper at ICLR 2026
3.2 CLASSIFIER-FREE GUIDANCE
Classifier-Free Guidance (CFG) (Ho & Salimans/ 2021) modifies the conditional sampling distribu- tion strengthen (or weaken) the influence of conditioning information while remaining anchored to the unconditional model. At timestep I, the effective distribution defined as
po [c)
(z c) po(()))
(4)
Do (xee)
where 1 recovers the standard conditional model, w amplifies the effect of the condition, and 0 < w < 1 dampens it.
In diffusion sampling, the model evolves states using the score function sæ t, V my log po (Ite c). Relating equation 4 to the score yields
Sg( c) s t) + w (sa (, - ())
(5)
where so (2 t) and se (Ze, c) denote the unconditional and conditional scores, respectively. Thus, CFG moves along the direction that separates the conditional and unconditional scores, with w controlling how far we step in that direction.
Under the DDPM parameterization Ho et al. 2020 Vāx xo + VI Oie e, the score and the noise prediction are linked by
(z), VI- at
Ez( t)
(6)
Substituting equation into equation gives the guided noise estimator
Ea (Z t, c) (2e, t) + (e) c0(x;,
In essence, CFG modifies the denoiser by amplifying the difference between the conditional and unconditional predictions, with w controlling how strongly the generation is steered toward the con- ditional distribution. The unconditional prediction (0(2,, t) serves reference point, and the correction term eœ (Zt, c) e (I f) shifts the generation trajectory away from regions favored by the unconditional model but inconsistent with the conditioning signal c. This allows CFG to suppress undesirable or generic outputs that the unconditional model might produce on its own, en- couraging samples that better reflect the intended conditioning, As result, CFG offers a principled and tunable mechanism to improve text-image alignment.
3.3 DIFFUSION DIRECT PREFERENCE OPTIMIZATION
Preference alignment often formalized using the Bradley-Terry (BT) model (Kendall Smith 1940). Building on this idea, Direct Preference Optimization (DPÓ) (Rafailov et a 2023) sidesteps the need to fit an explicit reward model by defining an implicit reward through likelihood ratios between the policy лe and a reference policy Teef
C (f) -EE -) log ar (3(10g log
(
To (x( | c) [ c)) Ther c) )
Te(=h c)
log
Here, 3 > is a temperature parameter that controls the strength of the preference signal, and a(.) denotes the sigmoid function. Extending this objective to diffusion models requires assigning preferences over entire trajectories To.r- In this context, Diff-DPO (Wallace et al. 2024 derives an upper bound on the exact Diff-DPO objective:
CDir-DrO(f:
(9)
with
(c) = ||e0 co(2)) t, c) - ( ,t,,t,,)ll t.c) A/(c)= = -Ile'
(10)
co(o, L, 1) 11 I|ec
(11)
Here, 2²" and 24 are noisy states obtained by applying the forward diffusion process to the clean samples 2) = with corresponding noise terms e ~ (0, I). The label E {+1, -1} encodes the preference direction and (t) is time-dependent weighting function.
4

---

Published as a conference paper at ICLR 2026
3.4 DIFFUSION NEGATIVE PREFERENCE OPTIMIZATION
Diffusion Negative Preference Optimization (Diff-NPO) (Wang et al. 2025) extends standard pref- erence learning by explicitly modeling undesirable behavior. It does so by training on an inverted preference dataset, where the roles of winners and losers are swapped. This yields negatively aligned model, 0-, which learns to assign higher likelihood to dispreferred samples. During sam- pling, Diff-NPO replaces the standard unconditional branch in classifier-free guidance with this negative model, leading to the following guidance formulation:
ANPO(Fe: c) = <g+ (Ia, c) + w [<a- (#t,, c) - ea- (1:, Ø)]
(12)
where Eø+ is the positively aligned model trained on standard preference data, and w is a guidance strength hyperparameter.
In practice, however, the positive and negative models that are trained independently often exhibit poor correlation, which undermines the effectiveness of cg- as meaningful contrastive signal dur- ing sampling. To mitigate this, Diff-NPO applies weight-merging procedure that combines the reference model Oef. the positive model 0+, and the negative model 0-:
i- Oee a(t+ Biet) + 3(0 Oget):
(13)
where and B are interpolation coefficients. In practice, the merged parameters Ô- replace eg- in the guidance formulation of Eq. 13] yielding improved generation quality. However, this benefit comes at the cost of higher training overhead and weakened negative alignment.
METHOD
4.1 LIMITATIONS OF NEGATIVE PREFERENCE OPTIMIZATION
Dual-model Negative Preference Optimization (NPO) Wang et al. 2025) trains separate models, 0+ and 0 for positive and negative preferences. However, maintaining two networks doubles memory, compute, and training time, severely limiting scalability for large diffusion backbones.
During inference, separate models preserve preference contrast but often degrade sample quality, particularly for 0- Diff-NPO mitigates this via merged parameter formulation, Å- (Eq. 13). interpolating between 0+, 0=, and the reference backbone 0zer. While merging improves generation quality (Wang et 2025). it heavily biases the result toward 0+,, diminishing the influence of 0- (Figure D]. This fidelity-al aidelit--lignment aliggmeen trade-off reveals that relying on separate networks and post-hoc merging ultimately undermines the preference contrast NPO is designed enforce. To resolve this, the next section introduces robust, scalable single-model formulation that unifies both preference signals within a shared architecture.
4.2 ISSUES WITH NAIVE SIFF-SNPO
Standard diffusion models inherently support both con- ditional and unconditional branches within single net- work-a design popularized by Classifier-Free Guid- ance (Ho Salimans, 202T). This built-in dual-branch structure presents natural alternative to the dual-modes Diff-NPO setup: rather than maintaining two separate networks, one can leverage the existing architecture by updating the conditional branch with preferred samples and the unconditional (or negatively conditioned) branch with dispreferred ones. This approach preserves the con- trast between preference signals while avoiding the dundancy and overhead training and managing sepa- rate models.
675
WhC/wi marge
NPO weigh
65.0
0.5
77.8
26.3
Figure 1: Negative implicit accuracy (left) and HPSv2 (right) on SD1.5. Weight merging lowers implicit accu- racy while increasing reward, revealing trade-off.
Specifically, let Y E [+1,-1) be a branch label with Pr(Y +1) = and Pr(Y -1) p, where
5

---

Published as a conference paper at ICLR 2026
Naive Diff-SNPO
Diff-SNPO
Iteration
Iteration
Figure 3: Generated samples across training iterations. Naive-SNPO produces progressively blurred outputs during training, while Diff-SNPO remains stable and yields increasingly preference- aligned images without blurring artifacts.
E [0, 1] is the CFG dropout probability. The effective
conditioning is
((Y) - c, Ø,
Y = =1 (conditional branch)
[
Y
(unconditional/null branch),
where e denotes the conditioning imput (c.g.. text prompt) and indicates null conditioning, in CFG. A Naive Diff-SNPO objective can thus be constructed by adapting the Diff-DPO objective in Eq.
LNavve Diff-SNPO (0) (u .b)) Y log dYT (AT(E(Y)) (A)()))))))))))))))))))))))))))) A(((()))))
(14)
While Naive Diff-SNPO provides a simple and intuitive way to incorporate flipped preferences within single network, it exhibits substantial degradation in generation quality over training. As shown in Fig. 3 this naive objective produces increasingly blurry outputs, with reduced contrast and diminished nigh-frequency detail. We attribute this behavior to known property of the DPO objective: improvements in pairwise margin often come from decreasing the likelihood of both the loser more amongly-ratho than consistently increasing the likelihood of the winner (Pal et al., 2024] Rafailov et al. 2024b Cho et al. 2025). This trend is reflected in our results as well, illustrated Fig.
To quantify this effect on preferred samples, we track the wim-sample likelihood ratio against fixed reference model, given by:
202A
Sive
Te (xr"))
Eng. [
Tret( Tret(2 (2:"0)
595%
Under Naive Diff-SNPO, we observe a consistent decrease the relative win probability of preferred samples over the course of training (Fig. 33. indicat- ing that the model suppresses likelihoods across both branches rather than reinforcing the preferred ones. This undesirable trend, combined with the structure of the optimization signals, leads to characteristic blurring effect. Specifically, Naive Diff-SNPO ap- plies symmetric but opposing updates to the conditional and unconditional branches encouraging one to increase the likelihood sample while the other decreases it. Since both branches share model parameters, these conflicting gradients interfere with each other, and the model resolves this
1856
Win probability ratio over training. Naive-SNPO's win probability decreases during training, whereas Diff- SNPO's improves steadily.
Figure

---

Published as a conference paper at ICLR 2026
conflict by averaging them. In generative settings, this averaging dampens contrast and degrades fine detail, resulting in blurry outputs. To overcome this limitation, we seek to break the destructive symmetry in the update rule. In the next section, we introduce asymmetric preference optimiza- tion approach that biases learning toward increasing win probability, thereby mitigating gradient interference and addressing the blurring effect.
4.3 DIFF-SNPO
Recently, Cho et al. 2025) introduced Bounded DPO (BDPO) to address a key shortcoming of standard DPO. As the model reduces the probability assigned to the losing sample, the corresponding lose sample log-likelihood term, log (y: x), in the objective grows disproportionately large, causing the loss to become dominated by the loser. This skews the gradient signal and can even drive updates that decrease the likelihood of the preferred (winning) output, despite the preference label.
To mitigate this issue, BDPO replaces the losing term with mixture distribution that includes non-vanishing contribution from the reference policy:
Tmix (y x) ( x) (1 A) mrerly x),
A E (0, 1),
which leads to the modified objective
[og of [ log mo(27]e) Tin (ur (e((=())))
EBDro(To: Fiet)
=/.g"=
Tarla [c] log
(15)
This modification bounds the contribution of the loser term and prevents it from overwhelming the loss, thereby preserving the intended effect of preference optimization: promoting the winning sample. In addition, BDPO retains the same global minimizers as DPO while enforcing lower bound on the winning likelihood, offering stronger stability guarantees throughout training (Cho let al. 2025).
To adapt BDPO to diffusion models, similar to Diff-DPO, we define the trajectory- reward
re (c, x0) Ex, r-pot-xx)) R(c, Xo:T)
(16)
The corresponding Diffusion BDPO objective is
Lpit BHir-DPP((F (f) = - R (x (x y DD [Log of log
pa (x log (x )
(17)
paal (a(z)
Parl o)
Following Diff- we -ound this objective via ELBO and then apply Jensen's 's inequality to obtain the result below (see Appendix A.2 for the full derivation):
log of - (m(aff mmi (zf)(
Cpi Bimo-DPO-UR uB ()) = E D.
(18)
Here the per-step terms are:
do,, = T (t) ||C t,) C,t, = T w() ||c tar
(19)
m(z,, = do( - ea
(20)
Mmix (Zt, c) = log do(oaaan)) + (1-x)e= e
e,t,
(21)
where CDIF-BDOOUB denotes the upper bound approximation of Diff-BDPO.
Building on this, we define our final Diff-SNPO objective, which applies our introduced Diff-BDPO- UB objective into a single model negative preference optimization framework:
LSNPO (f) (x
y [oo of B (m(E"(Y)) (Y),E(Y) mimxx ( (),
(22)
where (V) is given as:
w// if = +1 (conditional branch) y = (unconditional/nuul branch).
w//l =
(23)
a!/
As shown Fig. DIPO-SNPO avoids the decline in winner likelihood observed with Naive Diff- SNPO: the estimated likelihodoo of winning samples steadily improves throughout training. In line with this, Fig. Beg.Buuriin further shows that Diff- prevents the progressive low-contrast "blurring"
artifact characteristic of the naive training objective, effectively addressing its key shortcoming.

---

Published as a conference paper at ICLR 2026
Table 2: Comparison of Ditr-SNPO with baseline methods SDL.5 and SDXL backbones using Pick-a-Pic v2. All values are reported mean + 95% confidence interval over 4 random seeds. Diff-SNPO consistently achieves the highest scores across most human preference metrics, reflecting improved alignment and visual quality. For clarity, the best-performing method in each metric shown in bold, and the second-best is underlined.
Model
Method
HPSv2
Pick Score
Aesthetic Score
Image Reward
Baseline
26.24
20.64 0.09
5.2849 0.12
0.1221 = 0.08
Diff-DPO (Wallace al.
2024
26.55 0.04
21.01 + 0.04
5.3823 0.01 0.01
0.2968 + 0.07
SD1.5
Diff-NPO Wang ct al. 2025
26.92+0.00
21.46 0.04
5.5381 0.04
0.3786 + 0.09
CHATS (Fu ct al. 2025
0.11
21.05 0.06
5.6845 0.07
0.2995 + 0.07
Diff- Diff-BDPO(ON) (Ours)
26.64
21.15 0.10
5,4457 + 0.10
0.10
0.3165 + 0.11
Diff- (Ours)
27.23
22.24 1 0.03
5.6258 0.04
0.6936 + 0.07
Bascline
0.007
0.06
5.8850
0.7605 = 0.05
Diff-DPO WWlllcc
2024
28.09 0.03
5.8884 0001
0,9841 0.07
DifI-NPO (Wang al.
2025
28.30
22.67 1.05
0.02 5,9449
0.9847 1 0.01
SDXL
CHATS Fu el al. 2025
28.25
22.34
5,8785 0.03
1.0543 1 0.03
Diff-BDPO (Ours)
28.13
2.10
5.8037 0.07
0.9946 1 0.05
Diff-SNPO (Ours)
2.03
22.69 0.06
5.8129
1.0100 0.06
5 EXPERIMENTS
5.1
EXPERIMENTAL. SETUP
Datasets and Models. We fine-tune both Stable Diffusion v1.5 (SD1.5) (Rombach et al. 2022) and Stable Diffusion XL (SDXL) (Podell et al. 2024) using proposed Diffusion SNPO objective in Eq. 22 For training, we utilize the Pick-a-Pic v2 (Kirstain et al. 2023) corpus, large-scale human preference dataset comprising 851,293 image pairs across 58,966 unique prompts.
Training Details. Models are initialized from the publicly available SDL.5 (Rombach et al. 2022)
(CreativeML. Open RAIL-M license) and SDXL (Podell et al. 2024) (MIT license) checkpoints. Training is conducted using the Adam W (Loshchilov & Hutter 2019) optimizer with learning rate of 2.048 x 10-5, We use a batch size 512 and train for 3,000 steps on SD1.5, and larger batch size of 2,048 for 625 steps SDXL. All experiments are on NVNIIDI A6000 GPUs using distributed data parallelism. Total training time is approximately 12 hours for SD1.5 and 68 hours for SDXL. The regularization coefficient is set to B 2000 for SD1.5 and 3 5000 for SDXL. Unless stated otherwise, Diff-SNPO is trained with an interpolation parameter of = 0.9.
Baselines. We compare Diff-SNPO against both negative preference optimization methods and standard alignment baselines. Specifically, we evaluate Diff-NPO (Wang et al. (225),, CHATS (Fu jet al. 2025), and Diff-DPO (Wallace et al. 2024), along with the original pretrained models for SDI.5 (Rombach et al. 2022) and SDXL (Podell et al. 2024).
Evaluation Protocol. All models are evaluated using the DDIM (Song et al. 2021 sampler with 50 inference steps and a classifier-free guidance scale of 7.5. For DiO and CHATS, we adopt the hyperparameters specified in their released code, as these methods modify the sampling pipeline. Performance is assessed using five widely adopted reward models: PickScore (Kirstain et al. 2023 HPSv2 (Wu et al. 2023), ImageReward (Xu et al.) (023), and Aesthetics Score (Schuhmann 2023
To account for stochasticity, we report average results across four random seeds, with each model generating 2,000 images using prompts from the Pick-a-Pic v2 test set.
5.2
QUANTITATIVE RESULTS
Table
presents quantitative comparisons of Diff-SNPO against established baselines SD1.5 and SDXL Dackbones.
On SD1.5, Diff-SNPO consistently outperforms prior methods across most human preference met- rics. Notably, it achieves 27.23 on HPSv2 and 22.24 on PickScore, marking substantial improve- ment both Dif and the more stable Diff-BDPO. This indicates that the gains stem not just
8

---

Published as a conference paper at ICLR 2026
FMPO
D F--MPP
Makro DIT 5MP
5.55
Natvo CI SMPO
27.
26.5-
0.50
26.
535
0.
Figure 4: Ablation over À on SD 1.5 for both HPSv2 and Aesthetic score.
from BDPO's stabilization, but from negative preference optimization itself. While CHATS attains slightly higher Aesthetic Score, it lags behind on all other metrics, suggesting that more visually pleasing outputs come at the cost of semantic alignment. Diff-SNPO also surpasses its dual-model counterpart, Diff-NPO, across all metrics while using only half the computation. We attribute this to its single-model design, which preserves negative preference alignment better than Diff-NPO, where accuracy drops notably after model merging.
On SDXL, Diff-SNPO delivers strong and competitive results, with HPSv2 and ImageReward scores of 28.33 and 1.01, comparable to state-of-the-re methods. Its Aesthetic Score (5.81) is slightly below DPO (5.89), and the large advantage shows over Diff-NPO on SD1.5 becomes smaller SDXL. This stems from Diff-SNPO's bias toward optimizing "preferred" samples in the dataset. Although these samples are favored over their losing counterparts, they have been shown be less aesthetically pleasing than the already strong outputs of the base SDXL model. As a result, focusing on them may slightly reduce aesthetic quality and limit gains on stronger backbones. Similar trends appear with CHATS and Diff-BD BDPO, suggesting the limitation comes from the dataset rather than the alignment method itself. Even so, Diff-SNPO retains important practical benefits: it requires only half the computation of Diff-NPO and, with its single-model architecture, also enables faster compling-maken it both efficient and scalable.
5.3 IMPACT THE MIXING PARAMETERS À AND B ON DIFF-SNPO
In this section, examine the impact of the parameters A and 3 on Diff-SNPO using SD 1.5, while keeping all other hyperparameters identical to those the main experiments.
B
HPSv2
Aesthetic Score
1000
27.30
5.6682
2000
27.23
5.6258
3000
27.23
5.5886
From Fig. 4 we observe that Diff-SNPO mains stable as X varies, showing only minor Table 3: Effect of 3 on the Pick-a-Pic v2 dataset. fluctuations across both reward metries. This indicates that, within the tested range, the choice of has limited impact on performance. In contrast, the setting X = 1.0, corresponding to Naive- SNPO, shows a pronounced drop in both reward metrics. As discussed in Section 4.2 this decline stems from the blurring artifacts that emerge when the win likelihood decreases under single-model Negative Preference Optimization, ultimately degrading image quality.
When varying 8, Table 3 shows that performance is similarly stable across different choices. There- fore, we retain the 3 values from the original Diff-DPO configuration, as they provide a reliable default without introducing meaningful variability in performance.
5.4 COMPARING NEGATIVE PREFERENCE ALIGNMENT ACROSS METHODS
From Table 5 we observe that Dif and Diff-BDPO reach similar negative implicit accuracy in their unconditional branches, both consistently below 50%. This behavior is expected because their unconditional branches are trained only with positive preference alignment, which limits their ability to learn negative preference signals. In contrast, Diff-NPO experiences substantial drop in negative implicit accuracy after weight merging, decreasing by more than 10%. Its post-merge accuracy also falls below that of Diff-SNPO and lies several points behind its own pre-merged value.

---

Published as a conference paper at ICLR 2026
Table Training cost compartson. Experiments were conducted on 8xA6000 GPÜs with a total batch size of 512. Dual-model approaches require substantially more memory and incur slower training throughput. The best result in each column shown in bold.
Method
Memory (GB)
Time / Step (8)
Relative Speed f
Difi-NPO
44.2 x 2
12.25 x
1.00
CHATS
46.3
13.19
1.86x
Diff-SNPO (Ours)
12.25
2.00
Table 5: Negative preference implicit classification accuracy and loss. Parentheses denote Diff- NPO without weight merging. Diff-SNPO achieves higher negative implicit accuracy and lower negative preference loss than Diff-NPO, improving its negative alignment.
Method
Neg. Implicit Acc. (%)
Neg. Pref. Loss
Diff-DPO
31.86
0.759
Diff-BDPO
32.04
0.805
Diff-NPO
52.34 (63.80)
0.703 (0.648) (0.648)
Diff-SNPO
57.45
0.668
Although implicit accuracy is not definitive measure of overall performance, and can be influenced by reward-model artifacts (Rafailov et al. 2024a Amini et al. 2024). it is still useful for revealing the sampling and training mismatch present in Difi-NPO. This mismatch is further reflected in the negative preference alignment loss, which rises sharply after weight merging. Diff-SNPO does not suffer from this problem: its single-model design preserves the negative alignment signal and avoids the degradation observed in Diff-NPO. As result, Diff-SNPO maintains more reliable negative preference modeling and more effectively steers generation away from undesirable samples.
5.5 TRAINING COMPUTATIONAL COST
Beyond alignment performance, the practicality of preference optimization algorithm also depends on its computational efficiency. To assess this, we compare the training computational cost of dual- model approaches (CHATS and Diff-NPO) against our single-model Diff-SNPO. As reported in Table the single-model design of Diff-SNPO yields substantial efficiency gains in both memory usage and training time. By eliminating the need to train two separate networks, Diff-SNPO reduces memory consumption and achieves a 2 speedup in per-step training time relative to its dual-model counterpart, Diff-NPO. A similar advantage observed over CHATS: while both methods require comparable memory, Diff-SNPO trains faster because it optimizes the conditional and unconditional branches in parallel, whereas CHATS processes them sequentially. In summary, Diff- com- bines lower memory overhead with faster training, establishing it alternative for negative preference optimization. detailed comparison of inference cost, which further highlights the efficiency of our single-model approach, be found in Appendix A.7
efficient and scalable
CONCLUSION
In conclusion, we propose Diff-SNPO, a single-model framework for Negative Preference Opti- mization that achieves strong performance while simplifying the training pipeline. In contrast to prior approaches that require two separate models, Dif unifies conditional and unconditional branches within a single architecture, thereby eliminating redundant computation and substantially improving efficiency in both training and inference. Our experiments demonstrate that this stream- lined design not only matches, but often exceeds the performance of existing methods, highlighting its effectiveness in preserving negative preference alignment. Beyond performance, the reduced computational footprint and faster sampling make Diff-SNPO a practical and scalable solution, low- ering the barriers applying preference optimization in real-world generative modeling tasks.
10

---

Published as a conference paper at ICLR 2026
7 ACKNOWLEDGEMENTS
This work was supported by Institute for Information & communications Technology Planning & Evaluation (IITP) grant funded by the Korea government (MSIT) (No. RS-2021-11211381 Develop- ment of Causal Al through Video Understanding and Reinforcement Learning, and Its Applications Real Environments) and partly supported by Institute of Information & communications Technol- ogy Planning & Evaluation (IITP) grant funded by the Korea government (MSIT) (No. RS-201 RS-2022- 1122018 84, Development and Study of AI Technologies Inexpensively Conform Evolving Policy on Ethics).

---

Published as a conference paper at ICLR 2026
REFERENCES
Afra Amini, Tim Vieira, and Ryan Cotterell. Direct preference optimization with an offset. In Len Ku, Andre Martins, and Vivek Srikumar (eds.), Findings of the Association for Com- Bangkok, Thailand, August 2024. Associ-
putational Linguistics: ACL 2024, pp. ation for Computational Linguistics. doi: 10. 18653/v 1/t5:////2222../// 12
URL https:
7/ //cllanthology.oor /224. wt.....ffininino findings-----i accliingg-acl...... 5927
Kevin Black, Michael Janner, Yilun Du, Ilya Kostrikov, and Sergey Levine. Training diffusion models with reinforcement learning. In The Twelfth International Conference on Learning Rep- resentations, 2024. URL httttt:://pptrreeeeeee org.r//eweeeee.rrroooooooooooooo
net wttp:/rrrrmm/mmm (i=YYN
Jay Hyeon Cho, JunHyeok Oh, Myunsoo Kim, and Byung-Jun Lee. Rethinking dpo: The role of rejected responses in preference misalignment, 2025. URL https://arxiv..et abs/
Kevin Clark, Paul Vicol, Kevin Swersky, and David Fleet. Directly fine-tuning diffusion models on differentiable rewards. In The Twelfth International Conference on Learning Representations, 2024. [RLL:ttt:///peeeerraa http://www.eeeeotmmdddeee----- wttp:/mmmmmmm1mmaa1111111991
Ying Fan, Olivia Watkins, Yuqing Du Hao Liu, Moonkyung Ryu, Craig Boutilier, Pieter Abbeel, Mohammad Ghavamzadeh, Kangwook Lee, and Kimin Lee. Reinforcement learning for fine- tuning text-to-imape diffusion models. In Thirty-seventh Conference on Neural Information Pro- cessing Systems, 2023. URL http::::/ppeeeeeiiiiiea
net http:/eeeeettoooa111111--CCCCCC
Minghao Fu, Guo-Hua Wang, Liangfu Cao, Qing-Guo Chen, Zhao Xu, Weihua Luo, and Kaifu Zhang. CHATS: Combining human-aligned optimization and test-time sampling for text-to-image generation. In Forty- International Conference on Machine Learning, 2025. URL https: //openzeview. net htt.://www.....e.oommmdddddaad/.
Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. In NewrIPS 202/ Workshop on Deep Generative Models and Downstream Applicationx, 2021. URL.https:://pperrrr//
net forum? i qw8AKxfYbI
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neu-
ral Information Processing Systems, volume 33, pp. 6840-6851. Curran Associates, Inc.,
2020. URL htlee//ccocccc3Baaa0000111111111
copppper http://aarrffeespeeeeeeeeeee files/paper//000/ http://eww...eeeeooo/////o/e///a
Paper. paff
Jonathan Ho, Tim Salimans, Alexey A. Gritsenko, William Chan, Mohammad Norouzi, and David Fleet. Video diffusion models. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Neural Information Processing Systems, 2022. URL https: httt:::///penreview net wettt//orumiiddiie forumm?dd-faagggga_ee
Woo Hong, Tri Ton, Trung X Pham, Gwanhyeong Koo, Sunjae Yoon, and Chang D Yoo. Ita-mdt: Image-iinestep-adaptiee masked diffusion transformer framework for image-based virtual try-on. arXiv preprini arXiv:2503.204:
2025a
Jiwoo Hong, Sayak Paul, Noah Lee, Kashif Rasul, James Thore,, and Jongheon Jeong. Margin- aware preference optimization for aligning diffusion models without reference. In First Workshop on Scalable Optimization for Efficient and Adaptive Foundation Models, 2025b. URL https: 7/ /ppeneevieww net wwt.:eet/oooooe http:/ormmmm..////......3
Tero Karras, Mika Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion-based generative models. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Neural Information Processing Systems, 2022. URL wereeeeeeeeewe.eeeooorroe Faterummmd=x///PToMMMO
M. G. Kendall and B. Babington Smith. On the method of paired comparisons. Biometrika, 31 (3-4):324-344 345, 03 1940. ISSN 0006-3444. doi: 10.1093/biomee/31.....3333 URL https: 7/doi org. 00. 1093/b1 omet 3-4.3-4. 324
12

---

Published as a conference paper at ICLR 2026
Yuval Kirstain, Adam Polyak, Uriel Singer, Shahbuland Matiana, Joe Penna, and Omer Levy. Pick- a-pic: An open dataset of user preferences for text-to-image generation. 2023.
Shufan Li, Konstantinos Kallidromitis, Akash Gokul, Yusuke Kato, and Kazuki Kozuka. Aligning diffusion models by optimizing human utility. In The Thirty-eighth Annual Conference on Neu- ral Information Processing Systems, 2024. URL wtt.ttps::/ httpps://penrrevee..... nww. wareefooommmee MTMShU5QaC
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Confer- ence on Learning Representations, 2019. URL
net/ forum? i=
Bkg6RiCqY7
Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. DPM-solver: A fast ODE solver for diffusion probabilistic model sampling in around 10 steps. In Alice H. Oh, Alckh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Neural Information Processing Systems, 2022. URL
net wwwp::/wwwwwwww.weeeeeeeeemmeeee wtw.orummmnnaaaaaaaaaa 2uAaGw1F
Yunhong Lu, Qichao Wang, Hengyuan Cao, Xierui Wang. Xiaoyin Xu, and Min Zhang. Inpo: Inver- sion preference optimization with reparametrized ddim for efficient diffusion model alignment. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 28629-28633 June 2025.
Arka Pal, Deep Karkhanis, Samuel Dooley, Manley Roberts, Siddartha Naidu, and Colin White. Smaug: Fixing failure modes of preference optimisation with dpo-positive. arXiv preprint 2XXi:24402.13226 2024.
Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn Jonas Müller, Joe Penna, and Robin Rombach. SDXL: Improving latent diffusion models for high-resolution image synthesis. In The Twelfth International Conference Learning Representations, 2024. URL
httpp::/openrevvie net wtt.:/oooodmmd///
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model secretly a reward model. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL https:
// //oppnreeiiw.. foweet/fooummst www.ammiddduPPSSSSSeSe
Rafael Rafailov, Yaswanth Chittepu, Ryan Park, Harshit Sikchi, Joey Hejna, Bradley Knox, Chelsea Finn, and Scott Niekum. Scaling laws for reward model overoptimization direct alignment algorithms, 2024a. URL https:://aaxiiii wwwpexiwwv..r...o.m
www.ooo/////..oo abs// /ttaaa//2400 220. 02900
Rafael Rafailov, Joey Hejna, Ryan Park, and Chelsea Finn. From SrS to Sq Your language model is secretly q- function. In First Conference on Language Modeling, 2024b. URL https:
//openreview. wwt. htt//oormmm// dtw.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition (CVPR), pp. 10684-10695, June 2022.
Ludan Ruan, Yiyang Ma, Huan Yang. Huiguo He, Bei Liu, Jianlong Fu, Nicholas Jing Yuan, Qin Jin, and Baining Guo. Mm-diffusion: Learning multi-modas diffusion models for joint audio and video generation. CVPR, 2023.
Patrick Schramowski, Manuel Brack, Björn Deiseroth, and Kristian Kersting, Safe latent diffusion: Mitigating inappropriate degeneration in diffusion models. In Proceedings of the IEEE Confer- ence on Computer Vision and Pattern Recognition (CVPR),
Christoph Schuhmann. Laion-aestheties, https:/// thubbb...
October 2023.
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In Interna- tional Conference on Learning Representations, 2021. URL htt.:::///ppnreview..o
forum? I
13

---

Published as a conference paper at ICLR 2026
Joshua Tian Jin Tee, Hee Suk Yoon, Abu Hanif Muhammad Syarubany, Eunscop Yoon, and Chang D. Yoo. A gradient guidance perspective on stepwise preference optimization for diffusion Anmull net /tt/ forum? d=d6 1=d61TOnvOX2 TOnvox2
models. In The Thirty-ninth Annual Conference Neural Information Processing Systems, 2025. URL hht.tps///pperreeee
Tri Ton, Ji Woo Hong, and Chang D Yoo. Taro: Timestep-adaptive representation alignment with onset-aware conditioning for synchronized video-to to-ooio synthesis. In International Conference on Computer Vision (ICCV), 2025.
Bram Wallace, Meihua Dang. Rafael Rafailov, Linqi Zhou, Aaron Lou, Senthil Purushwalkam, Stefano Ermon, Caiming Xiong, Shafiq Joty, and Nikhil Naik. Diffusion model alignment using direct preference optimization. In Proceedings of the IEEE/CVF Conference Computer Viston June 2024.
and Pattern Recognition (CVPR), pp. 8228-82235
Fu- Yun Wang, Yunhao Shui, Jingtan Piao, Keqiang Sun, and Hongsheng Li. Diffusion-NPO: Negative preference optimization for better preference aligned generation of diffusion mod- els. In The Thirteenth International Conference on Learning Representations, 2025. URL
https:/ https::/opennst wwtp//eeeeeeeeeeeeeeeeeeeeeoeee. net f
Xiaoshi Wu, Yiming Hao, Keqiang Sun, Yixiong Chen, Feng Zhu, Rui Zhao, and Hongsheng Human preference score v2: A solid benchmark for evaluating human preferences of text-to- image synthesis. arXiv preprint arXiv:2306.093: 09341, 2023.
Xun Wu, Shaohan Huang, Lingjie Jiang, and Furu Wei. Rethinking dpo-style diffusion aligning frameworks. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 18068-18077, October 2025.
Jiazheng Xu, Xiao Liu, Yuchen Wu, Yuxuan Tong, Qinkai Li, Ming Ding, Jie Tang, and Yuxiao Dong. Imagereward: Learning and evaluating human preferences for text-to-image generation. In Thirty-seventh Conference Neural Information Processing Systems, 2023. URL https:
Hee Suk Yoon, Eunseop Yoon, Mark A. Hasegawa-Johnson Sungwoong Kim, and Chang D. Yoo. ConfPO: Exploiting policy model confidence for critical token selection in preference optimization. In Forty-second International Conference on Machine Learning, 2025a. URL fittss:/cpperevviet n wwwee//ooummaii forrmmidd227bbkpp6
Hee Suk Yoon, Eunscop Yoon, Joshua Tian Jin Tee, Kang Zhang, Yu-Jung Heo, Du-Seong Chang, and Chang D. Yoo. Bi-mdrg: Bridging image history in multimodal dialogue response generation. In Aleš Leonardis, Elisa Ricci, Stefan Roth, Olga Russakovsky, Torsten Sattler, and Gül Varol (eds.), Computer Vision ECCV 2024, pp. 578-396. Cham, 2025b. Springer Nature Switzerland. ISBN 978-3-031-72751-1.
Wenliang Zhao, Lujia Bai, Yongming Rao, Jie Zhou, and Jiwen Lu. UniPC: A unified predictor- corrector framework for fast sampling of diffusion models. In Thirty-sevntth Conference Neurul Information Processing Systems, 2023. URL https: / http:////peeeeeee.e..e net ferrfoum? id=hrkml u
14

---

Published as a conference paper at ICLR 2026
A APPENDIX
A.1 THE USE OF LLMS
We used LLMs solely for light editing such correcting grammatical errors and polishing some words. They did not contribute to research ideation, experiments, analysis, or substantive writing.
A.2 DERIVATION OF THE DIFFUSION BDOO UPPER BOUND
We begin by recalling the Diffusion-BDPP objective:
Cpir BDPO (0) (xw M -) log af (8) log Po (x c)
Pmix maix(x/:z7 (o)
log
(24)
Pret (xxxx
c) Pret x c)
where the mixture distribution defined as
Pmix(Xu:z) c) À 2 c) + (1 >) Prod(xber c).
A (0, 1).
(25)
To obtain a computationally tractable form, we first decompose log po (Nu:T process of diffusion models:
c) using the backward
log pa (xxer
cc log po (xr c) log II
po (x (),
(26)
t=1
log c) Dogpo(x:-1) x2,
(27)
Σ w(l) (,
(28)
where C' a parameter-indeepeddett constant. Next, consider the mixed log- na-probabiiltt::
log Pmixx Xo.T
c) = log > pe po(x +) A) Pro((xu c))
= C + log ( A log log xx.e) + - A) Eim log 10 (29)
For brevity, define
a; := log Po 1 X,, c):
b = Pre Pref (x/_1 x),c), t t= L...., T.
(30)
Then equation becomes
log Pmix X c) = = + log Actior a: + +11 (11 e
(31)
Applying the discrete n-factor Hölder inequality yields
T II
AePtat + (1 -A)ePibe 1/pc
de EE ae + A)e Ei= b <
(32)
==
for exponents PI...., PT 2 with Σt 1/p: = Taking logs and substituting back gives
log Pmix (xb. 1c) C+ Σ log XePiar + (1 2)ePtbs)
(33)
Pt
Choosing p. T for all & (so that E, 1/p: 1) produces the uniform bound
T
log Pmixl X 1)) SC T
log \e Toggulta .(xa,c) + (1 - A) e lug .lx.e)). Σ
(34)
15

---

Published as a conference paper at ICLR 2026
Table 6: Comparison of Difr-SNPO with baseline methods SDL.5 and SDXL backbones on HPDv2. All values are reported as mean + 95% confidence interval over 4 random seeds. Diff- SNPO achieves the best scores across most human preference metrics, indicating superior alignment and visual quality. For each metric, the top-perforning method is bolded, while the second-best is underlined.
Model
Method
HPSv2
Pick Score
Aesthetic Score
Image Reward
Baseline
26.52
20.80 = 0.06
5.3406 0.17
-0.0521 0.19
Diff-DPO Wallace
2024
26.85 + 0.03
21.26 0.02
5,4716 0.14
0.1620 + 0.08
Diff-NPO Wang ct al.
2025
27.18 0.06
21.71 0.05
5.5842 +
0.2287 + 0.14
SDL55
CHATS (Fu ct al. 2025
27.68 0.07
21.45 0.04
5.8605 + 0.07
0.3699 + 0.08
Diff-BDPO (Ours)
27.24
21.57 0.03
5.5677 - 0.06 0.06
0.3576 + 0.02
Diff-SNPO (Ours)
27.85 1 0.06
22.86 1 0.03
5.7737 0.06
0.8093 + 0.08
Baseline
28.01
22.80
5.9887 0.08
0.8781 = 0.11
Diff-DPO WWallace
2024
0.09
0.01
6.02
1.0788 + 0.09
Diff-NPO Wang el al. 2025
28.87 0.03
23.29
6.0816
1.1305 1 0.04
SDXL
CHATS (Fu el al. 2025
13.8210.00
22.85 1 0.011
5,9868 0.03
1.1165 1 0.02
Diff-BDPO (Ours)
28.62 1: 0.04
22.96 1 0.10
5.9562 0.09
1.091 + 0.10
Diff SNPO (Ours)
28.89 = 0.09
23.34 + 0.03
5.9876
1.1460 = 0.04
Upper bound on the objective. obtain
Substituting equation 34 and equation 28
into equation
24
DIHE-BDOO S - E (x ( log of [E [- B Bm(x" mmii (2' (2),
(35)
< (oy an e) (u log of (m(a²" mmix Aomix(ery (ct,,
= (O))
(by Jensen's inequality).
(36)
Here the per-step terms are defined as
do do [()( ||e (0(x,,,,,) ( -Tw(l) ||e 4,
(37)
m( c) = da e, c) - dherlar, t),, Mmix (Zt, c) = log Ae co( + A edattiitts)))))
(38)
(39)
In summary, by decomposing the diffusion likelihood and applying Hölder's inequality, we obtain tractable upper bound on the original Diffusion-BDPO objective, expressed in terms of per-step denoising eтrors equation 37 equation 39
A.3 ADDITIONAL QUANTITATIVE RESULTS
Table 6 presents the results on the HPDv2 test set, showing trends consistent with those observed on the Pick-a-Pic benchmark. On the SD1.5 backbone. Diff-SNPO delivers clear improvement, raising HPSv2 to 22.86 and IeageReward to 0.81, underscoring its effectiveness in capturing hu- man preferences. On the more advanced SDXL. backbone, continues to perform strongly, re- maining competitive with the leading approach, Diff-NPO. Crucially, Diff-SNPO achieves this per- formance with significantly lower computational cost and faster sampling, benefits enabled by its single-network design. This efficiency advantage highlights its practicality, offering fate-offthe-art alignment quality while maintaining scalability and runtime efficiency.
A.4 ABLATION STUDY ON DIFFERENT ODE SOLVERS
In this section, we conduct an ablation study to assess the performance DifT-SNPO across different ODE solvers. To this end, evaluate four widely used solvers: DDIM (Song et al., 2021). Euler Discrete Karrrs et al. 2022), UniPC (Zhao et al., 2023), and DPM Solver (Lu et al. 2022). The results, presented in Table show the performance of Diff-SNN SNPO the Pick-a-Pic v2 dataset with SD1.5. From these results, we observe that the choice solver has little impact on the performance metrics. This outcome aligns with the theory behind Diffusson ODE solvers, where different solvers
16

---

Published as a conference paper at ICLR 2026
Table 7: Performance comparison of different samplers across various reward metrics.
Sampler
Hpsv2 f
Aesthetic Score
DDIM
27.23
5.6258
Euler Discrete
27.22
5.6460
UniPC
27.25
5.6468
Dpm Solver
27.25
5.6466
are alternative numerical methods for solving the same ODE system (Lu et al. 2022 Karras et al. 2022). Consequently, with sufficiently large number of function evaluations (NFE), all solvers converge to the same image, leading to negligible differences in performance.
A.5
SAFETY ALIGNMENT RESULTS
Table 8: Comparison of IP values before and after finetuning CoProv2 dataset.
SafetyDPO
IP
Baseline
0.4308
Diff-DPO
0,5109
Diff-NPO
0.4203
Diff-SNPO
0.4719
After Finetuning CoProv2
Dio 0.1713
Dif 0.1318
Diff-SNPO
0.1100
Table 8 presents the results of training and evaluating models on the CoProv2 (Wu et al. 2025) for SD 1.5. This dataset contains 23,690 pairs of safe and unsafe images, spanning across 7 categories ((Hate, Harassment, Violence, Self-Harm, Sexual, Shocking, Illegal). Evaluation was performed using the Inappropriate Probability (IP) metric (Schramowski al. (023), which quantifies the model's ability to generate safe content when prompted with unsafe prompts.
From Table we observe that models trained the Pick-a-Pic v2 dataset exhibit weaker safety performance. Specifically, their IP scores increase relative to the baseline after finetuning, indicating decline safety. This is primarily due the nature of the Pick-a-Pic v2 dataset itself, which contains some unsafe images within its "win" samples. As a result, models trained on this dataset are inadvertently exposed unsafe content, causing them to generate unsafe outputs.
In contrast, when explicitly trained with safety-oriented oriented dataset, the safety performance improves significantly. particular,
outperforms Diff-DPO, achieving safety score of 0.11, which is slightly better than Diff-NPO's score of 0.13. This demonstrates that using safety-focused dataset, in combination with Diff-SNPO can lead improved alignment with safety objectives.
A.6 TOTAL TRAINING COMPUTE COMPARISONS
ssoos the total GPU hours required by different baseline methods and Diff-SNPO for both SD1.5 and SDXL using A6000 GPUs. The GPU hours for the baselines were calculated based on the hyperparameters specified in the respective papers. From Table 9 it is clear that Diff-SNPO outperforms its counterparts in computational efficiency, demonstrating its ability to achieve strong performance with significantly lower resource requirements.
17

---

Published as a conference paper at ICLR 2026
Table 9: Total GPU hours for different methods SD 1.5 and SDXL using A6000 GPUs..
Model
Method
Total GPU Hours
Diff-DPO CHATS
206
237
SD 1.5
Diff-NPO
2x206
DifI-SNPO
79
Diff-DPO
1592
CHATS
2162
SDXL
Dift-NPO
2x1592
Diff-SNPO
498
A.7 INFERENCE COMPUTATIONAL CoST
Table 10: Inference cost comparison. Single-image inference cost with SD1.5 on single A6000 GPU.
Table 10 reports the inference throughput of Diff- SNPO and other negative preference optimization methods SD1.5. The efficiency of the single- model design is particularly evident at inference: Diff-SNPO achieves the highest throughput among all methods, processing more than twice as many im- ages per second as the dual- dual-model baselines.
Method.
Throughput (img/s)
CHATS
0.11
DRINPO
0.27
DHV-SNPO (Ours)
0.48
By contrast, CHATS and Diff-NPO suffer from slower inference due to the need for two sequential forward passes-one for each model-combinee with additional passes required by their sampling schemes. These results underscore Diff-SNPO's ability to perform fast, parallel inference without compromising alignment quality.
Overall, our findings demonstrate that Diff-SNPO effectively balances preference alignment with computational efficiency. Its single-model design reduces training cost while delivering faster in- ference, making it sull-suited for large-scale training as well as real-time or resource-constraing deployment scenarios.
A.8 QUALITATIVE RESULTS
Figure 5 and Figure 6 present additional image samples generated by Diff-SNPO and various pref- erence alignment methods on SD 1.5 and SDXL using prompts from HPDy2 test set respectively.
18

---

Published as a conference paper at ICLR 2026
Prompt
SD1.5
DIfL-NPO
CHATS
DIft-SNPO (Ours)
small dog looking at white plate holding donsts..
A Ajellyfish sleeping space shation pod.
cat in turtu dancing to Swaa Lakes.
Spider-Wan holding a ginger cat.
Figure 5: Side-by-side comparison of images generated by related methods on HPDv2 using SD1.5.
Prompt
SDXL
Diff.NNPO
CHATS
DIfI-SNPO (Ours)
lemon wearing swit and tie, full body portmaith.
R
Side-view blue-ice sneaker inspired by Spiderman oreated by Wota FX
2B from NieR Automhata cating A bagel.
spaceship in empty landscapo.
Figure 6: Side-by-side comparison of images genkfated by related methods HPDv2 using SDXL..


Reviews:
[{"review_id": "9OG42ydk8t", "reviewer": "Reviewer_tGQL", "summary": "This paper proposes Diff-SNPO, a unified framework for diffusion models to learn from positive and negative preference examples. Diff-SNPO improves upon Diff-NPO by only using one model, removing the potentially undesirable model merging step and making the procedure more computationally efficient. Experiments show that Diff-SNPO outperforms Diff-NPO and other baselines.", "questions": "See weaknesses.", "limitations": "", "rating": 4, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper has extensive empirical verification, and good results showing that Diff-SNPO outperforms other baselines.\n2. The proposed method reduces training and inference cost by 2x, which can be significant when the models are large-scale. \n3. The paper is mostly clearly written.", "weaknesses": "1. Novelty and technical difficulty: Diff-SNPO seems to be a relatively straightforward combination of CFG and BDPO, which are both existing methods. Given that BDPO addresses the problem of preference learning procedures reducing the likelihood of the preferred examples, it is not very surprising that it would be effective here. Can the authors describe the technical challenges of combining these ideas?\n2. It would be great if the preliminary can include a dedicated section on CFG.\n3. The paper highlights that Diff-SNPO may suffer from conflicting gradients between positive and negative examples because it uses a single model. Since DPO doesn't have this problem (there are no inverted examples), how does BDPO, which only bounds the loss scale for dispreferred examples, specifically mitigate the problem of interfering gradients?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Qogv6nnVL3", "reviewer": "Reviewer_f2zF", "summary": "This paper proposes Diffusion Simple Negative Preference Optimization (Diff-SNPO), a single-network alternative to dual-model negative preference methods (e.g., Diff-NPO). The key idea is to train the conditional branch on preferred samples and the unconditional/null branch on inverted (negative) preferences within the same diffusion backbone (leveraging CFG’s two branches), thereby avoiding the cost and alignment dilution of maintaining and merging two models. The authors diagnose that a naive single-model version coupled with Diff-DPO blurs images by decreasing winner likelihood; they then adapt Bounded DPO (BDPO) to diffusion and derive an upper-bound (Diff-BDPO-UB) objective that mixes the loser term with a reference policy to prevent the winner likelihood collapse. The final objective (Eq. 18) instantiates SNPO with the BDPO upper bound and a mixing parameter λ. Experiments on Pick-a-Pic v2 and HPDv2 with SD1.5/SDXL show improved preference metrics (HPSv2, PickScore, ImageReward), higher negative implicit accuracy, and 2× training/inference speedups versus dual-model baselines.", "questions": "N/A", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. I think the approach that uses the native conditional/unconditional branches to encode positive vs. negative preferences is both elegant and practical.\n2. Adapting BDPO to diffusion and deriving a tractable upper bound yields a training signal that boosts the winner without exploding loser gradients.\n3. Good empirical results.\n4. Higher negative implicit accuracy than Diff-NPO post-merge, addressing a known weakness of merging", "weaknesses": "1. The main experiments use large batches and multi-GPU (e.g., 8×A6000, batch 512/2048), whereas Table 3's cost is a single-GPU batch-8 microbenchmark.\n2. Training negative signals into the unconditional branch might introduce risks altering the unconditional anchor used by CFG in safety contexts.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "urxwkZping", "reviewer": "Reviewer_cffW", "summary": "The paper proposes Diffusion Simple Negative Preference Optimization (Diff-SNPO), which trains a single diffusion model on a preference dataset using an upper bound of the Bounded Direct Preference Optimization (BDPO) objective and a modified classifier-free guidance (CFG) dropout training setup where the unconditional branch is the other example in the preference pair. The paper shows that training a single model with the Diff-DPO objective causes generated images to get blurrier over training (and positive example probabilities to go down), and that switching to their derived upper bound of the BDPO objective mitigates this failure.\n\nThe paper compares Diff-SNPO with 1. Diffusion DPO (Diff-DPO), which trains a single model on an upper bound of the DPO objective; 2. Diffusion Negative Preference Optimization (Diff-NPO), which uses the Diff-DPO objective to train two separate models, one with pairs flipped, followed by CFG during sampling using a merged weights model instead of the trained negative model directly; and 3. CHATS, which uses the Diff-DPO objective to train two separate models simultaneously, one for each reward term in the reward margin of the objective, followed by a variant of CFG using the two models and a proxy prompt for the negative model. Diff-SNPO is more efficient than #2 and #3 due to its use of a single model, and generally improves over Diff-DPO in quality.", "questions": "1. Are the quantitative quality results significant? Could the authors include some form of error bars, significance tests, etc.?\n2. Is there a reason why Diff-NPO is the main method being compared against in the story of the paper?\n3. Why should we care about implicit accuracy at all (e.g., as a downside of weight merging)?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The derivation of a bound of the BDPO loss, as well as the CFG-like training setup for training preference pairs on a single model, seem like clear contributions and are to my knowledge novel.\n2. The proposed method has a clear memory/computational wins over the two-model methods, and the move from DPO to BDPO when training a single model is also nicely showcased.\n3. The additional row showing Diff-BDPO (which if I understand correctly is meant to offer a direct comparison to Diff-DPO) is helpful as an ablation.", "weaknesses": "1. The quantitative results do not show report error bars or information about statistical significance. Also, there is not a clear winner across the quality metrics.\n2. The paper could benefit from some revisions of the writing (e.g., in the abstract, introduction, related work, figure 1) to better explain what currently exists and distinguish its contributions. For instance, lines 78-79 make claims about Diff-NPO that I first thought were just missing references to past works but later realized are actually new findings from the paper. Moreover, the abstract and figure 1 focus on Diff-NPO when it's not clear why that prior work should be privileged over the other methods for comparison. (The paper does compare to others, but it might be easier to, for instance, have a table comparing this method to various others rather than focus on Diff-NPO in multiple places.)\n3. The implicit classification results don't seem very relevant for any actual notion of performance that matters for these models. For instance, [Rafailov et al 2024](https://arxiv.org/abs/2406.02900) (as well as others) show the lack of correspondence between better rewards and better win rate. (This can be explained by that the reward looks at offline data that need mot characterize the model well.)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "CIMmG18xVB", "reviewer": "Reviewer_ascV", "summary": "This paper propose a new method called Diff-SNPO for improving preference alignment in diffusion models. It builds on Diff-NPO which uses two separate models for positive and negative preferences but that has high cost and need weight merging at inference. Instead, authors integrate both into single network using the conditional and unconditional branches from CFG. They identify problem with naive approach leading to blurry images due to likelihood collapse in DPO, and fix it by adapting Bounded DPO to diffusion setting. Experiments on SD1.5 and SDXL with Pick-a-Pic v2 show better performance and efficiency compare to baselines like Diff-NPO and CHATS.", "questions": "- In Eq18, how do you sample Y during training? Is p fixed or annealed?\n\n- Does Diff-SNPO work with other samplers beyond DDIM or only tested with that?\n\n- Have you try applying this to non-image domains, like video diffusion?", "limitations": "", "rating": 2, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- The idea of unifying positive and negative preference in one model is clever and adress the practical issues of dual-model approaches, like double compute and merging trade-offs. This make it more scalable for large models. \n\n- They provide good analysis of why naive single-model fail (blurring from conflicting gradients and DPO's winner likelihood decrease), and the bounded objective seem to solve it effectively based on figures.\n\n- Experiments are solid, evaluating on multiple reward models (PickScore, HPSv2 etc.) and showing gains in alignment metrics while reduce overhead. Code will be released which is plus.\n\n- The derivation of Diff-BDPO upper bound is detailled in appendix, and overall contribute to stable preference optimization in diffusion.", "weaknesses": "- The paper claim strong performance but evaluation is only on Pick-a-Pic v2 benchmark, which is aesthetics-focused. Would be better to test on other datasets like safety or bias alignment to show generality, especialy since abstract mention safety.\n\n- Hyperparameters like $\\lambda=0.9$ and $\\beta=2000/5000$ are tuned but not much ablation on them. How sensitive is method to these? Also, training time is given (12h for SD1.5) but no direct comparison of GPU hours vs baselines.\n\n- While they discuss blurring in naive approach, the qualitative examples in Fig3 are limited to one prompt. More diverse visuals would strengthen the claim.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)