Diffusion-NPO: Negative Preference Optimization for Better Preference Aligned Generation of Diffusion Models
Authors: Fu-Yun Wang
Yunhao Shui
Jingtan Piao
Keqiang Sun
Hongsheng Li
Venue: ICLR 2025 Poster (2025)
Keywords: Diffusion
Preference Optimization
Primary area: alignment, fairness, safety, privacy, and societal considerations

Abstract: Diffusion models have made substantial advances in image generation, yet models trained on large, unfiltered datasets often yield outputs misaligned with human preferences. Numerous methods have already been proposed to fine-tune pre-trained diffusion models, achieving notable improvements in aligning generated outputs with human preferences. However, we point out that existing preference alignment methods neglect the critical role of handling unconditional/negative-conditional  outputs, leading to a diminished capacity to avoid generating undesirable outcomes. This oversight limits the efficacy of classifier-free guidance (CFG), which relies on the contrast between conditional generation and unconditional/negative-conditional generation to optimize output quality. In response, we propose a straightforward but consistently effective approach that involves training a model specifically attuned to negative preferences. This method does not require new training strategies or datasets but rather involves minor modifications to existing techniques. Our approach integrates seamlessly with models such as SD15, SDXL, video diffusion models and models that have undergone preference optimization, consistently enhancing their ability to produce more human preferences aligned outputs.

Published as a conference paper at ICLR 2025
DIFFUSION-NPO: NEGATIVE PREFERENCE OPTI- MIZATION FOR BETTER PREFERENCE ALIGNED GEN-
ERATION OF DIFFUSION MODELS
Fu-Yun Wang
Yunhao Shui
Jingtan Piao
Kegiang Sun
Hongsheng Li
MMLab, CUHK, Hong Kong
Shanghai Jiang Tong University, Shanghai CPII under InnoHK, Hong Kong 115511 m0000link. link. cuhk. www..edu. eddu. hk
Fywango Link. cuhk. edu. hk, xi xilanhual22 jt..sstu.ddu......... cn, cn, kqaun wonll..cuhhhh edu. hk, hslidee cuhk. edu.hk
w/o NPO
w/ NPO
w/o NPO
w/ NPO
Figure 1: Diffusion-NPo enhances high-freug freeefrryenncy details, color and lighting, and low- www-freuency
structures in images by aligning human's negative preference.
ABSTRACT
Diffusion models have made substantial advances in image generation, yet mod- els trained on large, unfiltered datasets often yield outputs misaligned with hu- man preferences. Numerous methods have been proposed fine-tune pre-trained diffusion models, achieving notable improvements in aligning generated outputs with human preferences. However, we argue that existing preference alignment methods neglect the critical role of handling outputs, leading to a diminished capacity to avoid generating undesirable out- comes. This oversight limits the efficacy of classifier-free guidance (CFG), which relies on the contrast between conditional generation and unconditional/negatiee conditional generation to optimize output quality. In response, propose straightforward but versatile effective approach that involves training a model specifically attuned negative preferences. This method does not require new training strategies or datasets but rather involves minor modifications to exist- ing techniques. Our approach integrates seamlessly with models such SD1.5, SDXL, video diffusion models and models that have undergone preference opti- mization, consistently enhancing their alignment with human preferences.
INTRODUCTION
Diffusion models have made significant strides in image/vidco generation (Ho et al, 2020 Rombach
al. 2022 Podell al. 2023 Dhariwal & Nichol 2021 Singer al. 2022 Shi 2024
Wang

---

Published as a conference paper at ICLR 2025
et al. 2024f C b: Liang et al.) 2024a Liu al. 2022 Peebles & Xie 2023 Karras et al. 2022
Ke et al. 2024: Yin et al. 2024 However, diffusion models trained on massive unfiltered image- text pairs (Schuhhann 2022] Sun et al., 2024) often generate results that do not align with human preferences. To address this issue, many methods (Wu et al., 2023) 2024) have been proposed to align diffusion models with human preferences, aiming drive the generation to better match what users desire.
Human preference alignment methods typically require the prior collection of human preference dataset, such as Pick-a-pic (Kirstain et al. (023). The standard procedure involves gathering pairs of images generated from the same prompt and annotating them according to human preferences. Rather than assigning direct scores, these preferences are usually ranked in order. This ranking then utilized to train a scoring/reward model for text-image pairs using a contrastive loss func- tion (Ouyang et al. 2022). To explore this topic in depth, we first review existing approaches for aligning diffusion models with human preferences. In general, current methods can be categorized into three types:
a) Differentiable Reward (DR): These approaches directly feed multi-step generated images into a pretrained reward model, updating the diffusion models through gradient back prop-
agation (Xu et al. 2024 Prabhudesai et al. 2024 Zhang et al., [024bb] Wu et al.] 2023] 2024 While simple and direct, these methods are prone to reward leakage (Zhang et al. 2024b).
b) Reinforcement Learning (RL): In these approaches, the denoising process of diffusion models is formulated an equivalent Markov decision process (MDP) (Puterman 2014). PPO (Shhulann et al., 2017) and its variants are typically adopted for preference optimiza- tion. Images are generated and evaluated online based on the reward feedback, aiming to increase the probability of generating high-reward images. These approaches employ SDE solvers to achieve stochastic sampling and importance sampling (Sutton) 2018).
c) Direct Preference Optimization (DPO): These approaches simplify the reinforce- ment learning training objective into a straightforward simulation-free training objec- tive (Rafailov al. 2024, Wallace et al., 2024). They do not require training reward mod- els, nor do they need online generation and sampling, instead, they only require fine-tuning on pre-collected paired preference data. Although simple, these approaches often under- perform reinforcement Sarning-based methods, especially for disti--disttibuttin inputs.
Despite previous efforts to make models generate human-aligned images, we raise an important question: How can model know to avoid generating poor images if only knows how to generate good ones without understanding what is bad?
We identify crucial oversight in current diffusion model preference alignment efforts: most dif- fusion generation rely heavily on the classifier-free guidance (CFG) (Ho & Salimans) 2022, Karras ct al. 2024 Shen et al., 2024 Ahhn et al. 2024 Wang et al. 2024a d). CFG requires the model to si- multancously compute outputs under both conditional inputs and canatiieeooondiiiona//umoooooooo inputs at each denoising step. then linearly combine these outputs to bias the final prediction towards the conditional inputs and away from the cegative-oonditional inputs. Ideally, we expect the model's output under the conditional inputs to align closely with human preferences, while the output un- der the negative-conditional inputs should diverge from human preferences to maximize preference alignment. However, previous works focus exclusively on training models to generate outputs that align with human preferences, without considering the equally important task of teaching models to recognize and avoid generating outputs that humans do not favor. This oversight limits the ef- fectiveness of existing alignment strategies, particularly in scenarios where distinguishing between preferred and non-preferred outputs is crucial.
To address this issue, we propose Negative Preference Optimization (NPO): training an additional model that is aligned with preferences opposite to human. Importantly, our crucial insight is that training such a negative preference aligned model requires no new training strategies or datasets, only minor modifications to existing methods. 1) Approaches like differential reward and reinforce- ment learning. all need a reward model for training. We simply multiply the output of reward model train negative preference model using the same approaches. 2) For
by 11, which allows DaO-based methods, we reverse the order of the preferred image pairs. Notably, during the training of the reward model applied for differential reward and reinforcement learning approaches, the im-
2

---

Published as a conference paper at ICLR 2025
Diffusion Models
Preference Aligned
Noise, Prompt
Diffusion Models Eage
(Prompt, Image 1, Image
X T
Pretrained
Preference Optimization
Diffusion Models
6
(w + D) (x l, Cpos) Anog (xe, l, Cneg)
Diffusion Models
Negative Preference Aligned
(Prompt, Image Image 1)
Diffusion Models 6aa
Training
Switched Image Pairs
Inference
Human-Aligned Generation
Negative Preference Optimization
Figure 2: High-level overview of negative preference optimization (NPO). (Training) NPO needs no new training strategies and datasets. NPO training can be achieved through switching preference image pairs with existing preference optimization methods. (Inference) NPO trained models serve as the incondiioon/mngaatie-oottttioooo predictors in the classifier-free guidance.
age order can also be reversed to train the reward model. Therefore, in essence, all strategies can be perceived as reversing the order of image pairs in the collected preference data by adapting the same training procedure. Fig. provides overview of our method.
We validate the effectiveness of NPO on text-to-image generation with SD1.5 (Rombach and SDXL (Podell et al. 2023) and text-to-video generation with VidcoCrafter2 (Chen al. 2024). Our model can be used in a plug-and-plaa manner with these baseline models and their various preference-optimizzed versions, consistently improving generation quality. F
2022)
compar-
ative results. We evaluate our method using the widely adopted Pick-a-pic validation set, scoring with metrics including HPSv2, ImageReward, PickScore, and LAION-Aesthetic. Our approach significantly improves performance across all metrics.
UNDERSTANDING CLASSIFIER GUIDANCE
Preliminary CFG. CFG has became a necessary and important technique for improving gen- eration quality and text alignment of diffusion models. For convenience, we focus our discussion on the general formal of diffusion models, i.e., 3, a, 0,e (Kingma et al.) 2021). Sup- pose we learn a score estimator from epsilon prediction neural network 60 l). and we have
Vz, log Pe (ar/ |c; t) co( The sample prediction at timestep of the score estimator formulated
1 -a, + o o'Vn. log (e(xx(e;
ão
(1)
Applying the CFG is equivalent to add an additional score term (Krras et al. 2024), that is, we replace Va, log (o in Eq. with the following term,
Va, log (o(x, + Va, log Po(x,(d(tt)
where is to control the strength of CFG, e and e' are conditional and unconditional/neeatiee conditional inputs, respectively. is apparent that the generation will be pushed high probability region of Ie (xy|cc t) and relatively low probability region of Pe (xx (2) Write the above equation into the epsilon format, and then we have
c
(w 1)Ea (Fe, G, web (21,, d,
(3)
3

---

Published as a conference paper at ICLR 2025
Stable Diffusion
DreamShaper
Stable Diffusion XL
Diffusion-DPO
Diffusion-SPO
w/o NPO
w/ NPO
w/o NPO
w/ NPO
Figure 3: NPO works a plug and and-play inference enhancement strategy. It can be easily combined with base diffusion models and preference optimized diffusion models for better human preference aligned generation. Zoom out for better comparison in details.
Motivating example. To maximize human preference alignment, Eq. 3 the green component should guide the generated results to closely match human preferences, while the orange component should direct the results away from undesired outcomes. However, most preference optimization methods focus exclusively and thereby weakening its impact. To illustrate this point, we setup a motivating experiment to investigate the influence of the orange component. We emplogy two baselines:
optimizing the green component, neglecting the orange component
We use the DPO-optimized SD1.5 (Wallace et al. 2024 Rombach et al. 2022 for both the green component and the orange component.

---

Published as a conference paper at ICLR 2025
Prompt: "Foruritic vobot but weating medieval leed dothes and medieval carte, exvemneyy digital art, ambiant lightning. interion castle, medieval"
Prompt: "eeaiild portritt diskeveled hippe gli with bright gray eyes by anna dittmmenn digital painting 120k, dln Ed. hyper deta complimentar solors"
Prompt: "Mythological Hack cal on top ala mouniaat, digital art'
Phompt Huraa heart made of jewels
DresmS DrgsmShhoger lager
EBIB on HERR
DroamShopor NPO
Figure 4: Plug-and aae--nd--paa NPO DreamShaper. NPO not only works the base Stable Diffusion and its preference optimized variants, but also works on improving customized model finetuned on high-quality data.
2. We use the DPO-optimized SD1.5 for the green component and the model merged from weights DPO-optimized SD1.5 (0.6x) and original SD1.5 (0.4x) for the orange com- ponent, generating results with the same seed.
We compare the generated images from the two baslines one by one, score them using HPSv2, ImageReward, PickScore, and LAION-Acsthein and calculate their win probabilities. The results are shown in Fig. 5 We can observe significant improvement in human preference compared to only using the DPO-optimized model.
Analysis: the weight merge an approximated NPO. What is the meaning of the weight merged model? Suppose the weight of original SD 0, and then the DPO-optimized model weight can be de- noted as + 17 since it is further trained from the original weight O. The merged model weight is +1+ 01- ++ -+n--11-1000 (1-7)
Drababikky Compuriass
Strppd Micel
where 7 E [0, 1] is the merge factor. We can observe that after weight fusion, the weight offset obtained through DPO optimization r has decreased. Conse- quently, the DPO weight offset 7) has weaker im- pact on the generated results, enabling the model to output results that more contrary human pref- erences. Replace (1 ()(-)) with ö, and then the weight can be represented as
Figure 5: Motivating example results of Section 2 as the orange component (i.e.. predic- tion for mncondttionalnngaaiveeooiiiiiiii inputs) effectively improves the human pref- erence alignment.
Applying merged model
0 0 + 0
(5)
The above equation decomposes the weight applied for sraative-conditionall predictions into three parts: the original model weight 0, the preference alignment weight offset (direction) a weight offset opposite the preference alignment 8. Our paper aims to train suitable d and investigate its properties. Note that, once the 7) and ò are obtained, we can also flexibly change the influence of each weight offset by multiply simple scale factors. That is,
= + am + 36
(6)
where a {0,0)
B
5

---

Published as a conference paper at ICLR 2025
Table 1:
Quantitative performance comparison with stable diffusion vI-5 based models. means the metrics are copied from SPO papers. Other metrics are tested with official weights.
Method
PickScore
HPSv2
ImageReward
Aesthetic
SD-1.5
20.75
26,84
0.1064
5,539
"DDPO
21.06
24.91
0,0817
5.591
"D3PO
20.76
23.97
-0.1235
5.527
Dif
20.98
25.05
0.1115
5.505
DPO
21.41
26.85
0.1738
5,946
SD-1.5 NPO
21.26
27.36
0,2028
5.667
DiP + NPO
21.65
27.09
0.1939
5,999
Dift.-DPO NPO (reg- 500)
21.58
27.60
0.3101
S.762
Dif NPO (reg= 1000)
21.43
27.36
0.3472
5.773
DreamShaper
21.96
27.97
0.7131
6.085
DraamShaper + NPO (a = 1.0)
22.38
28.31
0.7396
6.169
DreamShaper + NPO (a 0.6)
22,46
28.08
0.6626
6.496
3 NEGATIVE PREFERENCE OPTIMIZATION
Previous approaches primarily focus on training single model weight that aligns with human prefer- ences. However, these methods often overlook the importance of unconditional outputs of classifier- free guidance in the diffusion generation process. Our approach seeks to train weight offset ō that opposes human preferences to fulfill the role of unconditional outputs. By integrating this offset predictor for unconditional inputs, thereby reducing
with the base model's 's weights, it functions the likelihood of generating outputs that conffict with human preference. The important motiva- model should not only learn
tion for negative preference optimization is that preference- to generate desirable outcomes but also understand what constitutes undesirable ones. This dual understanding is crucial for maximizing preference alignment while minimizing the occurrence of unwanted results.
3.1 TRAINING WITH NPO
An important insight in our work is that achieving negative preference optimization does not require new datasets, reward models, or even new training strategics. Standard preference optimization methods can be directly applied to negative preference optimization.
For methods based on reinforcement learning and differential rewards, which typically rely on reward model R(x, e) [0, 1] (can be easily scaled if not in this interval). This reward model can be transformed into the form required for negative preference optimization as follows:
Axpo(x, c) = 1 R(x, c).
Fo methods that utilize reward models, we can simply substitute the original R(x, c) in the algorithm with Rxpo(x; (x,c). c).
For methods that train on preference pairs r = (x, c). where Xo is less preferred and x1 is more preferred by humans, and e is the conditional information used for generation (indicating both images are generated from the same c), converting this to negative preference optimization algorithm requires simply reversing the order of the preference pair:
NNP = (x1+ Xg. c).
Beyond the fundamental implementation of negative preference optimization outlined above, it is important to recognize that many preference optimization methods may use CFG during training for sample collection, probability calculation, and gradient backpropagation. NPO can naturally extend to these methods as well. Although these methods might apply CFG during training to bridge the gap between training and inference, they train only a single weight offset, overlooking the fact that the conditional and unconditional (or negative-conditional objectives (i.e., preference-aligned and negative offset that is a compromise between the two opposite objectives, failing to fully achieve preference alignment. We propose optimize two distinet weight offsets simultaneously.
conditional) outputs in CFG have different optimization iereerencc-alignnede This could result in a weight

---

Published as a conference paper at ICLR 2025
Table Quantitative performance comparison with stable diffusion XL based models. All metrics are tested with official weights.
Method SDXL
PickScore
HPSv2
ImageReward
Aesthetic
22.06
27.89
0.6246
6.114 14
Diff.-DPO
22.57
28.58
0.8767
6.099
DPf
22.97
28.58
1.032
6.348
SDXL NPO
22.32
28.11
0.6831
6.136
Diff.-DPO NPO
22.69
28.78
0.9210
6.112
Diff.-SPO + NPO
23.08
28.70
1.047
6.438
3.2 INFERENCE WITH NPO
Let 0 denote the base model weight, 7 the weight offset after preference optimization, and the weight offset after negative preference optimization. straightforward strategy is to define 0pss = 0 + 77 and Oneg~ 0 + 8, and then apply classifier-free guidance as follows:
E = (w + 1) (I,, c, - WEB... (#t, c,t).
However, this approach often results in a significant output discrepancy between pos and Aagg- The outputs from classifier-free free guidance should maintain a necessary level of correlation; for example, two Gaussian noises are completely independent, the variance from the operation above would change from to 2 + 2w + 1. We find that it is typically necessary to incorporate the positive weight offset into the negative weights, such that:
neg 0 + on + 38, a, [0,
(10)
which aligns with our earlier motivating example and analysis.
4 EXPERIMENTS
4.1 VALIDATION SETUP
To validate the effectiveness and versatility of our approach, test three baseline methods:
a) Diffusion-DPO. Diffusion-DPP Wallace et al. 2024) is the first method to incorporate the Direct Preference Optimization (DPO) approach diffusion training. It introduces a simulation-free and reward model-free training strategy that enables direct training with preference pairs. The effectiveness of this method has been validated on popular text-to- image models, such as the 0.9B Stable Diffusion v1-5 and the 3B Stable Diffusion XL.
b) Diffusion-SPOL Diffusion-Sie
(Liang et al. 2024b) combines the DPO approach with re- inforcement learning. It involves online sample generation, stochastic solvers, and proba- bility calculations, while utilizing the DPO optimization objective for training. This method requires reward model to preferences for generated images online. Its effectiveness has also been demonstrated on the 0.9B Stable Diffusion v1-5 and the 3B Stable Diffusion XL for text-to-image generation.
c) VADER. VADER (Prabhudesai et al. 2024) is a differential reward-based approach that has shown effectiveness text-to-video generation, significantly enhancing the aesthetic quality of generated videos from raw models.
Therefore, our validation baselines include differential reward, reinforcement learning, and direct preference optimization (the three typical kinds of methods we mentioned), covering both text-to- image and text-to-video tasks. We believe our validation sufficiently convincing to demonstrate the effectiveness of our approach. Unless otherwise specified, ence configurations for all the aforementioned methods, including training data, number of training iterations, CFG strength, etc.
the default training and infer-
4.2 COMPARISON
Quantitative comparison. For text-to-image generation, we conduct the quantitative evaluation of our method by following previous work and using the Pick-a-pic 'test unique' split the test-
7

---

Published as a conference paper at ICLR 2025
Prompt:: "A person playing guitar by campiire under starty sky."
Prompt:: "A person harvesting fruit crrchand with sunlight filtering shrexugh the trees."
VideoCrafter2
VADER
VADER + NPO (HPSv2)
VADER NPO (PickScore)
Video comparison. The videos are trained using 12 frames. For better visualization, sample one key frame from every four frames..
Figure 6:
ing benchmark (Kirstain et al., 2023 We employ PickScore (Kirstain et al. 2023). HPSv2 (Wu al., (023), ImageReward (Xu et al. 2024), and Laion-Aesthetie (Schuhmann 2022) as evalu- ation metrics. The results of the quantitative evaluation are summarized in Tables and 2 The tables demonstrate that NPO, when combined with the base model and its preference-optimize sions, consistently enhances the aesthetic quality of the generated results. In addition to reporting the average scores, as illustrated in Fig. 7 we calculate the proportion of samples generated with the same prompt that achieve higher preference score. The results generated using NPO signif- icantly outperform those without NPO. For text-to-video generation, we compare four baselines: VideoCrafter2, VADER, VADER + NPO (HPSv2),, and VADER NPO (PickScore). Among these, VADER + NPO (HPSv2) is optimized using both HPSv2 and Laion- while VADER + NPO (PickScore) is optimized using PickScore as the reward model. We train the models using animal-related prompts, as was done with VADER, and evaluate on unseen animal- related prompts (same domain) and additional human prompts (out domain). The results, presented in Table reveal that VADER + NPO (HPSv2) shows significant improvements across all four met- rics, particularly the HPS and Aestheee metrics. VADER + NPO (PickScore) demonstrates greater improvement in the PickScore metric and, on animal-related prompts, even achieves better HPSv2 performance than VADER + NPO (HPSv2).
ver-
as reward models,
Qualitative comparison. Fig. 3 Fig. 4 Fig. 6 Fig. Fig. 2
Fig.
Fig. 14 and Fig. 15 present
a comparison of results generated with and without NPO across various scenarios. We observe that NPO significantly enhances high-frequency details, color and lighting, and low-frequency structures in images, consistently improving human preference scores.
User preference. We assess the generation quality in three specific areas: Color and Lighting, Fregh-Freyency Details, and now-Frequency Composition. For Color and Lighting, users evaluate whether the generated images display natural and visually pleasing color schemes and lightings. For
8

---

Published as a conference paper at ICLR 2025
Table 3: Quantitative performance comparison on text-to-video generation. All metries are tested the average score. Win means the average winning ratio to other
with official weights. Avg means methods. HPSv2 means we apply both aesthetic predictor and HPSv2 for training. PickScore means apply PickScore for training.
Method
Aesthetic Win
HPSv2
ImageReward Win
PickScure
Avg
Avg
Win
Avg
Avg
Win
Ammml
VideoCrafter2
5,527
0.00%
29.65
2.08%
1,368
30.73%
22.44
16.81 %
VADER
6,154
55.21%
32.24
46.88%
1.486
58.33%
22.97
34.23%
VADER NPO (PickScore)
6.110
50.00%
32.81
82.81 %
1.463
53.65%
24.16
98.44%
VADER NPO (HPSv2)
6.379
94.79 %
32.52
68.23%
1.492
59.96%
23.14
50.52%
Human
VideoCrafter2
5.726
1.04%
27.92
10.27%
0.9583
33.33%
22.41
27.75%
VADER
6.462
61 46%
29.74
51.71%
1.102
46.35%
22.55
34.23%
VADER NPO (PickScore)
6.244
39.58%
29.58
51.71%
1,086
55.73%
23.35
89.06%
VADER (HPSv2)
6.855
97.92%
30.76
86.98 %
1.164
64.58%
22.71
48.29%
Stable Diffusion v1-5
(b) Diffusion-DPO
Diffusion-SPO
(d) Stable Diffusion XL
(e) Difiusion-DPO-DDLL.
(l) Hillusion-SPO-SDX.
Figure Quantitative winning ratios.
High-Frequency Details, users assess the level detail textures and the sharpness of fine features, such as edges and small-scale elements. For Low-Frequency Composition, users examine the overall structure and balance of the images. We conduct the user study using the prompts from Pickapic 'validation "unatuuunn with different models generating images based on the same random seed. Users evaluate the models on the three aspects mentioned above and have three choices: "No Preference" (draw), "NPO is better," "NPO is worse." We distribute questionnaires to 15 volunteers online, with each questionnaire containing 50 pairs of generated images (randomly sampled from SDXL, SD15, and DreamShaper). A total of 750 votes collected. The final results presented in the Fig. g The user study indicates that NPO significantly enhances high-frequency details in the generated outputs, while also producing colors and lighting that align more closely with human preferences. Additionally, NPO can improve the compositional structure of the generated images to some extent.
Hyper-parameter sensitivity analysis. Negative preference optimization involves a crucial trade- off regarding how much the aandiitina//negaiiv-oondiiii outputs.f f the deviation is too small, the optimization becomes ineffective; if too large, it may result in blurred or unnatural images. During training, this trade-off managed by controlling how much the weights diverge from those of the base model. Preference optimization methods, such as Diffusion- DPO, often use a regularization factor (Beta) to control the degree deviation. For inference, this trade-off is determined by how much of the positive weight offset is incorporated into the negative weight a. We use the DPO algorithm to train NPO and systematically test this trade- off. Fig. and Fig. show examples of generated images with different parameter settings and the corresponding changes in quantitative metrics. The results indicate that choosing suitable parameters significantly improve performance.
outputs deviate from the conditional

---

Published as a conference paper at ICLR 2025
User Study Analysis
000
-00
200
Metrics
Figure 8: User study analysis.
Figure Visual example ablation study on hyper-parameter choice.
(a) Acsthetic win maio
(b) HPSv2
ImageRewan miño
SickSeure ratio
cc) Aeshetic mean
(f) HPSK3 mean score
ImageRRward
(h) PickScore mean
Figure 10: Heat map-based ablation study hyper-parameter choice.
Plug-and-play. Our method is not only applicable to the original stable diffusion-based models and their fine-tuned versions optimized through preference optimization but also directly extends to high-quality stylized models fine-tuned on proprietary data. To demonstrate the versatility of our approach, we use the validation_unia _nnique dataset as our test benchmark prompts. As shown in Table 1 we observe significant improvements across various metrics. By fine-tuning the inference parameters, we enhance the performance of the DreamShaper model with 0.9B parameters, enabling it to surpass the best-performing methods the 3B SDXL model in terms of aesthetic scores. Fig 4 presents several comparative results, with notable improvements in structural integrity, contrast, and texture details.
CONCLUSIONS
In this paper, investigate that previous preference optimization methods for diffusion models have overlooked the crucial role of unconditional//egativeooomm conniitiona outputs in classifier-free guidance. plug-and-plaa infer-
We innovatively propose the task of Negative Preference Optimization ence enhancement strategy to achieve better preference-aligned generation. We summarize existing preference optimization training strategies and provide straightforward but effective adaptation for Negative Preference Optimization. Extensive experimental results validate the effectiveness of Negative Preference Optimization.
Limitations: Diffusion-NPO requires the storage and loading of two different weight offsets for inference, which results in a higher storage cost. However, fortunately, preference optimization can typically be trained with LoRA, which requires only minimal amount of additional storage.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGEMENTS
This project is funded in part by National Key R&D Program of China Project 20222D0161 100, by the Centre for Perceptual and Interactive Intelligence (CPII) Ltd under the Innovation and Technol- ogy Commission (ITC)'s s InnoHK, by NSFC-RGC Project N. N.CUHK498/24. Hongsheng Li is a PI of CPII under the InnoHK.
REPRODUCIBILITY STATEMENT
We have undertaken substantial efforts to ensure that the results in this paper are reproducible. The training and evaluation code, along with detailed guidance, is made available hhttps: // github. com//- G-/U/GUUUU U- CG-NDDfffusiiee - We believe these resources will aid in replicating our findings and foster further research that builds upon our contributions.
REFERENCES
Donghoon Ahn, Hyoungwon Cho, Jaewon Min, Wooseok Jang, Jungwoo Kim, SeonHwa Kim, Hyun Hee Park, Kyong Hwan Jin, and Seungryong Kim. Self-rectifying diffusion sampling with perturbed-attention guidance. arXiv preprint arXiv:2403.1 1tlv:2403.17555 17377, 2024.
Kevin Black, Michael Janner, Yilun Du, Ilya Kostrikov, and Sergey Levine. Training diffusion models with reinforcement learning, arXiv preprint arXiv:2305. 13301, 2023.
Haoxin Chen, Yong Zhang, Xiaodong Cun, Menghan Xia, Xintao Wang, Chao Weng, and Ying Shan. Videocrafter2: Overcoming data limitations for high-quality video diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp- 7310-7320, 2024.
Kevin Clark, Paul Vicol, Kevin Swersky, and David Fleet. Directly fine-tuning diffusion models on differentiable rewards. arXiv preprint arXiv:2309. /7400, 2023.
Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans image synthesis. Advances in neural information processing systems, 34:8780-8794, 2021.
Euge. badhand, 2024. Uttpsttss///civitai..oo httpa//a/mmmmm/oe///////////eeee
Ying Fan, Olivia Watkins, Yuqing Du, Hao Liu, Moonkyung Ryu, Craig Boutilier, Pieter Abbeel, Mohammad Ghavamzadeh, Kangwook Lee, and Kimin Lee. Reinforcement learning for fine- tuning text-to-image diffusion models. Advances in Neural Information Processing Systems, 36, 2024.
Rinon Gal, Yuval Alaluf, Yuval Atzmon, Or Patashnik, Amit H Bermano, Gal Chechik, and Daniel Cohen-Or. An image is worth word: Personalizing text-to-image generation using textual inversion, arXiv preprint arXiv:2208.01611 2022.
Jonathan Ho and Tim Salimans. arXiv:2207. 12598, 2022.
Classifier-free diffusion guidance.
arXiv preprint
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-6851, 2020.
Susung Hong. Smoothed energy guidance: Guiding diffusion models with reduced energy curvature of attention. Advances in Neural Information Processing Systems, 37:66743-667722 2025.
Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion- based generative models. Advances in neural information processing systems, 35:26565-26577, 2022.
Tero Karras, Miika Aittala, Tuomas Kynkäänniemi, Jaakko Lehtinen, Timo Aila, and Samuli Laine. Guiding diffusion model with bad version of itself. arXiv preprint arXiv:2406.0550% 2024.

---

Published as a conference paper at ICLR 2025
Bingxin Ke, Anton Obukhov. Shengyu Huang, Nando Metzger, Rodrigo Caye Daudt, and Kon- rad Schindler. Repurposing diffusion-based image generators for monocular depth estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp- 9492-9502, 2024.
Diederik Kingma, Tim Salimans, Ben Poole, and Jonathan Ho. Variational diffusion models, Ad- vances in neural information processing systems, 34:21696-21 2021.
Yuval Kirstain, Adam Polyak, Uriel Singer, Shahbuland Matiana, Joe Penna, and Omer Levy. Pick- a-pic: An open dataset of user preferences for text-to-image generation. Advances in Neural Information Processing Systems, 36:36652-36663, 2023.
Youwei Liang, Junfeng He, Gang Li, Peizhao Li, Arseniy Klimovskiy, Nicholas Carolan, Jiao Sun, Jordi Pont-Tuset, Sarah Young. Feng Yang. et al. Rich human feedback for text-to-image genera- tion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 19401-19411, 2024a.
Zhanhao Liang, Yuhui Yuan, Shuyang Gu, Bohan Chen, Tiankai Hang. Li, and Liang Zheng- Step-aware preference optimization: Aligning preference with denoising performance at each step. arXiv preprint arXiv:2406.04314, 2024b.
Luping Liu, Yi Ren, Zhijie Lin, and Zhou Zhao. Pseudo numerical methods for diffusion models on manifolds. arXiv preprint arXiv:2202.09777 2022.
Nerfgun3. badprompt, 2023. URL http:://munglggaaeeee////rtttrtr
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to fol- low instructions with human feedback. Advances in neural information processing systems, 35: 27730-27744, 2022.
William Peebles and Saining Xie. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 4195 2023.
Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhom, Jonas Müller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion models for high-resolution image synthesis. arXiv preprint arXiv:2307.01952 2023.
Mihir Prabhudesai, Anirudh Goyal, Deepak Pathak, and Katerina Fragkiadaki. Aligning text-to- image diffusion models with reward backpropagation. arXiv preprint arXiv:2310.03739, 202..
Mihir Prabhudesai, Russell Mendonca, Zheyang Qin, Katerina Fragkiadaki, and Deepak Pathak.
Video diffusion alignment via reward gradients. arXiv preprint arXiv:2407.08737, 2024
Martin L. Puterman. Markov decision processes: discrete stochastic dynamic programming. John Wiley & Sons, 2014.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36, 2024.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings the IEEE/CVF confer- ence on computer vision and pattern recognition, pp. 10684-10695, 2022.
Christoph Schuhmann.
Laion-aesthetics.
https://laio.aai/llln
laion Melon-aese aesoonnisteeiccs// 2022. Accessed: 2023-11-10.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv: 1707.007703333 0777006377, 2017.
Dazhong Shen, Guanglu Song, Zeyue Xue, Fu- Yun Wang. and Yu Liu. Rethinking the spatial inconsistency in classifier free diffusion guidance. Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pp. 9370-9379, June 2024.
12

---

Published as a conference paper at ICLR 2025
Xiaoyu Shi, Zhaoyang Huang, Fu- Yun Wang, Weikang Bian, Dasong Li, Yi Zhang, Manyuan Zhang, Ka Chun Cheung, Simon See, Hongwei Qin, et al. Motion-i2v: Consistent and controllable image-to-video generation with explicit motion modeling. arXiv e-prints, pp. arXiv-2401, 2024.
Uriel Singer, Adam Polyak, Thomas Hayes, Xi Yin, Jie An, Songyang Zhang, Qiyuan Hu, Harry Yang, Oron Ashual, Oran Gafni, et al. Make-a-video: Text-to-vides generation without text-video data. arXiv preprint arXiv:2209. /4792, 2022.
Kegiang Sun, Junting Pan, Yuying Ge, Hao Li, Haodong Duan, Xiaoshi Wu, Renrui Zhang, Aojun Zhou, Zipeng Qin, Yi Wang, al. Journeydb: A benchmark for generative image understanding. Advances in Neural Information Processing Systems, 36, 2024.
Richard S Sutton. Reinforcement learning: An introduction. A Bradford Book, 2018.
Bram Wallace, Meihua Dang, Rafael Rafailov, Linqi Zhou, Aaron Lou, Senthil Purushwalkam, Stefano Ermon, Caiming Xiong, Shafiq Joty, and Nikhil Naik. Diffusion model alignment using direct preference optimization. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. $228-8238, 2024.
Fu-Yun Yun Wang, Zhaoyang Huang. Alexander William Bergman, Dazhong Shen, Peng Gao, Michael Lingelbach, Keqiang Sun, Weikang Bian, Guanglu Song, Yu Liu, et al. Phased consistency model. arXiv preprint (xX::24455184000 2024a.
Fu- Yun Wang. Zhaoyang Huang. Weikang Bian, Xiaoyu Shi, Keqiang Sun, Guanglu Song, Yu Liu, and Hongsheng Li. Animatelcm: Computation-effiiien personalized style video generation with- out personalized video data. In SIGGRAPH Asia 2024 Technical Communications, pp. 1-5. 2024b.
Fu- Yun Wang, Zhaoyang Huang, Qiang Ma, Guanglu Song. Xudong Lu, Weikang Bian, Yijin Li, Yu Liu, and Hongsheng Li. Zola: Zero-shot creative long animation generation with short video model. In European Conference on Computer Vision, pp. 329-345. Springer, 2024c.
Fu- Yun Wang, Zhaoyang Huang, Xiaoyu Shi, Weikang Bian, Guanglu Song, Yu Liu, and Hongsheng Li. Animatelcm: Accelerating the animation of personalized diffusion models and adapters with decoupled consistency learning. arXiv preprint arXiv:2402.00769 2024d.
Fu-Yun Wang, Xiaoshi Wu, Zhaoyang Huang, Xiaoyu Shi, Dazhong Shen, Guanglu Song, Yu Liu, and Hongsheng Li. Be-your-outpainter: Mastering video outpainting through input-specific adap- tation. In European Conference on Computer Vision, pp. 153-168. Springer, 2024c.
Yu-Yun Wang, Ling Yang, Zhaoyang Huang, Mengdi Wang, and Hongsheng Li. Rectified diffusion: Straightness is not your need in rectified flow. arXiv preprint arXiv:2410.07303, 2024f.
Max Woolf. made stable diffusion xl smarter by finetuning it on bad ai-generated images, 2023.
URL htttps://nmt //m ht////miimaa imaimiiiii.... com 2023 /tt/t///////.../..a///// e- fus -oriiionnxn--oooe
Xiaoshi Wu, Yiming Hao, Keqiang Sun, Yixiong Chen, Feng Zhu, Rui Zhao, and Hongsheng Li. Human preference score v2: A solid benchmark for evaluating human preferences of text-to- image synthesis. arXiv preprint S.tiv:2306003444 2023.
Xiaoshi Wu, Yiming Hao, Manyuan Zhang, Keqiang Sun, Zhaoyang Huang, Guanglu Song, Yu Liu, and Hongsheng Li. Deep reward supervisions for tuning text-to-image diffusion models. arXiv preprint wrxiv:2405.00720 2024.
Jiazheng Xu, Xiao Liu, Yuchen Wu, Yuxuan Tong, Qinkai Li, Ming Ding, Jie Tang, and Yuxiao Dong. Imagereward: Learning and evaluating human preferences for text-to-image generation. Advances in Neural Information Processing Systems, 36, 2024.
Kai Yang. Jian Tao, Jiafei Lyu, Chunjiang Ge, Jiaxin Chen, Weihan Shen, Xiaolong Zhu, and Xiu Li. Using human feedback to finettune diffusion models without any reward model. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, pp. 8941-8951, 2024.
13

---

Published as a conference paper at ICLR 2025
Tianwei Yin, Qiang Zhang, Richard Zhang. William T Freeman, Fredo Durand, Eli Shechtman, and Xun Huang. From slow bidirectional to fast causal video generators. arXiv preprint arXiv:2412.07772. 2024.
Ruiqi Zhang, Licong Lin, Yu Bai, and Song Mei. Negative preference optimization: From catas- trophic collapse to effective unlearning. arXiv preprint arXiv:2404400586, 2024a.
Yinan Zhang, Eric Tzeng, Yilun Du, and Dmitry Kislyuk. Large-scale reinforcement learning for diffusion models. arXiv preprint arXiv:2401. 12244 2024b.
14

---

Published as a conference paper at ICLR 2025
APPENDIX
Related works
Discussion
Dillusion NPO is fundamentally different from scaling DPO
11.2
Effectiveness of NPO compared to training-free CFG-streegthening techniques)
11.3
Impact of training data negative preference pptimization
11.4
Ethical Guidelines and Limitations for Responsible Diffusion-NPO Application
4
More results
I
RELATED WORKS
In this section, we give a brief introduction to previous efforts for diffusion-based preference opti- mization.
Preference datasets and reward models. Previous works including Pick-a-pic (Kirstain et al. 2023). ImageReward (Xu et al. 2024), HPSv2 (Wu et al. 2023) collect image pairs generated by diffusion models with the same prompts and label the human preference for each pair. Laion- Aesthetic (Schuhmann 2022) asks people to rate their preference for real images from to 10. They then train the preference score models based on the preference label collected. These works lay solid foundation for future human preference optimization works in diffusion models.
Differentiable reward. Some works including DRaFT (Clark et al. 2023), AlignProp (Prabhudesai al. 2023), and ReFL. (Xu et al. 2024) directly feed the generated images into pre-trained ImageRe- ward models and update the generative model through the gradient of differentiable reward model. These works are straightforward and effective. However, due to the imperfection reward models, these methods typically have reward leakage. For example, they may generate over-saturated images to cheat higher scores.
Reinforcement learning. Some works including DDPO (Black et al. 2023). and DPOK (Fan al. 2024) propose to perceive the diffusion denoising process as a Markov decision process and apply the reinforcement learning algorithms for preference alignment. Some works (Zhang al., (024b) scale up the training for better performance. Generally, they apply PPO or the variants for training.
Direct preference optimization. Diffusion-DPO (Wallace et al. 2024) proposes a simulation-fres training objective that enables direct preference optimization on preference-labeled image pairs. D3PO Yang et al. 2024) and SPO (Liang et al. 20246) combine reinforcement learning and direct preference optimization without the requirement to know specific score values for training.
Other works related negative preference optimization. Beyond preference optimization in the realm of diffusion models, we've ve've observed certain unpublished studies and efforts related to lan- guage models that have previously referenced or explored concepts akin to negative preference. We will present them in this discussion. Some community works (Woolf) 2023 Euge 2024 Nerfgun3) 2023) directly compress the negative preference representation into the negative-conditinall inputs via textual inversion (Gal et al. 2022) to form negative text embedding However, the negative preference is relatively complex, passively suppressing certain keyword- limited capability. And language model-related works (Zhang et al. 2024a) negative prefer- ence optimization aims to unlearn target data, usually bad concepts or privacy information, which optimizes negative forget loss.
features, exhibiting

---

Published as a conference paper at ICLR 2025
II DISCUSSION
II.1 DIFFUSION NPO IS FUNDAMENTALIY DIFFERENT FROM SCALING DPO
Empirical validation. The default training iteration for both DPO and NPO is defined To 2000 iterations. To demonstrate the effectiveness of NPO compared to solely scaling up the existing DPO method, adopt the official DPO code and extend its training iterations to k x To, where A: ranges from to 10 (i.e., up to 10 x To = 20, 000 iterations, equivalent to 10 times the training cost). In the
following table, denote the training iterations of each model as k 1, 2, ..... 10. We summarize the quantitative evaluation results in Table 4 Our findings can be summarized as follows:
Regardless of how long DPO is trained, the NPO weight offset, trained with only 1 x To iterations, consistently and significantly improves overall performance.
For example, DPO (k- 1) NPO (k 1) achieves an Aesthetic Score of 5.76, HPS Score of 27.46, and PickScore of 21.47, significantly outperforming DPO (A: = 10), which requires a training cost of 10 To equivaleen to 10/(1 + 1) 5 times longer than the combined cost of DPO (k: = 1) + NPO (k- = 1) with scores of 5.626, 27.10, and 21.05, respectively.
We believe this provides strong evidence of the effectiveness of diffusion-NPO.
Table 4: Quantitative performance comparison of DPO and DPO + NPO across multiple metries. All metrics are evaluated with official weights, where training iterations are denoted as & x To (Tu 2000). Avg denotes the average score, and Win denotes the average winning ratio against other methods.
Method
Aesthetic
HPS
Image
PickScore
Avg
Win
Avg
Win
Avg
Win
Avg
Win
DPO (A 1)
5.6320
32.2%
27.17
34.0%
0.2576
42.6%
21.00
18.6%
DPO (A 1) NPO (k
5.7667
67.8%
27.46
66.0%
0.3090
57.4%
21.47
81.4%
DPO (k
5.6178
30.8%
27.21
32.2%
0.2674
41.8%
21.06
19.6%
DPO (k- 2) + NPO (k- 1)
5.7662
69.2%
27.51
67.8%
0.3239
58.2%
21.50
80.4%
DPO (k 3)
5.6214
27.8%
27.28
33.4%
0.3322
49.2%
21.08
20.6%
DPO (& = 3) + NPO (k- 1)
5.7685
72.2%
27.60
66.6%
0.3362
50.8%
21.54
79.4%
DPO (k 4)
5.6399
31.8%
27.36
33.2%
0.3574
46.0%
21.12
21.6%
DPO (k- + NPO (k
5.7767
68.2%
27.60
66.8%
0.3612
54.0%
21.55
78.4%
DPO (k-
5.6718
33.4%
27.35
31.2%
0.3540
44.4%
21.15
18.6%
DPO (k 5) + NPO (k- 1)
5.7868
66.6%
27.65
68.8%
0.3688
55.6%
21.59
81.4%
DPO (6 6)
5.6632
34.6%
27.35
34.6%
0,3664
45.8%
21.18
20.6%
DPO (k- 6) + NPO (k- 1)
5.7685
65.4%
27.63
65.4%
0.3780
54.2%
21.59
79.4%
DPO (k
5.6648
36.4%
27.37
33.0%
0.3980
45.0%
21.19
21.0%
DPO (l 7) + NPO
5.7576
63.6%
27.65
67.0%
0.4141
55.0%
21.59
79.0%
DPO (A- 8)
5.6605
37.4%
27.32
31.4%
0,3840
49.8%
21.18
19.4%
DPO (k- 8) + NPO (I: 1)
5.7544
62.6%
27.62
68.6%
0,4122
50.2%
21.58
80.6%
DPO (A= 9)
3.6438
37.6%
27.22
27.2%
0.3679
44.8%
21.10
20.8%
DPO (k 9) + NPO (k-
5,7463
62.4%
27.61
72.8%
0.4121
55.2%
21.56
79.2%
DPO (k- 10)
5,6264
39.6%
27.10
28.8%
0.3214
42.0%
21.05
19.6%
DPO (k- 10) NPO (k 1)
5.7284
60.4%
27.51
71.2%
0.3986
58.0%
21.51
80.4%
Theoretical analysis. Please note that in our motivation example, the expression ((1 ()n) serves only approximation of the negative preference-optimiize weight offset 8. Consequently, simply scaling 7 cannot be expected to fully replicate the effect of NPO. In the paper, we demon- strate that the weight adopted for unconditional negative-conditional prediction can be expressed as
P + am + 36,
where represents the pretrained model weight, v is the weight offset derived from preference optimization, and is the weight offset from negative preference optimization, with a and 3 as their respective scaling factors. In our motivating example, we substitute with (1 - a)(n)) to provide a simplified illustration of the core concept of negative preference optimization. However, this substitution assumes a parallel relationship between 1) and 8, which is an oversimplification.
In more general framework, we can decompose ö into two components:
8 5jj 51,
2

---

Published as a conference paper at ICLR 2025
Table 5: Comparison of NPO with SEG and Autoguidance on Stable Diffusion XL.
Method
Aesthetic Avg
Win
HPSV2 Avg
Win
ImageKeward Ave
Win
PickScore Ave
Win
SDXL
27.89
0.62
22.06
SEG
6.15
55.2%
26.26
10.6%
-0.10
19.4%
20.99
9.0%
Autoguidance
5.80
28.8%
27.21
27.0%
0.45
40.0%
21.44
22 22.0%
NPO
51.4%
28.78
81.2%
0,92
73.6%
22,69
82.0%
where 5n is the component parallel to y,, and 8 i is the orthogonal component. The orthogonal component 81 cannot be captured by the simplified approximation. Specifically, according to the projection theorem, the parallel component can be computed as òj proj, ()) |/ 77, leaving 6, = 5 - -1- In the motivating example, we effectively set = (1 - a) (-)), which is a scalar multiple of -n. Since scalar multiplication does not alter the direction of vectors, this implies 81 = 0 in the simplified case. However, in the general representation, ô1 exists and cannot be approximated obtained through this scalar adjustment of n, highlighting the limitations of the simplified model and the nuanced contribution of NPO.
.
11.2 EFFECTIVENESS OF NPO COMPARED TO TRAINING-FREE CFG-STRENGTHENING TECHNIQUES
To more comprehensively show the effectiveness of Diffusion-NPO. we compared Diffusion-NPO with training-fre free CFG-strengthening methods like Autoguidance (Karras et al. 2024) and SEG (Hong 2025).
Stable Diffusion XL. We designed the following comparative tests: 1) SEG on Stable Diffusion XI. vs. Naive CFG on Stable Diffusion XI.. 2) DPO-optimized Stable Diffusion XI. as conditional predictors and original Stable Diffusion XL as unconditional predictors vs. Naive CFG Stable Diffusion XL. 3) DPO-optimized Stable Diffusion XL as conditional predictors and NPO-optimized Stable Diffusion XL unconditional predictors Naive CFG Stable Diffusion XL. The evalu- ation results are shown in Table SEG slightly improves Laion Aesthetic over SDXL but performs worse in other metries, while Autoguidance produces blurry outputs due the prediction gap be- tween original and DPO-optimized SDXL, with NPO outperforming both across all metrics.
Stable Diffusion v1-5. We designed the following comparative tests: 1) DPO-optimized Stable Dif- fusion v1-5 as conditional predictors and original Stable Diffusion v1-5 as unconditional predictors vs. Naive CFG on DPO-optimized Stable Diffusion v1-5. 2) DPO-optimized Stable Diffusion v1-5 as conditional predictors and NPO-optimized Stable Diffusion v1-5 as unconditional predictors vs. Naive CFG on OPO-optimized Stable Diffusion v1-5. The results are shown Table NPO con-
Table 6: Comparison of NPO with Autoguidance on Stable Diffusion v1-5.
Method
Aesthetic Avg
Win
HPSV2 Avg
Win
ImagoKeward Avg
Win
PickSeore Avg
Win
DPO
5.65
27.19
0.27
21.12
Autoguidance
$1.2% 68.8%
26.96
34.6% 76.6%
0.13
42.0% 59.2%
21.26
$8.2%
NPO
5.76
27.60
0.31
21.58
84.4%
sistently outperforms Autoguidance on Stable Diffusion v1-5 across all metrics, while SEG was not tested due to the lack of an open-source implementation.
11.3 IMPACT OF TRAINING DATA ON NEGATIVE PREFERENCE PPTIMIZATION
In our original implementation of Diffusion-NPO, we maintained the same training data pairs, pre- processing strategies, and configurations as the original preference optimization (PO) techniques. This choice ensured a controlled comparison, allowing to isolate the impact of the NPO method itself. However, key question arises: can a more carefully curated negative dataset further enhance NPO's performance?
To investigate this, we conducted additional experiments focusing on the impact training data quality in Diffusion-DPP-bbseee NPO. Specifically, explored two altermative approaches for gen- erating negative preference data: 1) Perturbed Text Prompts: We randomly shuffled text prompt sequences within batches to create mismatched image-text pairs. This approach tests whether
3

---

Published as a conference paper at ICLR 2025
Table 7: Performance comparison of NPO trained with different data corruption strategies. Win-
Rate represents the percentage of cases where the method outperforms the original NPO.
Method
Aesthetic
HPS
ImageReward
PickScore
Avg
Win
Avg
Win
Avg
Win
Avg
Win
Original NPO
5.7621
27.60
0.3102
21.58
NPO with Text Perturbation
5.7407
45%
27.52
37.6%
0,3148
49.2%
21.55
40.4%
NPO with Data Corruption
5.7676
54%
27.63
0.3222
54.0%
21.60
54.4%
more diverse but less semantically aligned negative dataset influences performance. 2) Random Im- age Corruptions: We applied controlled degradations, such as Gaussian blur, to negative images to analyze whether introducing noise helps refine NPO's learning process.
Our experimental results, summarized in Table 7 indicate that while text perturbations led to per- formance degradation, mild image corruptions slightly improved results. This suggests that while NPO benefits from a certain degree of data diversity, maintaining proper image-text alignment re- mains crucial. Notably, extreme corruptions resulted in significant output deviations, highlighting the trade-off between introducing variation and preserving meaningful training signals.
These findings reinforce the importance of data selection negative preference training. While NPO can operate effectively using standard preference optimization datasets, strategic modifications negative data can enhance its performance. This insight opens up new avenues for refining NPO through optimized data augmentation and curation strategies.
11.4 ETHICAL GUIDELINES AND LIMITATIONS FOR RESPONSIBLE DIFFUSION-NPO APPLICATION
Diffusion-NPO enhances model outputs by steering generative models away from undesirable sults, aligning with responsible Al principles. To ensure ethical deployment, the following guide- lines must be observed:
Preventing Malicious Use: Diffusion-NPP must be restricted to improving user experience and reducing harm, with safeguards against generating misinformation, bias, or deceptive content.
Transparent Disclosure: Its implementation should be clearly documented to foster trust and accountability, especially in sensitive domains.
Ethical Oversight: Regular audits and ethical reviews should assess potential unintended conse- quences to ensure responsible use.
Focused Scope: Diffusion-NPO should be applied selectively in high-stakes areas to prevent
overfitting and maintain broad applicability.
By adhering to these principles, Diffusion-NPO can effectively align generative models with ethical AI standards while minimizing risks.
III MORE RESULTS
4

---

Published as a conference paper at ICLR 2025
Stable Diffusion
Prompt: "a woman in silver suit with ponytall a detailed painting by WLOP, trending on Artstation, fantasy art. detailed painting, aristation hd. high detail"
Prompt: "A beautiful 25 yearol old whes mother from bong kong and father from turkey"
Prompt: "A house in the style of Escher"
Prompt:: "Wafencolour painting of an orange cat"
Ial
Prompt: 'Mili, pin' hair, that awesore time got edincarnatoi as slime"
Prompt "Hypereal lstic leng portrait gorgeous goddess standing fickd full of flowers ... lover 30 words?
Prompt: "female face, lct green eyes, long hair, slant eyes, cheeky cheeks, smiling, carefree, (over 20 words)"
Prompt gA eagle monster art"
Prompt: "Detailed painting of Attractive young women painting, model, detailed, cineraaic lightring.
Prompt: anime woman"
w/o NPO
w/ NPO
w/o NPO
w/ NPO
Figure 11: Comparison Stable Diffusion 1.5.

---

Published as a conference paper at ICLR 2025
DreamShaper
Prompt: mermaid playing chess with Golphin"
Prompt: "A living dragon on the world trade contorr"
"Dragon with six pairs wings stemoriking humans in. a village near volcano"
Prempt:
Prompt: Fist Fart
Prompt:
"Painting afa black 22 year old girl with long braids, she has her eyes openeú, highly detailed, style"
Prompt: "A panda rijing matorcyde"
Prompt:: "A drng and Santa Claus. Christraas trees in backgroung Black and white background"
Prompt:: "An evil villain holding mini Earth"
Prompt: "A jade statue an adorable att
Prompt: 'An anthopomorpiicaatmma
w/o NPO
w/ NPO
w/o NPO
w/ NPO
Figure 12: Comparison DreamShaper.

---

Published as a conference paper at ICLR 2025
Stable Diffsuion XL
Prompt: cat, fat, chubby, very fine wispy and extremely long swirly wavy for (over words?"
Prompt: "Human palm"
Prompt: "LeBron James slam dunking the planet saturn through its own Hings"
Prompt:: "Jessica alba, anime style"
Prompt: "A woman with blue eyes"
Prompt: gifinka black cat sushi chef
JJOS
Prompt: "A boss screarning at his empleyee for not working on the weekend by vincent van gogi""
Prompt: "Concept art, Disney, really cray cesture, colored pencils, cute, very creative drawing. ... Caver 50 words)"
Prompt: "Realistic photo with light pink background
Prompt: "A 20 yo girl in cyberpunk outfit"
color various shades, middle-aget ... Cover 30 wardss"
w/o NPO
w/ NPO
w/o NPO
w/ NPO
Figure 13: Comparison Stable Diffusion XI..

---

Published as a conference paper at ICLR 2025
Diffusion-DPO
Prampt:: "Wild man with breanee ase, ring armor and furs, wielding shield"
Prompt: "Fantasy warrior"
Prompt: "A beautiful natural woman"
Prompt: govllla"
Prompt: "God Hades Cotham like city. cyberpunk, up close, cinematic, neun"
Prompt "Sumeet reflecting on crystal ball, factory filled with android girls'
Prompt: "Smooth shading"
Prompt: "An attractive young woman rolling her cyes"
Prompt:: "A blue car"
Prompt:: "Heaven"
w/o NPO
w/ NPO
w/o NPO
w/ NPO
Figure 14: Comparison Diffusion-DPO.
8

---

Published as a conference paper at ICLR 2025
Diffusion-SPO
Prompt: "A bass scneaming at his employee not wearking on the weekend by vinsent van gagh"
Prompt: "Rachel Arberdd 5 wearing black skirt. Thin bady type, Young face, Sony Alphe A7 ill... Cover 30 words)"
Promot: PPhoooralistic style, photorealisiic pope francis wearing drip footwear. drip tents"
Prompt: Japaaeese children ballet school
Prompt: "Michael (ordan against bruce lee The strigght Hast round kick in the air niq basketball ball ... (ever 30 words)" Prompt: "Randor gidl hug: Henry Cavill superman"
Prompt: "Highly detailed realistic photograph a hand"
Prompt: "Sturdy and pink pickup truck"
Prompt: "3d render an ultrarcalitic exeature design, ONI entity with white long flowing hair"
Prompt: "A hot female Alex from Minecraft
w/o NPO
w/ NPO
w/o NPO
w/ NPO
Figure 15: Comparison Diffusion-SPO.


Reviews:
[{"review_id": "BlMDhvkwhl", "reviewer": "Reviewer_ka33", "summary": "This paper presents a simple and general method for aligning images generated from diffusion models with human preference. The key idea is to distinguish not only favorable images but also undesired ones. To accomplish this, the paper builds on various existing preference optimization approaches and propose negative preference optimization (NPO), which in essence reverses the order of the ranked image pairs for training a second set of diffusion weights that emphasize negative images. The experiments demonstrate strong qualitative results and favorable user study results in comparison to baselines without NPO.", "questions": "I am wondering whether it is optimal to share the same training data for PO and NPO. Humans prefer certain images for obvious reasons: higher visual quality and/or stronger semantic alignment with the text prompt. Oftentimes, the worse image in a training pair for PO still looks reasonable in quality and semantic alignment. The pool of negative images, however, can be larger and more diverse. For example, a negative image can be completely irrelevant to the text prompt, or can have arbitrary artifacts.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Simplicity and generality. The method is simple, intuitive, and can augment a variety of existing preference alignment approaches without additional training data and learning objectives.\n- Strong results. The method achieves superior qualitative and quantitative results in comparison to baselines not using NPO.\n- Efficiency. The method obtain stronger results without losing inference-time efficiency.", "weaknesses": "I did not find major weaknesses of the paper.\n\nDisclaimer: While I find the approach interesting and reasonable, I am not an expert on preference optimization of diffusion models, so I am not able to comment on the novelty of the method and the selection of baselines / benchmarks.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ACwNZiBqTS", "reviewer": "Reviewer_Fyov", "summary": "This paper introduces Negative Preference Optimization (NPO), a novel approach for enhancing the preference alignment of diffusion models by explicitly training the model to recognize and avoid generating outputs that are misaligned with human preferences.The proposed NPO aims to improve diffusion model outputs by creating a separate model component trained to avoid generating undesirable features.  By training models to understand both positive and negative preferences, NPO enhances the effectiveness of classifier-free guidance, which relies on balancing conditional and negative-conditional outputs to improve output quality.The writers designed the NPO to integrate easily with existing diffusion models, such as SD1.5, SDXL, and various preference-optimized versions, improving image and video quality across these models without requiring significant modifications.The paper provides quantitative and qualitative evaluations across several metrics (HPSv2, ImageReward, PickScore, Laion-Aesthetic) and demonstrates that NPO consistently enhances image quality, color accuracy, structural coherence, and alignment with human aesthetic preferences. Overall, this work addresses a gap in preference optimization by focusing on both desirable and undesirable outputs, improving human-aligned generation across diverse applications in image and video synthesis.", "questions": "1. How does NPO distinguish between truly negative attributes and those that are simply neutral or contextual?\n2. What are the computational costs of NPO, especially in terms of memory and processing time for real-time applications?\n3. How sensitive is NPO's performance to parameter settings, and what are the recommended heuristics for tuning them effectively?\n4. How well does NPO perform on domain-specific datasets like medical imaging or abstract art?\n5. Beyond quantitative metrics, what is the perceived quality improvement from NPO based on qualitative user evaluations?\n6. How does NPO mitigate the risk of reinforcing biases present in preference datasets?\n7. Have you discussed ethical guidelines and limitations on NPO's use to ensure responsible application?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This paper introduces a novel technique called Negative Preference Optimization (NPO) to improve the alignment of diffusion models with human preferences. Unlike traditional methods that focus solely on desirable features, NPO addresses the problem of undesirable outputs by training the model to recognize and avoid them. This innovative approach is both creative and practical, as it leverages existing preference data by simply reversing image pair rankings. The paper demonstrates that this simple technique can significantly improve the effectiveness of classifier-free guidance without requiring complex new datasets or training procedures.\n\nThe paper is well-executed, providing both theoretical insights and practical validation. It offers a clear and well-organized presentation, with a logical flow from problem statement to methodology and experimental results. The proposed method is grounded in a thoughtful analysis of classifier-free guidance, recognizing the critical role of conditional and negative-conditional outputs in achieving preference alignment. This theoretical foundation is reinforced by a comprehensive suite of quantitative evaluations (using metrics like HPSv2, ImageReward, and PickScore) and qualitative comparisons (sample images with and without NPO) that effectively demonstrate the improvements NPO brings to various diffusion models, including SD1.5, SDXL, and video diffusion models. The experiments are carefully structured to show the plug-and-play nature of NPO, further adding to the paper’s quality. Although some technical concepts may be challenging, the paper is structured to guide readers through them systematically.\n\nThe significance of this work is high, as it provides an effective, adaptable solution to a widely recognized issue in diffusion-based image and video generation: producing outputs that align well with human aesthetics and avoid undesired qualities. NPO addresses a core limitation in preference optimization approaches—specifically, the lack of attention to undesirable outputs—while remaining compatible with popular models and training frameworks. The paper’s contributions are likely to influence future work in human preference alignment, particularly in fields where aesthetic quality and user satisfaction are critical, such as digital art, content creation, and interactive AI. Furthermore, NPO’s plug-and-play compatibility makes it a practical choice for both researchers and developers aiming to improve generation quality without extensive re-training or model modification, boosting its applicability across the field.\n\nOverall, the paper presents an original, well-supported, and clearly articulated contribution that addresses a key gap in diffusion model preference optimization, with practical implications for a broad range of image and video generation applications.", "weaknesses": "1. While the paper's introduction of Negative Preference Optimization (NPO) is innovative, the simple reversal of preference pair rankings may oversimplify the complexity of human aesthetics. Negative preferences are not always straightforward opposites of positive preferences, and undesirable features can be subtle or context-dependent.\n\n2. Metrics-driven approach lacking user perspective\n\n3. The paper validates NPO primarily on general-purpose datasets and models like Stable Diffusion and DreamShaper. While these are commonly used in text-to-image generation, they may not represent the variety of domains where preference alignment is crucial, such as medical imaging, scientific visualization, or highly specialized art styles.\n\n4. The performance implications of NPO's dual weight system, especially for large-scale or real-time applications, are not fully explored.\n\n5. the paper's focus on CFG-based models limits the exploration of NPO's compatibility with other diffusion architectures.\nNPO's compatibility with non-CFG diffusion models remains unclear.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "xdUUB74OCQ", "reviewer": "Reviewer_5Mi5", "summary": "This paper tackles the task of aligning diffusion-based generative models with human preferences. The authors propose training an additional model that aligns with the opposite of human preferences. At inference, this model serves as the unconditional or negative-conditional component in classifier-free guidance. The method is simple to implement—requiring only a reward inversion (by multiplying it by -1) for reward-based methods or reversing the order of preferred image pairs for DPO-based methods. Evaluations on various text-to-image and text-to-video models show both qualitative and quantitative improvements across the board.", "questions": "Recently, several works have improved text-to-image models by changing the unconditional or negative-conditional component in classifier-free guidance (CFG). The general idea is to use a “worse” version of the model for the second part of CFG. For example, Autoguidance[1] uses an earlier checkpoint in training, and SEG[2] manually corrupts some internal features in the model’s attention layers. Some of these methods don’t need extra training, so it would be interesting to compare them to Diffusion-NPO. While the paper already makes a similar comparison summarized in Figure 5, expanding on this could help show how much improvement comes from using negative preferences versus just a “worse” model.\n\n[1] Karras, Tero, et al. \"Guiding a Diffusion Model with a Bad Version of Itself.\" *arXiv preprint arXiv:2406.02507* (2024).\n\n[2] Hong, Susung. \"Smoothed Energy Guidance: Guiding Diffusion Models with Reduced Energy Curvature of Attention.\" *arXiv preprint arXiv:2408.00760* (2024).", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "- The main idea is very intuitive, simple to implement, and highly effective.\n- The method is compared against several baseline alignment methods on multiple image and video diffusion models. The proposed algorithm is shown to be an improvement using various quality metrics and human user studies.\n- The proposed technique is quite general, and can be used alongside any alignment algorithm.", "weaknesses": "The paper doesn’t introduce a new alignment technique; instead, it builds on existing alignment algorithms to train the negative-aligned model. This has both pros and cons: on the plus side, it can work alongside any alignment method, but on the downside, its quality is limited by the performance of the alignment algorithm used.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "r5U2Ahxtul", "reviewer": "Reviewer_5t3C", "summary": "Many recent methods, such as diffusion DPO, fine-tune pretrained models using preference optimization. This paper highlights that while these methods focus on selecting preferred samples, they overlook the unconditional or negative outputs used in CFG. The paper addresses this by training a model using reverse DPO data for negative preference optimization, then incorporating this model into CFG to move outputs away from the negative values, resulting in improved image quality.", "questions": "- Could you compare the results of this method to those of more intensive DPO training (e.g., running DPO for a longer period)?\n- If the original pretrained model is used instead of the NPO-trained model to obtain unconditional/negative outputs, is there a significant performance drop?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper shows that using a model optimized with a preference for CFG's unconditional/negative term enhances performance.\n- Qualitatively, the images generated with NPO show fewer artifacts and are sharper.\n- This approach is simple to implement, as it only requires using the existing diffusion DPO method without additional data.", "weaknesses": "- The paper seems underdeveloped in certain areas. For instance, Table 2 displays scores but lacks discussion in the main text, and Figure 8’s caption does not specify the methods compared.\n- It does not clearly demonstrate the difference from scaling up the existing DPO method. The results might be similar to running DPO for a longer period. In the motivating example of Section 2, δ appears proportional to η. Subtracting the resulting model's output seems equivalent to scaling η in the DPO model’s unconditional parameter.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)