Diffusion-NPO: Negative Preference Optimization for Better Preference Aligned Generation of Diffusion Models

Authors: Fu-Yun Wang, Yunhao Shui, Jingtan Piao, Keqiang Sun, Hongsheng Li
Primary area: alignment, fairness, safety, privacy, and societal considerations
Keywords: Diffusion, Preference Optimization

Abstract:
Diffusion models have made substantial advances in image generation, yet models trained on large, unfiltered datasets often yield outputs misaligned with human preferences. Numerous methods have already been proposed to fine-tune pre-trained diffusion models, achieving notable improvements in aligning generated outputs with human preferences. However, we point out that existing preference alignment methods neglect the critical role of handling unconditional/negative-conditional  outputs, leading to a diminished capacity to avoid generating undesirable outcomes. This oversight limits the efficacy of classifier-free guidance (CFG), which relies on the contrast between conditional generation and unconditional/negative-conditional generation to optimize output quality. In response, we propose a straightforward but consistently effective approach that involves training a model specifically attuned to negative preferences. This method does not require new training strategies or datasets but rather involves minor modifications to existing techniques. Our approach integrates seamlessly with models such as SD15, SDXL, video diffusion models and models that have undergone preference optimization, consistently enhancing their ability to produce more human preferences aligned outputs.

Decision: accept (poster)
Reviewer scores: 8, 6, 8, 6

Reviews and discussion:
<thread>
creation date:
1730547593292

thread_type:
official_review

<thread_object>
creator: reviewer_5t3c

<content>
summary:
Many recent methods, such as diffusion DPO, fine-tune pretrained models using preference optimization. This paper highlights that while these methods focus on selecting preferred samples, they overlook the unconditional or negative outputs used in CFG. The paper addresses this by training a model using reverse DPO data for negative preference optimization, then incorporating this model into CFG to move outputs away from the negative values, resulting in improved image quality.

strengths:
- The paper shows that using a model optimized with a preference for CFG's unconditional/negative term enhances performance.
- Qualitatively, the images generated with NPO show fewer artifacts and are sharper.
- This approach is simple to implement, as it only requires using the existing diffusion DPO method without additional data.

weaknesses:
- The paper seems underdeveloped in certain areas. For instance, Table 2 displays scores but lacks discussion in the main text, and Figure 8’s caption does not specify the methods compared.
- It does not clearly demonstrate the difference from scaling up the existing DPO method. The results might be similar to running DPO for a longer period. In the motivating example of Section 2, δ appears proportional to η. Subtracting the resulting model's output seems equivalent to scaling η in the DPO model’s unconditional parameter.

questions:
- Could you compare the results of this method to those of more intensive DPO training (e.g., running DPO for a longer period)?
- If the original pretrained model is used instead of the NPO-trained model to obtain unconditional/negative outputs, is there a significant performance drop?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730691890288

thread_type:
official_review

<thread_object>
creator: reviewer_5mi5

<content>
summary:
This paper tackles the task of aligning diffusion-based generative models with human preferences. The authors propose training an additional model that aligns with the opposite of human preferences. At inference, this model serves as the unconditional or negative-conditional component in classifier-free guidance. The method is simple to implement—requiring only a reward inversion (by multiplying it by -1) for reward-based methods or reversing the order of preferred image pairs for DPO-based methods. Evaluations on various text-to-image and text-to-video models show both qualitative and quantitative improvements across the board.

strengths:
- The main idea is very intuitive, simple to implement, and highly effective.
- The method is compared against several baseline alignment methods on multiple image and video diffusion models. The proposed algorithm is shown to be an improvement using various quality metrics and human user studies.
- The proposed technique is quite general, and can be used alongside any alignment algorithm.

weaknesses:
The paper doesn’t introduce a new alignment technique; instead, it builds on existing alignment algorithms to train the negative-aligned model. This has both pros and cons: on the plus side, it can work alongside any alignment method, but on the downside, its quality is limited by the performance of the alignment algorithm used.

questions:
Recently, several works have improved text-to-image models by changing the unconditional or negative-conditional component in classifier-free guidance (CFG). The general idea is to use a “worse” version of the model for the second part of CFG. For example, Autoguidance[1] uses an earlier checkpoint in training, and SEG[2] manually corrupts some internal features in the model’s attention layers. Some of these methods don’t need extra training, so it would be interesting to compare them to Diffusion-NPO. While the paper already makes a similar comparison summarized in Figure 5, expanding on this could help show how much improvement comes from using negative preferences versus just a “worse” model.

[1] Karras, Tero, et al. "Guiding a Diffusion Model with a Bad Version of Itself." *arXiv preprint arXiv:2406.02507* (2024).

[2] Hong, Susung. "Smoothed Energy Guidance: Guiding Diffusion Models with Reduced Energy Curvature of Attention." *arXiv preprint arXiv:2408.00760* (2024).
</content>
</thread_object>

</thread>
<thread>
creation date:
1730717072846

thread_type:
official_review

<thread_object>
creator: reviewer_fyov

<content>
summary:
This paper introduces Negative Preference Optimization (NPO), a novel approach for enhancing the preference alignment of diffusion models by explicitly training the model to recognize and avoid generating outputs that are misaligned with human preferences.The proposed NPO aims to improve diffusion model outputs by creating a separate model component trained to avoid generating undesirable features.  By training models to understand both positive and negative preferences, NPO enhances the effectiveness of classifier-free guidance, which relies on balancing conditional and negative-conditional outputs to improve output quality.The writers designed the NPO to integrate easily with existing diffusion models, such as SD1.5, SDXL, and various preference-optimized versions, improving image and video quality across these models without requiring significant modifications.The paper provides quantitative and qualitative evaluations across several metrics (HPSv2, ImageReward, PickScore, Laion-Aesthetic) and demonstrates that NPO consistently enhances image quality, color accuracy, structural coherence, and alignment with human aesthetic preferences. Overall, this work addresses a gap in preference optimization by focusing on both desirable and undesirable outputs, improving human-aligned generation across diverse applications in image and video synthesis.

strengths:
This paper introduces a novel technique called Negative Preference Optimization (NPO) to improve the alignment of diffusion models with human preferences. Unlike traditional methods that focus solely on desirable features, NPO addresses the problem of undesirable outputs by training the model to recognize and avoid them. This innovative approach is both creative and practical, as it leverages existing preference data by simply reversing image pair rankings. The paper demonstrates that this simple technique can significantly improve the effectiveness of classifier-free guidance without requiring complex new datasets or training procedures.

The paper is well-executed, providing both theoretical insights and practical validation. It offers a clear and well-organized presentation, with a logical flow from problem statement to methodology and experimental results. The proposed method is grounded in a thoughtful analysis of classifier-free guidance, recognizing the critical role of conditional and negative-conditional outputs in achieving preference alignment. This theoretical foundation is reinforced by a comprehensive suite of quantitative evaluations (using metrics like HPSv2, ImageReward, and PickScore) and qualitative comparisons (sample images with and without NPO) that effectively demonstrate the improvements NPO brings to various diffusion models, including SD1.5, SDXL, and video diffusion models. The experiments are carefully structured to show the plug-and-play nature of NPO, further adding to the paper’s quality. Although some technical concepts may be challenging, the paper is structured to guide readers through them systematically.

The significance of this work is high, as it provides an effective, adaptable solution to a widely recognized issue in diffusion-based image and video generation: producing outputs that align well with human aesthetics and avoid undesired qualities. NPO addresses a core limitation in preference optimization approaches—specifically, the lack of attention to undesirable outputs—while remaining compatible with popular models and training frameworks. The paper’s contributions are likely to influence future work in human preference alignment, particularly in fields where aesthetic quality and user satisfaction are critical, such as digital art, content creation, and interactive AI. Furthermore, NPO’s plug-and-play compatibility makes it a practical choice for both researchers and developers aiming to improve generation quality without extensive re-training or model modification, boosting its applicability across the field.

Overall, the paper presents an original, well-supported, and clearly articulated contribution that addresses a key gap in diffusion model preference optimization, with practical implications for a broad range of image and video generation applications.

weaknesses:
1. While the paper's introduction of Negative Preference Optimization (NPO) is innovative, the simple reversal of preference pair rankings may oversimplify the complexity of human aesthetics. Negative preferences are not always straightforward opposites of positive preferences, and undesirable features can be subtle or context-dependent.

2. Metrics-driven approach lacking user perspective

3. The paper validates NPO primarily on general-purpose datasets and models like Stable Diffusion and DreamShaper. While these are commonly used in text-to-image generation, they may not represent the variety of domains where preference alignment is crucial, such as medical imaging, scientific visualization, or highly specialized art styles.

4. The performance implications of NPO's dual weight system, especially for large-scale or real-time applications, are not fully explored.

5. the paper's focus on CFG-based models limits the exploration of NPO's compatibility with other diffusion architectures.
NPO's compatibility with non-CFG diffusion models remains unclear.

questions:
1. How does NPO distinguish between truly negative attributes and those that are simply neutral or contextual?
2. What are the computational costs of NPO, especially in terms of memory and processing time for real-time applications?
3. How sensitive is NPO's performance to parameter settings, and what are the recommended heuristics for tuning them effectively?
4. How well does NPO perform on domain-specific datasets like medical imaging or abstract art?
5. Beyond quantitative metrics, what is the perceived quality improvement from NPO based on qualitative user evaluations?
6. How does NPO mitigate the risk of reinforcing biases present in preference datasets?
7. Have you discussed ethical guidelines and limitations on NPO's use to ensure responsible application?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730888763862

thread_type:
official_review

<thread_object>
creator: reviewer_ka33

<content>
summary:
This paper presents a simple and general method for aligning images generated from diffusion models with human preference. The key idea is to distinguish not only favorable images but also undesired ones. To accomplish this, the paper builds on various existing preference optimization approaches and propose negative preference optimization (NPO), which in essence reverses the order of the ranked image pairs for training a second set of diffusion weights that emphasize negative images. The experiments demonstrate strong qualitative results and favorable user study results in comparison to baselines without NPO.

strengths:
- Simplicity and generality. The method is simple, intuitive, and can augment a variety of existing preference alignment approaches without additional training data and learning objectives.
- Strong results. The method achieves superior qualitative and quantitative results in comparison to baselines not using NPO.
- Efficiency. The method obtain stronger results without losing inference-time efficiency.

weaknesses:
I did not find major weaknesses of the paper.

Disclaimer: While I find the approach interesting and reasonable, I am not an expert on preference optimization of diffusion models, so I am not able to comment on the novelty of the method and the selection of baselines / benchmarks.

questions:
I am wondering whether it is optimal to share the same training data for PO and NPO. Humans prefer certain images for obvious reasons: higher visual quality and/or stronger semantic alignment with the text prompt. Oftentimes, the worse image in a training pair for PO still looks reasonable in quality and semantic alignment. The pool of negative images, however, can be larger and more diverse. For example, a negative image can be completely irrelevant to the text prompt, or can have arbitrary artifacts.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732334147537

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General response [1/3]

comment:
We sincerely thank all reviewers for their valuable comments and suggestions. We are sincerely grateful to the reviewers for dedicating their time and effort to review our work. We are delighted to see reviewers commenting on our work with "`innovative`", "`high significance`", and "`intuitive, simple, general, and highly effective`".


In this rebuttal, we try our best to solve the concerns of reviewers. We summarize the important and common concerns in the following:


> **Comparison with training-free CFG enhancement approaches [1] [2] (Reviewer 5Mi5, 5t3C)**

Specifically, we test with the following comparisons:

1. SEG on Stable Diffusion XL  vs Naive CFG [3] on Stable Diffusion XL. 
2. DPO-optimized Stable Diffusion XL as conditional predictors and original Stable Diffusion XL as the unconditional predictors vs Naive CFG on Stable Diffusion XL.
3. DPO-optimized Stable Diffusion XL as conditional predictors and NPO-optimized Stable Diffusion XL as unconditional predictors vs Naive CFG on Stable Diffusion XL.

Note that the original Stable Diffusion XL can be viewed as an earlier version of DPO-optimized Stable Diffusion XL. Therefore, we adopt 2 as the baseline of Autoguidance.

We generate all the images with the same seed and the same prompts from the test_unique split as the text prompts. We score generated images with the HPSv2, PickScore, ImageReward, and Laion Aesthetic and compare their average scores and win-rates.

The results are shown in the following table. All the win-rate are compared with original SDXL.

| Method           | AES Mean Score | AES Win-Rate | HPS Mean Score | HPS Win-Rate | ImageReward Mean Score | ImageReward Win-Rate | PickScore Mean Score | PickScore Win-Rate |
| ---------------- | -------------- | ------------ | ------------------ | ------------ | ---------------------- | -------------------- | ------------------------ | ------------------ |
| SDXL             | 6.1142         | -            | 27.89              | -            | 0.6246                 | -                    | 22.06                    | -                  |
| SEG[2]           | 6.1498         | 55.2%        | 26.26              | 10.6%        | -0.1042                | 19.4%                | 20.99                    | 9.0%               |
| AutoGuidance [1] | 5.8049         | 28.8%        | 27.21              | 27.0%        | 0.4543                 | 40.0%                | 21.44                    | 22.0%              |
| NPO (Ours)       | 6.1116         | 51.4%        | 28.78              | 81.2%        | 0.9210                 | 73.6%                | 22.69                    | 82.0%              |

We can see that SEG only achieves minor improvement on Laion-Aesthetic compared to the original baseline of SDXL. All the other metrics show that SEG significantly underperforms the naive CFG strategy with SDXL. This can be attributed to that SEG was originally proposed to tackle the circumstances where the prompt is not used. When applying AutoGuidance, we find the outputs tend to be blurry. Our quantitative evaluation also proves our claim. This can be attributed to the prediction difference between original SDXL and DPO-optimized SDXL being too large (as analyzed in lines 340-344)

We also test on the Stable Diffusion v1-5 for further comparison:

1. DPO-optimized Stable Diffusion v1-5 as conditional predictors and original Stable Diffusion v1-5 as the unconditional predictors vs Naive CFG on DPO-optimized Stable Diffusion v1-5.

2. DPO-optimized Stable Diffusion v1-5 as conditional predictors and NPO-optimized Stable Diffusion v1-5 as the unconditional predictors vs Naive CFG on DPO-optimized Stable Diffusion v1-5.

We did not test SEG on Stable Diffusion v1-5 since we did not find open-source implementation of SEG on Stable Diffusion v1-5. 

The results are shown in the following table. All the win-rate are compared with dpo-optimized Stable Diffusion v1-5.

| Method           | AES Mean Score | AES Win Rate | HPS Mean Score | HPS Win Rate | ImageReward Mean Score | ImageReward Win Rate | PickScore Mean Score | PickScore Win Rate |
| ---------------- | -------------- | ------------ | ------------------ | ------------ | ---------------------- | -------------------- | ------------------------ | ------------------ |
| DPO              | 5.6484         | -            | 27.19              | -            | 0.2652                 | -                    | 21.12                    | -                  |
| AutoGuidance [1] | 5.6277         | 51.2%        | 26.96              | 34.6%        | 0.1251                 | 42.0%                | 21.26                    | 58.2%              |
| NPO              | 5.7621         | 68.8%        | 27.60              | 76.6%        | 0.3102                 | 59.2%                | 21.58                    | 84.4%              |

NPO still consistently outperforms the compared baselines.
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper addresses the problem of preference optimization for diffusion model and identifies an overlooked component: negative preference. The paper proposes to use standard PO techniques but specifically train the diffusion model to be able to control how far the sample is pushed against negative preference (defined as the negation of the normal reward model). Then at inference time, the model integrates this negative preference in the standard CFG formulation as part of the normally unconditional generation. Specifically a merged weight baseline is proposed where the DPO is used for the positive signal and a merged model of DPO and original for the negative signal. The paper then shows that the merged model performs better than a standard DPO model. The paper shows that the merged model performs better than the standard one. The paper also shows that this can improve text to video as well.

The reviewers were mostly positive on the paper mentioning few weaknesses. The reviewers mentioned that they found the paper innovative, simple, and effective.

I agree in general with the reviewers and advocate for acceptance of the paper. I find that the strengths of the paper (the simplicity and effectiveness as well as the very good presentation) outweigh the weaknesses (could be more thorough on the related works, could have done more exploration of the "negative" formulation – e.g. I think exploring things like corruptions or a different signal than the negation of the reward model would have been more interesting).

As a quick side-note, there have been some explorations into using "bad" images as a negative signal. For example, I remember this post from 2023 (https://minimaxir.com/2023/08/stable-diffusion-xl-wrong/). Additionally there is actually a fairly rich literature of work exploring this exact topic (explicitly modeling negative preference) for language. However, I do not think that the authors should be penalized for work in the general public or different fields. I think in the interest of academic honesty / transparency, I would recommend including them on a website in a section named "other work" or "similar ideas", but since they're not published and/or in different fields, I don't think this is 100% required.

Note: THIS IS REQUIRED. The bottom right example of Figure 11 "preteen girls with no underware..." and the top left of Figure 15 "boy climbing into oven, lewd" MUST BE REPLACED for the paper to be accepted.