Dynamic Negative Guidance of Diffusion Models

Authors: Felix Koulischer, Johannes Deleu, Gabriel Raya, Thomas Demeester, Luca Ambrogioni
Primary area: generative models
Keywords: Classifier-free guidance, Negative prompting, Diffusion model guidance

Abstract:
Negative Prompting (NP) is widely utilized in diffusion models, particularly in text-to-image applications, to prevent the generation of undesired features. In this paper, we show that conventional NP is limited by the assumption of a constant guidance scale, which may lead to highly suboptimal results, or even complete failure, due to the non-stationarity and state-dependence of the reverse process. Based on this analysis, we derive a principled technique called ***D**ynamic **N**egative **G**uidance*, which relies on a near-optimal time and state dependent modulation of the guidance without requiring additional training. Unlike NP, negative guidance requires estimating the posterior class probability during the denoising process, which is achieved with limited additional computational overhead by tracking the discrete Markov Chain during the generative process. We evaluate the performance of DNG class-removal on MNIST and CIFAR10, where we show that DNG leads to higher safety, preservation of class balance and image quality when compared with baseline methods. Furthermore, we show that it is possible to use DNG with Stable Diffusion to obtain more accurate and less invasive guidance than NP.

Decision: accept (poster)
Reviewer scores: 8, 5, 6, 6

Reviews and discussion:
<thread>
creation date:
1729971900310

thread_type:
official_review

<thread_object>
creator: reviewer_jr45

<content>
summary:
The paper presents Dynamic Negative Guidance (DNG), a novel technique for improving Negative Prompting (NP) in diffusion models, specifically addressing limitations in text-to-image applications. Conventional NP assumes a fixed guidance scale to suppress undesired features, which can result in poor performance due to the non-stationary and state-dependent nature of the reverse process in diffusion models. DNG overcomes this by introducing an adaptive approach that adjusts the guidance dynamically based on time and state, thereby refining the model’s ability to avoid generating unwanted features.

strengths:
1. the DNG method in novel and has strong practical significance
2. paper did a comprehensive evaluation on safety, class balance, and image quality on MNIST and CIFAR10.
3. the paper is straightforward and easy to follow

weaknesses:
1. terms like Dynamic Negative Guidance and guidance scale could better deserve a brief contextual note, as not all readers may be familiar with their meaning in this context. 
2. the paper needs to avoid jargon without explanation.
3. the paper needs to improve flow and conciseness.

questions:
see weakness
</content>
</thread_object>

</thread>
<thread>
creation date:
1730011100314

thread_type:
official_review

<thread_object>
creator: reviewer_v6iu

<content>
summary:
In text-to-image generation, negative prompts are used to guide the generative model not to create something. This paper presents a method called dynamic negative guidance that aims to be safer, less invasive than regular negative prompting in guiding a T2I model to create desirable images while avoiding unwanted components. The dynamic negative guidance relies on a near-optimal time and state dependent modulation of the guidance without requiring additional training. The method has been tested in MNIST and CIFAR-10.

strengths:
A strength of the proposed method, DNG, is to estimate the posterior by tracking the discrete Markov chain during the denoising process. The strength of the guidance is dynamically related to how close the negative prompt is related to the positive prompt. This seems to a strength of the method since it can adaptively determine whether the negative prompt is even relevant at all. To the contrary, existing negative prompting methods may not be able to ignore irrelevant negative prompts. The proposed method may overcome the weakness of existing NP methods that blankly try to invert the force field to move away from the positive prompts without precisely moving away from the negative prompts. This is reflected in the factor pt(c-|x)/(1−pt(c-|x)) in Eq 10.

weaknesses:
Writing can be better and consistent. For example, the paper has both c_- and c-, it should be consistent.
The biggest weakness with the paper is the example images given at the end, starting from page 22. First, the pictures are so small, it is hard to appreciate the difference between NP results and DNG results. Second, I could be missing something, but it seems there is no big difference between the two types of results. In particular, it seems NP results are not bad or accidently include the undesirable features given in the negative prompts. 
Similar comment for Figure 6, as it does not seem that NP results in the presence of negative prompt "view of skyline" is incorrect.

questions:
While the mathematical derivation of DNG looks reasonable, the generated image examples are hard to understand. 
Can authors point out the difference between NP results and DNG results in the example figures, such as Figure 6 and figures starting on page 22? 
The purpose of experiments of Section 4.1 and corresponding Figure 5 are not clear. I don't understand why the experiments need to remove one class in MNIST and one class in CIFAR-10. If the purpose of the method is to avoid generating undesirable features, shouldn't the model be trained on all classes and only in practical use of the method, one can prompt the model to not generate something, for example, not to generate number zero or an airplane. Or am I missing something here.
While Table 2 lists the positive prompts and related and unrelated negative prompts, can authors give examples on how exactly a full prompt was written up in English and fed to a T2I model? 
In Figure 10(b), as the authors explained, because SLD may have less invasiveness, it gave better FID than DNG, which is reasonable, but why from the figure it seems NP also had a lower FID than DNG as % of wrong images went up? Can authors elaborate on this?
From both Figure 10(a) and (b), it seems that NP had overall worse performance than SLD and DNG, but it also appears that NP performance in KL divergence and FID had a decreasing pattern while those of DNG appeared to pick up at higher % of wrong images, what could be the reason behind this observation? Can authors discuss about this?
Figure 11, for illogical prompts, the guidance scales were larger at later diffusion time than at early diffusion time, can authors provide intermediate generative results corresponding to this change of the guidance scale?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730474297317

thread_type:
official_review

<thread_object>
creator: reviewer_ueks

<content>
summary:
The authors argue that conventional negative prompting methods are constrained by the assumption of a constant guidance scale. To address this limitation, they propose a novel dynamic negative guidance technique that adapts the guidance scale based on both time and state, aiming for near-optimal modulation. Notably, this approach does not require additional training. The authors evaluate their method on MNIST and CIFAR10, comparing it against various baselines, and demonstrate its effectiveness. They also show that the technique integrates well with Stable Diffusion, offering improved accuracy in defining negative prompts.

strengths:
• The proposed method demonstrates promising results on MNIST and CIFAR10, outperforming standard negative prompting techniques and safe latent diffusion methods. This improvement suggests that the dynamic guidance approach offers a meaningful advantage in generating more accurate outputs for image datasets with complex features.

• Preliminary results with Stable Diffusion also appear promising, indicating that the method may effectively enhance prompt accuracy within more sophisticated generative models. However, additional evaluation would further substantiate these findings and provide more insight.

• Though a minor detail, the use of color highlights in algorithms and formulas is a thoughtful touch that enhances readability.

weaknesses:
• There is a lot of research happening in the field of negative prompting, yet this paper lacks a comprehensive comparison with many  leading methods. An in-depth comparison would have more clearly illustrated the strengths and limitations of this approach relative to existing techniques, helping to clarify its unique contributions.

• Overall, the evaluation of the proposed method lacks depth. A more thorough and systematic assessment across various scenarios and metrics would strengthen the validity of the results and give a clearer picture of the method’s real-world applicability and robustness.

• As acknowledged by the authors, a significant limitation of this manuscript is the limited evaluation of text-to-image generation, which is the primary application area for this method. Without a comprehensive exploration of T2I use cases, the potential impact of this work is somewhat undermined, leaving much of its promise unexplored.

• Additionally, the absence of quantitative metrics is a notable gap. Deferring these metrics to future work is a missed opportunity, as they would have added rigor to the analysis and allowed for a more objective assessment of the method's effectiveness.

questions:
Given that you acknowledge these limitations in the manuscript, could you clarify why they were not addressed in the current version? Including these aspects seems essential to strengthen the manuscript's rigor and completeness.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730553637286

thread_type:
official_review

<thread_object>
creator: reviewer_xp5a

<content>
summary:
In this paper, a new concept called Dynamic Negative Guidance (DNG) is proposed, which is an improvement on the existing Negative Prompting (NP) method in diffusion models.

strengths:
1. By dynamically adjusting the intensity of negative prompts, DNG solves the problem that the traditional NP method may encounter suboptimal results or complete failures in the generation process.
2. The structure of the paper is clear and the content is well organized.
3. DNG combines the needs of diffusion model (DMs) and condition generation, and estimates the posterior probability by tracking discrete Markov chains in the generation process. This method is an innovative extension of the existing technology.

weaknesses:
1. Although the paper has been experimentally verified on MNIST and CIFAR10 datasets, the performance of DNG for more complex tasks (Text to image) has not been fully verified.
2. DNG is compared with NP and SLD methods in this paper, but the comparison of each method may lack in-depth analysis, especially the performance comparison under different parameter settings.

questions:
1. You have demonstrated the effectiveness of DNG on the MNIST and CIFAR10 datasets. How does DNG perform on more complex and diverse data sets, such as ImageNet, especially on different image semantics and complexity?
2. The paper contains some visualizations, but can you provide more detailed visualizations to show the progressive impact of DNG in the image generation process, especially how it dynamically adjusts in the denoising step?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732810132008

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Summary of reviews and changes made during rebuttal

comment:
We would like to thank the reviewers for their detailed feedback, which has helped us significantly improve the quality of the manuscript. 

The reviewers agreed both on the innovativeness and the practical significance of the posterior estimation scheme obtained by tracking the Markov Chain. The reviewers also all found the paper clear and well-organized. 

They argued that generalizability of the results could have been better demonstrated.  In response, we extended the class removal experiments on CIFAR and MNIST from one to four classes (see Fig. 4), and the reported results confirm the robustness of our dynamic negative guidance scheme.

Some concerns were expressed over the choices of hyperparameters for concurrent baseline approaches, in particular that of safe latent diffusion (SLD).  In response, we performed a sweep over the SLD threshold parameter, confirming that the choice already proposed by the authors in the context of T2I remains applicable in the class removal setting.

Furthermore, in answer to some reviewers' suggestion to make the dynamic negative guidance scheme better understandable, we have added a new figure (Fig. 5) showing diffusion trajectories with their according dynamic guidance scales. We believe that these figures both provide great insight and demonstrate the relevance of our scheme. We thank the reviewers for this valuable suggestion.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper proposed dynamic negative guidance for diffusion models instead of previous negative prompt with a constant scale. Overall, the paper is technically sound. However there are also some issues with the paper, such as the lack of validation in more complex and diverse applications, lack of depth in evaluation, etc. While the reviewers maintain their scores after rebuttal, the responses seem to address these concerns partially. Given that the authors need some space for the theoretical analysis of the problem, I believe it is hard for the authors to include a very diverse evaluation in the results. Nevertheless, this is a conference that has very limited space. I would suggest the authors to further validate the work in more applications in the future works.