VSF: Simple, Efficient, and Effective Negative Guidance in Few-Step Image Generation Models By Value Sign Flip
Authors: Wenqi Marshall Guo
Shan Du
Venue: ICLR 2026 Poster (2026)
Keywords: Image Generation
Diffusion Models
Negative Guidance
Primary area: generative models

Abstract: We introduce Value Sign Flip (VSF), a simple and efficient method for incorporating negative prompt guidance in few-step (1-8 steps) diffusion and flow-matching image and video generation models. Unlike existing approaches such as classifier-free guidance (CFG), NASA, and NAG, VSF dynamically suppresses undesired content by flipping the sign of attention values from negative prompts. Our method requires only a small computational overhead and integrates effectively with MMDiT-style architectures such as Stable Diffusion 3.5 Turbo and Flux Schnell, as well as cross-attention-based models like Wan. We validate VSF on a proposed challenging dataset, NegGenBench, with complex prompt pairs. Experimental results on our proposed dataset show that VSF significantly improves negative prompt adherence (reaching 0.420 negative score for quality settings and 0.545 for strong settings) compared to prior methods in few-step models (scored 0.320-0.380 negative score) and even CFG in non-few-step models (scored 0.300 negative score), while maintaining competitive image quality and positive prompt adherence. Our method also suppressed a generate-then-edit pipeline, while also having a much faster runtime. Code, ComfyUI node, and dataset are available in https://github.com/weathon/VSF/tree/main.

Published as a conference paper at ICLR 2026
VSF: SIMPLE, EFFICIENT, AND EFFECTIVE NEGATIVE GUIDANCE IN FEW-STEP IMAGE GENERATION MOD- ELS BY VALUE SIGN FLIP
Wenqi Guo && Shan Du
Department of Computer Science, Mathematics, Physics and Statistics
University of British Columbia Kelowna, BC VIV IV7, Canada
wg25r@student, ubc ca, shan. du@ du@ubc. ca
ABSTRACT
We introduce Value Sign Flip (VSF), simple and efficient method for incorporat- ing negative prompt guidance in few-step (1-8 steps) diffusion and flow-matching image and video generation models. Unlike existing approaches such as classifier- free guidance (CFG), NASA, and NAG, VSF dynamically suppresses undesired content by flipping the sign of attention values from negative prompts. Our method requires only a small computational overhead and integrates effectively with MMDiT-style architectures such Stable Diffusion 3.5 Turbo and Flux Schnell, as well as cross-attention-based models like Wan. We validate VSF on proposed challenging dataset, NegGenBench, with complex prompt pairs. Ex- perimental results on our proposed dataset show that VSF significantly improves negative prompt adherence (reaching 0.420 negative score for quality settings and 0.545 for strong settings) compared to prior methods in few-step models (scored 0.320-0.380 negative score) and even CFG in non-few-step models (scored 0.300 negative score), while maintaining competitive image quality and positive prompt adherence. Our method also suppressed senerate-then-ede pipeline, while also having much faster runtime. Code, ComfyUI node, and dataset are available in https: //githtu. com wea thon reee /main.
from lust hen dovel mooiing gif
typortter deterve morouti morlle bedies
ciecs de tncoli weteed veai
mahld
witting the mold Anglend mars dinng cont
dlsst porte warch
Low plaa Nomaaha
segarved Lanters
Figure 1: Original image without negative guidance and image generated using our VSF negative guidance on Stable Diffusion 3.5 Large Turbo. The green prompt the positive prompt, and the red one is the negative prompt. These examples have significant challenges as they are removing essential parts of an object. The "hands" the last image mean clock hands.
AAsso affilated to Department of MEOW, Weathon Software "Correppnniigg author

---

Published as a conference paper at ICLR 2026
INTRODUCTION
Diffusion models (including flow matching models) have demonstrated their ability to produce di- verse and high-quality images (Black Forest Lab, 2025; Woolf, 2022: Stability AI, 2024) and videos (Wan Team et al., 2025; Yin et al., 2025). However, longstanding issue remains: the challenge of effectively applying negative guidance in image and video generation. Addressing this problem is crucial for improving content control, moderation (Schramowski et al., 2023), quality assurance, and reducing biases when generating general concepts (Chen et al., 2025a). However, vision lan- guage models (VI.Ms) have difficulties interpreting negations (Park et al., 2025; Alhamoud et al., 2025; Singh et al., 2025; 2024), rendering prompts containing negations ineffectively or made the negative prompt appears
more (e.g.. prompt like "a scientist who is not wearing glasses" will often generate a scientist with glasses-
even more frequently than a simple prompt like "a scientist"). Classifier-free guidance (CFG) (Ho & Salimans, 2022) can be used to address this issue when substituting unconditional generation with negative guidance.
However, to enhance efficiency in image and video generation, merous models have been distilled to support inference in just few steps (1-8 steps), such Flux Schnell (Black Forest Lab, 2025), Stable Diffusion 3.5 Large Turbo (Stability AI, 2024), SDXL Light- ing (Lin et al., 2024), SNOOPI (Nguyen al., 2024), and Caus (Seppanen; Yin et al., 2025). However, CFG is incompatible with these models. These models are usually distilled and run in CFG- disabled mode, which means only the positive guidance is used, and there is no extrapolation. When CFG is applied forcefully, the resulting image often becomes oversaturated, particularly when the CFG scale is set high enough to suppress unwanted concepts. More- over, if the number of diffusion steps is too low, the output may re- flect features from both the positive and negative prompts (Nguyen al., 2024), rather than excluding the negative prompt entirely. This occurs due to a divergence between the positive and negative guidance signals (Chen al., 2025a)) An example is shown in Fig- ure 2. Additionally, even if CFG works, it requires two forward passes, one for positive guidance and one for negative guidance. which doubles the run time.
Figure 2: forcefully applying CFG to step-distilled model is shown using a guidance scale of 2.8 and only 4 steps on SD- 3.5- Turbo. The posi- tive prompt describes a Cana- dian winter landscape, while the negative prompt includes the word "lake." The result- ing image does not cancel the lake and exhibits severe over- saturation artifacts (trees in the background).
An example of
To address this, two methods, Negative Steer Away Attention (NASA) (Nguyen et al., 2024) and Normalized Attention Guidance (NAG) (Chen et al., 2025a), have been introduced, employing neg- ative guidance within attention final output space rather than the output space. NASA currently limited to cross-attention models (though it can be re-implemented into other models), while NAG primarily targets quality control rather than negative prompt avoid- ance. Both methods calculate positive and negative attentions separately and subtract them using a prefixed scale (same as CFG), resulting in a fixed guidance strength throughout the generation, across different areas of the image, and at different layers of the model. This approach lacks adapt- ability to various time steps, layers, or image regions, limiting effectiveness in negative prompt adherence compared to a more adaptive method (Schramowski al., 2023; Koulischer al., 2025; Ban et al., 2024).
In this study, we introduce Value Sign Flip (VSF), method that dynamically adjusts the guidance strength by flipping the sign of negative prompt values within the attention calculation (i.e.. not the attention output). This enables the model to steer away from negative concepts adaptively based on their current presence strength, similar to the approach of Koulischer et al. (2025). VSF has small computational overhead and, when combined with few-step models, facilitates extremely fast image or video generation (< 3 seconds). Our contributions in this work are: (1) we proposed new method for better negative guidance; (2) we constructed a dataset, NegGenBench, consisting of challenging Dositive-negative prompt pairs; (3) collected images generated using these three methods (VSF, NAG, NASA) and labeled their negative and quality score. We further fine-tuned a VLM on it for future work to better evaluate negative prompt following.
2

---

Published as a conference paper at ICLR 2026
2
RELATED WORK
2.1 CLASSIFIER FREE GUIDANCE
Vision language models struggle to understand negation (Yuksekgonul al., 2023; Singh et al., 2024: Alhamoud et al., 2025; Park et al., 2025) (We discussed more about this in the Appendix). Original classifier-free guidance (CFG) (Ho & Salimans, 2022) generates a conditioned noise pre- diction and an unconditioned noise prediction. In flow matching (Lipman et al., 2023), the predicted targets are the velocity (we). Thus, the original flow matching CFG prediction can be written as
U = f00, + X(f(p .I+11. f(0, xx+1,
where pt is the positive prompt. Itz is the latent time t (where higher t means more toward the noise distribution), f(-) is the trained model, and À is the guidance scale. Later, the community finds out that by replacing the unconditional generation with a negative prompt (e.g., description of an unwanted image), the model will avoid the prompt due to the negative sign. This is the common implementation of negative prompt. This turns the above equation into
U (( + Xf(x*, ,e411. () f(p , ()),
where p is the negative prompt.
2.2 RECENT WORKS ON DYNAMIC NEGATIVE GUIDANCE
The studies on dynamic negative guidance are very limited (only (Ban et al., 2024; Koulischer et al., 2025; Schramowski et al., 2023)). Ban al. (Ban al., 2024) found that the negative prompts affect the model by delayed effects and ncutralization. After the model has generated unwanted contents, the negative guided output (up-) will neutralize the content. They also observed the reverse activation effect, where the negative prompt introduced early in the diffusion processes could actually induce the unwanted concepts. To address this, they proposed applying the negative guidance later in the diffusion process and found it effective.
Schramowski et al. (2023) used very similar idea CFG to avoid unwanted (NSFW) content. They generated an unsafe vector and purposely avoided it by subtracting it from the predicted noise. They also added pixel-level guidance scale that depends on the pixel-wise distance between the positive predicted noise and the unwanted noise, making it adaptive to different regions in the image.
Koulischer al. (2025) used similar ideas of both and proposed a temporal dynamic guidance scale method. They calculated a probability that the generated concept contains negative content and adjusted the guidance scale accordingly. However, their adaptive scale only changes throughout the steps and does not adapt to different regions in the image.
2.3 FEW-STEP IMAGE GENERATION MODELS
Traditional diffusion or flow-matching image generation models typically require many inference steps. However, with improved sampler, this can be reduced to around 20 steps. Recent approaches go further by using step distillation to reduce the number of steps to fewer than 8, or even a single step, demonstrated in Flux Schnell (Black Forest Lab, 2025), SDXL Lightning (Lin al., 2024), CausVid (Seppanen; Yin et al., 2025), SNOOPI (Nguyen et al., 2024), and Stable Diffusion 3.5 Turbo (Stability AI, 2024). Since these models are distilled, they generally do not use classifier- free guidance (CFG) during inference; when CFG is forcibly applied, the results are significantly degraded to the point that completely unusable (Nguyen et al., 2024), see Figure 2 for an example.
2.4 RECENT WORKS ON NEGATIVE GUIDANCE IN FEW-STEP MODELS
Recently, two approaches have specifically targeted negative guidance techniques for few-shot mod- els: Negative-Away Steer Attention (NASA) (Nguyen al., 2024) and Normalized Attention Guid- ance (NAG) (Chen et al., 2025a). Although they both focused on avoiding unwanted content and improving quality (using negative prompt that describes bad quality), NASA mainly focused on avoiding unwanted content, while NAG focused on improving quality.
3

---

Published as a conference paper at ICLR 2026
The authors of the NASA study found that neither standard CFG CFG applied directly text embeddings yields desirable results in few-step scenarios, particularly in single-step settings. Specif- ically, the regular CFG independently computes positive and negative guidance signals, preventing the negative guidance from effectively neutralizing unwanted concepts. As a result, the produced images merely appear mixture of both positive and negative prompts unnaturally (an average image of the positive prompt generated image and the negative prompt independently generated im- age) rather than excluding negative prompt elements. Furthermore, the authors noted that applying CFG to text embeddings produces minimal benefits. For detailed examples and further illustration, readers could refer the original paper introducing NASA (Nguyen al., 2024).
NASA applies the guidance in intermediate states (attention outputs) instead of the final predicted noise or velocity. Specifically, they calculate a positive attention output Z and a negative attention output Z- and the final attention Z. NASA is obtained by subtracting the two with factor a, as shown in Equation The value is usually between and 1.
z NASA Zz -ab
(3)
Normalized Attention Guidance (NAG) used a similar approach. But instead of subtracting the negative attention map from the positive, it CFG,
Origital Atsention Coelg
uses similar extrapolation approach as shown in Equation 4. The starting point Z+ could also be replaced with Z they are equiv- alent if é is increased by 1.
inf
Madked Attention
NoyWeler
NM
-ief
Blzsed Aftenticn
NO
L. Nan podlie
NAG Z + +(= --)
(4)
N
Eomen Lice cnly saa ualiity
However, to maintain the stability of the atten- tion output space, they also applied normaliza- tion to AAG to limit its norm relative to Z+ with scale v per token, resulting Z. Then it used blending factor to blend it with the positive attention result, as shown in Equa- tion 5.
Queraa
The attention mechanism of our method. We pass in image tokens (!), positive prompt to- kens (P), and negative prompt tokens (N) into at- tention. For key and values, N is duplicated, with values of one copy (N(() scaled by -a. Some areas are masked avoid interference. An bias -3 is added to I + (1) attention.
Figure 3:
(5)
The normalization and blending ensure the attention output of the NAG does not drift away from what the model usually sees during training, improving the quality of generated images. However, if the constraint is set to be too tight (i.e., high and low r), it might also limit the model's ability to follow negative prompts.
2.5 OTHER RELATED WORKS
There is previous work that controls attention to manipulate images. Attend-and-HE Exeiie (Chefer et al., 2023) forces the attention on all key tokens to avoid the generation missing some terms the prompt. Self-Guidance (Epstein al., 2023) mantluplates attention to changing elements' properties in the image. BoxDiff (Xie et al., 2023) controls the attention map such that the object appears on the desired location in the image. However, our work is different in that we used it in way to cancel unwanted elements by simply flipping the sign of values in the negative prompt.
Our work also related debiasing in image generation, since can be used way to move away from learned associations (which could be about demography bias or object pairings). Previous work mainly targeted on the input, such as using reference images as input (Zhang et al., 2023) or using leamed prompt. FairQueue (Teo et al., 2024). Our method is different such that we control the value in the negative prompt. Additionally, our method can work against very strong associations like bike without wheels.
3 PROPOSED METHODS
Our proposed method is built on top of NASA (Nguyen et al., 2024), Koulischer et al. (2025), and Schramowski et al. (2023). NASA has fixed guidance for every attention calculation, Koulischer
4

---

Published as a conference paper at ICLR 2026
al. (2025) does not have token-level modulation, and Schramowski al. (2023)'s approach of taking the differences between the positive and negative noise predictions is too simple for complex situations, mentioned in Koulischer et al. (2025). In NAG's (Chen al., 2025a) future work section, they also mentioned the possibility of token-level modulation but they did not propose a specific solution.
3.1 VALUE SIGN FLIP ADAPTIVE ATTENTION
We propose to expand Koulischer et al. (2025) idea to token-level modulation in few-step models. Let W be a per-token weight each attention calculation for how strongly the token is associated with positive concept compared to the negative one. We can modify the NASA attention to Eq. 7. W is obtained by a function with positive prompt, a negative prompt, and an image as input.
= = W-g(p" I),,
(6)
then we can rewrite the equation in NASA as
z = Wz a(1 - w)z
An intuitive method calculate W involves using the model's attention map: when the image attends more to the negative prompt compared to the positive one, it should be steered away strongly accordingly. Thus, we can calculate the attention map between the image and the positive tokens A+ and the image and the negative tokens A before softmax calculation, then calculate their ratios to their sum. Q the image query tokens and K+ and K keys.
are the positive and negative prompt
QKK+T ),A- = exp( Q(K-)T W = Σ(A++A-) Vd
ΣA+
A+ = exp(
(8)
Vd
This approach involves complex attention calculation and two attention passes, but it can imple- mented by simpler approach. We can concatenate the values and keys of the positive and negative prompts, then flip the sign the negative prompt values. This enables that when the image attends to the negative prompt, the flipped value of the negative prompt can cancel the unwanted content. The equation of our method cross attention models, written in the matrix calculation, is shown in Equation 9, where @ means matrix concatenation the sequence length dimension, is the softmax function on the sequence length dimension, and V+ and V- are the positive and negative prompt values.
VSF = =0 QKKK
--v-)
Vd
This is similar to noise-canceling headphones, where a "flipped" wave played to cancel the noise. Note that the key the negative prompt is not flipped keep the original meaning of the unwanted concept to match image patches. Mathematically, this is equivalent to Z Proof is in the Appendix.
This approach gives dynamic weight for the positive and negative prompts, and it varies for differ- ent layers, steps, and tokens.
3.2 ATTENTION MASKING AND DUPLICATION OF NEGATIVE EMBEDDING
The above method works well for cross-attention-ben methods, where attention only exists be- tween image-to-imant in self-attention layers and image-to-text in cross- tross-attention layers. However, it requires modification, including masking and duplication, to work in MMDiT-style models such as SD3.5 (Stability AI, 2024), where all image and text tokens concatenated into single sequence before attention.
In the standard MMDiT-style setup without method (e.g., using CFG, NASA, NAG), the sequence inputs for the attention module are: [I, P and II N]. If we concatenate all tokens into a single sequence without any modification, we will get: [I, P, N], where represents image tokens,
5

---

Published as a conference paper at ICLR 2026
P represents positive prompt tokens, and N is the negative prompt. During attention, queries, keys, and values are all projected from this combined sequence.
If apply sign flip to the negative prompt values by scaling Vvv VN with -a (where V is the value projection), this flipped content affects all attention paths involving VN. That includes not only the intended interaction between image and negative prompt (I + N) but also undesired interactions such as positive-to-negative (P + N) and regative-to-negatte the value will cancel itself). These unintended interactions can distort the behavior of the model since the flipped signal influences more than just the image.
(N + N) (in which
To address this, we introduce duplication of the negative prompt. One copy remains unflipped and unscaled, denoted by N(0), and the value (and only value) of the other is flipped and scaled, denoted by VN -a VNO, The sequence becomes: II, P. N(),, NO), where N(1) does not act as query in attention calculation.
Additionally, inspired by Wang et al. (2025), where blocking some attention directions could im- prove quality, we apply attention masks to isolate the effect of (()) to only I. Specifically. No(0) is only allowed to attend to I and to itself, while N(1) is only attended to by I. Figure 3 shows the attention mask. Since (1) does not act as a key or value in any attention query, it doesn't 't produce associated output. Instead, O() serves as the negative prompt tokens passed to the subsequent MLP layer and into the next attention layer, where it will be flipped again. It acts as information collec- tor from images to collect unwanted elements and also keeps updating itself from attention to itself, matching the prompt updating in a positive prompt.
This setup allows updates to the negative prompt based on attention from the image and from itself, and keeps the unflipped form active in the MLP path. It also prevents interference between positive and negative prompts and ensures that the flipped negative content affects only the intended image- to-negative attention path.
To preserve the high quality of generated images, we also applied attention bias (-3) to N(1) (also shown in Figure 3) and we removed the padding tokens from the negative prompt. Details and pseudo-code of our method are in the Appendix.
In our implementation, negative embeddings are duplicated once due to an implementation detail; however, this has a negligible impact on the final results after offsetting the scaling factor with 2
EXPERIMENTS
4.1 DATASET
Following Park et al. (2025), we use ChatGPT o3 (Open Al) to generate pairs of prompts and nega- tive prompts to construct our dataset NegGenBench. Unlike prior work, our prompts are intention- ally more challenging: the negative prompt is typically related to the positive one, and as critical component-ce c.g. the positive prompt of a bike could have negative prompt of "wheels". However, the positive prompt sometimes also uses non non-negation method to imply the item is missing, such as using terms like "empty" and "exposed" to make it more natural. Besides prompts, two questions generated at the same time for later evaluation, object, either with or without the negative element, and the other one queries if the negative prompt element missing. Prompts are generated in batches. There are 200 prompts generated, and we run them with 2 different seeds for the main results.
question asking if the image has the main
4.2 BASELINE AND METRICS
We chose NAG (Chen et al., 2025a) and NASA (Nguyen et al., 2024) as our baseline for few-step models. We also used a base model without negative guidance vanilla baseline, aiming to show the lower bound of the dataset. (i.e., how likely the positive prompt alone will help avoid negative concepts, if there is no negative guidance. This could happen either because the model is following
'The arrow direction is the attention direction, or the opposite direction of the information flow Phtpps: / :
/g/ github. com woom/weathon//aee /issuss/sss

---

Published as a conference paper at ICLR 2026
To abcrect pairting
Nunnal
Figure 4: (Left) Style Avoidance Tests, (Right) (Semi)-abstradl art generated by mentioning the main object "car" in a negative prompt. The car semi-canceled and thus still present but in an abstract form.
the implication in the positive prompt, such as the word 'missing', or simply by chance.) Because NASA's original source code was not publicly available at the time of writing, we reimplemented based on NAG's codebase. Specifically, we replaced the guidance equation from NAG (Eq. 4) with NASA's equation (Eq. 3), removed normalization and blending, and enabled guidance when the scale is greater than 0 (instead of 1). Additionally, we compared our method in non-few-step models with CFG and used other models as external baselines (External baseline results are in Table 1, but the experiment details in the Appendix). We included Flux Kontext Labs al. (2025) as an image-editing baseline. We first generate an image using SD-3.5-Large-Turbo and then edit the image with Flux Kontext using the prompt Remove [negative prompt]. Since NAG was focused on quality instead of negative prompt avoidance, we re-tuned its hyperparameter such that it has stronger negation following in trade-off of quality and positive prompt following. We name this variance as NAG Strong. Same for our VSF method, we provided two different variations with different hyperparameters, focusing on quality (VSF Quality) and negative prompt following (VSF Strong). Hyperparameter details are in the Appendix.
Following Park et al. (2025); Wei al. (2025), we used multimodal large language models (MLLM), specifically 11 amama-- -4 -w--aaaiiiieo -h- 1 rucctffctee fp8, to evaluate if the generated image follows the positive prompt and the negative prompt using the two questions generated during prompt generation. We did not use previous negation-aware CL.IP-based work because they do not focus on missing an essential component, but simple meaning (e.g.. a dog that is not on the grass). We did not use HPSv2 (Wu et al., 2023) or ImageReward (Xu et al., 2023) because they might give low quality score for unusual objects (essential part being removed). Instead, MLLM is used to rate the image quality at the same time. At the end of our experiment, VL (Bai et al., 2025) model using data we generated by VSF, NAG, and NASA for better negation understanding. More details about the metries and comparison with human validation are in the Appendix.
also fine-tuned a Qwen-2.5-
5 RESULTS
Quantitative results from using ILaMAA as a judge evaluation are shown in Table 2, and qualitative results are shown in Figure 12 in the Appendix. Human validation is shown in Table 3 Auto- matic evaluation using the better negation-aware MLLM Qwen-2.5-VL. is shown in Table 8 in the Appendix. Both the human validation and Qwen-2.5-VL results are aligned with our LLaMA evalu- ation relative ranking. is important to highlight that the LLaMA assigns relatively generous quality scores; a score lower than 90 usually means the image already has degraded quality. Examples are in the Appendix Figure 11.
Based on the guantitative results, VSF Strong shows a significantly higher negative score than other methods, while maintaining comparable better quality scores. Our more conservative method, VSF Quality, still achieved the second-highest negative score, with the highest quality score. Both VSF Strong and VSF Quality even achieve higher negative score than traditional CFG in non- few-step models, demonstrating a stronger ability to avoid negative elements, even relative to the
'The results here might be negligibly different than our code due to implementation differences. https:
/i thub. ccm/ wttpw/tww/wweeeeeeeeeee/ee weathon/VSS /ttp///F//s///// ssues/19
7

---

Published as a conference paper at ICLR 2026
Table 1: External Baselines Comparsion
Positive Score (1)
Negative Score (1)
Quality Score (1)
Runtime ())
Oeen-weight Models
VSF Strong
0.870
0.545
0,952
3s
VSF Quality
0,980
0,420
0,986
3s
Generate+Edit
0.875
0,488
0,958
55s
Janus-4o
0.925
0.225
0,944
20s
owen-Image NP
0.973
0.190
0.935
110s
Owen- Image Negation
0.990
0.100
0.937
110s
wlighh Models
GPT-4o
0.978
0.705
0.954
47s
Nano Banana
0.985
0.498
0,980
established strong baseline. When compared with the external baseline, VSF also gets the high- est performance in open source methods and only lags behind GPT-4o and achieves comparable performance with Nano Banana.
We also tested concepts/style avoidance, as shown in Figure 4 left. In the Starry Night example, VSF completely removed any signature elements Vincent van Gogh's 's style, including the town, and generated generic starry night image. Figure 4 also illustrates how our method can produce abstract art, which is typically discouraged during a model's finetuning since reward models favor realism. This is achieved through using the same word the main object for both positive and negative scores in VSF, as detailed in the Appendix. VSF also has the ability to generate "anti- aestheties" (unconventionall, including abstract) art. Details of these results are provided in Figure ?? and Figure 17 (in the Appendix).
6 DISCUSSION
6.1 TRADE OFF CURVE
To systematically evaluate how effectively each model balanced positive prompt adherence, negative prompt adherence, and image quality, we conducted a hyperparameter sweep across each model. Specifically, we performed 66 runs VSF and 287 runs for NAG, and 10 runs for NASA, with respect to their hyperparameter counts (2 for VSF, 3 for NAG, and for NASA). A random sweep was executed besides for NASA, on which a single variable "grid" search used, and evaluations were conducted using LLaMA, following the same criteria previously described. Due to the large volume of runs, we limited our evaluation to the first 100 prompts with single generation seed, potentially resulting in minor differences from carlier outcomes. Results are shown in Figure 5 Left.
From both plots, we observe that as the negative score increases, NAG and NASA both exhibit significantly steeper and earlier decline compared to VSF in both positive and quality scores. In terms of positive score, VSF maintains scores above 90 even when the negative score rises to approximately 60. Regarding image quality, VSF similarly retains scores above 90 until negative score of around 60, after which quality declines. In contrast, NAG and NASA both experience sharp and early decline, with their quality score rapidly dropping to nearly 60 even before the negative score reaches 50. Keep in mind that a quality score under 90 means the image is already degraded, and if an image is rated 60, is usually completely distorted. See Figure 11 in the Appendix for example.
Additionally, VSF demonstrates broader operational range in negative scores. When necessary, can achieve negative scores exceeding 70 while still preserving acceptable positive prompt adher- ence and image quality. Conversely. NAG and NASA become unacceptable in quality negative scores below 50, limiting their practical effectiveness.
6.2 ATTENTION MAPS
Since our proposed method performs adaptive steering based on a negative attention map. we visu- alize the attention maps generated during the diffusion process in Figure in Appendix. Extracting the full attention maps is difficult because efficient implementations, such as FlashAttention, do not explicitly store these maps, and storing and computing them will require large amount of mem-
8

---

Published as a conference paper at ICLR 2026
Positive vs Negative Trade off
Quality vs Negative Trade off
100
VSF
NA4
90
Quality Score
40
80-
85
VSF
80
NAC
NASA
75
60
Negetiive Storee
Negalive Score
Pos Score frentier
Quallty frontier
0.50
F3 25-33000
0.95
0.57
0.50
0.00
10 rAaD:
0.34
0.90
0.2
0.2
033
0.4
0.5
0.5
0.2
0.2
0.3
0.4
0.5
0.6
Regg Score
Neg Score
Figure 5: (Top) Trade-off plot of positive-negative score and quality -igittive score. follows "higher is better." (Down) Trade-off plot of the ablation study.
Both axies
ory. Therefore, we computed only the unnormalized attention values between the image tokens and negative prompt tokens. Figure 6 demonstrates that when the scale is set 0, umbrellas appear, whereas setting the scale to 3 effectively avoids them. As indicated in the attention maps, image to- kens corresponding regions where umbrellas might exist (e.g., above human heads) exhibit higher attention toward the negative prompt tokens. Specifically, in steps 4 and 5, regions above the indi- viduals on the left and right show strong negative attention, aligning with areas visually identified as umbrellas in = 0 image. In the final image, these highlighted regions no longer contain umbrel- las, confirming that our method effectively suppresses the presence of undesired objects specific locations.
6.3 ABLIATION STUDY
To evaluate the effectiveness of each component of our approach, we conducted an ablation study using the following settings. For each setting, scanned across scales for all 200 prompts using the same seed. Similar to before, we plotted the tradc-off curve for each setting.
Rather than altering the attention values, we explored simpler and more intuitive approach: flipping the text embedding prior to input into the DiT (Whole Embedding Flip, WEF). This is similar to applying the CFG on text embeddings studied in Nguyen al. (2024), but keeps the positive and negative tokens separated. Specifically, the negative text embedding is scaled by -a, concatenated with the positive prompt embedding in the sequence length dimension, and used as the prompt embedding for the DIT. We did not remove the padding for the negative prompt, as we found out that removing it causes the negative prompts to have no effect at all.
We also tested our approach with no bias, no mask (but still duplication), and no duplication no mask. The trade-off plot is shown in Figure 5 Right. The simpler and more intuitive WEF approach appears to have almost no effect at all. We hypothesize that this is because it is similar to flipping both the key and the value, causing regions most similar to the flipped key (i.e., least similar to the original negative prompt) to be pushed away, rather than pushing away regions most similar to the original negative prompt (i.e., unflipped key). From the figure, we can see that the confi configuration guniiuurations without masking have sharp positive score drop the negative score increases. The WEF has very limited range of negative scores. Our methods and the one without attention bias have similar results; this could be due the MLLM not being sensitive enough see the minor changes in quality.
Ablation study on hyperparameters shown in the Appendix.

---

Published as a conference paper at ICLR 2026
Positive scores (how well the model follows the positive prompts) and negative scores (how well the model avoids the negative prompts) of our model (VSF), NAG (Chen et al., 2025a), and
Table 2:
NAG with hyperparameter re-tuned (NAG Strong).
Positive Score (f) Negative Score (1)
Quality Score (f)
VSF Strong
0.870
0.545
0,952
VSF Quality
0.980
0.420
0.986
NAG (Chen et al., 2025a)
0,993
0.220
0,968
NAG Strong NASA(Nguyen et al., 2024)
0.975
0.320
0,901
0,970
0.380
0,867
None
0.990
0.195
0,968
CFG (Ho & Salimans, 2022) (28 steps)
1.000
0.300
0.956
Table 3: Human Labelled Metric For 10 Selected Prompts with 2 Seeds
Positive Score (f)
Negative Score (f)
Quality Score (f)
NAG Strong NAG
0.950
0.250
0.675
1.000
0.100
0.895
NASA VSF Quality
0.950
0.150
0.685
0,900
0.550
0.823
CONCLUSION
In this paper, we introduced VSF, novel approach for enhancing negative prompt adherence in image and video generation models. Our method involves flipping the sign of attention values and duplicating negative prompts and attention masking, effectively suppressing unwanted content. Experimental results indicate that VSF significantly outperforms previous methods, NAG (Chen al., 2025a), NASA (Nguyen et al., 2024), and even CPG in terms of negative prompt adherence, with much lower trade-offs in overall quality and positive prompt following. We also showed that VSF can be applied to create more creative (by style avoidance, abstract images, and anti-aesthetics styles) images. VSF also only has one main hyperparameter and one minor hyperparameter, making it easier to tune them downstream tasks. Future work directions are discussed the Appendix.
ACKNOWLEDGEMENTS
This work was supported by the NFRF under grant GR024801 and the CFI under grant GR024473. The authors acknowledge Weathon Software (https:/ /ttp///wwtff....m/ooo/o com) and Lambda. Inc. (https: wla.maaaaaa......... ai/) for providing computing resources via Google Colab and Lambda Cloud, respectively.
We also want to thank our cat Marshall the Huai for helping us babysitting our experiments.
Following the completion of this research and paper submission, became of NegPiP memuiinnegipp)) a community project that
(https: /itthb.. utilizes approach similar to ours. While this work developed entirely independently from NegPiP, we recognize its contribution and suggest as relevant resource for readers.
REFERENCES
Kumail Alhamoud, Shaden Alshammari, Yonglong Tian, Guohao Li, Philip Torr, Yoon Kim, and Marzyeh Ghassemi. Vision-Language Models Do Not Understand Negation, May 2025, URL 09425. trXiv:2501.09411
http:://rxxt
Anne Arzberger, Stefan Buijsman, Maria Luce Lupetti, Alessandro Bozzon, and Jie Yang. Nothing Comes Without Its World Practical Challenges of Aligning LLMs to Situated Human Values through RLHF, Proceedings the AAAI/ACM Conference on AI, Ethics, and Society, 7:61-73, October 2024. ISSN 3065-8365 doi: 111.9///1ie..7...111...... URL https: //jjs.....///////////////// ss. aaai. arg/index..o http:p////////atrreeee/e//ee///e
10

---

Published as a conference paper at ICLR 2026
Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang. Humen Zhong, Yuanzhi Zhu, Mingkun Yang, Zhaohai Li, Jianqiang Wan, Pengfei Wang, Wei Ding. Zheren Fu, Yiheng Xu, Jiabo Ye, Xi Zhang, Tianbao Xie, Zesen Cheng, Hang 7hang. Zhibo Yang, Haiyang Xu, and Junyang Lin. Qwen2.5-VL. Technical Report, Febru- ary 2025. URL http: /aaxiixivv org/ abs 2502 13923. arXiv:2502.1392.
Yuanhao Ban, Ruochen Wang, Tianyi Zhou, Minhao Cheng, Boqing Gong, and Cho-Jui Hsieh. Understanding the Impact of Negative Prompts: When and How Do They Take Effect?, June 2024. URL
arXiv:2406.02%65.
Black Forest Lab. back-forestlbb//LUUXX---mmo Hugging Face, June 2025. URL https: http://lacccforst-t--eeeeeee- /ttpgggiiff.e...oppooooooootoeaa
Hila Chefer, Yuval Alaluf, Yael Vinker, Lior Wolf, and Daniel Cohen-Or. Attend-and-Excite Attention-Based Semantic Guidance for Tex--tolmmae Diffusion Models, May 2023, URL
http://arxis/ wwwp:/s/////////ee ab///30ss/2111...///// 13826. WXXiv2301.13822 [es].
Dar-Yen Yen Chen, Hmrishav Bandyopadhyay. Kai Zou, and Yi-Zhe Song. Normalized Attention Guidance: Universal Negative Guidance for Diffusion Models, June 2025a. URL http:
21179. arXiv:2505.21179)
Junying Chen, Zhenyang Cai, Pengeheng Chen, Shunian Chen, Ke Ji, Xidong Wang, Yunjin Yang, and Benyou Wang- ShareGPT-4o-Imagee Aligning Multimodal Models with GPT- 4o-Level Image Generation, June 2025b. URL http: buttp://arxiy arxiv. wttp//ggg///b///eeeeee 20w..b/.....00000 18095. DeXiv:2506.1809.
Xiaokang Chen, Zhiyu Wu, Xingchao Liu, Zizheng Pan, Wen Liu, Zhenda Xie, Xingkai Yu, and Chong Ruan. Janus-Pro Unified Multimodal Understanding and Generation with Data and Model Scaling, January 2025c. URL http://////ax org/ abs/ /2511 17811. arXiv:2501.17811
Tri Dao. FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning, July 2023. URL http:::taaarrte org/ htt////b//3.....// 08691. #tXiv:2307008691.
Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. QLoRA: Efficient Fine- tuning of Quantized I.LMs, May 2023. URL arXiv:2305.14314
14314
Dave Epstein, Allan Jabri, Ben Poole, Alexei A. Efros, and Aleksander Holynski. Diffusion Self- Guidance for Controllable Image Generation, June 2023. URL http:///rxxiv.orr/tttt 2306. 988. arXiv:2306.00986 [es].
Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, Dustin Podell, Tim Dockhorn, Zion English, Kyle Lacey, Alex Goodwin, Yannik Marek, and Robin Rombach. Scaling Rectified Flow Trans- formers for High-Resolution Image Synthesis, March 2024. 2403. 03206 6. arXiv:2403.03206
Rohit Gandikota, Joanna Materzynska, Jaden Fiotto-Kaufman, and David Bau. Erasing Con- cepts from Diffusion Models, June 2023. URL pettp://arxisn org/ abs/2303. 07345. arXiv:2303.07345.
Ashish Goswami, Satyam Kumar Modi, Santhosh Rishi Deshineni, Harman Singh, Prathosh A. P. and Parag Singla. GraPE: A Generate-Plan-Eddi Framework for Compositional T2I Synthesis, March 2025. URL http: /ttp///aavvvv.//aa///aaeeaaaaaaa wwtexivvvrr//ie org. halgg//as/// 22.... 06089. arXiv:2412.0608,.
Wenqi Marshall Guo, Yiyang Du, Heidi J. S. Tworek, and Shan Du. Position: The Pitfalls of Over- Alignment: Overly Caution Health-Related Responses From LLMS are Unethical and Dangerous, August 2025, URL http: /ttp://////ivia///a www.ariwi..../raeaaooooaa /2099. 008..08939.... arXiv:2509.0883 008333.
Jonathan Ho and Tim Salimans. Classifier-Free Diffusion Guidance, July 2022. URL http: / //waaiivorgg/aaeeeeeeeea wttpt//.rrrabbb//////// 12598. arXiv:2207.12558.
11

---

Published as a conference paper at ICLR 2026
Felix Koulischer, Johannes Deleu, Gabriel Raya, Thomas Demeester, and Luca Ambrogioni. Dy- namic Negative Guidance of Diffusion Models, January 2025. URL http://axxiv.orr abs/2410 14398. arXiv:2410.14338.
Black Forest Labs, Stephen Batifol, Andreas Blattmann, Frederic Boesel, Saksham Consul, Cyril Diagne, Tim Dockhorn, Jack English, Zion English, Patrick Esser, Sumith Kulal, Kyle Lacey, Yam Levi, Cheng Li, Dominik Lorenz, Jonas Müller, Dustin Podell, Robin Rombach, Harry Saini, Axel Sauer, and Luke Smith. FLUX.1 Kontext: Flow Matching for In-Context Image Generation and Editing in Latent Space, June 2025. URL http: attpaaxiivvoooo wttp://wwwwgg////......../ arXiv:2506.1572
Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. BLIP: Bootstrapping Language-Iman Pre-training for Unified Vision-Language Understanding and Generation, February 2022. URL
httt:///rxiv..omm
12086. arXiv:2201.1206
Shanchuan Lin, Anran Wang, and Xiao Yang. SDXL-Lightning Progressive Adversar-
ial Diffusion Distillation, March 2024. URL http://aaxxvv.or/abbs//// arXiv:2402.13929.
Yaron Lipman, Ricky T. Q. Chen, Heli Ben-Hamu, Maximilian Nickel, and Matt Le. Flow Matching for Generative Modeling, February 2023. URL http://arxiv..rrg wttp://g/./....././// arXiv:2210.02747.
Anne-Sofie Maerten, Li-Wei Chen, Stefanie De Winter, Christophe Bossens, and Johan Wagemans. LAPIS: A novel dataset for personalized image aesthetic assessment, 2025. URL https:// arxiv. org/ abs 2504 07670. Version Number:
Viet Nguyen, Anh Nguyen, Trung Dao, Khoi Nguyen, Cuong Pham, Toan Tran, and Anh Tran. SNOOPI: Supercharged One-step Diffusion Distillation with Proper Guidance, December 2024.
URL
wttp:oo/gabbbbaaaaeaa 02687. arXiv:2412.02687.
Open AL.
Introducing OpenAl and 04-mini.
URL https: httpp://oppeeaa..cccomm http///ppaiiccmmmm/.
Junsung Park, Jungheom Lee, Lee, Jongyoon Song, Sangwon Yu, Dahuin Jung, and Sungroh Yoon. Know
"No" Better: A Data-Driven Approach for Enhancing Negation Awareness CLIP, March 2025.
URL
attp.///5...bbb./...l.... 10913. arXiv:2501.10913.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and llya Sutskever. Learning Transferable Visual Models From Natural Language Supervision, February 2021. URL httt:://aaaaee
www.:orrggaaaaaae /210333.. C0020. arXiw:2103.00020.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer, September 2023, URL htttt:tt/aaxiiiii wttp::/...rraab//ll.i1 10683. arXiv:1910.. 10683
version: 4.
Patrick Schramowski, Manuel Brack, Björn Deiseroth, and Kristian Kersting. Safe Latent Diffusion: Mitigating Inappropriate Degeneration in Diffusion Models, April 2023. URL URL:ttt:///aat org/abs 2211 05105. arXiv:2211.0SS 1.05105.
Jukka Seppanen. Kijai/Wan Video comfy Hugging Face. URL URLhhttss://huggigggees
Ki jiijai/aaiiiees Wtwaaaaadeooooof)
Jaisidh Singh, Ishaan Shrivastava, Mayank Vatsa, Richa Singh, and Aparna Bharati. Learn "No" to Say "Yes" Better: Improving Vision- Language Models via Negations, March 2024. URL org/ ats//bb//2e/ 20312. 2. arXiv:2403.20312.
Jaisidh Singh, Ishaan Shrivastava, Mayank Vatsa, Richa Singh, and Aparna Bharati. Learning the Power of "No": Foundation Models with Negations. In 2025 JEEE/CVF Winter Confer- Applications of Computer Vision (WACV), pp. 8002-8012, February 2025. doi: 10. ieee. org/ document/
DWWWAAC66041.202..0000 10943641. ISSN: 2642-9381.
URL
12

---

Published as a conference paper at ICLR 2026
Stability AI. Introducing Stable Diffusion 3.5, 2024. URL UR/https://stbbbittyyaaaa
Margit Sutrop. Challenges of Aligning Artificial Intelligence with Human Values. Acta Baltica
Historiae et Philosophiae Scientiarum, 8(2):54-72, December 2020. ISSN 22282009, 22282017.
doi: 10. 10t.15/0//bsss202000222222
URL https://ww..ess.ee/bbbaap.. wwtpee/eeaaaaaaaaaaaaeaaeeeeaae pdf.
wew.wies.ee. eerss.eebbaate
Christopher T. Teo, Milad Abdollahzadeh, Xinda Ma, and Ngai-man Cheung. FairQueue: Rethinking Prompt Learning for Fair Text-to-lmage Generation, October 2024. URL http:
18615. arXiv:2410.16155 [es].
Alexey Turchin. Ai alignment problem: Human values don't actually exist. 2019. URL https: //philarchive. wwwooorg//o... retg////C/TRRA
Patrick von Platen, Suraj Patil, Anton Lozhkov, Pedro Cuenca, Nathan Lambert, Kashif Ra- sul, Mishig Davaadorj, Dhruv Nair, Sayak Paul, Steven Liu, William Berman, Yiyi Xu, and com/
Thomas Wolf. Diffusers: State-of-the-att diffusion models. URL https: huggingface/ /id//diffi difffsers.
Wan Team, Ang Wang, Baole Ai, Bin Wen, Chaojic Mao, Chen-Wei Xie, Di Chen, Feiwu Yu, Haiming Zhao, Jianxiao Yang, Jianyuan Zeng, Jiayu Wang, Jingfeng Zhang. Jingren Zhou, Jinkai Wang, Jixuan Chen, Kai Zhu, Kang Zhao, Keyu Yan, Lianghua Huang, Mengyang Feng. Ningyi Zhang, Pandeng Li, Pingyu Wu, Ruihang Chu, Ruili Feng. Shiwei Zhang, Siyang Sun, Tao Fang, Tianxing Wang. Tianyi Gui, Tingyu Weng, Tong Shen, Wei Lin, Wei Wang, Wei Wang, Wenmeng Zhou, Wente Wang, Wenting Shen, Wenyuan Yu, Xianzhong Shi, Xiaoming Huang, Xin Xu, Yan Kou, Yangyu Lv. Yifei Li, Yijing Liu, Yiming Wang, Yingya Zhang, Yitong Huang, Yong Li, You Wu, Yu Liu, Yulin Pan. Yun Zheng, Yuntao Hong, Yupeng Shi. Yutong Feng. Zeyinzi Jiang, Zhen Han, Zhi-Fan Wu, and Ziyu Liu. Wan: Open and Advanced Large- Video Generative Models, April 2025. URL htwp::ttt/// /tattp:///axvv../// http://awwwwv....aabbooooo/
wwwo/////aaaoo. abs 2503. 20314 4. arXiv:2503.20311
Luozhou Wang, Yijun Li, Zhifei Chen, Jui-Hsien Wang, Zhifei Zhang, He Zhang, Zhe Lin, and Yingcong Chen. TransPixeler: Advancing Text-to-Vides Generation with Transparency, January 2025. URL http: wtt.aaixvvvv...... pttg/abb//501111 0300 m.Xvv2501.033066.
Xinyu Wei, Jinrui Zhang, Zeqing Wang. Hongyang Wei, Zhen Guo, and Lei Zhang. TIIF-Bench: How Does Your T21 Model Follow Your Instructions?, June 2025. URL BRLhttp:// http://aaxvvvvooiaeeee June
abs/2506. 02161. arXiv:2506.02161.
Max
Woolf. Stable Diffusion 2.0 and the Importance of Negative Prompts for Good Results,
November 2022. URL https: http://mmiimaxxiiim http://///mimmiir...o//////11111 ctttcc/2///// weopppppp/.....
Chenfei Wu, Jiahao Li, Jingren Zhou, Junyang Lin, Kaiyuan Gao, Kun Yan, Sheng-ming Yin, Shuai Bai, Xiao Xu, Yilei Chen, Yuxiang Chen, Zecheng Tang, Zekai Zhang, Zhengyi Wang, An Yang, Bowen Yu, Chen Cheng, Dayiheng Liu, Deqing Li, Hang Zhang, Hao Meng, Hu Wei, Jingyuan Ni, Kai Chen, Kuan Cao, Liang Peng. Lin Qu, Minggang Wu, Peng Wang, Shuting Yu, Tingkun Wen, Wensen Feng, Xiaoxiao Xu, Yi Wang, Yichang Zhang, Yongqiang Zhu, Yujia Wu, Yuxuan Cai, and Zenan Liu. QweneImage Technical Report, August 2025. URL http://arxtt org/abs/ 220s/ 02324. 2...022508.02324.
Xiaoshi Wa, Yiming Hao, Keqiang Sun, Yixiong Chen, Feng Zhu, Rui Zhao, and Hongsheng Li. Human Preference Score v2: Solid Benchmark for Evaluating Human Preferences of Text-to- Text-to-Image Synthesis, September 2023. URL http://arxi org/ absp/////////// 09341. arXiv:2306.0934.
Jinheng Xie, Yuexiang Li, Yawen Huang, Haozhe Liu, Wentian Zhang, Yefeng Zheng, and Mike Zheng Shou. BoxDiff: Text-to-Image Synthesis with Training-Free Box-Constrained Dif- fusion, August 2023. URL http: //txxaaaivv///////////eea//aaaa/ arggabb/2337 10816. arXiv:2307.18816
[cs].
13

---

Published as a conference paper at ICLR 2026
Jiazheng Xu, Xiao Liu, Yuchen Wu, Yuxuan Tong, Qinkai Li. Ming Ding, Jie Tang, and Yuxiao Dong. ImageReward: Learning and Evaluating Human Preferences for Text-to-Iman Imoge Generation, December 2023. URL http: / aaxivv www.aiiiiir.iooooooooooooo abs 05977. arXiv:2304.05977.
Jiazheng Xu, Yu Huang, Jiale Cheng, Yuanming Yang, Jiajun Xu, Yuan Wang. Wenbo Duan, Shen Yang. Qunlin Jin, Shurun Li, Jiayan Teng, Zhuoyi Yang. Wendi Zheng, Xiao Liu, Ming Ding, Xiaohan Zhang, Xiaotao Gu, Shiyu Huang, Minlie Huang, Jie Tang, and Yuxiao Dong. Vi- sionReward: Fine-Grae Grained Generation, March 2025. [es].
Human Preference Learning for Image and Video 9.9. arXiv:2412.21059
Junyan Ye, Dongzhi Jiang, Zihao Wang, Leqi Zhu, Zhenghao Hu, Zilong Huang, Jun He, Zhiyuan Yan, Jinghua Yu, Hongsheng Li, Conghui He, and Weijia Li. Echo-4o: Harnessing the Power of GPT-4o Synthetic Images for Improved Image Generation, August 2025. URL http://
a arxiv. org/ abs 2508 09987. 1XXi:25508009983
Tianwei Yin, Qiang Zhang, Richard Zhang. William T. Freeman, Fredo Durand, Eli Shechtman, and Xun Huang. From Slow Bidirectional to Fast Autoregressive Video Diffusion Models, January 2025. URL http::tpp/... / rxii. .erivi.ore org/ abs 2412, C7772. arXiv:2412.07772.
Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, Cong Wei, Botao Yu, Ruibin Yuan, Renliang Sun, Ming Yin, Boyuan Zheng, Zhenzhu Yang. Yibo Liu, Wenhao Huang, Huan Sun, Yu Su, and Wenhu Chen. MMMU: Massive Multi-discipline Multimodal Understanding and Reason- ing Benchmark for Expert AGI, June 2024. URL http://////// www.iiivooooooooooo.oooooo abs /2112 16502.
arXiv:2311.1650
Mert Yuksekgonul, Federico Bianchi, Pratyusha Kalluri, Dan Jurafsky, and James Zou. When and why lingun-language models behave like bags-of-words, and what do about it?, March 2023.
URL
org/ abs/22111 01 936. arXiv:2210.01936.
Cheng Zhang, Xuanbai Chen, Siqi Chai, Chen Henry Wu, Dmitry Lagun, Thabo Beeler, and Fernando De la Torr. ITI-GEN: Inclusive Text-to-Image Generation, September 2023. URL
hetp://arker wwt/abs//////ooooo...tt..pt....r absaabss/000 /2009 05569. arXiv:2309.05566 [es].
14


Reviews:
[{"review_id": "ZaAlKluDb8", "reviewer": "Reviewer_DjfS", "summary": "This paper proposes a simple approach for negative guidance of the text-to-image (T2I) models. Current approaches flip the sign of the attention output for this, but this ends up on applying the same scale of the negative guidance across different areas of the image, and all different layers of the model. Instead, they use the attention map to calculate a per-token wieght for this negative guidance. They first develop this for approaches that use cross-attention mechanism (like latent diffuoin architecture), and then propose a new mechanism to adapt this to multimodal diffusion transformer (MMDiT)-based models like SD3, and others.", "questions": "Please check the weaknesses.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The idea of this paper is simple, but I like how they have distilled knowledge from the literature, and based on that—as well as their solid understanding of the attention mechanism—they have proposed this simple idea.", "weaknesses": "1) There are some grammatical errors and confusing parts in the paper that need to be addressed:\n- Should $x_{t-1}$ be $x_{t+1}$ in Eq. (1)?\n- line 166: *\"The method NASA applies the guidance in intermediate states instead of the predicted noise or velocity.\"* — this statement is somewhat ambiguous.\n- line 188: \"*However, it also limits the model’s ability to follow negative prompt guidance if the constraint is set to be too tight ...*\" — this sentence could be improved for clarity and readability.\n\n2) The concepts used to illustrate the issue with generation quality in Figure 2, under the presence of negative and positive prompts, are not optimal. Since winter and snow are strongly related concepts, they may be entangled in the diffusion model’s learned distribution. Using one as a positive and the other as a negative prompt may not clearly demonstrate the intended issue with current models, as the observed effect could stem from this conceptual dependence rather than the model’s capability to interpret negative guidance.\n\n3) This paper mentions that \"*rendering prompts containing negations ineffectively or made the negative prompt appears even more (e.g., a prompt like “a scientist who is not wearing glasses” will often generate a scientist with glasses—sometimes even more frequently than a simple prompt like “a scientist”).*\".  \nThere are some fairness-oriented approaches, such as ITI-Gen [1] and FairQueue [2], that discuss related issues. They point out that this cannot be addressed using **hard prompts**, but can be mitigated through prompt learning. While I understand that your setup is different, discussing the similarities and differences between your approach and these works could strengthen the paper.\n\n4. I believe the task of **negative guidance** can be viewed as a special case of **image editing**, where the prompt explicitly describes the removal of a concept while no input image is provided. From this perspective, using Qwen-Image as an external baseline (Table 1) is an interesting choice. Given that it can edit images while keeping other regions intact, it could serve as an informative upper bound for the negative guidance task. The performance gap could also highlight directions for future research. However, the details of how Qwen-Image is used, including the experimental setup, should be included in the main paper (at least at a high level).\n\n$ $\n\n*References:*\n\n[1] ITI-GEN: Inclusive Text-to-Image Generation, ICCV'23\n\n[2] FairQueue: Rethinking Prompt Learning for Fair Text-to-Image Generation, NeurIPS'24", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "FXEE8j82Jb", "reviewer": "Reviewer_emZq", "summary": "This paper proposes Value Sign Flip (VSF): a negative guidance.\n\nNegative guidance is useful in generative pipelines: progressively adjusting the results by removing something from the image.\n\nProblem statement:\n* CFG does not work in few-step configurations.\n* Negative Steer Away Attention (NASA) is currently limited to cross-attention models.\n* Normalized Attention Guidance (NAG) primarily targets quality control rather than avoiding negative prompts.\n* NASA and NAG subtract negative attention.\n* They do not generalize to various timesteps, layers, or image regions.\n\nMethod (VSF)\n* (Cross-attention-based models) Flipping the sign of negative prompt values within the attention calculation.\n* (DiT models) Duplicating negative prompts, one remains unflipped, another is flipped.\n\nAdvantages\n* VSF removes some concepts from generated images: wheels from bicycle, hands from clock, etc.\n    * maintaining image quality and adherence to the positive prompts\n* VSF works in few-step (1~8 steps) diffusion and flow matching models.\n* VSF generalizes to SD 3.5 turbo, FLUX schunell, and Wan.\n* VSF is computationally cheap.\n\nNew dataset: NegGenBench\n* Positive-negative prompt pairs from ChatGPT o3.\n* Negative prompts are core components from positive prompt; it is intentionally challenging.\n\nEvaluation: fine-tune a VLM (Qwen) for measuring faithfulness to negative prompts", "questions": "1. Why does NASA have points with negative score below 50 only in Figure 6?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Originality:\n1. The proposed method is new.\n\nQuality:\n1. The related work section covers relevant literature: CFG, Negative Guidance, and Few-step generators\n2. The competitors are aggressively chosen, even Nano Banana.\n3. Discussion is thorough\n    1. trade-off between positive and negative prompts\n    2. trade-off between quality and negative prompts\n    3. attention maps\n    4. ablation study\n\nClarity:\n1. The explanations are kind to the readers, step-by-step from NASA to the proposed method.\n\nSignificance:\n1. The method is simple and effective.\n    1. simple: concatenate the values and keys of the positive and negative prompts, then flip the sign of the negative prompt values\n    2. effective: Table 2", "weaknesses": "minor\n1. Please properly use \\citet and \\citep\n2. Fonts are too small in the figures.\n3. Is “unbrulla” a typo? or is there a message?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "kwBjOVkJ1g", "reviewer": "Reviewer_qk8d", "summary": "- Since previous negative prompts approach generally employ score matching, they are not compatible with few-step generation models.\n- To address the problem, they simply flip the value in the attention layer, which results in effectively removing the undesired contents in the final images.", "questions": "- Compared to the previous works using attention manipulation, does the proposed method contain a specific technique or contribution for being compatible with few-step models?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- The proposed method is simple but effective\n- The proposed method can be applied to a few-step model.", "weaknesses": "- The proposed method is not novel. Manipulating attention has been employed for image editing with diffusion models and the flow-matching model. (e.g., [Attend-and-exit], [self-guidance], [BoxDiff])\n- In Figure 5, VFS not only eliminates the undesired contents but also changes the other components. Specifically, in the starry night examples, the city has gone. Also, in Figure 5(right), the car is still reflected in the image with the proposed method.\n- Figures should be well illustrated. ( font size of figure 3,5 is too small, and figure 6 is too small)\n\n[Attend-and-exit]: Attention-Based Semantic Guidance for Text-to-Image Diffusion Models\n[self-guidance]: Diffusion Self-Guidance for Controllable Image Generation\n[BoxDiff]: Text-to-Image Synthesis with Training-Free Box-Constrained Diffusion", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ODYxATiGFf", "reviewer": "Reviewer_2X36", "summary": "The paper proposes Value Sign Flip (VSF), a lightweight, training-free negative-guidance method for few-step diffusion/flow models that flips the sign of negative-prompt value vectors inside attention. For cross-attention models VSF concatenates positive/negative keys/values and applies a \\alpha scaling to the negative values; for MMDiT-style models (e.g., SD 3.5 Turbo, FLUX Schnell) it further duplicates the negative tokens and masks attention paths so only image -> negative interactions are affected, optionally adding a bias to stabilize quality. VSF adapts token-, layer- and step-wise, aiming to avoid oversaturation and “mixing” failures seen when forcing CFG or using fixed-strength attention methods like NASA and NAG. On a new negation benchmark (NegGenBench) built from challenging positive/negative prompt pairs, VSF improves negative adherence while keeping quality and positive adherence competitive, and runs in ~3 s with few steps; human and MLLM-based evaluations broadly agree with these trends.", "questions": "see weaknesses", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The proposed VSF is simple and plug-and-play. And it does not include many runtime overhead.\n- It performs better than baseline methods on the collected benchmarks. It achieves better negative-prompt compliance at similar quality vs. NAG/NASA and even CFG (multi-step) in reported settings. The visualization results look visually good.\n- Clear ablations/trade-offs (α, β, masking/duplication, Whole-Embedding Flip)", "weaknesses": "- Evaluation relies heavily on MLLM judges (LLaMA/Qwen variants) for pos/neg/quality scoring; such metrics can be biased or insensitive to artifacts, and the dataset/hyperparameters are author-curated. \n- Some quality trade-offs remain at higher negative strength; masking/bias choices introduce extra knobs and implementation complexity in MMDiT stacks. \n\nWriting:\n-  Figure 3, 5, 6 is too small, could be scaled up for better organization.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)