Title: Normalized Attention Guidance: Universal Negative Guidance for Diffusion Models
Venue: NeurIPS 2025
Accepted: yes
Keywords: Diffusion models

negative guidance

guidance

text2image

text2video
Abstract: Negative guidance -- explicitly suppressing unwanted attributes -- remains a fundamental challenge in diffusion models, particularly in few-step sampling regimes. While Classifier-Free Guidance (CFG) works well in standard settings, it fails under aggressive sampling step compression due to divergent predictions between positive and negative branches. We present Normalized Attention Guidance (NAG), an efficient, training-free mechanism that applies extrapolation in attention space with L1-based normalization and refinement. NAG restores effective negative guidance where CFG collapses while maintaining fidelity. Unlike existing approaches, NAG generalizes across architectures (UNet, DiT), sampling regimes (few-step, multi-step), and modalities (image, video), functioning as a \textit{universal} plug-in with minimal computational overhead. Through extensive experimentation, we demonstrate consistent improvements in text alignment (CLIP Score), fidelity (FID, PFID), and human-perceived quality (ImageReward). Our ablation studies validate each design component, while user studies confirm significant preference for NAG-guided outputs. As a model-agnostic inference-time approach requiring no retraining, NAG provides effortless negative guidance for all modern diffusion frameworks -- pseudocode in the Appendix!


Peer reviews:
comments: SUMMARY:
This paper introduces Normalized Attention Guidance (NAG), a novel method for negative guidance in diffusion models. It addresses the limitations of existing approaches, such as Classifier-Free Guidance (CFG), which fail in few-step sampling regimes. NAG operates directly within the attention feature space, applying controlled extrapolation complemented by L1-norm-based normalization and feature refinement to prevent feature drift issues. Extensive experiments demonstrate its robustness and ability to generalize effectively.

STRENGTHS AND WEAKNESSES:
Strengths
- This paper is well written and easy to follow.
- The authors conduct comprehensive experiments across a wide range of models (UNet, DiT), sampling regimes (few-step to 25-step), and modalities (image and video). Metrics used include CLIP Score, FID, PFID, and ImageReward, providing multi-faceted validation.
- NAG adds minimal computational overhead and does not require retraining, making it highly practical for deployment.

Weaknesses
- As shown in Figure 8, the effect of normalization does not appear to be significant. Additionally, the experiment "w/o Norm & with Refine" is missing.
- While generally effective, the paper acknowledges that NAG struggles with very aggressive prompts or stylistic suppression, which may lead to instability or visual artifacts (Figure 11).

QUESTIONS:
None

LIMITATIONS:
- While NAG operates effectively in attention feature space, its applicability may be limited to models or architectures that leverage attention mechanisms. It may not generalize to non-attention-based architectures or other types of generative models.
TITLE: Official Review
CLARITY: 2
IMPACT: 2
ORIGINALITY: 2
DATE: 2025-06-16

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces Normalized Attention Guidance (NAG), a plug-in technique for attention-based diffusion models that achieves effects similar to negative prompting through simple attention modulation. The method employs L1 normalization and linear combination to process attention maps from both unconditional and conditional branches.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths\n- This paper provides a straightforward approach through normalization and linear combination of attention maps.\n- Since the method is based on simpler assumptions, this paper shows broad applicability across diffusion model variants, including single-step and video generation tasks.\n- NAG can handle edge cases where CFG with negative prompts cannot.\n\n## Weaknesses\n- Incremental contribution is of major concern. The core innovation appears to be adding a normalization step to NASA, which represents an incremental rather than fundamental contribution.\n- Modulating self-attention outputs fundamentally alters the learned data manifold, yet the paper lacks rigorous analysis of why these modifications preserve generation quality. Figure 4 somewhat addresses this, and I understand that L1 normalization constraints may limit extreme deviations, but this doesn't fully justify why such modulation is admissible.\n- Unlike Classifier-Free Guidance (CFG), which derives from well-established Classifier Guidance principles enabling principled extrapolation of noise predictions, this approach lacks comparable theoretical grounding.\n- The technique still requires CFG integration rather than serving as a standalone replacement, raising questions about its fundamental necessity.\n- Ablation studies comparing L2 normalization and max normalization variants are absent, limiting understanding of the method's sensitivity to normalization choices.\n\nQUESTIONS:\n- Why does modulating self-attention outputs preserve generation quality and not corrupt the learned data manifold?\n- Why specifically L1 normalization? Do you have any ablation studies comparing other normalization variants? There are also variants that have as benign an effect as L1 normalization on low magnitude signals.\n- If NAG still requires CFG, what fundamental classes of problems does it solve that CFG alone cannot? If there's no such distinction, why not study the prediction of classifier-free guidance and then modulate attention layers within it?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 1, "DATE": "2025-06-28"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper builds on NASA [1], which is an attention guidance/extrapolation method for negative prompting, by applying two additional operations: (1) feature normalization by clipping and re-scaling the extrapolated feature, and (2) feature blending via interpolating between the re-scaled feature and the positive guidance feature. These constraints restrict the features to be within the manifold of meaningful representations, while maintaining the effectiveness of negative guidance. It is a training-free, simple, and effective method. Experiments are conducted on different architectures (UNet and DiT), different sampling steps, and on both image and video generations, demonstrating the adaptability of the proposed method.\n\n[1] SNOOPI: Supercharged One-step Diffusion Distillation with Proper Guidance. Arxiv 2024.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n\n1. The writing of the paper is clear and well-structured, making the paper easy to follow and understand. The visual examples are helpful in illustrating the limitation of the standard CFG and the baseline NASA method.\n2. The proposed NAG is training-free, simple and effective according to several metrics such as CLIP Score, FID, PFID, and user study.\n3. The proposed NAG is adaptable to different settings of diffusion models, such as different architectures (UNet and DiT), different sampling steps (one-step and few-steps), and different modality (image and video generation).\n\nWeaknesses\n\n1. The improvements over the NASA [1] baseline is somewhat incremental, as only the last two operations (normalization and blending) are additionally proposed in NAG compared to NASA. Also, these two operations are more like tricks instead of technical improvements. Specifically, the normalization step is essentially doing a clipping/rescaling, while the blending step (Equation 10) seems to up-weight the positive branch from Equation 7, whose effect may be achieved by additional hyperparameter tuning in Equation 7. In such case, I would suggest the authors conduct more analysis (include but not limited to why L1 can retain essential structure for precise guidance as claimed) to show the technical contribution of these two operations, which should be emphasised more than the design of attention-based guidance by applying extrapolation in attention feature space, which NASA uses. Currently, such analysis is quite insufficient, as shown in the short methodology section 4.1.\n2. The proposed method introduces an additional blending scale hyperparameter $\\alpha$. It would be good to see additional analysis of the performance sensitivity to this when it is set to 0.1, 0.3, … 0.9, which is currently missing. Also, during the normalization step, an additional clipping threshold $τ$ is employed to limit feature magnitude. The sensitivity analysis of such hyperparameter is also missing.\n3. In Figure 1, the caption includes “CFG fails in few-step models”, which is, however, not reflected in the Figure. It’d be better if the authors could include the failure examples of CFG for these specific examples for qualitative comparison, or simply remove the caption to reduce the emphasis on comparing with CFG and make it a standalone plot for demonstrating the result of the proposed NAG.\n4. The denominator R in Equation 9 is not defined. Should it be the previously defined R[i]?\n\nQUESTIONS:\nI am happy to raise the score if the authors can sufficiently address the following questions and the concerns in the weaknesses section:\n\n1. Any performance gaps when the guidance scale $\\phi<1$ (i.e., doing interpolation)? From Figure 2, it seems the gap only exists when the scale is large, while one can still have effective results by using the baseline NASA method by simply setting this hyperparameter properly. \n2. Have you experimented with the proposed method on stronger few-step generation DiT and UNet backbones to show consistent performance enhancement to support the claim that NAG can be uniformly applied?\n3. Can you provide analyses of the hyperparameters additionally introduced by NAG?\n4. Can you provide additional analysis of the proposed two stabilization mechanisms to demonstrate the technical contribution in this paper in addition to NASA [1]?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes Normalized Attention Guidance (NAG), a novel inference-time approach for negative prompting in diffusion models. Unlike traditional Classifier-Free Guidance (CFG) or Perturbed Attention Guidance (PAG) methods that operate in the output space, NAG directly performs extrapolation within the attention feature space between positive and negative conditioned attention maps. To ensure stable extrapolation, the authors introduce L1-based normalization and refinement blending, constraining attention vectors within stable manifolds. Comprehensive experiments demonstrate the effectiveness and generalizability of NAG across multiple model architectures (UNet, DiT), sampling regimes (few-step, multi-step, guidance distilled models), and modalities (image, video).\n\nSTRENGTHS AND WEAKNESSES:\n\\** Strength\n\n\\#1 Attention-space Extrapolation:\nNAG’s core innovation, extrapolating directly in attention feature space, offers rich semantic control over diffusion models, surpassing output-space extrapolation approaches in terms of stability and flexibility.\n\n\\#2 Strong Empirical Generalization:\nExtensive experiments show NAG's generalizability across architectures, sampling scenarios (including aggressive few-step regimes and distilled models), and modalities such as images and videos, which are clearly valuable contributions.\n\n\\#3 Practicality and Ease of Integration:\nNAG is computationally lightweight, training-free, and easily integrated into existing diffusion pipelines. These practical benefits significantly enhance the approach's applicability.\n\n\\** Major Weakness\n\n\\#1 Lack of Theoretical Grounding beyond Geometric Interpretation\n\n- While the authors provide a helpful geometric interpretation of attention-space extrapolation, they overly rely on this intuition without offering formal theoretical analysis or probabilistic justification.\n\n- CFG and PAG clearly interpret extrapolation in output space within a well-understood diffusion probabilistic framework. In contrast, extrapolation in attention space fundamentally differs and lacks an equivalent probabilistic interpretation.\n\nThe authors exclusively rely on empirical observations and geometric explanations without theoretical guarantees or detailed formal analysis, making the underlying principles unclear.\n\n\\#2  Inaccurate Novelty Claim regarding Attention-Space Extrapolation \nThe authors claim NAG is the first method performing extrapolation in attention feature space. However, the method presented in:\n- PLADIS: Pushing the Limits of Attention in Diffusion Models at Inference Time by Leveraging Sparsity (ICCV 2025)\nalready introduced attention-space extrapolation explicitly. Thus, novelty claims must be significantly narrowed or revised clearly.\n\n- Indeed, NAG seems conceptually a special case of PLADIS, where positive attention maps are sparse and negative attention maps are dense. Explicit citation of PLADIS and detailed methodological differences are crucial, not just differences in tasks or applications.\n\n\\#3 Insufficient Justification of Normalization and Refinement Components\nWhile the normalization and blending (refinement) steps are central to NAG’s success, explicit hyperparameter details—particularly the alpha (α) value for refinement blending—are not clearly reported in the main text.\n\n- The manuscript fails to explain intuitively or theoretically why normalization and refinement specifically enhance stability.\n\n- Although the authors argue these mechanisms allow higher guidance scales, results in Figure 8 show performance peaks around relatively modest scales (2–4), questioning the practical value of further increasing the guidance scale (e.g., to 16). Qualitatively, Figure 9 similarly indicates diminishing returns after moderate scales, undermining this core claimed advantage.\n\n\\#4 No Standalone Performance Evaluation for NAG\nAll provided experimental results combine NAG with other guidance methods (CFG or PAG).\n\n- Critical information on NAG's standalone performance without additional guidance methods is absent, limiting understanding of NAG’s intrinsic value and effectiveness.\n\n\\** Minor Weakness\n\n\\#1  Lack of Specialized Quantitative Metrics:\nThe current evaluation relies heavily on general-purpose metrics like ImageReward. More specialized metrics specifically designed for diffusion and text-image alignment (e.g., PickScore, HPSv2) would provide deeper insights into alignment quality.\n\n\\#2 Lack of Quantitative Video Evaluation:\nVideo generation results rely solely on qualitative assessments, lacking quantitative metrics such as VBench. Providing quantitative results would strengthen reliability and interpretability.\n\nQUESTIONS:\nQ1. Beyond geometric interpretation, can authors provide a formal theoretical justification or probabilistic interpretation for extrapolating directly in attention feature space? Is there any formal rationale for why this manipulation maintains semantic coherence and stability?\n\nQ2.Considering PLADIS (ICCV 2025) already introduced attention-space extrapolation, how precisely does NAG differ methodologically from PLADIS? Is NAG essentially a special case of PLADIS, with sparse positive and dense negative attention maps, or are there fundamental differences beyond the application task?\n\nQ3. What specific hyperparameter values (particularly the refinement blending factor α) are used across experiments, and why were these particular choices made? How sensitive is performance to variations in α?\n\nQ4. Given the empirical observations in Figures 8 and 9 showing peak performance at relatively low guidance scales (around 2–4), what is the practical advantage of employing the additional complexity of normalization and refinement mechanisms to enable higher guidance scales?\n\nQ5. How does NAG perform independently without CFG or PAG? Can standalone NAG achieve competitive results, or is the method inherently complementary rather than independently effective?\n\nQ6. While NAG is designed explicitly for negative prompting, can the authors extend or generalize the NAG approach to standard (positive-only) text-to-image generation scenarios, without explicitly provided negative prompts?\n\nLIMITATIONS:\n- Theoretical underpinnings beyond geometric intuition are completely absent, weakening interpretability.\n\n- Novelty claims must be revised considering PLADIS’s prior introduction of attention-space extrapolation.\n\n- Lack of standalone evaluations limits the clarity of intrinsic method effectiveness.\n\n- Incomplete quantitative assessments (particularly video evaluation) and specialized metrics are significant gaps.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-06"}