Normalized Attention Guidance: Universal Negative Guidance for Diffusion
  Model
Authors: name: Dar-Yen Chen; name: Hmrishav Bandyopadhyay; name: Kai Zou; name: Yi-Zhe Song
Published: 2025-05-27 13:30:46

Abstract: Negative guidance -- explicitly suppressing unwanted attributes -- remains a
fundamental challenge in diffusion models, particularly in few-step sampling
regimes. While Classifier-Free Guidance (CFG) works well in standard settings,
it fails under aggressive sampling step compression due to divergent
predictions between positive and negative branches. We present Normalized
Attention Guidance (NAG), an efficient, training-free mechanism that applies
extrapolation in attention space with L1-based normalization and refinement.
NAG restores effective negative guidance where CFG collapses while maintaining
fidelity. Unlike existing approaches, NAG generalizes across architectures
(UNet, DiT), sampling regimes (few-step, multi-step), and modalities (image,
video), functioning as a universal plug-in with minimal computational
overhead. Through extensive experimentation, we demonstrate consistent
improvements in text alignment (CLIP Score), fidelity (FID, PFID), and
human-perceived quality (ImageReward). Our ablation studies validate each
design component, while user studies confirm significant preference for
NAG-guided outputs. As a model-agnostic inference-time approach requiring no
retraining, NAG provides effortless negative guidance for all modern diffusion
frameworks -- pseudocode in the Appendix!