Eliminating Oversaturation and Artifacts of High Guidance Scales in
  Diffusion Models
Authors: name: Seyedmorteza Sadat; name: Otmar Hilliges; name: Romann M. Weber
Published: 2024-10-03 12:06:29

Abstract: Classifier-free guidance (CFG) is crucial for improving both generation
quality and alignment between the input condition and final output in diffusion
models. While a high guidance scale is generally required to enhance these
aspects, it also causes oversaturation and unrealistic artifacts. In this
paper, we revisit the CFG update rule and introduce modifications to address
this issue. We first decompose the update term in CFG into parallel and
orthogonal components with respect to the conditional model prediction and
observe that the parallel component primarily causes oversaturation, while the
orthogonal component enhances image quality. Accordingly, we propose
down-weighting the parallel component to achieve high-quality generations
without oversaturation. Additionally, we draw a connection between CFG and
gradient ascent and introduce a new rescaling and momentum method for the CFG
update rule based on this insight. Our approach, termed adaptive projected
guidance (APG), retains the quality-boosting advantages of CFG while enabling
the use of higher guidance scales without oversaturation. APG is easy to
implement and introduces practically no additional computational overhead to
the sampling process. Through extensive experiments, we demonstrate that APG is
compatible with various conditional diffusion models and samplers, leading to
improved FID, recall, and saturation scores while maintaining precision
comparable to CFG, making our method a superior plug-and-play alternative to
standard classifier-free guidance.