Adaptive Guidance: Training-free Acceleration of Conditional Diffusion
  Models
Authors: name: Angela Castillo; name: Jonas Kohler; name: Juan C. Pérez; name: Juan Pablo Pérez; name: Albert Pumarola; name: Bernard Ghanem; name: Pablo Arbeláez; name: Ali Thabet
Published: 2023-12-19 17:08:48

Abstract: This paper presents a comprehensive study on the role of Classifier-Free
Guidance (CFG) in text-conditioned diffusion models from the perspective of
inference efficiency. In particular, we relax the default choice of applying
CFG in all diffusion steps and instead search for efficient guidance policies.
We formulate the discovery of such policies in the differentiable Neural
Architecture Search framework. Our findings suggest that the denoising steps
proposed by CFG become increasingly aligned with simple conditional steps,
which renders the extra neural network evaluation of CFG redundant, especially
in the second half of the denoising process. Building upon this insight, we
propose "Adaptive Guidance" (AG), an efficient variant of CFG, that adaptively
omits network evaluations when the denoising process displays convergence. Our
experiments demonstrate that AG preserves CFG's image quality while reducing
computation by 25%. Thus, AG constitutes a plug-and-play alternative to
Guidance Distillation, achieving 50% of the speed-ups of the latter while being
training-free and retaining the capacity to handle negative prompts. Finally,
we uncover further redundancies of CFG in the first half of the diffusion
process, showing that entire neural function evaluations can be replaced by
simple affine transformations of past score estimates. This method, termed
LinearAG, offers even cheaper inference at the cost of deviating from the
baseline model. Our findings provide insights into the efficiency of the
conditional denoising process that contribute to more practical and swift
deployment of text-conditioned diffusion models.