Fast High-Resolution Image Synthesis with Latent Adversarial Diffusion
  Distillation
Authors: name: Axel Sauer; name: Frederic Boesel; name: Tim Dockhorn; name: Andreas Blattmann; name: Patrick Esser; name: Robin Rombach
Published: 2024-03-18 17:51:43

Abstract: Diffusion models are the main driver of progress in image and video
synthesis, but suffer from slow inference speed. Distillation methods, like the
recently introduced adversarial diffusion distillation (ADD) aim to shift the
model from many-shot to single-step inference, albeit at the cost of expensive
and difficult optimization due to its reliance on a fixed pretrained DINOv2
discriminator. We introduce Latent Adversarial Diffusion Distillation (LADD), a
novel distillation approach overcoming the limitations of ADD. In contrast to
pixel-based ADD, LADD utilizes generative features from pretrained latent
diffusion models. This approach simplifies training and enhances performance,
enabling high-resolution multi-aspect ratio image synthesis. We apply LADD to
Stable Diffusion 3 (8B) to obtain SD3-Turbo, a fast model that matches the
performance of state-of-the-art text-to-image generators using only four
unguided sampling steps. Moreover, we systematically investigate its scaling
behavior and demonstrate LADD's effectiveness in various applications such as
image editing and inpainting.