FlexiDiT: Your Diffusion Transformer Can Easily Generate High-Quality
  Samples with Less Compute
Authors: name: Sotiris Anagnostidis, name: Gregor Bachmann, name: Yeongmin Kim, name: Jonas Kohler, name: Markos Georgopoulos, name: Artsiom Sanakoyeu, name: Yuming Du, name: Albert Pumarola, name: Ali Thabet, name: Edgar Schönfeld

Despite their remarkable performance, modern Diffusion Transformers are
hindered by substantial resource requirements during inference, stemming from
the fixed and large amount of compute needed for each denoising step. In this
work, we revisit the conventional static paradigm that allocates a fixed
compute budget per denoising iteration and propose a dynamic strategy instead.
Our simple and sample-efficient framework enables pre-trained DiT models to be
converted into flexible ones -- dubbed FlexiDiT -- allowing them to
process inputs at varying compute budgets. We demonstrate how a single
flexible model can generate images without any drop in quality, while
reducing the required FLOPs by more than 40\% compared to their static
counterparts, for both class-conditioned and text-conditioned image generation.
Our method is general and agnostic to input and conditioning modalities. We
show how our approach can be readily extended for video generation, where
FlexiDiT models generate samples with up to 75\% less compute without
compromising performance.