SiT: Exploring Flow and Diffusion-based Generative Models with Scalable
  Interpolant Transformers
Authors: name: Nanye Ma; name: Mark Goldstein; name: Michael S. Albergo; name: Nicholas M. Boffi; name: Eric Vanden-Eijnden; name: Saining Xie
Published: 2024-01-16 18:55:25

Abstract: We present Scalable Interpolant Transformers (SiT), a family of generative
models built on the backbone of Diffusion Transformers (DiT). The interpolant
framework, which allows for connecting two distributions in a more flexible way
than standard diffusion models, makes possible a modular study of various
design choices impacting generative models built on dynamical transport: using
discrete vs. continuous time learning, deciding the objective for the model to
learn, choosing the interpolant connecting the distributions, and deploying a
deterministic or stochastic sampler. By carefully introducing the above
ingredients, SiT surpasses DiT uniformly across model sizes on the conditional
ImageNet 256x256 benchmark using the exact same backbone, number of parameters,
and GFLOPs. By exploring various diffusion coefficients, which can be tuned
separately from learning, SiT achieves an FID-50K score of 2.06.