SwiftBrush v2: Make Your One-step Diffusion Model Better Than Its
  Teacher
Authors: name: Trung Dao; name: Thuan Hoang Nguyen; name: Thanh Le; name: Duc Vu; name: Khoi Nguyen; name: Cuong Pham; name: Anh Tran
Published: 2024-08-26 10:42:53

Abstract: In this paper, we aim to enhance the performance of SwiftBrush, a prominent
one-step text-to-image diffusion model, to be competitive with its multi-step
Stable Diffusion counterpart. Initially, we explore the quality-diversity
trade-off between SwiftBrush and SD Turbo: the former excels in image
diversity, while the latter excels in image quality. This observation motivates
our proposed modifications in the training methodology, including better weight
initialization and efficient LoRA training. Moreover, our introduction of a
novel clamped CLIP loss enhances image-text alignment and results in improved
image quality. Remarkably, by combining the weights of models trained with
efficient LoRA and full training, we achieve a new state-of-the-art one-step
diffusion model, achieving an FID of 8.14 and surpassing all GAN-based and
multi-step Stable Diffusion models. The evaluation code is available at:
https://github.com/vinairesearch/swiftbrushv2.