On Architectural Compression of Text-to-Image Diffusion Models
Authors: name: Bo-Kyeong Kim, name: Hyoung-Kyu Song, name: Thibault Castells, name: Shinkook Choi

Exceptional text-to-image (T2I) generation results of Stable Diffusion models
(SDMs) come with substantial computational demands. To resolve this issue,
recent research on efficient SDMs has prioritized reducing the number of
sampling steps and utilizing network quantization. Orthogonal to these
directions, this study highlights the power of classical architectural
compression for general-purpose T2I synthesis by introducing block-removed
knowledge-distilled SDMs (BK-SDMs). We eliminate several residual and attention
blocks from the U-Net of SDMs, obtaining over a 30% reduction in the number of
parameters, MACs per sampling step, and latency. We conduct distillation-based
pretraining with only 0.22M LAION pairs (fewer than 0.1% of the full training
pairs) on a single A100 GPU. Despite being trained with limited resources, our
compact models can imitate the original SDM by benefiting from transferred
knowledge and achieve competitive results against larger multi-billion
parameter models on the zero-shot MS-COCO benchmark. Moreover, we demonstrate
the applicability of our lightweight pretrained models in personalized
generation with DreamBooth finetuning.