SVDQunat: Absorbing Outliers by Low-Rank Components for 4-Bit Diffusion
  Models
Authors: name: Muyang Li, name: Yujun Lin, name: Zhekai Zhang, name: Tianle Cai, name: Xiuyu Li, name: Junxian Guo, name: Enze Xie, name: Chenlin Meng, name: Jun-Yan Zhu, name: Song Han

Diffusion models have been proven highly effective at generating high-quality
images. However, as these models grow larger, they require significantly more
memory and suffer from higher latency, posing substantial challenges for
deployment. In this work, we aim to accelerate diffusion models by quantizing
their weights and activations to 4 bits. At such an aggressive level, both
weights and activations are highly sensitive, where conventional post-training
quantization methods for large language models like smoothing become
insufficient. To overcome this limitation, we propose SVDQuant, a new 4-bit
quantization paradigm. Different from smoothing which redistributes outliers
between weights and activations, our approach absorbs these outliers using a
low-rank branch. We first consolidate the outliers by shifting them from
activations to weights, then employ a high-precision low-rank branch to take in
the weight outliers with Singular Value Decomposition (SVD). This process eases
the quantization on both sides. However, na\"{\i}vely running the low-rank
branch independently incurs significant overhead due to extra data movement of
activations, negating the quantization speedup. To address this, we co-design
an inference engine Nunchaku that fuses the kernels of the low-rank branch into
those of the low-bit branch to cut off redundant memory access. It can also
seamlessly support off-the-shelf low-rank adapters (LoRAs) without the need for
re-quantization. Extensive experiments on SDXL, PixArt-Sigma, and FLUX.1
validate the effectiveness of SVDQuant in preserving image quality. We reduce
the memory usage for the 12B FLUX.1 models by 3.5times, achieving
3.0times speedup over the 4-bit weight-only quantized baseline on the 16GB
laptop 4090 GPU, paving the way for more interactive applications on PCs. Our
quantization library and inference engine are open-sourced.