SVDQuant: Absorbing Outliers by Low-Rank Component for 4-Bit Diffusion Models

## Abstract

Abstract Diffusion models have been proven highly effective at generating high-quality images. However, as these models grow larger, they require significantly more memory and suffer from higher latency, posing substantial challenges for deployment. In this work, we aim to accelerate diffusion models by quantizing their weights and activations to 4 bits. At such an aggressive level, both weights and activations are highly sensitive, where conventional post-training quantization methods for large language models like smoothing become insufficient. To overcome this limitation, we propose SVDQuant , a new 4-bit quantization paradigm. Different from smoothing which redistributes outliers between weights and activations, our approach absorbs these outliers using a low-rank branch. We first consolidate the outliers by shifting them from activations to weights, then employ a high-precision low-rank branch to take in the weight outliers with Singular Value Decomposition (SVD). This process eases the quantization on both sides. However, naïvely running the low-rank branch independently incurs significant overhead due to extra data movement of activations, negating the quantization speedup. To address this, we co-design an inference engine Nunchaku that fuses the kernels of the low-rank branch into those of the low-bit branch to cut off redundant memory access. It can also seamlessly support off-the-shelf low-rank adapters (LoRAs) without the need for re-quantization. Extensive experiments on SDXL, PixArt- Σ Σ \Sigma roman_Σ , and FLUX.1 validate the effectiveness of SVDQuant in preserving image quality. We reduce the memory usage for the 12B FLUX.1 models by 3.5×, achieving 3.0× speedup over the 4-bit weight-only quantized baseline on the 16GB laptop 4090 GPU, paving the way for more interactive applications on PCs. Our quantization library * * * Quantization library: github.com/mit-han-lab/deepcompressor and inference engine † † † Inference Engine: github.com/mit-han-lab/nunchaku are open-sourced.

## 1 Introduction

Diffusion models have shown remarkable capabilities in generating high-quality images (Ho et al., ), with recent advances further enhancing user control over the generation process. Trained on vast data, these models can create stunning images from simple text prompts, unlocking diverse image editing and synthesis applications (Meng et al., ; Ruiz et al., ; Zhang et al., ).

Figure: Figure 2: Computation *vs*. parameters for LLMs and diffusion models. LLMs’ computation is measured with 512 context and 256 output tokens, and diffusion models’ computation is for a single step. Dashed lines show trends.
Refer to caption: x2.png

To pursue higher image quality and more precise text-to-image alignment, researchers are increasingly scaling up diffusion models. As shown in Figure 2, Stable Diffusion (SD) (Rombach et al., ) 1.4 only has 800M parameters, while SDXL (Podell et al., ) scales this up to 2.6B parameters. AuraFlow v0.1 (fal.ai, ) extends this further to 6B parameters, with the latest model, FLUX.1 (Black-Forest-Labs, ), pushing the boundary to 12B parameters. Compared to large language models (LLMs), diffusion models are significantly more computationally intensive. Their computational costs(^‡^‡‡Computational cost is measured by number of Multiply-Accumulate operations (MACs). 1 MAC=2 FLOPs.) increase more rapidly with model size, posing a prohibitive memory and latency barrier for real-world model deployment, particularly for interactive use cases that demand low latency.

As Moore’s law slows down, hardware vendors are turning to low-precision inference to sustain performance improvements. For instance, NVIDIA’s Blackwell Tensor Cores introduce a new 4-bit floating point (FP4) precision, doubling the performance compared to FP8 (NVIDIA, ). Therefore, using 4-bit inference to accelerate diffusion models is appealing. In the realm of LLMs, researchers have leveraged quantization to compress model sizes and boost inference speed (Dettmers et al., ; Xiao et al., ). However, unlike LLMs–where latency is primarily constrained by loading model weights, especially with small batch sizes–diffusion models are heavily computationally bound, even with a single batch. As a result, weight-only quantization cannot accelerate diffusion models. To achieve speedup, both weights and activations must be quantized to the same bit width; otherwise, the lower-precision side will be upcast during computation, negating potential performance enhancements.

In this work, we focus on quantizing both the weights and activations of diffusion models to 4 bits. This challenging and aggressive scheme is often prone to severe quality degradation. Existing methods like smoothing (Xiao et al., ; Lin et al., ), which attempt to transfer the outliers between the weights and activations, are less effective since both sides are highly vulnerable to outliers. To address this issue, we propose a new general-purpose quantization paradigm, SVDQuant. Our core idea is to introduce a low-cost branch to absorb outliers on both sides. To achieve this, as illustrated in Figure 3, we first aggregate the outliers by migrating them from activation ${\bm{X}}$ to weight ${\bm{W}}$ via smoothing. Then we apply Singular Value Decomposition (SVD) to the updated weight, $\hat{{\bm{W}}}$, decomposing it into a low-rank branch ${\bm{L}}_{1}{\bm{L}}_{2}$ and a residual $\hat{{\bm{W}}}-{\bm{L}}_{1}{\bm{L}}_{2}$. The low-rank branch operates at 16 bits, allowing us to quantize only the residual to 4 bits, which has significantly reduced outliers and magnitude. However, naively running the low-rank branch separately incurs substantial memory access overhead, offsetting the speedup of 4-bit inference. To overcome this, we co-design a specialized inference engine Nunchaku, which fuses the low-rank branch computation into the 4-bit quantization and computation kernels. This design enables us to achieve measured inference speedup even with additional branches.

SVDQuant can quantize various text-to-image diffusion architectures, including both UNet (Ho et al., ; Ronneberger et al., ) and DiT (Peebles & Xie, ) backbones, into 4 bits, while maintaining visual quality. It supports both INT4 and FP4 data types, and integrates seamlessly with pre-trained low-rank adapters (LoRA) (Hsu et al., ) without requiring re-quantization. To our knowledge, we are the first to successfully apply 4-bit post-training quantization to both the weights and activations of diffusion models, and achieve measured speedup on NVIDIA GPUs. On the latest 12B FLUX.1, we largely preserve the image quality and reduce the memory footprint of the original BF16 model by 3.5× and deliver a 3.0× speedup over the NF4 weight-only quantized baseline, measured on a 16GB laptop-level RTX4090 GPU. See Figure 1 for visual examples.

Figure: Figure 3: Overview of SVDQuant. (a) Originally, both the activation ${\bm{X}}$ and weight ${\bm{W}}$ contain outliers, making 4-bit quantization challenging. (b) We migrate the outliers from the activation to weight, resulting in the updated activation $\hat{{\bm{X}}}$ and weight $\hat{{\bm{W}}}$. While $\hat{{\bm{X}}}$ becomes easier to quantize, $\hat{{\bm{W}}}$ now becomes more difficult. (c) SVDQuant further decomposes $\hat{{\bm{W}}}$ into a low-rank component ${\bm{L}}_{1}{\bm{L}}_{2}$ and a residual $\hat{{\bm{W}}}-{\bm{L}}_{1}{\bm{L}}_{2}$ with SVD. Thus, the quantization difficulty is alleviated by the low-rank branch, which runs at 16-bit precision.
Refer to caption: x3.png

## 2 Related Work

Diffusion models. Diffusion models (Sohl-Dickstein et al., ; Ho et al., ) have emerged as a powerful class of generative models, known for their ability to generate high-quality samples by modeling the data distribution through an iterative denoising process. Recent advancements in text-to-image diffusion models (Balaji et al., ; Rombach et al., ; Podell et al., ) have already revolutionized content generation. Researchers further shifted from convolution-based UNet architectures (Ronneberger et al., ; Ho et al., ) to transformers (*e.g*., DiT (Peebles & Xie, ) and U-ViT (Bao et al., )) and scaled up the model size (Esser et al., ). However, diffusion models suffer from extremely slow inference speed due to their long denoising sequences and intense computation. To address this, various approaches have been proposed, including few-step samplers (Zhang & Chen, ; Zhang et al., ; Lu et al., ) or distilling fewer-step models from pre-trained ones (Salimans & Ho, ; Meng et al., ; Song et al., ; Luo et al., ; Sauer et al., ; Yin et al., ; ; Kang et al., ). Another line of works choose to optimize or accelerate computation via efficient architecture design (Li et al., ; ; Cai et al., ; Liu et al., ), quantization (Shang et al., ; Li et al., ), sparse inference (Li et al., ; Ma et al., ; ), and distributed inference (Li et al., ; Wang et al., ; Chen et al., ). This work focuses on quantizing the diffusion models to 4 bits to reduce the computation complexity. Our method can also be applied to the few-step diffusion models to further reduce the latency (see Section 5.2).

Quantization. Quantization has been recognized as an effective approach for LLMs to reduce the model size and accelerate inference (Dettmers et al., ; Frantar et al., ; Xiao et al., ; Lin et al., ; ; Kim et al., ; Zhao et al., ). For diffusion models, Q-Diffusion (Li et al., ) and PTQ4DM (Shang et al., ) first achieved 8-bit quantization. Subsequent works refined these techniques with approaches like sensitivity analysis (Yang et al., ) and timestep-aware quantization (He et al., ; Huang et al., ; Liu et al., ; Wang et al., ). Some recent works extended the setting to text-to-image models (Tang et al., ; Zhao et al., ), DiT backbones (Wu et al., ), quantization-aware training (He et al., ; Zheng et al., ; Wang et al., ; Sui et al., ), video generation (Zhao et al., ), and different data types (Liu & Zhang, ). Among these works, only MixDQ (Zhao et al., ) and ViDiT-Q (Zhao et al., ) implement low-bit inference engines and report measured 8-bit speedup on GPUs. In this work, we push the boundary further by quantizing diffusion models to 4 bits, supporting both the integer or floating-point data types, compatible with the UNet backbone (Ho et al., ) and recent DiT architecture (Peebles & Xie, ). Our co-designed inference engine, Nunchaku, further ensures on-hardware speedup. Additionally, when applying LoRA to the model, existing methods require fusing the LoRA branch to the main branch and re-quantizing the model to avoid tremendous memory-access overhead in the LoRA branch. Nunchaku cuts off this overhead via kernel fusion, allowing the low-rank branch to run efficiently as a separate branch, eliminating the need for re-quantization.

Low-rank decomposition.
Low-rank decomposition has gained significant attention in deep learning for enhancing computational and memory efficiency (Hu et al., ; Zhao et al., ; Jaiswal et al., ). While directly applying this approach to model weights can reduce the compute and memory demands (Hsu et al., ; Yuan et al., ; Li et al., ), it often leads to performance degradation. Instead, Yao et al. () combined it with quantization for model compression, employing a low-rank branch to compensate for the quantization error. Low-Rank Adaptation (LoRA) (Hu et al., ) introduces another important line of research by using low-rank matrices to adjust a subset of pre-trained weights for efficient fine-tuning.
This has sparked numerous advancements (Dettmers et al., ; Guo et al., ; Li et al., ; He et al., ; Xu et al., ), which combines quantized models with low-rank adapters to reduce memory usage during model fine-tuning. However, our work differs in two major aspects. Firstly, our goal is different, as we aim to accelerate model inference through quantization, while previous works focus on model compression or efficient fine-tuning. Thus, they primarily consider weight-only quantization, resulting in no speedup. Secondly, as shown in our experiments (Figure 6 and ablation study in Section 5.2), directly applying these methods not only degrades the image quality, but also introduces significant overhead. In contrast, our method yields much better performance due to our joint quantization of weights and activations and our inference engine Nunchaku minimizes the overhead by fusing the low-rank branch kernels into the low-bit computation ones.

## 3 Quantization Preliminary

Quantization is an effective approach to accelerate linear layers in networks. Given a tensor ${\bm{X}}$, the quantization process is defined as:

$$ ${\bm{Q}}_{\bm{X}}=\text{round}\left(\frac{{\bm{X}}}{s_{{\bm{X}}}}\right),s_{{ \bm{X}}}=\frac{\max(|{\bm{X}}|)}{q_{\max}}.$ (1) $$

Here, ${\bm{Q}}_{\bm{X}}$ is the low-bit representation of ${\bm{X}}$, $s_{{\bm{X}}}$ is the scaling factor, and $q_{\max}$ is the maximum quantized value. For signed $b$-bit integer quantization, $q_{\max}=2^{b-1}-1$. For 4-bit floating-point quantization with 1-bit mantissa and 2-bit exponent, $q_{\max}=6$. Thus, the dequantized tensor can be formulated as $Q({\bm{X}})=s_{{\bm{X}}}\cdot{\bm{Q}}_{\bm{X}}$. For a linear layer with input ${\bm{X}}$ and weight ${\bm{W}}$, its computation can be approximated by

$$ ${\bm{X}}{\bm{W}}\approx Q({\bm{X}})Q({\bm{W}})=s_{{\bm{X}}}s_{{\bm{W}}}\cdot{ \bm{Q}}_{\bm{X}}{\bm{Q}}_{\bm{W}}.$ (2) $$

The same approximation applies to convolutional layers. To speed up computation, modern arithmetic logic units require both ${\bm{Q}}_{\bm{X}}$ and ${\bm{Q}}_{\bm{W}}$ using the same bit width. Otherwise, the low-bit side needs to be upcast to match the higher bit width, negating the speed advantage. Following the notation in QServe (Lin et al., ), we denote $x$-bit weight, $y$-bit activation as W$x$A$y$. “INT” and “FP” refer to the integer and floating-point data types, respectively.

In this work, we focus on W4A4 quantization for acceleration, where outliers in both weights and activations place substantial obstacles.
Traditional methods to suppress these outliers include quantization-aware training (QAT) (He et al., ) and rotation (Ashkboos et al., ; Liu et al., ; Lin et al., ). QAT requires massive computing resources, especially for tuning models with more than 10B parameters (*e.g*., FLUX.1). Rotation is inapplicable due to the usage of adaptive normalization layers (Peebles & Xie, ) in diffusion models. The runtime-generated normalization weights preclude the offline integration of the rotation matrix with the weights of projection layers. Consequently, online rotation of both activations and weights incurs significant runtime overhead.

## 4 Method

In this section, we first formulate our problem and discuss where the quantization error comes from.
Next, we present SVDQuant, a new W4A4 quantization paradigm for diffusion models. Our key idea is to introduce an additional low-rank branch that can absorb quantization difficulties in both weights and activations.
Finally, we provide a co-designed inference engine Nunchaku with kernel fusion to minimize the overhead of the low-rank branches in the 4-bit model.

Figure: Figure 4: Example value distribution of inputs and weights in PixArt-$\Sigma$. ${\bm{\lambda}}$ is the smooth factor. Red indicates the outliers. Initially, both the input ${\bm{X}}$ and weight ${\bm{W}}$ contain significant outliers. After smoothing, the range of $\hat{{\bm{X}}}$ is reduced with much fewer outliers, while $\hat{{\bm{W}}}$ shows more outliers. Once the SVD low-rank branch ${\bm{L}}_{1}{\bm{L}}_{2}$ is subtracted, the residual ${\bm{R}}$ has a narrower range and is free from outliers.
Refer to caption: x4.png

### 4.1 Problem Formulation

Consider a linear layer with input ${\bm{X}}\in{\mathbb{R}}^{b\times m}$ and weight ${\bm{W}}\in{\mathbb{R}}^{m\times n}$. The quantization error can be defined as

$$ $E({\bm{X}},{\bm{W}})=\left\lVert{\bm{X}}{\bm{W}}-Q({\bm{X}})Q({\bm{W}})\right \rVert_{F},$ (3) $$

where $\|\cdot\|_{F}$ denotes Frobenius Norm.

###### Proposition 4.1 (Error decomposition) .

The quantization error can be decomposed as follows:

$$ $\displaystyle E({\bm{X}},{\bm{W}})\leq\left\lVert{\bm{X}}\right\rVert_{F}\left \lVert{\bm{W}}-Q({\bm{W}})\right\rVert_{F}+\left\lVert{\bm{X}}-Q({\bm{X}}) \right\rVert_{F}(\left\lVert{\bm{W}}\right\rVert_{F}+\left\lVert{\bm{W}}-Q({ \bm{W}})\right\rVert_{F}).$ (4) $$

See Appendix A.1 for the proof. From the proposition, we can see that the error is bounded by four elements – the magnitude of the weight and input, $\left\lVert{\bm{W}}\right\rVert_{F}$ and $\left\lVert{\bm{X}}\right\rVert_{F}$, and their respective quantization errors, $\left\lVert{\bm{W}}-Q({\bm{W}})\right\rVert_{F}$ and $\left\lVert{\bm{X}}-Q({\bm{X}})\right\rVert_{F}$. To minimize the overall quantization error, we aim to optimize these four terms.

###### Proposition 4.1 (Error decomposition) .

### 4.2 SVDQuant: Absorbing Outliers via Low-Rank Branch

Migrate outliers from activation to weight.
Smoothing (Xiao et al., ; Lin et al., ) is an effective approach for reducing outliers. We can smooth outliers in activations by scaling down the input ${\bm{X}}$ and adjusting the weight matrix ${\bm{W}}$ correspondingly using a per-channel smoothing factor ${\bm{\lambda}}\in{\mathbb{R}}^{m}$. As shown in Figure 4(a)(c), the smoothed input $\hat{{\bm{X}}}={\bm{X}}\cdot\text{diag}({\bm{\lambda}})^{-1}$ exhibits reduced magnitude and fewer outliers, resulting in lower input quantization error. However, in Figure 4(b)(d), the transformed weight $\hat{{\bm{W}}}={\bm{W}}\cdot\text{diag}({\bm{\lambda}})$ has a significant increase in both magnitude and the presence of outliers, which in turn raises the weight quantization error. Consequently, the overall error reduction is limited.

Absorb magnified weight outliers with a low-rank branch. Our core insight is to introduce a 16-bit low-rank branch and further migrate the weight quantization difficulty to this branch. Specifically, we decompose the transformed weight as $\hat{{\bm{W}}}={\bm{L}}_{1}{\bm{L}}_{2}+{\bm{R}}$, where ${\bm{L}}_{1}\in{\mathbb{R}}^{m\times r}$ and ${\bm{L}}_{2}\in{\mathbb{R}}^{r\times n}$ are two low-rank factors of rank $r$, and ${\bm{R}}$ is the residual. Then ${\bm{X}}{\bm{W}}$ can be approximated as

$$ ${\bm{X}}{\bm{W}}=\hat{{\bm{X}}}\hat{{\bm{W}}}=\hat{{\bm{X}}}{\bm{L}}_{1}{\bm{L }}_{2}+\hat{{\bm{X}}}{\bm{R}}\approx\underbrace{\hat{{\bm{X}}}{\bm{L}}_{1}{\bm {L}}_{2}}_{\text{16-bit low-rank branch}}+\underbrace{Q(\hat{{\bm{X}}})Q({\bm{ R}})}_{\text{4-bit residual}}.$ (5) $$

Compared to direct 4-bit quantization, i.e., $Q(\hat{{\bm{X}}})Q({\bm{W}})$, our method first computes the low-rank branch $\hat{{\bm{X}}}{\bm{L}}_{1}{\bm{L}}_{2}$ in 16-bit precision, and then approximates the residual $\hat{{\bm{X}}}{\bm{R}}$ with 4-bit quantization. Empirically, $r\ll\min(m,n)$, and is typically set to 16 or 32. As a result, the additional parameters and computation for the low-rank branch are negligible, contributing only $\frac{mr+nr}{mn}$ to the overall costs. However, it still requires careful system design to eliminate redundant memory access, which we will discuss in Section 4.3.

From Equation 5, the quantization error can be expressed as

$$ $\left\lVert\hat{{\bm{X}}}\hat{{\bm{W}}}-(\hat{{\bm{X}}}{\bm{L}}_{1}{\bm{L}}_{2 }+Q(\hat{{\bm{X}}})Q({\bm{R}}))\right\rVert_{F}=\left\lVert\hat{{\bm{X}}}{\bm{ R}}-Q(\hat{{\bm{X}}})Q({\bm{R}})\right\rVert_{F}=E(\hat{{\bm{X}}},{\bm{R}}),$ (6) $$

where ${\bm{R}}=\hat{{\bm{W}}}-{\bm{L}}_{1}{\bm{L}}_{2}$. According to Proposition 4.1, since $\hat{{\bm{X}}}$ is already free from outliers, we only need to focus on optimizing the magnitude of ${\bm{R}}$, $\left\lVert{\bm{R}}\right\rVert_{F}$ and its quantization error, $\left\lVert{\bm{R}}-Q({\bm{R}})\right\rVert_{F}$.

###### Proposition 4.2 (Quantization error bound) .

For any tensor ${\bm{R}}$ and quantization method described in Equation 1 as $Q({\bm{R}})=s_{\bm{R}}\cdot{\bm{Q}}_{\bm{R}}$. Assuming the element of ${\bm{R}}$ follows a normal distribution, we have

$$ $\displaystyle\mathbb{E}\left[\left\lVert{\bm{R}}-Q({\bm{R}})\right\rVert_{F} \right]\leq\frac{\sqrt{\log\left(\text{size}({\bm{R}})\right)\pi}}{q_{\max}} \mathbb{E}\left[\left\lVert{\bm{R}}\right\rVert_{F}\right],$ (7) $$

where $\text{size}({\bm{R}})$ denotes the number of elements in ${\bm{R}}$.

Figure: Figure 5: First 64 singular values of ${\bm{W}}$, $\hat{{\bm{W}}}$, and ${\bm{R}}$. The first 32 singular values of $\hat{{\bm{W}}}$ exhibit a steep drop, while the remaining values are much more gradual.
Refer to caption: x5.png

See Appendix A.2 for the proof. From this proposition, we obtain the intuition that the quantization error $\left\lVert{\bm{R}}-Q({\bm{R}})\right\rVert_{F}$ is bounded by the magnitude of the residual $\left\lVert{\bm{R}}\right\rVert_{F}$. Thus, our goal is to find the optimal ${\bm{L}}_{1}{\bm{L}}_{2}$ that minimizes $\left\lVert{\bm{R}}\right\rVert_{F}=\left\lVert\hat{{\bm{W}}}-{\bm{L}}_{1}{\bm
{L}}_{2}\right\rVert_{F}$, which can be solved by simple Singular Value Decomposition (SVD). Given the SVD of $\hat{{\bm{W}}}={\bm{U}}{\bm{\Sigma}}{\bm{V}}$, the optimal solution is ${\bm{L}}_{1}={\bm{U}}{\bm{\Sigma}}_{:,:r}$ and ${\bm{L}}_{2}={\bm{V}}_{:r,:}$. Figure 5 illustrates the singular value distribution of the original weight ${\bm{W}}$, transformed weight $\hat{{\bm{W}}}$ and residual ${\bm{R}}$. The singular values of the original weight ${\bm{W}}$ are highly imbalanced. After smoothing, the singular value distribution of $\hat{{\bm{W}}}$ becomes even sharper, with only the first several values being significantly larger. By removing these dominant values, Eckart–Young–Mirsky theorem(^§^§§[https://en.wikipedia.org/wiki/Low-rank_approximation](https://en.wikipedia.org/wiki/Low-rank_approximation)) suggests that the magnitude of the residual
${\bm{R}}$ is dramatically reduced, as $\left\lVert{\bm{R}}\right\rVert_{F}=\sqrt{\sum_{i=r+1}^{\min(m,n)}\sigma_{i}^{
2}}$, compared to the original magnitude $\left\lVert\hat{{\bm{W}}}\right\rVert_{F}=\sqrt{\sum_{i=1}^{\min(m,n)}\sigma_{
i}^{2}}$, where $\sigma_{i}$ is the $i$-th singular value of $\hat{{\bm{W}}}$. Furthermore, empirical observations reveal that ${\bm{R}}$ exhibits fewer outliers with a substantially compressed value range compared to $\hat{{\bm{W}}}$, as shown in Figure 4(d)(e). In practice, we can further reduce quantization errors by iteratively updating the low-rank branch through decomposing ${\bm{W}}-Q({\bm{R}})$ and adjusting ${\bm{R}}$ accordingly for several iterations, and then picking the result with the smallest error.

Figure: Figure 6: (a) Naïvely running low-rank branch with rank 32 will introduce 57% latency overhead due to extra read of 16-bit inputs in Down Projection and extra write of 16-bit outputs in Up Projection. Our Nunchaku engine optimizes this overhead with kernel fusion. (b) Down Projection and Quantize kernels use the same input, while Up Projection and 4-Bit Compute kernels share the same output. To reduce data movement overhead, we fuse the first two and the latter two kernels together.
Refer to caption: x6.png

###### Proposition 4.2 (Quantization error bound) .

### 4.3 Nunchaku: Fusing Low-Rank and Low-Bit Branch Kernels

Although the low-rank branch introduces theoretically negligible computation, running it as a separate branch would incur significant latency overhead—approximately 50% of the 4-bit branch latency, as shown in Figure 6(a).
This is because, for a small rank $r$, even though the computational cost is greatly reduced, the data sizes of input and output activations remain unchanged, shifting the bottleneck from computation to memory access. The situation deteriorates, especially when the activation cannot fit into the GPU L2 cache. For example, the up projection in the low-rank branch for QKV projection is much slower since its output exceeds the available L2 cache and results in the extra load and store operations to DRAM. Fortunately, we observe that the down projection ${\bm{L}}_{1}$ in the low-rank branch shares the same input as the quantization kernel in the low-bit branch, while the up projection ${\bm{L}}_{2}$ shares the same output as the 4-bit computation kernel, as illustrated in Figure 6(b). By fusing the down projection with the quantization kernel and the up projection with the 4-bit computation kernel, the low-rank branch can share the activations with the low-bit branch, eliminating the extra memory access and also halving the number of kernel calls. As a result, the low-rank branch adds only 5$\sim$10% latency, making it nearly cost-free.

## 5 Experiments

### 5.1 Setups

Models.
We benchmark our methods using the following text-to-image models, including both the UNet (Ronneberger et al., ; Ho et al., ) and DiT (Peebles & Xie, ) backbones:

- •
FLUX.1 (Black-Forest-Labs, ) is the SoTA open-sourced DiT-based diffusion model. It consists of 19 joint attention blocks (Esser et al., ) and 38 parallel attention blocks (Dehghani et al., ), totaling 12B parameters. We evaluate on both the 50-step guidance-distilled (FLUX.1-dev) and 4-step timestep-distilled (FLUX.1-schnell) variants.
- •
PixArt-$\Sigma$ (Chen et al., ) is another DiT-based model. Instead of using joint attention, it stacks 28 attention blocks composed of self-attention, cross-attention, and feed-forward layers, amounting to 600M parameters. We evaluate it on the default 20-step setting.
- •
Stable Diffusion XL (SDXL) is a widely-used UNet-based model with 2.6B parameters (Podell et al., ). It predicts noise with three resolution scales. The highest-resolution stage is processed entirely by ResBlocks (He et al., ), while the other two stages jointly use ResBlocks and attention layers. Like PixArt-$\Sigma$, SDXL employs cross-attention layers for text conditioning. We evaluate it in the 30-step setting, along with its 4-step distilled variant, SDXL-Turbo (Sauer et al., ).

Datasets. Following previous works (Li et al., ; Zhao et al., ; ), we randomly sample the prompts in COCO Captions 2024 (Chen et al., ) for calibration. To assess the generalization capability of our method, we adopt two distinct prompt sets with varying styles for benchmarking:

- •
MJHQ-30K (Li et al., ) consists of 30K samples from Midjourney with 10 common categories, 3K samples each. We uniformly select 5K prompts from this dataset to evaluate model performance on artistic image generation.
- •
Densely Captioned Images (DCI) (Urbanek et al., ) is a dataset containing $\sim$8K images with detailed human-annotated captions, averaging over 1,000 words. For our experiments, we use the summarized version (sDCI), where captions are condensed to 77 tokens using large language models (LLMs) to accommodate diffusion models. Similarly, we randomly sample 5K prompts for efficient evaluation of realistic image generation.

Baselines.
We compare SVDQuant against the following post-training quantization (PTQ) methods:

- •
4-bit NormalFloat (NF4) is a data type for weight-only quantization (Dettmers et al., ). It assumes that weights follow a normal distribution and is the information-theoretically optimal 4-bit representation. We use the community-quantized NF4 FLUX.1 models (Lllyasviel, ) as the baselines.
- •
ViDiT-Q (Zhao et al., ) uses per-token quantization and smoothing (Xiao et al., ) to alleviate the outliers across different batches and tokens and achieves lossless 8-bit quantization on PixArt-$\Sigma$.
- •
MixDQ (Zhao et al., ) identifies the outliers in the begin-of-sentence token of text embedding and protects them with 16-bit pre-computation. This method enables up to W4A8 quantization with negligible performance degradation on SDXL-Turbo.
- •
[TensorRT](https://developer.nvidia.com/blog/tensorrt-accelerates-stable-diffusion-nearly-2x-faster-with-8-bit-post-training-quantization/) contains an industry-level PTQ toolkit to quantize the diffusion models to 8 bits. It uses smoothing and only calibrates activations over a selected timestep range with a percentile scheme.

Metrics. Following previous work (Li et al., ; ), we mainly benchmark image quality and similarity to the results produced by the original 16-bit models. For the image quality assessment, we use Fréchet Inception Distance (FID, lower is better) to measure the distribution distance between the generated images and the ground-truth images (Heusel et al., ). Besides, we employ Image Reward (higher is better) to approximate the human rating of the generated images (Xu et al., ). We use LPIPS (lower is better) to measure the perceptual similarity (Zhang et al., ) and Peak Signal Noise Ratio (PSNR, higher is better) to measure the numerical similarity of the images from the 16-bit models. Please refer to our Appendix B.1 for more metrics (CLIP IQA (Wang et al., ), CLIP Score (Hessel et al., ) and SSIM(^¶^¶¶[https://en.wikipedia.org/wiki/Structural_similarity_index_measure](https://en.wikipedia.org/wiki/Structural_similarity_index_measure))).

**Table 1: Quantitative quality comparisons across different models. IR means ImageReward. Our 8-bit results closely match the quality of the 16-bit models. Moreover, our 4-bit results outperform other 4-bit baselines, effectively preserving the visual quality of 16-bit models.**
|  |  |  |  | MJHQ | sDCI |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Backbone | Model | Precision | Method | Quality | Similarity | Quality | Similarity |  |  |  |  |
|  |  |  |  | FID ($\downarrow$) | IR ($\uparrow$) | LPIPS ($\downarrow$) | PSNR( $\uparrow$) | FID ($\downarrow$) | IR ($\uparrow$) | LPIPS ($\downarrow$) | PSNR ($\uparrow$) |
|  | FLUX.1<br>-dev<br>(50 Steps) | BF16 | – | 20.3 | 0.953 | – | – | 24.8 | 1.02 | – | – |
|  | INT W8A8 | Ours | 20.4 | 0.948 | 0.089 | 27.0 | 24.7 | 1.02 | 0.106 | 24.9 |  |
|  | W4A16 | NF4 | 20.6 | 0.910 | 0.272 | 19.5 | 24.9 | 0.986 | 0.292 | 18.2 |  |
|  | INT W4A4 | Ours | 19.9 | 0.932 | 0.254 | 20.1 | 24.7 | 0.992 | 0.273 | 18.8 |  |
|  | FP W4A4 | Ours | 21.0 | 0.933 | 0.247 | 20.2 | 25.7 | 0.995 | 0.267 | 18.7 |  |
|  | FLUX.1<br>-schnell<br>(4 Steps) | BF16 | – | 19.2 | 0.938 | – | – | 20.8 | 0.932 | – | – |
|  | INT W8A8 | Ours | 19.2 | 0.966 | 0.120 | 22.9 | 20.7 | 0.975 | 0.133 | 21.3 |  |
| DiT | W4A16 | NF4 | 18.9 | 0.943 | 0.257 | 18.2 | 20.7 | 0.953 | 0.263 | 17.1 |  |
|  | INT W4A4 | Ours | 18.4 | 0.969 | 0.292 | 17.5 | 20.1 | 0.988 | 0.299 | 16.3 |  |
|  | FP W4A4 | Ours | 19.9 | 0.956 | 0.279 | 17.5 | 21.5 | 0.967 | 0.278 | 16.6 |  |
|  | PixArt-$\Sigma$<br>(20 Steps) | FP16 | – | 16.6 | 0.944 | – | – | 24.8 | 0.966 |  |  |
|  | INT W8A8 | ViDiT-Q | 15.7 | 0.944 | 0.137 | 22.5 | 23.5 | 0.974 | 0.163 | 20.4 |  |
|  | INT W8A8 | Ours | 16.3 | 0.955 | 0.109 | 23.7 | 24.2 | 0.969 | 0.129 | 21.8 |  |
|  | INT W4A8 | ViDiT-Q | 37.3 | 0.573 | 0.611 | 12.0 | 40.6 | 0.600 | 0.629 | 11.2 |  |
|  | INT W4A4 | ViDiT-Q | 412 | -2.27 | 0.854 | 6.44 | 425 | -2.28 | 0.838 | 6.70 |  |
|  | INT W4A4 | Ours | 20.1 | 0.898 | 0.394 | 16.2 | 25.1 | 0.922 | 0.434 | 14.9 |  |
|  | FP W4A4 | Ours | 18.3 | 0.946 | 0.326 | 17.4 | 23.7 | 0.978 | 0.357 | 16.1 |  |
| UNet | SDXL<br>-Turbo<br>(4 Steps) | FP16 | – | 24.3 | 0.845 | – | – | 24.7 | 0.705 | – | – |
| INT W8A8 | MixDQ | 24.1 | 0.834 | 0.147 | 21.7 | 25.0 | 0.690 | 0.157 | 21.6 |  |  |
| INT W8A8 | Ours | 24.3 | 0.845 | 0.100 | 24.0 | 24.8 | 0.701 | 0.110 | 23.7 |  |  |
| INT W4A8 | MixDQ | 27.7 | 0.708 | 0.402 | 15.7 | 25.9 | 0.610 | 0.415 | 15.7 |  |  |
| INT W4A4 | MixDQ | 353 | -2.26 | 0.685 | 11.0 | 373 | -2.28 | 0.686 | 11.3 |  |  |
| INT W4A4 | Ours | 24.5 | 0.816 | 0.265 | 17.9 | 25.7 | 0.667 | 0.278 | 17.8 |  |  |
| FP W4A4 | Ours | 24.1 | 0.822 | 0.250 | 18.5 | 24.7 | 0.699 | 0.261 | 18.4 |  |  |
| SDXL<br>(30 Steps) | FP16 | – | 16.6 | 0.729 | – | – | 22.5 | 0.573 | – | – |  |
| INT W8A8 | TensorRT | 20.2 | 0.591 | 0.247 | 22.0 | 25.4 | 0.453 | 0.265 | 21.7 |  |  |
| INT W8A8 | Ours | 16.6 | 0.718 | 0.119 | 26.4 | 22.4 | 0.574 | 0.129 | 25.9 |  |  |
| INT W4A4 | Ours | 20.7 | 0.609 | 0.298 | 20.6 | 26.3 | 0.494 | 0.314 | 20.4 |  |  |
| FP W4A4 | Ours | 19.0 | 0.607 | 0.294 | 21.0 | 25.4 | 0.480 | 0.312 | 20.7 |  |  |

Implementation details.
For the 8-bit setting, we use per-token dynamic activation quantization and per-channel weight quantization with a low-rank branch of rank 16. For the 4-bit setting, we adopt per-group symmetric quantization for both activations and weights, along with a low-rank branch of rank 32. INT4 quantization uses a group size of 64 with 16-bit scales. FP4 quantization uses a group size of 32 with FP8 scales (Rouhani et al., ). For FLUX.1 models, the inputs of linear layers in adaptive normalization are kept in 16 bits (*i.e*., W4A16). For other models, key and value projections in the cross-attention are retained at 16 bits since their latency only covers less than 5% of total runtime.

### 5.2 Results

Quality results.
We report the quantitative quality results in Table 1 across various models and precision levels, and show some corresponding 4-bit qualitative comparisons in Figure 7. Among all models, our 8-bit results can perfectly mirror the 16-bit results, achieving PSNR higher than 21, beating all other 8-bit baselines. On FLUX.1-dev, our INT8 PSNR even reaches 27 on MJHQ.

For 4-bit quantization, on FLUX.1, our SVDQuant surpasses the NF4 W4A16 baseline regarding Image Reward. On the schnell variant, our Image Reward even exceeds that of the original BF16 model, suggesting stronger human preference. On PixArt-$\Sigma$, while our INT4 Image Reward shows slight degradation, our FP4 model achieves an even higher score than the FP16 model. This is likely due to PixArt-$\Sigma$’s small model size (600M parameters), which is already highly compact and benefits from a smaller group size. Remarkably, both our INT4 and FP4 results consistently outperform ViDiT-Q’s(^∥^∥∥Our FP16 PixArt-$\Sigma$ model is slightly different from ViDiT’s, though both offer the same quality. For fair comparisons, ViDiT-Q’s similarity results are calculated using their FP16 results.) W4A8 results by a large margin across all metrics. For UNet-based models, on SDXL-Turbo, our 4-bit models significantly outperform MixDQ’s W4A8 results, and our FID scores are on par with the FP16 models, indicating no loss in performance. On SDXL, both our INT4 and FP4 results achieve comparable quality to TensorRT’s W8A8 performance, which represents the 8-bit SoTA. As shown in Figure 15 in the Appendix, our visual quality only shows minor degradation.

Figure: Figure 7: Qualitative visual results on MJHQ. Image Reward is calculated over the entire dataset. On FLUX.1 models, our 4-bit models outperform the NF4 W4A16 baselines, demonstrating superior text alignment and closer similarity to the 16-bit models. For instance, NF4 misinterprets “dinosaur style,” generating a real dinosaur. On PixArt-$\Sigma$ and SDXL-Turbo, our 4-bit results demonstrate noticeably better visual quality than ViDiT-Q’s and MixDQ’s W4A8 results.
Refer to caption: x7.png

Figure: Figure 8: SVDQuant reduces the model size of the 12B FLUX.1 by 3.6×. Additionally, our engine, Nunchaku, further cuts memory usage of the 16-bit model by 3.5× and delivers 3.0× speedups over the NF4 W4A16 baseline on both the desktop and laptop NVIDIA RTX 4090 GPUs. Remarkably, on laptop 4090, it achieves in total 10.1× speedup by eliminating CPU offloading.
Refer to caption: x8.png

Figure: Figure 9: Our INT4 model seamlessly integrates with off-the-shelf LoRAs without requiring requantization. When applying LoRAs, it matches the image quality of the original 16-bit FLUX.1-dev. See Appendix C for the text prompts.
Refer to caption: x9.png

Figure: Figure 10: Ablation study of SVDQuant on PixArt-$\Sigma$. The rank of the low-rank branch is 64. Image Reward is measured over 1K samples from MJHQ. Our results significantly outperform the others, achieving the highest image quality by a wide margin.
Refer to caption: x10.png

Memory save & speedup.
In Figure 8, we report measured model size, memory savings, and speedup for FLUX.1. Our INT4 quantization reduces the original transformer size from 22.2 GiB to 6.1 GiB, including a 0.3 GiB overhead due to the low-rank branch, resulting in an overall 3.6× reduction. Since both weights and activations are quantized, compared to the NF4 weight-only-quantized variant, our inference engine Nunchaku even saves more memory footprint, and offers 3.0× speedup on both desktop- and laptop-level NVIDIA RTX 4090 GPUs. Notably, while the original BF16 model requires per-layer CPU offloading on the 16GB laptop 4090, our INT4 model fits entirely in GPU memory, resulting in a 10.1× speedup by avoiding offloading. We anticipate even greater speedups for FP4-quantized models on NVIDIA’s next-generation Blackwell GPUs, as they inherently support microscaling for group quantization without the need for specialized GPU kernels.

Integrate with LoRA. Previous quantization methods require fusing the LoRA branches and re-quantizing the model when integrating LoRAs. In contrast, our Nunchaku eliminates redundant memory access, allowing adding a separate LoRA branch. In practice, we can fuse the LoRA branch into our low-rank branch by slightly increasing the rank, further enhancing efficiency. In Figure 9, we exhibit some visual examples of applying LoRAs of five different styles ([Realism](https://huggingface.co/XLabs-AI/flux-RealismLora), [Ghibsky Illustration](https://huggingface.co/aleksa-codes/flux-ghibsky-illustration), [Anime](https://huggingface.co/alvdansen/sonny-anime-fixed), [Children Sketch](https://huggingface.co/Shakker-Labs/FLUX.1-dev-LoRA-Children-Simple-Sketch), and [Yarn Art](https://huggingface.co/linoyts/yarn_art_Flux_LoRA)) to our INT4 FLUX.1-dev model. Our INT4 model successfully adapts to each style while preserving the image quality of the 16-bit version. For more visual examples, see Appendix B.2.

Ablation study.
In Figure 10, we present several ablation studies of SVDQuant on PixArt-$\Sigma$. First, both SVD-only and naïve quantization perform poorly in the 4-bit setting, resulting in a severe degradation of image quality. While applying smoothing to the quantization slightly improves image quality compared to naïve quantization, the overall results remain unsatisfactory. LoRC (Yao et al., ) introduces a low-rank branch to compensate for quantization errors, but this approach is suboptimal. Quantization errors exhibit a smooth singular value distribution. Consequently, low-rank compensation fails to effectively mitigate these errors, as discussed in Section 4.2. In contrast, we first decompose the weights and quantize only the residual. As demonstrated in Figure 5, the first several singular values are significantly larger than the rest, allowing us to shift them to the low-rank branch, which effectively reduces weight magnitude. Finally, smoothing consolidate the outliers, further enabling the low-rank branch to absorb outliers from the activations and substantially improving image quality.

Trade-off of increasing rank. Figure 11 presents the results of different rank $r$ in SVDQuant on PixArt-$\Sigma$. Increasing the rank from 16 to 64 significantly enhances image quality but increases parameter and latency overhead. In our experiments, we select a rank of 32, which offers a decent quality with minor overhead.

Figure: Figure 11: Increasing the rank $r$ of the low-rank branch in SVDQuant can enhance image quality, but it also leads to higher parameter and latency overhead.
Refer to caption: x11.png

## 6 Conclusion & Discussion

In this work, we introduce a novel 4-bit post-training quantization paradigm, SVDQuant, for diffusion models. It adopts a low-rank branch to absorb the outliers in both the weights and activations, easing the process of quantization. Our inference engine Nunchaku further fuses the low-rank and low-bit branch kernels, reducing memory usage and cutting off redundant data movement overhead. Extensive experiments demonstrate that SVDQuant preserves image quality. Nunchaku further achieves a 3.5× reduction in memory usage over the original 16-bit model and 3.0× speedup over the weight-only quantization on an NVIDIA RTX-4090 laptop. This advancement enables the efficient deployment of large-scale diffusion models on edge devices, unlocking broader potential for interactive AI applications.

Limitations. In this work, we do not report the speedups for our FP4 models. This is because we have no access to Blackwell GPUs, which natively support the precision and microscaling for group quantization. On Blackwell hardware, we anticipate greater speedups compared to our INT4 results on 4090 GPUs.

## Acknowledgments

We thank MIT-IBM Watson AI Lab, MIT and Amazon Science Hub, MIT AI Hardware Program, National Science Foundation, Packard Foundation, Dell, LG, Hyundai, and Samsung for supporting this research. We thank NVIDIA for donating the DGX server.

## Appendix A Missing Proofs

### A.1 Proof of Proposition 4.1

###### Proof.

$$ $\displaystyle\left\lVert{\bm{X}}{\bm{W}}-Q({\bm{X}})Q({\bm{W}})\right\rVert_{F}$ $\displaystyle=$ $\displaystyle\left\lVert{\bm{X}}{\bm{W}}-{\bm{X}}Q({\bm{W}})+{\bm{X}}Q({\bm{W} })-Q({\bm{X}})Q({\bm{W}})\right\rVert_{F}$ $\displaystyle\leq$ $\displaystyle\left\lVert{\bm{X}}({\bm{W}}-Q({\bm{W}}))\right\rVert_{F}+\left \lVert({\bm{X}}-Q({\bm{X}}))Q({\bm{W}})\right\rVert_{F}$ $\displaystyle\leq$ $\displaystyle\left\lVert{\bm{X}}\right\rVert_{F}\left\lVert{\bm{W}}-Q({\bm{W}} )\right\rVert_{F}+\left\lVert{\bm{X}}-Q({\bm{X}})\right\rVert_{F}\left\lVert Q ({\bm{W}})\right\rVert_{F}$ $\displaystyle\leq$ $\displaystyle\left\lVert{\bm{X}}\right\rVert_{F}\left\lVert{\bm{W}}-Q({\bm{W}} )\right\rVert_{F}+\left\lVert{\bm{X}}-Q({\bm{X}})\right\rVert_{F}\left\lVert{ \bm{W}}-({\bm{W}}-Q({\bm{W}}))\right\rVert_{F}$ $\displaystyle\leq$ $\displaystyle\left\lVert{\bm{X}}\right\rVert_{F}\left\lVert{\bm{W}}-Q({\bm{W}} )\right\rVert_{F}+\left\lVert{\bm{X}}-Q({\bm{X}})\right\rVert_{F}(\left\lVert{ \bm{W}}\right\rVert_{F}+\left\lVert{\bm{W}}-Q({\bm{W}})\right\rVert_{F}).$ $$

∎

###### Proof.

### A.2 Proof of Proposition 4.2

###### Proof.

$$ $\displaystyle\left\lVert{\bm{R}}-Q({\bm{R}})\right\rVert_{F}$ $\displaystyle=$ $\displaystyle\left\lVert{\bm{R}}-s_{\bm{R}}\cdot{\bm{Q}}_{\bm{R}}\right\rVert_ {F}$ $\displaystyle=$ $\displaystyle\left\lVert s_{\bm{R}}\cdot\frac{{\bm{R}}}{s_{{\bm{R}}}}-s_{\bm{R }}\cdot\text{round}\left(\frac{{\bm{R}}}{s_{{\bm{R}}}}\right)\right\rVert_{F}$ $\displaystyle=$ $\displaystyle|s_{\bm{R}}|\left\lVert\frac{{\bm{R}}}{s_{{\bm{R}}}}-\text{round} \left(\frac{{\bm{R}}}{s_{{\bm{R}}}}\right)\right\rVert_{F}.$ $$

So,

$$ $\displaystyle\mathbb{E}\left[\left\lVert{\bm{R}}-Q({\bm{R}})\right\rVert_{F}\right]$ $\displaystyle\leq$ $\displaystyle\mathbb{E}\left[|s_{\bm{R}}|\right]\sqrt{\text{size}({\bm{R}})}$ $\displaystyle=$ $\displaystyle\frac{\sqrt{\text{size}({\bm{R}})}}{q_{\max}}\cdot\mathbb{E}\left [\max(|{\bm{R}}|)\right]$ $\displaystyle\leq$ $\displaystyle\frac{\sqrt{\text{size}({\bm{R}})}}{q_{\max}}\cdot\sigma\sqrt{2 \log\left(\text{size}({\bm{R}})\right)},$ (8) $$

where $\sigma$ is the std deviation of the normal distribution. Equation 8 comes from the maximal inequality of Gaussian variables (Lemma 2.3 in Massart ()).

On the other hand,

$$ $\displaystyle\mathbb{E}\left[\left\lVert{\bm{R}}\right\rVert_{F}\right]$ $\displaystyle=$ $\displaystyle\mathbb{E}\left[\sqrt{\sum_{x\in{\bm{R}}}x^{2}}\right]$ $\displaystyle\geq$ $\displaystyle\mathbb{E}\left[\frac{\sum_{x\in{\bm{R}}}|x|}{\sqrt{\text{size}({ \bm{R}})}}\right]$ (9) $\displaystyle=$ $\displaystyle\sigma\sqrt{\frac{2\text{size}({\bm{R}})}{\pi}},$ (10) $$

where Equation 9 comes from Cauchy-Schwartz inequality and Equation 10 comes from the expectation of half-normal distribution.

Together, we have

$$ $\displaystyle\mathbb{E}\left[\left\lVert{\bm{R}}-Q({\bm{R}})\right\rVert_{F}\right]$ $\displaystyle\leq$ $\displaystyle\frac{\sqrt{\text{size}({\bm{R}})}}{q_{\max}}\cdot\sigma\sqrt{2 \log\left(\text{size}({\bm{R}})\right)}$ $\displaystyle\leq$ $\displaystyle\frac{\sqrt{\log\left(\text{size}({\bm{R}})\right)\pi}}{q_{\max}} \mathbb{E}\left[\left\lVert{\bm{R}}\right\rVert_{F}\right].$ $$

∎

###### Proof.

## Appendix B Additional Results

### B.1 Quality Results

We report extra quantitative quality results with additional metrics in Table 2. Specifically, CLIP IQA (Wang et al., ) and CLIP Score (Hessel et al., ) assesses the image quality and text-image alignment with CLIP (Radford et al., ), respectively. Structural Similarity Index Measure (SSIM) is used to measure the luminance, contrast, and structure similarity of images produced by our 4-bit model against the original 16-bit model. We also visualize more qualitative comparsions in Figures 12, 13, 14, 15 and 16.

**Table 2: Additional quantitative quality comparisons across different models. C.IQA means CLIP IQA, and C.SCR means CLIP Score.**
|  |  |  |  | MJHQ | sDCI |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Backbone | Model | Precision | Method | Quality | Similarity | Quality | Similarity |  |  |
|  |  |  |  | C.IQA ($\uparrow$) | C.SCR ($\uparrow$) | SSIM( $\uparrow$) | C.IQA ($\uparrow$) | C.SCR ($\uparrow$) | SSIM ($\uparrow$) |
|  | FLUX.1<br>-dev<br>(50 Steps) | BF16 | – | 0.952 | 26.0 | – | 0.955 | 25.4 | – |
|  | INT W8A8 | Ours | 0.953 | 26.0 | 0.748 | 0.955 | 25.4 | 0.697 |  |
|  | W4A16 | NF4 | 0.947 | 25.8 | 0.748 | 0.951 | 25.4 | 0.697 |  |
|  | INT W4A4 | Ours | 0.950 | 25.8 | 0.773 | 0.953 | 25.3 | 0.721 |  |
|  | FP W4A4 | Ours | 0.950 | 25.8 | 0.780 | 0.952 | 25.3 | 0.727 |  |
|  | FLUX.1<br>-schnell<br>(4 Steps) | BF16 | – | 0.938 | 26.6 | – | 0.932 | 26.2 | – |
|  | INT W8A8 | Ours | 0.938 | 26.6 | 0.844 | 0.932 | 26.2 | 0.811 |  |
| DiT | W4A16 | NF4 | 0.941 | 26.6 | 0.713 | 0.933 | 26.2 | 0.674 |  |
|  | INT W4A4 | Ours | 0.939 | 26.5 | 0.693 | 0.932 | 26.2 | 0.647 |  |
|  | FP W4A4 | Ours | 0.938 | 26.5 | 0.703 | 0.933 | 26.2 | 0.667 |  |
|  | PixArt-$\Sigma$<br>(20 Steps) | FP16 | – | 0.944 | 26.8 | – | 0.966 | 26.1 | – |
|  | INT W8A8 | ViDiT-Q | 0.948 | 26.7 | 0.815 | 0.966 | 26.1 | 0.756 |  |
|  | INT W8A8 | Ours | 0.947 | 26.8 | 0.849 | 0.967 | 26.0 | 0.800 |  |
|  | INT W4A8 | ViDiT-Q | 0.912 | 25.7 | 0.356 | 0.917 | 25.4 | 0.295 |  |
|  | INT W4A4 | ViDiT-Q | 0.185 | 13.3 | 0.077 | 0.176 | 13.3 | 0.080 |  |
|  | INT W4A4 | Ours | 0.927 | 26.6 | 0.602 | 0.952 | 26.1 | 0.519 |  |
|  | FP W4A4 | Ours | 0.935 | 26.7 | 0.652 | 0.957 | 26.1 | 0.574 |  |
| UNet | SDXL<br>-Turbo<br>(4 Steps) | FP16 | – | 0.926 | 26.5 | – | 0.913 | 26.5 | – |
| INT W8A8 | MixDQ | 0.922 | 26.5 | 0.763 | 0.907 | 26.5 | 0.750 |  |  |
| INT W8A8 | Ours | 0.925 | 26.5 | 0.821 | 0.912 | 26.5 | 0.808 |  |  |
| INT W4A8 | MixDQ | 0.893 | 25.9 | 0.512 | 0.895 | 26.1 | 0.493 |  |  |
| INT W4A4 | MixDQ | 0.556 | 13.1 | 0.289 | 0.548 | 11.9 | 0.296 |  |  |
| INT W4A4 | Ours | 0.916 | 26.5 | 0.630 | 0.894 | 26.8 | 0.610 |  |  |
| FP W4A4 | Ours | 0.919 | 26.4 | 0.640 | 0.901 | 26.7 | 0.620 |  |  |
| SDXL<br>(30 Steps) | FP16 | – | 0.907 | 27.2 | – | 0.911 | 26.5 | – |  |
| INT W8A8 | TensorRT | 0.905 | 26.7 | 0.733 | 0.901 | 26.1 | 0.697 |  |  |
| INT W8A8 | Ours | 0.912 | 27.0 | 0.843 | 0.910 | 26.3 | 0.814 |  |  |
| INT W4A4 | Ours | 0.916 | 26.5 | 0.630 | 0.894 | 26.8 | 0.610 |  |  |
| FP W4A4 | Ours | 0.919 | 26.4 | 0.640 | 0.901 | 26.7 | 0.620 |  |  |

Figure: Figure 12: Qualitative visual results of FLUX.1-dev on MJHQ.
Refer to caption: x12.png

Figure: Figure 13: Qualitative visual results of FLUX.1-schnell on MJHQ.
Refer to caption: x13.png

Figure: Figure 14: Qualitative visual results of PixArt-$\Sigma$ on MJHQ.
Refer to caption: x14.png

Figure: Figure 15: Qualitative visual results of SDXL on MJHQ.
Refer to caption: x15.png

Figure: Figure 16: Qualitative visual results of SDXL-Turbo on MJHQ.
Refer to caption: x16.png

### B.2 LoRA Results

In Figure 17, we showcase more visual results of applying the aforementioned five community-contributed LoRAs of different styles ([Realism](https://huggingface.co/XLabs-AI/flux-RealismLora), [Ghibsky Illustration](https://huggingface.co/aleksa-codes/flux-ghibsky-illustration), [Anime](https://huggingface.co/alvdansen/sonny-anime-fixed), [Children Sketch](https://huggingface.co/Shakker-Labs/FLUX.1-dev-LoRA-Children-Simple-Sketch), and [Yarn Art](https://huggingface.co/linoyts/yarn_art_Flux_LoRA)) to our INT4 quantized models.

Figure: Figure 17: Additional LoRA results on FLUX.1-dev. When applying LoRAs, our INT4 model matches the image quality of the original BF16 model. See Appendix C for the detailed used text prompts.
Refer to caption: x17.png

## Appendix C Text Prompts

Below we provide the text prompts we use in Figure 9 (from left to right).

a man in armor with a beard and a swordGHIBSKY style, a fisherman casting a line into a peaceful village lake surrounded by quaint cottagesgirl, neck tuft, white hair, sheep horns, blue eyes, nm22 stylesketched style, A squirrel wearing glasses and reading a tiny book under an oak treea panda playing in the snow, yarn art styleThe text prompts we use in Figure 17 are (in the rasterizing order):

A male secret agent in a tuxedo, holding a gun, standing in front of a burning buildingA handsome man in a suit, 25 years old, cool, futuristicA knight in shining armor, standing in front of a castle under siegeA knight fighting a fire-breathing dragon in front of a medieval castle, flames and smokeA male wizard with a long white beard casting a lightning spell in the middle of a stormA young woman with long flowing hair, standing on a mountain peak at dawn, overlooking a misty valleyGHIBSKY style, a cat on a windowsill gazing out at a starry night sky and distant city lightsGHIBSKY style, a quiet garden at twilight, with blooming flowers and the soft glow of lanterns lighting up the pathGHIBSKY style, a serene mountain lake with crystal-clear water, surrounded by towering pine trees and rocky cliffsGHIBSKY style, an enchanted forest at night, with glowing mushrooms and fireflies lighting up the underbrushGHIBSKY style, a peaceful beach town with colorful houses lining the shore and a calm ocean stretching out into the horizonGHIBSKY style, a cozy living room with a view of a snow-covered forest, the fireplace crackling and a blanket draped over a comfy chaira dog wearing a wizard hat, nm22 anime stylea girl looking at the stars, nm22 anime stylea fish swimming in a pond, nm22 stylea giraffe with a long scarf, nm22 stylea bird sitting on a branch, nm22 minimalist stylea girl wearing a flower crown, nm22 stylesketched style, A garden full of colorful butterflies and blooming flowers with a gentle breeze blowingsketched style, A beach scene with kids building sandcastles and seagulls flying overheadsketched style, A hot air balloon drifting peacefully over a patchwork of fields and forests belowsketched style, A sunny meadow with a girl in a flowy dress chasing butterfliessketched style, A little boy dressed as a pirate, steering a toy ship on a small streamsketched style, A small boat floating on a peaceful lake, surrounded by trees and mountainsa hot air balloon flying over mountains, yarn art stylea cat chasing a butterfly, yarn art stylea squirrel collecting acorns, yarn art stylea wizard casting a spell, yarn art stylea jellyfish floating in the ocean, yarn art stylea sea turtle swimming through a coral reef, yarn art style

## Peer Reviews

confidence: 3
contribution: 2
initial_rating: 6
presentation: 3
questions: 1- Using outliers and low rank for quantizations is a very well-known technique. The authors need to have a section in related work and highlight the difference/novelty between the technique and all related work. Otherwise, the novelty seems incremental.
Maybe you can provide a table for all related work including the mentioned one here and show all similarities and differences with the proposed technique. 

[A] QNCD: Quantization Noise Correction for Diffusion Models

[B] Q-DiT: Accurate Post-Training Quantization for Diffusion Transformers


2- The choice of rank (e.g., rank 32) appears somewhat arbitrary, without adequate theoretical or empirical justification for why this setting was chosen over others.  Can the author put more comments about it? Maybe more ablation studies and more detailed experiments about it will give more insight to the reviewer. 

3- If a neural network exhibits skewed distributions that may not align well with the low-rank assumption used here (e.g., nongaussian behavior), how does it work? Please discuss the robustness of the method to different weight distributions.

4-With the advent of mixed-precision computation, some architectures might benefit from a mix of 4-, 8-, and. How would LoRunner perform in such configurations, and could it be adapted to handle this flexibility?

5-What are the challenges to extending the method below 4-bit? For example 2w2a or 4w2a or 2w4a?

6-I am interested to see a breakdown of LoRunner’s impact on speed and memory, perhaps by comparing SVDQuant with and without LoRunner.

7- Minor typos:
"Quantizes both the weights and activations" – quantizes both weights and activations.
“on NVIDIA RTX-4090 laptop" – "on an NVIDIA RTX-4090 laptop."
soundness: 3
strengths: 1-SVDQuant’s strategy to manage outliers using low-rank decomposition offers an improvement over standard smoothing techniques.

2-LoRunner effectively reduces memory access overhead and enhances performance, particularly for GPU inference, addressing practical deployment constraints.

3-Pushing boundaries for low-bit quantization of both weight and activation
summary: This paper presents SVDQuant quantizes weights and activations to 4 bits to accelerate inference of large-scale diffusion models. Through a low-rank decomposition branch, the authors introduce SVDQuant, a method for mitigating the impact of outliers on quantization.
They absorb outliers on weights/activations. To do so, they first migrate the outliers from activation to weight. Then they apply SVD to the updated weight, decomposing it into a low-rank branch and a residual.  Additionally, they incorporate an inference engine, LoRunner, which combines low-rank and low-bit kernels to maximize computation speed and minimize memory overhead. As demonstrated in empirical results for different large diffusion models, SVDQuant offers significant memory savings and performance improvements without affecting image quality significantly. This demonstrates that it is a promising approach for deploying diffusion models on consumer-grade hardware.
weaknesses: 1-The level of contribution is not great. Using outliers and low rank for quantizations is a very well-known technique. The authors need to have a section in related work and highlight the difference/novelty between the technique and all related work. Otherwise, the novelty seems incremental.

2- The paper lacks a theoretical analysis of why the low-rank decomposition approach can consistently outperform other outlier-handling techniques beyond empirical observations.

3- Code is not provided.

---

confidence: 3
contribution: 4
initial_rating: 8
presentation: 3
questions: Please see above.
soundness: 3
strengths: The paper is well-written and clear. Though the idea is generally intuitive, the observation of the low-rank outliers and the fusion from the low-rank branch to low-bit branch makes this work a strong submission. Experiments are conducted on the latest diffusion models, and the results are generally strong.
summary: This paper proposes a 4-bit PTQ method for diffusion models that absorbs the outliers between weights and activations using a low-rank branch. To augment this method, the paper also presents an efficient inference engine to avoid the redundant memory access of the activations. Experimental results on latest diffusion models validate the superior accuracy and memory efficiency of the proposed method.
weaknesses: 1. Can the authors include some results on sub 4-bit quantization with the proposed method? It would be good to know the limitations of the method.

2. The authors should include more comparisons in their experimental results. While they compared with MixDQ and ViDiT-Q, some prominent works have been left out, such as Q-Diffusion [1] and QUEST [2].

[1] https://arxiv.org/pdf/2302.04304

[2] https://arxiv.org/pdf/2402.03666v1

---

confidence: 4
contribution: 3
initial_rating: 8
presentation: 4
questions: 1. Could you explain the calculation method for the smoothing factor in more detail?
2. In the ablation study, what is meant by "SVD-only" and "naive quantization"? Does "SVD-only" indicate that no quantization is applied? And what is the setting for naive quantization?
soundness: 4
strengths: 1. The paper introduces a full-precision low-rank adapter and Singular Value Decomposition (SVD) to effectively compensate for quantization errors.
2. The authors have implemented a novel kernel that efficiently fuses low-rank and low-bit branches, minimizing computational overhead.
3. The extensive experiments conducted on state-of-the-art diffusion models, such as FLUX, provide strong evidence of the method's effectiveness and robustness.
summary: The paper introduces a post-training quantization method for diffusion models called SVDQuant that successfully quantizes both weights and activations to 4 bits without sacrificing visual quality. By integrating the low-rank branch kernels into the low-bit branch, SVDQuant minimizes overhead, significantly accelerating model inference. Tested on the 12B parameter FLUX.1-schnell model, it reduces memory usage by 3.6 times compared to the BF16 model and achieves a 3.6 times speedup over the NF4 W4A16 baseline on a laptop equipped with an RTX-4090 GPU.
weaknesses: 1. Adding a W8A8 baseline to the latency comparison would provide valuable insights and a clearer performance reference.
2. Since the authors use group quantization for weights and activations in the 4-bit setting, it would be beneficial to include methods that also use group quantization, such as Atom [1], as baselines.

References: 
[1] Zhao et al. "Atom: Low-bit Quantization for Efficient and Accurate LLM Serving" in MLSys'24

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: In the experiment part, this work almost evaluates on 4/8-bit quantization; what is the best trade-off between bit-width and performance for diffusion models? 

Also, adding more best practice of the diffusion model quantization will be fine.
soundness: 2
strengths: - It is an interesting topic to combine the low-bit quantization and low-rank decomposition, previous work including Loft-Q (ICLR 24), et al.

- This work proposes a kernel fusion implementation to speed up on-device inference.

- The method is reasoning and the writing is easy to follow.
summary: This work accelerate diffusion models by low-bit quantization and low-rank decomposition. The motivation of this work is the outlier quantization error in activations for recent PTQ methods. The authors propose a multi-branch architecture with both low-bit and low-rank operations. To speed up the inference, a kernel fusion implementation is also proposed for the two branches. In experiments, the DiT and UNet diffusion models are evaluated in 4/8-bit conditions.
weaknesses: - I don't think splitting low-rank and low-bit branches is a good idea to overcome quantization errors. Different from the QLLM (ICLR 24) et al.,  the computation of the two branches can not merge after training, leading to 5~10% overheads in the paper (line 314).

- Experiments parts are not very solid: the baseline quantization methods are kind-of weak. Recent works including SpinQuant [1], AffineQuant [2], et al. achieve much higher performance than the baseline NF4 in the paper and also without the additional branch.

[1] SpinQuant: LLM quantization with learned rotations
[2] AffineQuant: Affine Transformation Quantization for Large Language Models

---

confidence: 4
contribution: 3
initial_rating: 8
presentation: 4
questions: Please clarify how the diag lambda is determined
if possible provide an estimate of the speedup expected by the mixed kernel - possibly using mixed kernel ad higher precision and see how the speedup degrades from "pure" low precision kernel.
soundness: 3
strengths: The paper presents clearly the flow of ideas: statements are clear and detailed proofs are moved to the appendix, hence they do not distract from the flow f the key messages.
The motivation for the work is clearly described as well as the gaps in the SoA, and the claims of novelty. 
The reader is led step by step to the final solution: this flow helps understanding the technical motivations of the various steps. 
Experiments are clear and the KPI used are well defined.
summary: The paper proposes an aggressive quantization method (4-bit) for both activations and weight in diffusion models. To mitigate quantization-induced quality losses, the author propose to  migrate outlayers from activations to weights and then to add low-rank branches at high accuracy 16b (as opposed to the traditional smoothing approach which is not sufficient at 4-bit quantization level especially for diffusion models): i.e. the network is modified also for inference by the presence of the branches A baseline implementation of the low-rank branches in inference is slow and negates the speed advantages of the quantized computations, The authors then propose a remedy to this shortcoming which merges the branches, to regain speed (while also retaining memory reduction advantages)
weaknesses: It is not clear ho lambda for the migration of outlayers from activations and weights si computed (it is per-channel - so the array can be quite big). This is in my view a bit of a problem, because lamda migh be impacted by the dataset used for calibration (if it is decided offline) or may be hard to determine efficiently online. 

Inference time results are missing because the authors have no access to modern gpus (Blackwell) with native 4b support. This is a minor weakness, but it must be noted that it's not fully clear if the merged kernels will benefit from the same speedup claimed for pure 4b kernels. 

the key ideas are not novel per se, but the combination is interesting - the key intuition being the merged low-rank and low-precision kernel to recover speed

---

confidence: 3
contribution: 2
initial_rating: 6
presentation: 2
questions: 1. Some strong statements need to be properly supported by evidence:

- "Weight-only quantization *cannot* accelerate diffusion models" - For modern GPUs, maybe, but this lacks concrete evidence. Also, different hardware platforms have different bottlenecks. 

 - "Weights and activations *must* be quantized to the same bit width" - Can custom hardware support direct mixed precision operations?

 - "they primarily consider weight-only quantization..." For example, the authors have cited Q-diffusion and EfficientDM which quantize both activations and weights. This statement needs further justification. 

2. The authors mention that the "lower-precision side will be upcast during computation, negating potential performance boosts". However, as illustrated in Figure 6(b), the XL1L2 branch appears to retain 16-bit full precision before being combined with the quantized residual. This approach seems to be different from the authors' initial claim.

3. It would be very helpful if the authors could elaborate on the similarities and differences between their methods and LoRC (Yao et al.)
 
Minor:

- How is quantization level defined in Figure 3? And why can it take fractional numbers? I vaguely understand after smoothing, the peak of |X| drops from 10 to 2, but why the peak of |W| also drops? I thought smaller peaks means easier to quantize.

- QKV projection, presumably, is an LLM concept, and it may look abrupt in 4.3 without preliminary discussion.
soundness: 3
strengths: 1. I like the idea of translating the weight quantization into residual quantization to eliminate outliers in weights. 
2. The figures are well illustrated, and the math presentations are insightful. 
3. The model with a dinosaur head is hilarious.
summary: The authors propose SVDQuant to enable 4-bit quantization for diffusion models in terms of both weights and activations, with SVD approach utilized to enable quantization of residual (R=W-L1L2) rather than directly quantizing weight matrices. SVDQuant first use a smoothing technique proposed in previous work to transfer the outliers in activations to weights, then use SVD to enable 16-bit low-rank approximation of the weights and quantize the residual between the two weights, absorbing the weight outliers in the low rank branches L1 and L2.
weaknesses: 1. There are a few strong statements in this work with insufficient reasoning. 
2. The method section mainly focuses on justifying the minimization of quantization error but lacking discussion of the computation flow. 
3. The residual quantization approach looks similar to the quantization of error matrix in LoRC.

Meta-review: This paper proposed a novel 4-bit post-quantization paradigm called SVDQuant for diffusion models. It is well written. The experimental parts, a variety of latest diffusion models were evaluated with about 3.5x speedup , demonstrating the universal effectiveness of the proposed method. The main weakness is that the novelty (or contribution) of the key idea per se is not that new, but the combinations of them are indeed really interesting and useful, and important. I think it is a strong submission.

Decision: Accept (Spotlight)