SVDQuant: Absorbing Outliers by Low-Rank Component for 4-Bit Diffusion Models
Authors: Muyang Li
Yujun Lin
Zhekai Zhang
Tianle Cai
Xiuyu Li
Junxian Guo
Enze Xie
Chenlin Meng
Jun-Yan Zhu
Song Han
Venue: ICLR 2025 Spotlight (2025)
Keywords: Quantization
Diffusion Models
Efficiency
Acceleration
Primary area: generative models
TL;DR: 4-bit Post-Training Quantization for Diffusion Models

Abstract: Diffusion models can effectively generate high-quality images. However, as they scale,  rising memory demands and higher latency pose substantial deployment challenges. In this work, we aim to accelerate diffusion models by quantizing their weights and activations to 4 bits. At such an aggressive level, both weights and activations are highly sensitive, where existing post-training quantization methods like smoothing become insufficient. To overcome this limitation, we propose *SVDQuant*, a new 4-bit quantization paradigm. Different from smoothing, which redistributes outliers between weights and activations, our approach *absorbs* these outliers using a low-rank branch. We first consolidate the outliers by shifting them from activations to weights. Then, we use a high-precision, low-rank branch to take in the weight outliers with Singular Value Decomposition (SVD), while a low-bit quantized branch handles the residuals. This process eases the quantization on both sides. However, naively running the low-rank branch independently incurs significant overhead due to extra data movement of activations, negating the quantization speedup. To address this, we co-design an inference engine *Nunchaku* that fuses the kernels of the low-rank branch into those of the low-bit branch to cut off redundant memory access. It can also seamlessly support off-the-shelf low-rank adapters (LoRAs) without re-quantization. Extensive experiments on SDXL, PixArt-$\Sigma$, and FLUX.1 validate the effectiveness of SVDQuant in preserving image quality. We reduce the memory usage for the 12B FLUX.1 models by 3.5×, achieving 3.0× speedup over the 4-bit weight-only quantization (W4A16) baseline on the 16GB laptop 4090 GPU with INT4 precision. On the latest RTX 5090 desktop with Blackwell architecture, we achieve a 3.1× speedup compared to the W4A16 model using NVFP4 precision. Our quantization library and inference engine are available at https://github.com/mit-han-lab/deepcompressor/ and https://github.com/mit-han-lab/nunchaku/, correspondingly.

Published as a conference paper at ICLR 2025
SVDQUANT: ABSORBING OUTLIERS BY LOW-RANK COMPONENTS FOR 4-BIT DIFFUSION MODELS
Muyang Li
Yujun Lin'
Zhekai Zhang' If
Tianle Cai
Xiuyu Li
Junxian Guo
Enze Xie
Chenlin Meng'
Jun- Zhu
Song Han 1.2
MIIT NVIDDA ²CMU
*princeton UC Berkeley
SJTU
"Pika Labs
https://hanlat
wtt.laaa............ wwwp:teeedpooooooo.ooooo svdquant
FLUX.I-do BF16
NE4 (W4A16) LPIPS: 0 0.272
Naive INT4 (WAM4) LPIPS: 0.322
SVDQuant INT4 (W4MI LPISS: 0.223
Steps) DIT Memory; 22.7 GiB E2E Latency: 111.7
DIT Memory; GiB (3.3x Less)
DIT Memory GiB (3. Lese] E2E Latency; 12.5 a (8.9% Faster)
DIT Memory: GiB (3.5x Less) EIE Latency: 12.9 (8.7x Faster)
E2E Latency: 38.6 (2.9x Faster)
SYDGMMM LITTE AND COd FAST
SVDQuant is and fast
SVDQuant
SVDQuant lite and fast
and fast
Prompt irrforyan cot Inbleyy Ange dhat SWPDglast and fand" manving Sance prapohr Mlack Aushheraaeee.
PixAm-- FP16 Steps)
VIDIT-Q ViDiT-Q(WAAB) LPIPS:
Nuivve INT4 (W4A4) LPIPS:
SVDQuant INT4 (W4A4)
LPIPS:
Pronett madhas tenderlatt
Figure SVDQuant post-training quantization technique for weights and activations that well maintains visual fidelity. 12B FLUX. 1-dev, achieves eliminating CPU offloading, it offers 8.7x speedup over the 16-bit model when on a 16GB laptop 4090 GPU, 3x faster than the NF4 W4A 16 baseline. On PixArt-E, it demonstrates significantly superior visual quality over other W444 even W4A8 baselines. "E2E" means the end-to latency including the text encoder and VAE decoder.
memory reduction compared
BF16 model. By
ABSTRACT
Diffusion models can effectively generate high-quality images. However, as they scale, rising memory demands and higher latency pose substantial deployment challenges. this work, we aim to accelerate diffusion models by quantizing their weights and activations to 4 bits. At such an aggressive level, both weights and activations are highly sensitive, where existing post-training quantization methods like smoothing become insufficient. To overcome this limitation, we propose SVDQuant, new 4-bit quantization paradigm. Different from smoothing, which redistributes outliers between weights and activations, our approach absorbs these outliers using low-rank branch. We first consolidate the outliers by shifting them from activations to weights. Then, we use high- precisic,,,,, low-rank branch to take in the weight outliers with Singular Value Decomposition (SVD), while low-bit quantized branch handles the residuals. This process eases the quantization both sides. However, naïvely running the low-rank branch independently incurs significant overhead due to extra data movement of activations, negating the quantization speedup. To address this, we co-design an inference engine Nunchaku that fuses the kernels of the low-rank branch into those of the low-bit branch to cut
A Agorithm co-lead. System lead. Part of the work done during internship at NYIDIA.

---

Published as a conference paper at ICLR 2025
off redundant memory access. It can also seamlessly support off-the-shelf low-rank adapters (LoRAs) without re-quantization. Extensive experiments on SDXL, PixArt-E, and FLUX.1 validate the effectiveness of SVDQuant in preserving image quality. We reduce the memory usage for the 12B FLUX.1 models by 3.5x, achieving 3.0x speedup over the 4-bit weight-only quantization (W4A16) baseline on the 16GB laptop 4090 GPU with INT4 precision. On the latest RTX 5090 desktop with Blackwell architecture, we achieve 3.1x speedup compared to the W4A16 model using NVFP4 precision. Our quantization library" and inference engine' are open-sourced.
INTRODUCTION
Diffusion models have shown remarkable capabilities in generating high-quality images (Ho et al., 2020), with recent advances further enhancing user control over the generation process. Trained on vast data, these models can create stunning images from simple text prompts, unlocking diverse image editing and synthesis applications (Meng et al., 2022b; Ruiz et al., 2023; Zhang et al., 2023).
To pursue higher image quality and more precise text-to-image alignment, researchers are scaling up diffusion models. As shown Figure 2, Stable Diffusion (SD) (Rombach al., 2022) 1.4 only has 800M parameters, while SDXL (Podell et al., 2024) scales this up to 2.6B parameters. AuraFlow v0.1 (fal.ai, 2024) extends this further 6B parameters, with the latest model, FLUX.1 (Black-Foree 2024), pushing the boundary to 12B parameters. Compared to large language models (LLMs), diffusion models are significantly more computationally intensive. Their computational costs increase more rapidly with model size, posing a prohibitive memory and latency barrier for real-world model deployment, particularly for interactive use cases that demand low latency.
Diffesion Model L.M
48
Computation (TMAGS)
Ulaad 12
12
Parmmeters
Figure 2: Computation param- eters for LLMs and diffusion mod- els. L.MMs' computation is mea- sured with 512 context and 256 output tokens, and diffusion mod- els' computation is for single step. Dashed lines show trends.
As Moore's law slows down, hardware vendors are turning low-precision inference to sustain performance improvements. For instance, NVIDIA's Blackwell Tensor Cores introduce a new 4-bit floating point (FP4) precision, doubling the performance compared to FP8 (NVIDIA, 2024). Therefore, using 4-bit inference to accelerate diffusion models is appealing. In the realm of LLMs, researchers have leveraged quantization to compress model sizes and boost inference speed (Dettmers et al., 2022; Xiao et al., 2023). However, unlike Where latency is primarily constrained by loading model weights on modern GPUs, especially with small batch diffs-diffusin models are heavily computationally bounded, even with a single batch. As a result, weight-only quantization cannot accelerate diffusion models. To achieve speedup on these devices, both weights and activations must be quantized to the same bit width; otherwise, the wwwrisireiiioone weight will be upcast during computation, negating potential performance enhancements.
In this work, we focus on quantizing both the weights and activations of diffusion models bits. This challenging and aggressive scheme is often prone to severe quality degradation. Existing methods like smoothing (Xiao et al., 2023; Lin et al., 2024), which transfer the outliers between the weights and activations, are less effective since both sides are highly vulnerable to outliers. To address this issue, we propose a to use low-cost branch to absorb outliers on both sides. To achieve this, as illustrated in Figure 3, we first aggregate the outliers by migrating them from activation X to weight W via smoothing. Then we apply Singular Value Decomposition (SVD) to the updated weight, W, decomposing it into a low-rank branch L, L2 and a residual W - L, L2. The low-rank branch operates at 16 bits, allowing us to quantize only the residual to 4 bits, significantly reducing outliers and magnitude. However, naively running the low-rank branch separately incurs substantial memory access overhead,
quantization paradigm. SVDQuant. Our core idea is
Quantization library: github.com/nnee sattuu..oo/miithhnn.ooe hamm://iihaan-bbbddeoooopsssssss
'Inference Engine: wwwphh.oo/mm-taa--aabbiaaa Mcasured by the number of Multiply-Accmmulat operations (MACs). 1 FLOPS.
2

---

Published as a conference paper at ICLR 2025
Musie dhiteddy with Stroehing
Murie Uhilindly with SVD
Outiaz
[X]
[W]
W]
Rick
16-8it E,
Loy Trleeto Bax
malk-122
1e-Bie
Cheed
Daand! Hard to Quoeitze
Chand
Chazd
Clardd Eayy tn Quention
Chand
Very Bard to Quettoe
Eacy to Quentie
Ilaneer to Quenttr
Tass to Quantine
No Need tu Quantine
(0) Ortginal
(b) Stift Outliers from Autivattin X Weight W
(o SV
Figure 3: Overview of SVDQuant. (a) Originally, both the activation X and weight W contain outliers, making 4-bit quantization challenging. (b) We migrate the outliers from the activation to weight, resulting the updated activation X and weight W. While X becomes casier quantize, W becomes more difficult. (c) SVDQuant further decomposes W into a low-rank component L1 L and residual W L1 Le with SVD. Thus, the quantization difficulty is alleviated by the low-rank branch, which runs at 16-bit precision.
offsetting the speedup of 4-bit inference. To overcome this, we co-design a specialized inference engine Nunchaku, which fuses the low-rank branch computation into the 4-bit quantization and computation kernels. This design enables us to achieve measured inference speedup even with additional branches.
SVDQuant can quantize various text-to-image diffusion architectures into bits, including both UNet (Ho et al., 2020; Ronneberger et al., 2015) and DiT (Peebles & Xie, 2023) backbones, while maintaining visual quality. supports both INT4 and FP4 data types and integrates seamlessly with pre-trained low-rank adapters (LoRA) (Hsu et al., 2022) without requiring re-quantization. To our knowledge, we are the first to successfully apply 4-bit post-training quantization to both the weights and activations of diffusion models, and achieve measured speedup NVIDIA GPUs. On the latest 12B FLUX.1, our b-t models largely preserve the image quality and reduce the memory footprint of the original BF16 model by 3.5x. Furthermore, 3. Ix speedup over the NF4 weight-only quantized baseline on the 1 16GB laptop-level RTX 4090 and desktop-level RTX 5090 GPU, respectively. See Figure for visual examples.
INT4 and FP4 model delivers a 3.0x and
RELATED WORK
Diffusion models (Sohl-Dickstein al., 2015; Ho al., 2020) have emerged as a powerful class of generative models, known for generating high-quality samples by modeling the data distribution through an iterative denoising process. Recent advancements in text-to-image diffusion models (Bal- aji et al., 2022; Rombach et al., 2022; Podell et al., 2024) have already revolutionized content generation. Researchers further shifted from convolution-base
UNet architectures (Ronneberger et al., 2015; Ho et al., 2020) to transformers (Peebles & Xie, 2023; Bao et al., 2023) and scaled up the model size (Esser et al., 2024). However, diffusion models suffer from extremely slow inference speed due to their long denoising sequences and intense computation. To address this, various approaches have been proposed, including few-step samplers (Zhang & Chen, 2022; Zhang et al., 2022; Lu et al., 2022) distilling fewer-step models from pre-trained ones (Salimans & Ho, 2021; Meng et al., 2022a; Song et al., 2023; Luo et al., 2023; Sauer et al., 2024; Yin et al., 2024b;a; kan, 2024). Another line of works choose optimize or accelerate computation via efficient architecture design (Li et al., 2023b; 2020; Cai et al., 2024; Liu et al., 2024a), quantization (Shang et al., 2023; Li et al., 2023a), sparse inference (Li et al., 2022; Ma et al., 2024c;b), and distributed inference (Li et al., 2024b; Wang et al., 2024c; Chen et al., 2024b). This work focuses on quantizing the diffusion models to 4 bits to reduce the computation complexity. Our method can also applied to few-step diffusion models to further reduce the latency (see Section 5.2).
Quantization has been recognized as an effective approach for LLMS to reduce the model size and accelerate inference (Dettmers et al., 2022; Frantar et al., 2023; Xiao et al., 2023; Lin al., 2025; 2024; Kim et al., 2024; Zhao et al., 2024d). For diffusion models, Q-Diffusion (Li et al., 2023a) and PTQ4DM (Shang et al., 2023) first achieved 8-bit quantization. Subsequent works refined these techniques with approaches like sensitivity analysis (Yang et al., 2023) and timestep-aware quantization (He et al., 2023; Huang et al., 2024; Liu et al., 2024b; Wang et al., 2024a). Some recent works extended the setting to text-to-image models (Tang et al., 2024; Zhao et al., 2024c), DiT backbones (Wu et al., 2024), quantization aware training (He et al., 2024; Zheng et al., 2024; Wang et al., 2024b; Sui et al., 2024), video generation (Zhao et al., 2024b), and different data types (Liu & Zhang, 2024). Among these works, only MixDQ (Zhao et al., 2024c) and ViDiT-Q (Zhao et al., 2024b) implement low-bit inference engines and report measured 8-bit speedup on GPUs. In this
3

---

Published as a conference paper at ICLR 2025
work, we push the boundary further by quantizing diffusion models to 4 bits, supporting both the integer or floating-por -oint data types, compatible with the UNet backbone (Ho et al., 2020) and recent DiT architecture (Peebles & Xie, 2023). Our co-designed inference engine, Nunchaku, further ensures on-hardware speedup. Additionally, when applying LoRA to the model, existing methods require fusing the LoRA branch the main branch and re-quantizing the model avoid tremendous memory- access overhead in the LoRA branch. Nunchaku cuts off this overhead via kernel fusion, allowing the low-rank branch run efficiently as a separate branch, eliminating the need for re-quantization.
Low-rank decomposition has gained significant attention in deep learning for enhancing computa- tional and memory efficiency (Hu et al., 2022; Zhao et al., 2024a; Jaiswal et al., 2024). While directly applying this approach to model weights can reduce the compute and memory demands (Hsu et al., 2022; Yuan et al., 2023; Li et al., 2023c), it often leads to performance degradation. Instead, Yao et al. (2024) combined it with quantization for model compression, employing low-rank branch com- pensate for the quantization error. Low-Rank Adaptation (LoRA) (Hu et al., 2022) introduces another active line of research using low-rank matrices adjust a subset of pre-trained weights for efficient fine-tuning. This has sparked numerous advancements (Dettmers et al., 2023; Guo et al., 2024; Li et al., 2024c; Xu al., 2024b; Meng al., 2024), which combines quantized models with low-rank adapters to reduce memory usage during model fine-tuning. However, our work differs in two major aspects. First, our goal is different, as we aim to accelerate model inference through quantization, while previ- works focus model compression only quantization, resulting in no speedup. Second, as shown in our experiment (Figure and ablation study in Section 5.2), directly applying these methods not only degrades the image quality, but also in- troduces significant overhead. In contrast, our method yields much better performance due to our joint quantization of weights and activations and overhead reduction of our inference engine Nunchaku.
efficient fine-tuning. Thus, they primarily consider weight-
QUANTIZATION PRELIMINARY
Quantization is effective approach accelerate linear layers in networks. Given tensor X, the quantization process is defined as:
max([X])
Qx x = round
8x
Jmax
Here, Qx is the low-bit representation of X, 8x is the scaling factor, and Gmax is the maximum quantized value. For signed k-bit integer quantization, qmax = 2 1. For 4-bit floating-pos -pintt quantization with 1-bit mantissa and 2-bit exponent, Imax = 6. Thus, the dequantized tensor can be formulated as Q(X) 8x Qx. For linear layer with input X and weight W, its computation can be approximated by
XW Q())Q(W) = -x QxQw
The same approximation applies to convolutional layers. To speed up computation, modern commer- cial GPUs require both Qx and Qw using the same bit width. Otherwise, the low-bit weights need be upcast to match the higher bit width of activations, or vice versa, negating the speed advantage. Following the notation in QServe (Lin et al., 2025), we denote z-bit weight, y- bit activation as Wz "INT" and "FP" refer to the integer and floating-port point data types, respectively.
In this work, we focus on W4A4 quantization for acceleration, where outliers in both weights and activations place substantial obstacles. Traditional methods to suppress these outliers include quantization- aware training (QAT) (He et al., 2024) and rotation (Ashkboos et al., 2024; Liu et al., 2024c; Lin et al., 2025). QAT requires massive computing resources, especially for tuning models with more than 10B parameters such 1 Rotation is inapplicable due to the usage of adaptive normalization layers (Peebles & Xie, 2023) in diffusion models. The runtime- generated normalization weights preclude the offline rotation with the weights of projection layers, while online rotation of both activations and weights incurs significant runtime overhead.
4 METHOD
In this section, we first formulate our problem and discuss where the quantization error comes from. Next, present SVDQuant, new W4A4 quantization paradigm for diffusion models. Our key
4

---

Published as a conference paper at ICLR 2025
Oudlies
Original
201
Ouliiens
After Smoothing
More Outliers
After SVD
Input Activation Value
Inpii Tigittii Adt Eivatio Valii
Wolgnn Value
Weigin Valuee
Magin Nalle
bonk
8
192 70
76c
Activation Group Index
Weight Group Index (5)
Imput Activatioo Group Index ding(A)
Weight Group Index
Weight Group Indes
|X]
[X]
(d)
dingtA))
(0) |R| L,4,1
Figure 4: Example value distribution of inputs and weights in PixArt-E (Chen et al., 2024a) À the smooth factor. Red indicates the outliers. Initially, both the input X and weight W contain sigmificant outliers. After smoothing, the range of X is reduced with much fewer outliers, while W shows more outliers. Once the SVD low-rank branch L2 L2 is subtracted, the residual R has narrower range and is free from outliers.
idea is to introduce an additional low-rank branch that can absorb quantization difficulties in both weights activations. Finally, we provide a co-designed inference engine Nunchaku with kernel fusion to minimize the overhead of the low-rank branches in the 4-bit model.
4.1 PROBLEM FORMULATION
Consider linear layer with input X E Rx and weight W Rm where b represents the batch size, and m and n denote the input and output channels, respectively. The quantization error can be defined as
E(X, W)
|X - QQ(X)Q(W)Il.
where
||e denotes Frobenius Norm.
Proposition 4.1 (Error decomposition). The quantization error can be decomposed as follows:
E(X, W) < |X] W - Q(W)l + X - Q( ( + W - Q(W)ll))
(4)
See Appendix A. for the proof. From the proposition, we can see that the error is bounded by four elements the magnitude of the weight and input, || and |X|] ll and their respective quantization errors, (W-Q(W)lll and X - Q(X)ll To minimize the overall quantization error, we aim optimize these four terms.
4.2 SVDQUANT: ABSORBING OUTLIERS VIA LOW-RANK BRANCH
Migrate outliers from activation to weight. Smoothing (Xiao et al., 2023; Lin et al., 2024) is an effective approach for reducing outliers. We can smooth outliers in activations by scaling down the input X and adjusting the weight matrix W correspondingly using a per-channel smoothing factor A R As shown in Figure 4(a)c).. the smoothed input X X diag (A) exhibits reduced magnitude and fewer outliers, resulting in lower input quantization error. However, in Figure (())
the transformed weight W = W diag(A) has a significant increase both magnitude and the presence of outliers, which in turn raises the weight quantization error. Consequently, the overall error reduction is limited.
Absorb magnified weight outliers with a low-rank branch. Our core insight is to introduce 16-bit low-rank branch to further migrate the weight quantization diffieulty. Specifically, decompose the transformed weight W- L, L2 + R, where L, E mm and L2 Rrxn low-rank factors of rank r, and R is the residual. Then XW can be approximated as
XW XW XL XR
XL
Q(X)Q(R)
(5)
4-bit residual
Compared to direct 4-bit quantization, i.c., Q())) our method first computes the low-rank branch L L2 in 16-bit precision, and then approximates the residual X R with 4-bit quantization. Empirically, r & min(m, n), and is typically set to 16 or 32. As a result, the additional parameters and computation for the low-rank branch are negligible, contributing only mrter to the overall costs. mn However, it still requires careful system design to eliminate redundant memory access, which we will discuss in Section 4.3.
5

---

Published as a conference paper at ICLR 2025
500
Lalency (us)
400
Down Proji
kt,
Up Proj.
Down Proj.
KL,4
300
300
Up Proj. 4-bit Compate Quamtize
200
Quantize
E-ltt Compute Q
Fused
Fused Kernel
Naive
Nunchaku (Ours)
Shared Input
Shared Output
(a) Latency Breakdown QKV projection
Nunchaku Kernel Fusion
Figure 6: (a) Naively running low-rank branch with rank 32 will introduce 57% lateney overhead due to extra read of 16-bit inputs in Down Projection and extra write of 16-bit outputs in Up Projection. Our Nunchaku engine optimizes this overhead with kernel fusion. (b) Down Projection and Quantize kernels use the same input, while Up Projection and 4-Bit Compute kernels share the same output. To reduce data movement overhead, fuse the first two and the latter two kernels together.
From Equation 5, the quantization error can be expressed as
XW (XL, + F XR Q())Q(R) FX R):
where R W - L2 According to Proposition 4.1, since X is already free from outliers, only need to focus on optimizing the magnitude of R, || and its quantization error, || - (R) p- Proposition 4.2 (Quantization error bound). For any tensor and quantization method described in Equation I QRR) SR QR. Assuming the elements of R. follow distribution that satisfies the following regularity condition: There exists a constant e such that
E [max [maa( cc [[]]])
Then, we have
sizel R.
E 11 R Q(R))))
E
R
(8)
9max
where size( denotes the number of elements in R. Especially if the elements of R follow a normal distribution, Equation holds for c [logaaize(B) ()))
(ize(R)
400
Extremely High
W
See Appendix A.2 for the proof. From this proposition, we obtain the intuition that the quantization error
R - Q(R)l is bounded by the magnitude the residual p Thus, our goal is find the
100
Singular Values
Low Singular
optimal 1 Ly that minimizes be solved by Singular Value Decomposition (SVD) (Eckart & Young, 1936; Mirsky, 1960). Given the SVD of W UEV, the optimal solution is L1 = UΣ and L2 = V.rg- Figure 5 illustrates the singular value distribution of the original weight W, transformed weight W and residual R. The singular values of the original weight W are highly imbalanced. After smoothing, the singular value distribution of W becomes even sharper, with only the first several values being significantly larger. By removing these dominant values, the magnitude of the residual R is dramatically reduced, as [| F
R
w L2 which can
Values
10
0
16
32
48
64
Figure 5: First 64 singular values af W, W, R. The first 32 singular values of W exhibit a steep drop. while the remaining values much more gradual.
min((m.n) o, compared to the original magnitude
[w]]
Σ D min(m,.)) o?, where o, is the
F
i-th singular value of W. Furthermore, Figure 4(d)(e) show that R exhibits fewer outliers with substantially compressed value range compared to W. In practice, we further reduce quantization errors by iteratively updating the low-rank branch through decomposing W - Q(R) and adjusting R accordingly for several iterations, and then picking the result with the smallest error.
4.3 NUNCHAKU: FUSING LOW-RANK AND Low-Bi BIT BRANCH KERNELS
Although the low-rank branch introduces negligible computation in theory, running it separate branch incurs large latency overheal-aaproximmeee approximately 50% of the 4-bit branch latency, as shown in Figure 6(a). This occurs because, for small rank r, even though the computational cost decreases

---

Published as a conference paper at ICLR 2025
significantly, the input and output activation sizes remain unchanged, shifting the bottleneck from computation to memory access. This issue worsens, especially when the activation cannot fit into the GPÚ L2 cache. For example, in the diffusion transformer block, the up projection in the low-rank branch for QKV projection is much slower since its output exceeds the available 1.2 cache, resulting in the extra DRAM load and store operations. Fortunately, the down projection L1 in the low-rank branch shares the same input as the quantization kernel in the low-bit branch, while the up projection L2 shares the same output as the 4-bit computation kernel, as illustrated in Figure 6(b). By fusing the down projection with the quantization kernel and the up projection with the 4-bit computation kernel, the low-rank branch can share the activations with the low-bit branch, eliminating the extra memory access and halving the number of kernel calls. As a result, our low-rank branch adds only 5~10 latency, making nearly cost-free.
EXPERIMENTS
5.1 SETUPS
Models. We benchmark our methods using FLUX.1 (Black-Forest-Labs, 2024), PixArt-E (Chen et al., 2024a), SANA (Xie et al., 2025), Stable Diffusion XL (SDXL) (Podell et al., 2024) and SDXL-Turbo (Sauer al., 2024), including both the UNet (Ronneberger al., 2015; Ho et al., 2020) and DiT (Peebles & Xie, 2023) backbones. See Appendix B for more details.
Datasets. Following previous works (Li et al., 2023a; Zhao et al., 2024c;b), we randomly sample the prompts in COCO Captions 2024 (Chen al., 2015) for calibration. To evaluate the generalization capability of our method, we sample 5K prompts from the MJHQ-30K (Li et al., 2024a) and the summarized Densely Captioned Images (sDCI) (Urbanek et al., 2024) for benchmarking. See Appendix C for more details.
Baselines. We compare SVDQuant against the following post-training quantization (PTQ) methods:
4-bit NormalFloat (NF4) is an infommtioo--therrttialll optimal 4-bit data type for weight-only quantization (Dettmers et al., 2023), which assumes that weights follow a normal distribution. We use the
NF4 FLUX.1 models (Lllyasviel, 2024) as the baselines.
ViDiT-Q (Zhao et al., 2024b) uses per-token quantization and smoothing (Xiao et al., 2023) to alleviate the outliers across different batches and tokens and achieves lossless 8-bit quantization on PixArt-
MixDQ (Zhao et al., 2024c) identifies the outliers in the begin-of-sentence token of text embedding and protects them with 16-bit pre-computation. This method enables up to W4A8 quantization with negligible performance degradation on SDXL-Turbo.
TensorRT contains industry-level PTQ toolkit to quantize the diffusion models to 8 bits. It uses smoothing and only calibrates activations over a selected timestep range with a percentile scheme.
Metrics. Following previous works (Li et al., 2022; 2024b), we evaluate image quality and image similarity with respect to the 16-bit models' results. For image quality assessment, we use Fréchet Inception Distance (FID, lower is better) to measure the distribution distance between the generated images and the ground-truth images (Heusel et al., 2017; Parmar al., 2022). Besides, we employ Image Reward (higher is better) to approximate the human rating of the generated images (Xu et al., 2024a). We use LPIPS (lower is better) measure the perceptual similarity (Zhang et al., 2018) and Peak Signal Noise Ratio (PSNR, higher is better) to measure the numerical similarity of the images from the 16-bit models. Please refer to our Appendix E. for more metrics (CLIP IQA (Wang et al., 2023b), CLIP Score (Hessel et al., 2021) and SSIM*).
Implementation details. Please refer to Appendix D fore more details.
5.2
RESULTS
Visual quality results. We report the quantitative results in Table across various models and precision levels, and show some corresponding 4-bit qualitative comparisons in Figure 7. Among all models, our 8-bit results can perfectly mirror the 16-bit results, achieving PSNR higher than 21, beating all other 8-bit baselines. On FLUX. our INTS PSNR even reaches 27 on MJHQ.
https::/en wi www.pediia.ccomm httpaaa.orrgggiit//ttttrrrrraaae htttt//ttttuaaaaeeeooeeeaaaaaaat Liimi lari tetityiindedd inddinneesuurreeeeeeeeeeeeeaaaee mes_mmasuuee
7

---

Published as a conference paper at ICLR 2025
Table Quantitative quality comparisons across different models. RTN stands for round no-nearest. IR means ImageReward. Our 8-bit results closely match quality the 16-bit models. Moreover, our 4-hit results outperform other 4-hit baselines, effectively preserving the visual quality of 16-bit models.
MJHQ
sDCI
Backbone
Model
Precision
Method
Quality
Similarity
Quality
Similarity
FID (U)
IR (1)
LPIPS (J)
PSNR t)
FID (L)
IR (?)
LPIPS (L)
PSNR (1)
RFI6
20.3
0.953
24.8
1.02
FLUX.1
WSA8
27.0
24.7
-dev
W4A16
NF4
20.6
0.910
0.272
19.5
24.9
0.986
0.292
18.2
Steps)
INT W4N4
Ours
19.9
0.935
0.223
21.0
24.2
1.01
0.240
19.7
NVEP W444
Ours
20.4
0.937
0.208
21.4
24.7
1.01
0,218
20.2
BF16
19.2
0.938
20.8
0.932
FLUX.1
INT WSA8
Ours
19.2
0.966
0.120
22.9
20.7
0.975
0.133
21.3
schnell
W4A16
NF4
18.9
0.943
0.257
18.2
20.7
0.953
0.263
17.1
Steps)
INT W4A4
Ours
18.3
0.951
0.258
18.3
20.1
0.979
0.260
17.2
NVEP W4A4
Ours
0.968
0.227
19.0
20.5
0.226
FP16
16.6
0.944
24.8
0.966
DIT
INT W8A8
VIDIT-Q
15.7
0.944
0.137
22.5
23.5
0.974
0.163
20.4
PixArt-
(20 Steps)
INT W4A8
VIDIT-Q
37.3
0.573
0.611
12.0
40.6
0.600
0.629
11.2
INT W444
VIDIT Q
412
-2.27
0.854
6.44
425
-2,28
0,838
6.70
INT W444
Ours
19.2
0.878
0.323
17.6
25.9
0.918
0.352
16.5
NVEP W444
Ours
0.940
18.5
22.9
0.298
17.2
0.952
29.9
SANA
INT W4A4
RTN
20.5
0.894
0.339
15.3
28.6
0.807
0.371
13.8
-1.6B (20 Steps)
INT W444
Ours
19.3
0.935
0,220
17.8
28.1
0.846
0,242
16.2
NVEP W444
RTN
19.7
0.932
0.237
17.3
29.0
0.829
0.265
15.6
29.3
FP16
24.3
0.845
24.7
0.705
INT W8A8
MixDQ
24.1
0.834
0.147
21.7
25.0
0.690
0.157
21.6
WSAS
Ours
0.845
24.0
24.8
0.110
23.7
-Turbo
INT W4A8
MixDQ
27.7
0.708
0.402
15.7
25.9
0.610
0.415
15.7
Steps)
INT W444
MixDQ
353
-2.26
0.685
11.0
373
-2.28
0.686
11.3
INT W4A4
Ours
24.6
0.816
0,262
18.1
26.0
0.671
0.272
18.0
UNet
0.832
FP16
16.6
0.729
22.5
0.573
SDXL
INT W8A8
TensorRT
20.2
0.591
0.247
22.0
25.4
0.453
0.265
21.7
(30 Steps)
INT W8A8
Ours
0.718
0.119
26.4
22.4
0.574
0.129
25.9
W444
0.601
0.288
21.0
26.2
0.477
0.307
NVEP W4A4
Ours
18.3
0,640
0.250
21.8
23.9
0.502
0.261
21.7
For 4-bit quantization, NVFP4 outperforms INT4, thanks to the native hardware support of smaller microscaling group size on Blackwell. On FLUX.1, our SVDQuant consistently surpasses the NF4 W4A16 bascline regarding all metrics. For the dev variant, our method even exceeds the original BF16 model regarding Image Reward, suggesting stronger human preference. On PixArt-E. while our INT4 method shows slight degradation, our NVFP4 model achieves a comparable score to the FP16 model. This is likely due to PixArt-E's highly compact model size (600M parameters), which benefits from a smaller group size. Remarkably, our INT4 and NVFP4 models significantly outperform ViDiT-Q's W4A8 results by a large margin across all metrics. Note that our FP16 PixArt-E model differs slightly from ViDiT's, though both offer the same quality. For fair comparisons, ViDIT-Q's similarity results are calculated using their FP16 results.
For UNet-based models, on SDXL-Turbo. our 4-bit models substantially outperform MixDQ W4A8, and our FID scores INT4 and NVFP4 results achieve comparable quality to TensorRT's W8A8 performance, which represents the 8-bit SoTA. As shown in Figure 14 in the Appendix, our visual quality only shows minor degradation.
par with the FP16 models, indicating quality loss. On SDXL, our
Memory save and speedup. In Figure 8, we report measured model size, memory savings, and speedup for 1LUX.1. Our INT4 and NVFP4 quantization reduce the original transformer size from 22.2 GiB to 6.1 GiB, including a 0.3 GiB overhead due to the low-rank branch, resulting in an overall 3.6x reduction. Since both weights and activations are quantized, compared to the NF4
8

---

Published as a conference paper at ICLR 2025
4Lu BF16 Reward: 0.955
SLUX.esched 0.905
NEW Reward 0.551
INT
Reward:
0.335
0.937
usss
AI
AI
AI
AI
Pusppe Anuate www. vkke creand od TOKEN. Fano,, Lasinng 30 masiral Cool Splet.
Froeps Ashostar esumesltr Auerke dedge, ausiler pata woth massg see of mysicll bace, daysha Aghing mnttv maiging cksic
Phorpe solgeetu undemane dr only convcttion wdo alemettr
Prrnge: euslngg plartin reatlagr
alameess and Rocso inbrno logsh photseaply protle Aaaaniii of
young vomman, long
koask in Iaslanerrl nut snsk ring night redtonada malt colemate lastamnnd
pargarcta, Phokigophy Shnee proce
FLUX.
FPI6 Image Reward M496
VIDNT-Q Imagee Reward: lingue 0.578
Wold
SDNL--Tubb
MiaDQ
W4A4
Imagee 0.940
Image Rewand Image
Image
Phooge
Pnoeptt aod Jusaiid pratty Jurk Enchgmmand absaild phote
graws sangle
clalber Mark nik Aillyy Alpor cosluis
cimmatic lighiey
Poxrpe: Cunsnnggher Alvor NAI eddeeee siti raslute Protag dhaasss Aphiar PocAri-E
Pscepe: ohhypast cil satad drne
(4) SDXL-Turbo
Figure 7: Qualitative visual results on MIHQ. Image Reward is calculated over the entire dataset. On FLUX..I models, our 4-hit models outperform the NE4 W4A16 baselines, demonstrating superior text alignment and closer similarity to the 16-bit models. For instance, NF4 misses the swinging chair in the top right example. On PixArt-E E and SDXL-Turbo, 4-bit results demonstrate noticeably better visual quality than ViDiT-Q's MixDQ's W4A8 results.
BF16
NF4 (W1A16)
SVDQuant INTANVEP (W4A4)
$25
657
1500
4182
375
-8-4
496
2.5%
3.5x
350
1000
F320
10.3x
3,1X
5.8
6.1
6.9
218
162
B016
N14
18.14
18:14
B0716 (c) Single Step Lakency on Desktop 1090 (ms)
IN14
1916 (d) Single Step Latency on Laptop 4090 (ms)
IND4
NF4 N911
(a) Model Size (GIB)
(b) DiT Infermnce Memory (GiB)
Single Step Latency on Desktop 5090 (ms)
Figure 8: SVDQuant reduces the 12B FLUX.1 model size 3.6x and cuts the 16-bit model's memory usage by 3.5x. With Nunchaku, our INT4 model runs faster than NF4 W4A16 baseline on both desktop and laptop NVIDIA RTX 4090 GPUs. Notably, on the laptop 4090, achieves a total 10.1x speedup by eliminating CPU offloading. Our NVFP4 model is also 3.1x faster than both BF16 and NF4 on the RTX 5090 GPU.
weight-only-quantized variant, our inference engine Nunchaku even saves more memory footprint. offers 3.0x speedup on both desktop- and laptop-level NVIDIA RTX 4090 GPUs with INT4 precision and 3.1x speedup on the RTX 5090 GPU with NVFP4 precision, compared to both NF4 and the original 16-bit models. Notably, while the original BF16 model requires per-layer CPU offloading on the 16GB laptop 4090, our INT4 model fits entirely in GPU memory, resulting in 10.1x 1x speedup by avoiding offloading.
Integrate with LoRA. Previous quantization methods require fusing the LoRA branches and re- re-quantizin the model when integrating LoRAs. In contrast, our Nunchaku climinates redundant memory access, allowing adding separate LoRA branch. practice, we can fuse the LoRA branch into our low-rank branch by slightly increasing the rank, further enhancing efficiency. In Figure 9, we exhibit some visual examples of applying LoRAs of five different styles (Realism, Ghibsky Illustration, Anime, Children Sketch, and Yarn Art) to our INT4 FLUX. 1-dev model. Our INT4 model successfully adapts to each style while preserving the image quality of the 16-bit version. For more visual examples, Appendix E.2. For -schnell further support LoRAs from one-step conditional model pix2pix-turbo (Parmar et al., 2024), enabling additional controls like sketch. An interactive demo available here.

---

Published as a conference paper at ICLR 2025
Realism LoRA
Giibsky Illustration LoRA
Anime LoRA
Children Sketch LoRA
Yarn Art LoRA
F-Ux.I-dev BF16
Our INT4
Figure 9: Our 4-bit model seamlessly integrates with off-the-shelf LoRAs without requiring requantization. When applying LoRAs, it matches the image quality the original 16-bit FLUX.1-dev. See Appendix for the text prompts.
P:uMn-:: FP16 Image Reward: 0.931
Nalve Quanizztion
Smoothnng
LoRC
OLes Smoodiing Image 0,600
Reward:
Reward:
Image 0.508
Reward:
0.878
Prump:: Besoh vich image popaler read pisnner contect
Prompt merppe (mage, ongry croh vallad, soiiaater daii egole photoggapphed by doold topeelle evrte resinaisnnce colors, aleard wiisiing recipe on white bootgrowed
Figure 10: Ablation study of SVDQuant Pi The rank of the low-rank branch is 64. Image Reward is measured over IK samples from MJHQ. Our results significantly outperform the others, achieving highest image quality by a wide margin.
Ablation study. Figure 10, we present several ablation studies of SVDQuant on PixArt-E. First, both SVD-only and naïve quantization perform poorly in the 4-bit setting, resulting in severe quality degradation. While applying smoothing to the quantization slightly improves image quality compared to naīve quantization, the results remain unsatisfactory. LoRC (Yao et al., 2024) introduces low-rank branch to compensate for quantization errors, but this approach is suboptimal, as quantization errors exhibit well-spread distribution of singular values. Consequently, low-rank compensation fails to effectively mitigate these errors, as discussed in Section 4.2. In contrast, we first decompose the weights and quantize only the residual. As demonstrated in Figure 5, the first several singular values are significantly larger than the rest, allowing us to shift them to the low-rank branch, effectively reducing weight magnitude. Finally, smoothing consolidates the outliers, enabling the low-rank branch to absorb outliers from the activations and substantially improving image quality.
Trade-off of increasing rank. Please refer to Appendix E.5 for more details.
6 CONCLUSION
In this work, we introduce a novel 4-bit post-training quantization paradigm, SVDQuant, for diffusion models. It adopts a low-rank branch to absorb the outliers in both the weights and activations, easing the process of quantization. Our inference engine Nunchaku further fuses the low-rank and low-bit branch kernels, reducing memory usage and cutting off redundant data movement overhead. Exten- sive experiments demonstrate that SVDQuant preserves image quality. Nunchaku further achieves a the W4A16 on an NVIDIA RTX 4090 and 5090 GPUs. This advancement enables the efficient deployment of large- scale diffusion models edge devices, unlocking broader potential for interactive AI applications.
3.5x reduction in memory usage over the original 16-bit model and 3.0x speedup
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
We thank NVIDIA for donating the DGX server and Blackwell GPUs. We thank MIT-IBM Watson AI Lab, MIT and Amazon Science Hub, MIT AI Hardware Program, National Science Foundation, Sloan foundation, Packard Foundation, Dell, LG, Hyundai, and Samsung for supporting this research. We thank Paulius Micikevicius for his support and discussion.
REFERENCES
Distilling Diffusion Models into Conditional GANs, 2024. 3
Salch Ashkboos, Amirkeivan Mohtashami, Maximilian Croci, Bo Li, Pashmina Cameron, Martin Jaggi, Dan Alistarh, Torsten Hoefler, and James Hensman. Quarot: Outlier-free 4-bit inference in rotated llms. NeurlPS, 2024.
Yogesh Balaji, Seungjun Nah, Xun Huang, Arash Vahdat, Jiaminn Song, Karsten Kreis, Miika Aittala, Timo Aila, Samuli Laine, Bryan Catanzaro, et al. ediffi: Text-to-image diffusion models with an ensemble of expert denoisers. arXiv preprint arXiv:2211.0332 2022.
Fan Bao, Shen Nic, Kaiwen Xue, Yue Cao, Chongxuan Li, Hang Su, and Jun Zhu. All are worth words: vit backbone for diffusion models. In CVPR, 2023. 3
Black-Forest-Labs. Flux.1, 1, 2024. URL htl.tt/s://// //1 forest labs a ai/. 2,7, 18
Han Cai, Muyang Li, Qinsheng Zhang, Ming- Yu Liu, and Song Han. Condition-aware neural network for controlled image generation. CVPR, 2024.
Junsong Chen, Chongjian Ge, Enze Xie, Yue Wu, Lewei Yao, Xiaozhe Ren, Zhongdao Wang, Ping Luo, Huchuan Lu, and Zhenguo Li. Pixart-o: Weak-to-strong, training of diffusion transformer for 4k text-to-image generation. ECCV, 2024a. 5, 7,
Junyu Chen, Han Cai, Junsong Chen, Enze Xie, Shang Yang. Haotian Tang, Muyang Li, Yao Lu, and Song Han. Deep compression autoencoder for efficient nigh-resolution diffusion models. JCLR, 2025. 18
Xinlei Chen, Hao Fang, Tsung-Y Yi Lin, Ramakrishna Vedantam, Saurabh Gupta, Piotr Dollár, and Lawrence Zitnick. Microsoft coco captions: Data collection and evaluation server. arXiv preprint 00000:/500400325. 2015.
Zigeng Chen, Xinyin Ma, Gongfan Fang, Zhenxiong Tan, and Xinchao Wang. Asyncdiff: Parallelizing diffusion models by asynchronous denoising. arXiv preprint Stdi:22006069111 2024b. 3
Mostafa Dehghani, Josip Djolonga, Basil Mustafa, Piotr Padlewski, Jonathan Heek, Justin Gilmer, Andreas Peter Steiner, Mathilde Caron, Robert Geirhos, Ibrahim Alabdulmohsin, et al. Scaling vision transformers 22 billion parameters. In ICML. PMLR, 2023. 18
Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer. Gpt3, int8 O): 8-bit matrix multiplication for transformers at scale. NewrlPS, 2022. 2, 27
Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. QLoRA: Efficient finetuning of quantized LLMs. In NeurIPS, 2023. 4,7
Carl Eckart and Gale Young. The approximation of one matrix by another of lower rank. Psychome- trika, (1)))21-2218, 1936. 6
Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari. Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, et al. Scaling rectified flow transformers for high-resolution image synthesis. In JCML, 2024. 3,
fal.a. Auraflow v0.1, 2024. URL https: /blog fal. ai/uua///
Elias Frantar, Salch Ashkboos, Torsten Hoefler, and Dan Alistarh. GPTQ: Accurate post-training compression for generative pretrained transformers. ICLR, 2023. 19, 25
11

---

Published as a conference paper at ICLR 2025
Han Guo. Philip Greengard, Eric Xing, and Yoon Kim. Lq-lora: Low-rank plus quantized matrix decomposition for efficient language model finetuning. ICLR, 2024. 4
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, 2016. 18
Yefei He, Luping Liu, Jing Liu, Weijia Wu, Hong Zhou, and Bohan Zhuang. Ptqd: Accurate
post-training quantization for diffusion models. NeurlPS, 2023. 3
Yefei He, Jing Liu, Weijia Wu, Hong Zhou, and Bohan Zhuang. Efficientdm: Efficient quantization- aware fine-tuning of low-bit diffusion models. In ICLR, 2024. 3,4
Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. Clipscore: A reference- free evaluation metric for image captioning. arXiv preprint arXiv:2 104.08718, 2021. 7, 20
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. NeurIPS, 2017. 7
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. NeurlPS, 2020. 2.3.4, 7. 7
Yen-Chang Hsu, Ting Hua, Sungen Chang, Qian Lou, Yilin Shen, and Hongxia Jin. Language model compression with weighted low-rank factorization. In ICLR, 2022. 3.4
Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang. and Weizhu Chen. Lora: Low-rank adaptation of large language models. In ICLR, 2022. 4
Yushi Huang, Ruihao Gong, Jing Liu, Tianlong Chen, and Xianglong Liu. Tfmq-dm: Temporal feature maintenance quantization for diffusion models. In CVPR, 2024.
Ajay Jaiswal, Lu Yin, Zhenyu Zhang, Shiwei Liu, Jiawei Zhao, Yuandong Tian, and Zhangyang Wang. From galore to welore: How low-rank weights non-uniformly emerge from low-rank gradients. arXiv preprint arXiv: 2407.11238 2024. 4
Sehoon Kim, Coleman Richard Charles Hooper, Amir Gholami, Zhen Dong. Xiuyu Li, Sheng Shen, quantization. In ICML,
Michael W. Mahoney, and Kurt Keutzer. SqueezeLLM:
2024. 3
Daiqing Li, Aleks Kamko, Ehsan Akhgari, Ali Sabet, Linmiao Xu, and Suhail Doshi. Playground v2.5: Three insights towards enhancing aesthetic quality in text-to-image generation, 2024a. 7, 18, 25
Muyang Li, Ji Lin, Yaoyao Ding, Zhijian Liu, Jun- Yan Zhu and Song Han. Gan compression: Efficient architectures for interactive conditional gans. In CVPR, 2020. 3
Muyang Li, Ji Lin, Chenlin Meng, Stefano Ermon, Song Han, and Jun- Yan Zhu. Efficient spatially sparse inference for conditional gans and diffusion models. In NeurlPS, 2022. 3,7
Muyang Li, Tianle Cai, Jiaxin Cao, Qinsheng Zhang, Han Cai, Junjie Bai, Yangqing Jia, Ming Yu Liu, Kai Li, and Song Han. Distrifusion: Distributed parallel inference for high-resolution diffusion models. In CVPR, 2024b. 3,7
Xiuyu Li, Yijiang Liu, Long Lian, Huanrui Yang, Zhen Dong, Daniel Kang, Shanghang Zhang, and Kurt Keutzer. Q-diffusion: Quantizing diffusion models. In ICCV, 2023a. 3, 7, 27
Yanyu Li, Huan Wang, Qing Jin, Ju Hu, Pavlo Chemerys, Yun Fu, Yanzhi Wang, Sergey Tulyakov, and Jian Ren. Snapfusion: Text-to-image diffusion model on mobile devices within two seconds. NewrIPS, 2023b. 3
Yixiao Li. Yifan Yu, Qingru Zhang, Chen Liang, Pengcheng He, Weizhu Chen, and Tuo Zhao. LoSparse: Structured compression of large language models based low-rank and sparse approx- imation. In ICML, volume 202. PMLR, 2023c. 4
Yixiao Li, Yifan Yu, Chen Liang, Nikos Karampatziakis, Pengcheng He, Weizhu Chen, and Tuo Zhao. Loftq: Lora-fine-tuning-wware quantization for large language models. In ICLR, 2024c. 4
12

---

Published as a conference paper at ICLR 2025
Ji Lin, Jiaming Tang, Haotian Tang, Shang Yang, Wei-Ming Chen, Wei-Chen Wang, Guangxuan Xiao, Xingyu Dang. Chuang Gan, and Song Han. Awq: Activation-wware weight quantization for on-device Ilm compression and accelcration. In MLSys, 2024. 2, 2,
Yujun Lin, Haotian Tang, Shang Yang, Zhekai Zhang, Guangxuan Xiao, Chuang Gan, and Song Han. Qserve: W4a8kv4 quantization and system co-design for efficient Ilm serving. In MLSys, 2025. 3, 27
Songhua Liu, Weihao Yu, Zhenxiong Tan, and Xinchao Wang. Linfusion: image. arXiv preprint arXiv:2409.0200 02099, 2024a. 3
gpu, minute, 16k
Wenxuan Liu and Saiqian Zhang. Hq-dit: Efficient diffusion transformer with fp4 hybrid quantization. arXiv preprint arXiv:2405. 2024. 3
Xuewen Liu, Zhikai Li, Junrui Xiao, and Qingyi Gu. Enhanced distribution alignment for quantization of diffusion models. arXiv preprint arXiv:2401. 044585, 2024b.
Zechun Liu, Changsheng Zhao, Igor Fedorov, Bilge Soran, Dhruv Choudhary, Raghuraman Krish- namoorthi, Vikas Chandra, Yuandong Tian, and Tijmen Blankevoort. Spinquant- IIm quantization with learned rotations. arXiv preprint arXiv:2405. /6406, 2024c.
Lllyasviel. [major update] bitsandbytes guidelines and flux Illyasviel stable-diffusion-
webui-forge discussion #981, 2024. URL https: /gittub.tt///g/////////. www//llyysviee//////
webuii
Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. Dpm-solver: A fast ode solver for diffusion probabilistie model sampling in around 10 steps. NeurIPS, 2022.
Simian Luo, Yiqin Tan, Longbo Huang. Jian Li, and Hang Zhao. Latent consistency models: Synthesizing high-resolution images with few-step inference. arXiv preprint arXiv: 2310.04378,
2023. 3
Shuming Ma, Hongyu Wang. Lingxiao Ma, Lei Wang. Wenhui Wang, Shaohan Huang, Li Dong, Ruiping Wang, Jilong Xue, and Furu Wei. The era 1-bit llms: All large language models are in 1.58 bits. arXiv preprint arXiv:2402. 17764, 2024a. 27
Xinyin Ma, Gongfan Fang, Michael Bi Mi, and Xinchao Wang. Learning-to-cachen Accelerating diffusion transformer via layer caching. NeurIPS, 2024b. 3
Xinyin Ma. Gongfan Fang, and Xinchao Wang. Deepcache: Accelerating diffusion models for free. In CVPR, 2024c.
Pascal Massart. Concentration inequalities and model selection: Ecole d' d'Elé de Probabilités de Saint-Flour XXXIII-2000 Springer, 2007. 18
Chenlin Meng, Ruiqi Gao, Diederik P Kingma, Stefano Ermon, Jonathan Ho, and Tim Salimans. On distillation of guided diffusion models. arXiv preprini arXiv:2210.03142, 2022a. 3
Chenlin Meng, Yutong He, Yang Song, Jiaming Song, Jiajun Wu, Jun-Yan Zhu, and Stefano Ermon. SDEdit: Guided image synthesis and editing with stochastic differential equations. In ICLR, 2022b. 2
Fanxu Meng, Zhaohui Wang, and Muhan Zhang. Pissa: Principal singular values and singular vectors adaptation of large language models. NeurlPS, 2024. 4
Leon Mirsky. Symmetric gauge functions and unitarily invariant norms. The quarterly journal of mathematics, 11(1):50-59, 1960.
NVIDIA. Nvidia blackwell architecture technical brief, 2024. URL htttss://resourree nvidia cww/en-us-bileeeeeeeeooooo-
NVIDIA Corporation. Block Scaling in cuDNN Frontend API, 2025.
URL https:
/ docs nvidia. com/ B ockSca ing. html. 19
13

---

Published as a conference paper at ICLR 2025
Gaurav Parmar, Richard Zhang, and Jun-Yan Zhu. On aliased resizing and surprising subtleties in gan evaluation. In CVPR, 2022.
Gaurav Parmar, Taesung Park, Srinivasa Narasimhan, and Jun-Yan Zhu. One-step image translation with text-to-image models. arXiv preprint arXiv:2403. 12036, 2024. 9
William Peebles and Saining Xic. Scalable diffusion models with transformers, In ICCV, 2023. 3, 4,
Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion models for high-resolution image synthesis. In ICLR, 2024. 2, 3, 18
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In ICML, 2021. 20
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In CVPR, 2022. 2,
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. Medical Image Computing and Conputer-Assisted Intervention- 2015: 18th International Conference, Munich, Germany, October 5-9, 2015, Proceedings, Part III 18, pp. 234-24--2222 241. Springer, 2015. 3,7
Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman. Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation. In CVPR, 2023. 2
Tim Salimans and Jonathan Ho. Progressive distillation for fast sampling of diffusion models. In ICLR, 2021.
Axel Sauer, Dominik Lorenz, Andreas Blattmann, and Robin Rombach. Adversarial diffusion distillation. In ECCV, 2024. 3, 18
Yuzhang Shang, Zhihang Yuan, Bin Xie, Bingzhe Wu, and Yan Yan. Post-training quantization on diffusion models. In CVPR, 2023.
Jascha Sohl- Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In JCML, 2015. 3
Yang Song, Prafulla Dhariwal, Mark Chen, and Ilya Sutskever. Consistency models. ICML, 2023.
Yang Sui, Yanyu Li, Anil Kag, Yerlan Idelbayev, Junli Cao, Ju Hu, Dhritiman Sagar, Bo Yuan, Sergey Tulyakov, and Jian Ren. Bitsfusion: 1.99 bits weight quantization diffusion model. In NeurIPS, 2024. 3
Siao Tang, Xin Wang, Hong Chen, Chaoyu Guan, Zewen Wu, Yansong Tang, and Wenwu Zhu. Post-training quantization with progressive calibration and activation relaxing for text-to-image diffusion models. In ECCV, 2024. 3
Jack Urbanek, Florian Bordes, Pietro Astolfi, Mary Williamson, Vasu Sharma, and Adriana Romero- Soriano. A picture is worth more than 77 text tokens: Evaluating clip-style models on dense captions. In CVPR, 2024. 7, 18
Changyuan Wang, Ziwei Wang, Xiuwei Xu, Yansong Tang. Jie Zhou, and Jiwen Lu. Towards accurate post-training quantization for diffusion models. In CVPR, 2024a.
Haoxuan Wang, Yuzhang Shang, Zhihang Yuan, Junyi Wu, and Yan Yan. Quest: Low-bit diffusion model quantization via efficient selective finetuning. arXiv preprint arXiv:2402.03666 2024b. 3
Hongyu Wang, Shuming Ma, Li Dong, Shaohan Huang. Huaijie Wang, Lingxiao Ma, Fan Yang, Ruiping Wang, Yi Wu, and Furu Wei. Bitnet: Scaling 1-bit transformers for large language models. arXiv preprint a11iv:2310.1 /453, 2023a. 27
14

---

Published as a conference paper at ICLR 2025
Jiannan Wang, Jiarui Fang, Aoyu Li, and PengCheng Yang. Pipefusion: Displaced patch pipeline parallelism for inference of diffusion transformer models. arXiv preprint 244.:22000.144430, 2024c.
Jianyi Wang, Kelvin CK Chan, and Chen Change Loy. Exploring clip for assessing the look and feel of images. In AAAI, 2023b. 7, 20
Junyi Wu, Haoxuan Wang, Yuzhang Shang, Mubarak Shah, and Yan Yan. Ptq4dit: Post-training quantization for diffusion transformers. In NeurlPS, 2024. 3
Guangxuan Xiao, Ji Lin, Mickael Seznec, Hao Wu, Julien Demouth, and Song Han. Smoothquant: Accurate and efficient post-trat post-training quantization for large language models. In ICML, 2023. 2, 3, 5, 7, 29
Enze Xie, Junsong Chen, Junyu Chen, Han Cai, Haotian Tang, Yujun Lin, Zhekai Zhang, Muyang Li, Ligeng Zhu, Yao Lu, et al. Sana: Efficient high-resolution image synthesis with linear diffusion transformers. ICLR, 2025. 7, 18
Jiazheng Xu, Xiao Liu, Yuchen Wu, Yuxuan Tong, Qinkai Li, Ming Ding, Jie Tang, and Yuxiao Dong. Imagereward: Learning and evaluating human preferences for text-to-image generation. NewrIPS, 2024a. 7
Yuhui Xu, Lingxi Xie, Xiaotao Gu, Xin Chen, Heng Chang, Hengheng Zhang, Zhengsu Chen, Xiaopeng Zhang, and Qi Tian. Qa-lora: Quantization-aware low-rank adaptation of large language models. In ICLR, 2024b. 4
Yuewei Yang, Xiaoliang Dai, Jialiang Wang, Peizhao Zhang, and Hongbo Zhang. Efficient quantiza- tion strategies for latent diffusion models. arXiv preprint arXiv:23/2.05431, 2023. 3
Zhewei Yao, Xiaoxia Wu, Cheng Li, Stephen Youn, and Yuxiong He. Exploring post-training quantization in ilms from comprehensive study to low rank compensation. In AAAI, 2024. 4, 10
Tianwei Yin, Michaël Gharbi, Taesung Park, Richard Zhang, Eli Shechtman, Fredo Durand, and William T Freeman. Improved distribution matching distillation for fast image synthesis. In NeurlPS, 2024a. 3
Tianwei Yin, Michaël Gharbi, Richard Zhang, Eli Shechtman, Fredo Durand, William T Freeman, and Taesung Park. One-step diffusion with distribution matching distillation. In CVPR, 2024b.
Zhihang Yuan, Yuzhang Shang, Yue Song, Qiang Wu, Yan Yan, and Guangyu Sun. Asvd: Activation- aware singular value decomposition for compressing large language models. arXiv preprint arXiv: 2312.05821, 2023.
Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. ICCV, 2023. 2
Qinsheng Zhang and Yongxin Chen. Fast sampling of diffusion models with exponential integrator. In ICLR, 2022. 3
Qinsheng Zhang, Molei Tao, and Yongxin Chen. gddim: Generalized denoising diffusion implicit models. In ICLR, 2022. 3
Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as perceptual metric. In CVPR, 2018.
Jiawei Zhao, Zhenyu Zhang, Beidi Chen, Zhangyang Wang, Anima Anandkumar, and Yuandong Tian. GaLore: Memory-efficient LLM training by gradient low-rank projection. In ICML, 2024a.
4
Tianchen Zhao, Tongcheng Fang, Enshu Liu, Wan Rui, Widyadewi Soedarmadji, Shiyao Li, Zinan Lin, Guohao Dai, Shengen Yan, Huazhong Yang, et al. Vidit-q: Efficient and accurate quantization of diffusion transformers for image and video generation. arXiv preprint arXiv:2406.0 02540, 2024b.
3, 7
15

---

Published as a conference paper at ICLR 2025
Tianchen Zhao, Xuefei Ning, Tongcheng Fang. Enshu Liu, Guyue Huang, Zinan Lin, Shengen Yan, Guohao Dai, and Yu Wang. Mixdq: Memory-efficien ftw-step text-to-image diffusion models with metric-decoupled mixed precision quantization. In ECCV, 2024c. 3.7 7
Yilong Zhao, Chien-Yo Yu Lin, Kan Zhu, Zihao Ye, Lequn Chen, Size Zheng, Luis Ceze, Arvind Krishnamurthy, Tianqi Chen, and Baris Kasikci. Atom: Low-bit quantization for efficient and accurate Ilm serving. MLSys, 2024d. 27
Xingyu Zheng, Haotong Qin, Xudong Ma, Mingyuan Zhang, Haojie Hao, Jiakai Wang. Zixiang Zhao, Jinyang Guo, and Xianglong Liu. Binarydm: Towards accurate binarization of diffusion model. arXiv preprint arKiv:2404.0555 006622, 2024. 3
16

---

Published as a conference paper at ICLR 2025
A PROOFS
A. PROOF OF PROPOSITION 4.1
Proposition 4.1. The quantization error EXX, W) || - Q(X)Q |I in Equation 3 can
be decomposed as follows:
E(X, W) X W Q(W) X - Q(x) | (11))
W - Q(W))) (9)
Proof.
XW Q(X)Q(W))I
XW XQ(W) +QQ()) - Q(X)Q
< | W -Q(W) Q(X))Q(W))
X (W QQ())) [[X
Q(X)II IQ(W)
XXI W Q(W)] X Q(X) W - -W Q((W))l
X W -Q(W) F+ X Q(X) | ([ F I - Q(W)))))
A.2 PROOF OF PROPOSITION 4.2
Proposition 4.2. For any tensor R and quantizaatin method described in Equation as Q(R) Sn Qn. Assuming the elements of R follow a distribution that satisfies the following regularity condition: There exists a constant such that
E mmx [|)) c- [|]]]]
(10)
Then, we have
E [|| Q(R) CV size R
R
(11)
qmax
where Rize(R) denotes the number of elements in R. Especially if the elements of R follow a normal distribution, Equation 10 holds for c Toogsizel R)
size
Proof.
R Q(R)I
R
SR
QnlIF
R
R
SR
SR
round
SR
SR
F
R
RR
S
round
So,
[1] R Q(R) p]
LE [[ size R)
/size(R) R)
maxx R[)
Tmax
size R
R
Xmax
Especially, if the elements of R follows normal distribution, we have
E
max
) < a / (size(R)
(12)
17

---

Published as a conference paper at ICLR 2025
where is the std deviation of the normal distribution. Equation 12 comes from the maximal inequality of Gaussian variables (Lemma 2.3 in Massart (2007)). On the other hand,
E
R
F
x
ZER
Σ MEE xc
(13)
/size(
2size(R)
(14)
π
where Equation 13 comes from Cauchy- Cauchy-Schwaard Schwwatzz inequality and Equation 14 comes from the expectation of half-normal distribution.
Together, we have that for normal distribution,
[max [[]
log (size R))
log (size( R2) . R
size Rt)
log(sizo RT sizel R)
In other words, Equation 10 holds for c=
B BENCHMARK MODELS
We benchmark our methods using the following six text-to-image models:
FLUX.1 (Black-Forest-l) consists of 19 joint attention blocks (Esser et al., 2024) and 38 parallel attention blocks (Dehghani al., 2023), totaling 12B parameters. We evaluate both the 50-step guidance-distilld (FLUX.1- (FLUX. 1 --schnell) variants.
2024) is the SoTA open-sourced DiT-based diffusion model. It
dev) and 4-step timestep-distile
PixArt-E (Chen et al., 2024a) another DiT-based model. Instead of using joint attention, it stacks 28 attention blocks composed of self-attention, cross-attention, and feed-forward layers, amounting to 600M parameters. We evaluate it on the default 20-step setting.
SANA (Xie et al., 2025) is a 1.6B DiT model. utilizes 32x compression autoencoder (Chen et al., 2025) and replaces Softmax attention with linear attention to accelerate image generation.
Stable Diffusion XI. (SDXL.) is widely-used UNet-based model with 2.6B parameters (Podell al., 2024). It predicts noise with three resolution scales. The highest-resolution stage is processed entirely by ResBlocks (He et al., 2016), while the other two stages jointly use ResBlocks and attention layers. Like Pi SDXL uses cross-attention layers for text conditioning. We evaluate it in the 30-step setting, along with its 4-step distilled variant, SDXL-Turke (Sauer et al., 2024).
C BENCHMARK DATASETS
To assess the generalization capability of our method, we adopt two distinct prompt sets with varying styles for benchmarking:
MJHQ-30K (Li et al., 2024a) consists of 30K samples from Midjourney with 10 common categories, 3K samples each. We randomly select 5K prompts from this dataset evaluate model performance on artistic image generation.
Densely Captioned Images (DCI) (Urbanek et al., 2024) is a dataset containing ~8K images with detailed human-annotated captions, averaging over 1,000 words. For experiments,
18

---

Published as a conference paper at ICLR 2025
the summarized version (sDCI), where captions condensed to 77 tokens using large language models (LLMs) to accommodate diffusion models. Similarly, we randomly sample 5K prompts for realistic image generation.
D IMPLEMENTATION DETAILS
For the 8-bit setting, quantization with a low-rank branch of rank 16. For the 4-bit setting, we adopt per-group symmetric quantization for both activations and weights, along with low-rank branch of rank 32. INT4 quantization uses a group size of 64 with 16-bit scales. We use NVFP4 for FP4 quantization, which has native hardware support of group size of 16 with FP8 scales Blackwell GPUs (NVIDIA Corporation, 2025). We use GPTQ (Frantar et al., 2023) to quantize the residual weights. For FLUX.1 models, the imputs of linear layers in adaptive normalization are kept in 16 bits (i.e., W4A16), For other models, key and value projections in the cross-attention are retained at 16 bits since their latency only covers less than 5% of total runtime.
per-token dynamic activation quantization and per-channel weight
The smoothing factor À IR is per-channel vector whose i-th element is computed as A max([x...))" max( (..)) 1-a following SmoothQuant (Xiao al., 2023) Here, X E Rxx and W Rmxx, It is decided offline by searching for the best migration strength for each layer to minimize the layer output mean squared error (MSE) after SVD on the calibration dataset.
19

---

Published as a conference paper at ICLR 2025
E ADDITIONAL RESULTS
E VISUAL QUALITY RESULTS
We report extra quantitative quality results with additional metrics in Table Specifically, CLIP IQA (Wang et al., 2023b) and CLIP Score (Hessel et al., 2021) assesses the image quality and text-image alignment with CLIP (Radford al., 2021), respectively. Structural Similarity Index Measure (SSIM) is used to measure the luminance, contrast, and structure similarity of images produced by our 4-bit model against the original 16-bit model. We also visualize more qualitative comparsions in Figures 11, 12, 13, 14 and 15.
Table 2: Additional quantitative quality comparisons different models. RTN stands for round-to-ncarest. C.IQA means CLIP IQA, and C.SCR means CLIP Score
MJIQQ
<DC
Model
Precision
Method
Quality
Similarity
Quality
Similarity
C.IQA (t)
C.SCR (t)
SSIM 1)
C.QA (1)
C.SCR (1)
SSIM (1)
BF16
0.952
26.0
0.955
25.4
FLUX.I
INT W8A8
Ours
0.953
26.0
0.748
0.955
25.4
0.697
dev
W4A16
NF4
0.947
25.8
0.748
0,951
25.4
0.697
(50 Steps)
INT W4A4
Ours
0.950
25.8
0.797
0.951
25.3
0.751
BF16
0.938
26.6
0.932
26.2
FLUX.
INT W8A8
Ours
0.938
26.6
0.844
0.932
26.2
0.811
-cchnell
W4A16
NF4
0.941
26.6
0,713
0.933
26.2
0.674
(4 Steps)
INT W4A4
Ours
0.937
0.720
0.932
26.2
0.681
NVEP W444
Ours
0.939
26.6
0.745
0,932
26.1
0.712
0.944
0.966
DIT
WSAS
VIDITQQ
0.966
0.756
PixArt-E (20 Steps)
INT WSA8
Ours
0.947
26.8
0.849
0.967
26.0
0.800
INT W4A8
VIDIT-Q
0.912
25.7
0.356
0.917
25.4
0.295
INT W444
VIDIT-Q
0.185
13.3
0.077
0.176
13.3
0.080
NVEP W444
Ours
0.938
26.7
0.692
0.956
26.1
0.618
BF16
0.934
26.8
0.958
26.4
SANA
-1.6B
INT W4A4
Owrs
0.926
26.9
0.710
0.951
26.4
0.649
(20 Steps)
NVEP W4A4
0.929
0.694
0.953
26.4
0.626
NVEP W4A4
Ours
0.932
26.9
0.755
0.955
26.4
0.701
INT WSAS
MixDO
0.922
26.5
0.763
0.907
26.5
0.750
SDXL .Turbo
INT WSAS
Ours
0.925
26.5
0.821
0.912
26.5
0.808
W4A8
MixDQ
0.893
0.895
0.493
(4 Steps)
INT W4A4
MixDO
0.556
13.1
0.289
0,548
11.9
0.296
FP W444
Ours
0.919
26.5
0.663
0.902
26.6
0.649
FP16
0.907
27.2
0.911
26.5
SDXI.
WSAS
TemorRT
0.901
0.697
(30 Steps)
INT WSAS
Ours
0.912
27.0
0.843
0.910
26.3
0.814
INT W4A4
Ours
0.878
26.7
0.717
0.862
26.2
0.672
NVEP W4A4
Ours
0.892
26.8
0.739
0.877
26.4
0.701
20

---

Published as a conference paper at ICLR 2025
FLUX dev BF16
Our INT W&AS Image Reward: 0 9:18
NFS W4A16 Image Reward: 0,910
Our IN' WAM
NYFP W4A4
Image Rewand: 0.953
Image Rewand: 0.935
Image Rewand: 0.937
END
END
END
END
END
Prompe: photogmaply of
sign google Sunlding by night
wanng dutiio Cloar facial
Propt A relentist anatycing requential data with recuront ncural actwork revorrc laboratory withh computer screes and graphs the background F Phooresenn lighting 35mm, Canon EOS Mark DSLR !125 second shotter 180 160
Prompt Lifell lover landed the moon, fromt noon perspeett, carth in background no towtt
Figure 11: Qualitative visual results of 1-U on MJHQ.
FLUX. L-ccinell BF16
Our INT WAAAS Image Reward: a 9666
NF4 W4A16 Image Rewand: 0.943
Our INT W444 Image Revond: 0.951
Our NVPP W444
Image Resemds 0.068
Image Rewanf: 0 968
gyheynoo Jon glowing eyer pongle Ayprovalinic
Prompt Cal de aupplasses relaws on hnsaocc
Promptt Ludnig Beethoven woden electonic mullkkeybardd Jamatz DIGITAL CINEM PRAPTOR MK Slarier lons. spper mugpieg, Meganiaels
Prompt: the wond FLORIDA pelled our with each letter hoving own beach conntal shent
Figure 12: Qualitative visual results of 1...X.1-scchell MJHQ.
21

---

Published as a conference paper at ICLR 2025
P FP16
VIDIT-Q INT WHAS
Our INT WWAS
VIDIT-Q INT W4M
ON W4A4
Our WAA4
Image Rewani: 0,944
Image Rewand: 0,944
Image Reward: 0.955
Image Reward: 0,573
Imiage Rowand: 0878
Image Reward: 0,540
Poompt: 2950s atyle hambunger reziarrnt Carione wil sull and fanms contuarn with Ad usie white baclground
Prompt: willigl phonogrossty photogngpe, tigh qualty wilipe, f.A., Aicu, Mr national geograpelic swardd wilning phooogrogs by vicit alcl
Prompti lake Poredi or muense Dromatic Jushnyy vitl are shingg oner the marls. Snll uaier. Rushon photaguphh Brooriniinn landeape Ar 12
Browtthst rhreinn
Jow' apo auoo
grodiog Cancn
awazitg breaniuting asezing agprr
super accent Nigning Illenitation, productioo gualip:
depsh offedd profosstoni graling drssi, ngt liphting lighting fighting Sgiting
Sharp Diffuc Suckliphing,
Phatogruhh by
Trasing Iilanination, Optis,
Bounk. inmollily dennled sapple au poir, orange pisk, yullorr
Showsering, Bofestions Screen Soace Reflction,, Grating Diptovement pacing, aliain 5540 ASS EQAA
ATX, CGI YEX shaden, nuw supping ohoomnn
Figure 13: Qualitative visual results PixArt- E on MJHQ.
SDXL
TensoRT WBAS Rewand: 591
Our W8A8
Our INT WIM
Our NVFP WAM
Image Rewand:
Image
Image
Prompt: Anny poncaber epic and ronisticc photo, Difoted on dark barkgoummi, photograply f2
Prompe of young pw0
Procupt: profezzional photo of negront cocktaii Juiiin ansosphere.
Prompt: murgue hyporealoate, mearing modempttoon inskug
Figure 14: Qualitative visual results of SDXL on MJHQ.
22

---

Published as a conference paper at ICLR 2025
SDXII-Turbo FP16
MisnQ INT WSAR
Our INT WKAS
MisDQ INT W4AS
Our INT W4A4
Our NVFP W444
Imtge Reward: 0,845
Imsage Reward 0.854
Image Reward: 0.845
Image Roward: 0,708
Image Reward: 0886
Image Reward: 0.832
ponntt-potuull a sthee after Aud mork in doal mind ligtt connast fot of disunis good light, Miling shaf tackgrount Canon EOS less, der Aishnng of nataral light and antifical tiyhtngg creatage dhamatic and intouse offect
Pronpe furcelone conference dHoN Macichals Sammn
POOTUCE o OUT sliing d monid Ahune enting paysoon
Prompu: Al, flaming witk busion body warnor fightng poss, PIC. a photomalsste while ripor snurgony from the pungle makking prgy soor
Figure 15: Qualitative visual results of SDXL-Turbo on MJHQ.
23

---

Published as a conference paper at ICLR 2025
E.2 LORA RESULTS
In Figure 16, we showcase more visual results of applying the aforementioned five community- contributed LoRAs of different styles (Realism, Ghibsky Illustration, Anime, Children Sketch, and Yarn Art) to our INT4 quantized models.
FLUX11der
BF16
FLUXCIder BF16
BE16
Anime LoRAA
FLUX I-dce BE16
(c) Children Sketch LoRA
FLUX I-dcx HF16
(c) Yam Art LoRA
Figure 16: Additional LoRA results on FLUX.1-6 I-dv.. When applying LoRAs, our INT4 model matches the image quality of the original BF16 model. See Appendix F for the detailed used text prompts.
24

---

Published as a conference paper at ICLR 2025
E.3 ADDITIONAL ABLATION OF SVDQUANT
Table 3: Quantitative comparisons of different SVDQuant settings MJHQ. NVFP4 outperforms INT4.
SVDQuant leverages low-rank branch apply GPTQ to quantize the weight residual, further improving quality.
quantization, significantly enhancing image quality. can further
Model
Precision
Low ronk Branch
GPTQ
Image Reward (f)
LPIPS (U)
PSNR
X
0,908
0.322
18.5
0,933
0.297
19.1
0.926
0.256
20.1
FLUX.I-dev
0.935
0.223
NVFP4
X
0.936
0.204
21.5
0.935
0.223
20.9
0.937
0.208
0.962
0.345
16.3
0.962
0.317
16.8
INT4
0.957
0.289
17.6
FLUX Fl.hel--chnell
0.957
0,280
17.5
0.956
0.247
18.5
0.968
0.247
18.4
0.227
BF16
0.944
-1.226
0.762
9.1
INT4
0.858
0.356
17.0
PixAn--
0.660
0.517
14.8
0.696
0.480
15.6
NVFP4
0.945
0,290
18.0
BF16
0.952
0.894
0.339
15.3
0.881
0.288
INT4
0.922
0,234
17.4
SANA-1
0.935
0,220
17.8
X
0.932
0.237
17.3
NVFP4
0.957
0,188
18.7
In Table 3, we provide additional quantitative ablation results of SVDQuant on the MJHQ prompt set (Li et al., 2024a). Across all models, NVFP4 outperforms INT4 due to its native support for smaller microscaling group sizes on Blackwell. SVDQuant leverages low-rank branch to absorb outliers, significantly enhancing image quality all settings. Additionally, it can incorporate GPTQ (Frantar et al., 2023) instead of round-to-nearest for weight quantization, further improving quality in most cases. Notably, combining SVDQuant with NVFP4 precision achieves the best results, reaching PSNR of 21.5 on FLUX. I-de, closely matching the image quality of the original 16-bit model. In Figure 17, we provide qualitative comparisons across different precision settings.
25

---

Published as a conference paper at ICLR 2025
FLUX BF16
INT4 LPIPS 0.322
INTISSVDQUI LPIPS 022
NYTPN LPIPS: 0224
NVPRR-VVDQQI LPIPS: 0.208
www.ciiee///aajaa.....aa Sakayin föx aad Appednatedd
SaaSE4 LOVE. SESIICA Vasia okoumd Siy ophacced Instictoon ruticl g'onddrr
here Sepers socde wigh of Cemtho secerdeee and Sowe Soryoum tnniintio wrinng you vihoomo ahe Levenr entante wetee gah worch Merieaye Noneyy
PiANE
XVFRS-SVDQuat
Funge Goo Ouganatoo Clark Cranori, Sapleekke T'ole will walorda rgci du Serrgee Kooppegeeee opcedia Sen Smrings
SANA-I 66
INT4 LIPIS 0.330
INT4FSVDUQaR EPPSS: 0,220
NYTI LPOPS-0237
LIPSS: 0.177
yang Mec geree www.ea. mng, Joomoon, pendlp muchict Aowengragy
Figure 17: Qualitative comparisons of different precisions on MJHQ. NVFP44SVDQuant yields the highest image fidelity.
26

---

Published as a conference paper at ICLR 2025
E.4 LATENCY RESULTS
In Table 4, we compare FLUX latency on a laptop-level 4090 GPU across different precisions. Compared to INT8, 4-bit quantization delivers a 1.3x speedup. However, without optimization, SVDQuant incurs an 18% overhead due to the low-rank branch. By eliminating redundant memory access, Nunchaku achieves latency comparable to naive INT4.
Table 4: Single-step latency comparisons of FLUX desktop-level 4090 GPU.
Method
BF16
INT8
INT4
SVDQuant
SVDQuant +Nunchaku
Latency (ms)
657
282
212
250
218
E.5 TRADE-OFF OF INCREASING RANK
Figure 18 presents the results different rank r in SVDQuant on PixArt-E. Increasing the rank from 16 64 significantly enhances image quality but increases parameter and latency overhead. In our experiments, we select a rank of 32, which offers a decent quality with minor overhead.
PixAm E: FP16
SVDQuamt Rank 16 Image Reward: 0.787
SVDQuamt Rank-32
Reward: 0.829
SVDQuaat Rank-64
Reward: 0.859
Rank-18
Rank-332
Rankm64
17%
10.8
MudiiSSee Ovethed
11.39
5.8%
Procmpt: aoond wisinng pholography of beannjfu andic smiling
Figure 18:
Increasing the rank y the low-rank branch in SVDQuant can enhance image quality, but it also leads to higher parameter and latency overhead.
E.6 TRADE-OFF BETWEEN QUALITY AND BITWIDTHI
We evaluate LPIPS across different bitwidths for various quantization methods on PixArt-E and FLUX.I-schnell using the MJHQ dataset in Figure 19, with weights and activations sharing the same bitwidth. Following the convention (Xiao et al., 2023; Lin et al., 2024; 2025; Li et al., 2023a; Zhao al., 2024d; Dettmers et al., 2022), for bitwidths above 4, we apply per-channel quantization; for or below, we use per-group quantization (group size 64). SVDQuant consistently outperforms naive quantization and SmoothQuant. Notably, on PixArt- and FLUX. I-schnell, our 4-bit results match 7-bit and 6-bit naive quantization, respectively.
Our SVDQuant can still generate images in the 3-bit settings on both PixArt-E and FLUX.I-schnit I-shhnell, performing much better than SmoothQuant. Below this precision (c.g., W2A4 or W4A2), SVDQuant cannot produce images either since 2-bit symmetric quantization is essentially a ternary quantization. Prior work (Ma et al., 2024a; Wang al., 2023a) has shown that ternary neural networks require quantization-awee training even for weight-only quantization to adapt the weights and activations to the low-bit distribution.
Naive
SmaothQuant
SVDQuant
6.5
0.7
LPIPS
0.4
0.3
0.5
0.0
Nisoith PixArt-E
(b) Fschel
Figure 19: LPIPS of different quantization methods PixArt-2 and FLUX.I-sch 1-schnell across different bitwidths.
27

---

Published as a conference paper at ICLR 2025
F
TEXT PROMPTS
Below we provide the text prompts we use in Figure (from left right).
man armor with a beard and a sword
GHIBSKY style, a fishorman casting a 1ine into a peaceful village lake surrounded by quaint cottages
girl, neck tuft, white hair, sheep horns, blue eyes, nm22 style
sketched style, A squirrel wearing glasses and reading tiny book under an oak tree a panca playing in the snow, yarn art style
The text prompts we use in Figure 16 are (in the rasterizing order):
A male secret agent in a tuxedo, helding à gun, standing in front of a burning building
handsome man in suit, 25 years cld, cool, futuristic
knight in shining armor, standing in front of castle under siege knight fighting fire-breathing dragon front of a redieval castle, flames and smake
A male wizard with long white beard casting lightning spell in the
micale of storm
young woman with long flowing hair, standing on a mountain peak at dawn, overlocking a misty valley
starry night sky and GHIBSKY style, a quiet garden at twilight, with blooming flowers and the soft glow of lanterns lighting up the path GHIBSKY style, serene mountain lake with crystal-clear water, surrounded by towering pine trees and rocky eliffs
GHIBSKY style, distant city lights
windowsill gazing
GHibsKy style, an enchanted forest at night, with glowing mushrooms and fireflies lighting up the underbrush
GH GHiSSKY style, peaceful beach town with colorful houses lining the shore and calm ocean stretching out into the herizon GHIBSKY style, cozy living
with view of snow-covered forest, the fireplace crackling and a blanket draped over a cenfy chair
dog wearing vizard hat, nm22 anime style
girl looking at the stars, nm22 anime style
fish swimming in pond, nm22 style
a giraffe with long scarf, nm22 style
bird sitting on branch, nm22 minimalist style
girl wearing flower crown, nm22 style
shetchea style, A garden full of celorful butterflies and blooming flowers with a gentle preeze blowing
sketchea style, A beach scene with kids building sandcastles and seagulls flying overhead
sketched style, A hot air balloon drifting peacefully over patchwork fields and forests below
sketched style, A sunny meadow with a girl in a flowy dress chasing butterflies
sketched style, A little boy dressed pirate, steering toy ship small stream
sketched style, A small boat floating on a peaceful lake, surrounded by trees and mountains
hot air bal leen flying over mountains, yarn art style
a cat chasing a butterfly, yarn art style
squirrel collecting acorns, yarn art style wizard casting spell, yarn art style jellyfish floating in the ocean, yarn art style
a sea turtle swimming through coral reef, yarn art style
28


Reviews:
[{"review_id": "zC0XuQzUqJ", "reviewer": "Reviewer_R3Nw", "summary": "This paper presents SVDQuant quantizes weights and activations to 4 bits to accelerate inference of large-scale diffusion models. Through a low-rank decomposition branch, the authors introduce SVDQuant, a method for mitigating the impact of outliers on quantization.\nThey absorb outliers on weights/activations. To do so, they first migrate the outliers from activation to weight. Then they apply SVD to the updated weight, decomposing it into a low-rank branch and a residual.  Additionally, they incorporate an inference engine, LoRunner, which combines low-rank and low-bit kernels to maximize computation speed and minimize memory overhead. As demonstrated in empirical results for different large diffusion models, SVDQuant offers significant memory savings and performance improvements without affecting image quality significantly. This demonstrates that it is a promising approach for deploying diffusion models on consumer-grade hardware.", "questions": "1- Using outliers and low rank for quantizations is a very well-known technique. The authors need to have a section in related work and highlight the difference/novelty between the technique and all related work. Otherwise, the novelty seems incremental.\nMaybe you can provide a table for all related work including the mentioned one here and show all similarities and differences with the proposed technique. \n\n[A] QNCD: Quantization Noise Correction for Diffusion Models\n\n[B] Q-DiT: Accurate Post-Training Quantization for Diffusion Transformers\n\n\n2- The choice of rank (e.g., rank 32) appears somewhat arbitrary, without adequate theoretical or empirical justification for why this setting was chosen over others.  Can the author put more comments about it? Maybe more ablation studies and more detailed experiments about it will give more insight to the reviewer. \n\n3- If a neural network exhibits skewed distributions that may not align well with the low-rank assumption used here (e.g., nongaussian behavior), how does it work? Please discuss the robustness of the method to different weight distributions.\n\n4-With the advent of mixed-precision computation, some architectures might benefit from a mix of 4-, 8-, and. How would LoRunner perform in such configurations, and could it be adapted to handle this flexibility?\n\n5-What are the challenges to extending the method below 4-bit? For example 2w2a or 4w2a or 2w4a?\n\n6-I am interested to see a breakdown of LoRunner’s impact on speed and memory, perhaps by comparing SVDQuant with and without LoRunner.\n\n7- Minor typos:\n\"Quantizes both the weights and activations\" – quantizes both weights and activations.\n“on NVIDIA RTX-4090 laptop\" – \"on an NVIDIA RTX-4090 laptop.\"", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1-SVDQuant’s strategy to manage outliers using low-rank decomposition offers an improvement over standard smoothing techniques.\n\n2-LoRunner effectively reduces memory access overhead and enhances performance, particularly for GPU inference, addressing practical deployment constraints.\n\n3-Pushing boundaries for low-bit quantization of both weight and activation", "weaknesses": "1-The level of contribution is not great. Using outliers and low rank for quantizations is a very well-known technique. The authors need to have a section in related work and highlight the difference/novelty between the technique and all related work. Otherwise, the novelty seems incremental.\n\n2- The paper lacks a theoretical analysis of why the low-rank decomposition approach can consistently outperform other outlier-handling techniques beyond empirical observations.\n\n3- Code is not provided.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "x0ADsyq9IN", "reviewer": "Reviewer_2HuL", "summary": "This paper proposes a 4-bit PTQ method for diffusion models that absorbs the outliers between weights and activations using a low-rank branch. To augment this method, the paper also presents an efficient inference engine to avoid the redundant memory access of the activations. Experimental results on latest diffusion models validate the superior accuracy and memory efficiency of the proposed method.", "questions": "Please see above.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "The paper is well-written and clear. Though the idea is generally intuitive, the observation of the low-rank outliers and the fusion from the low-rank branch to low-bit branch makes this work a strong submission. Experiments are conducted on the latest diffusion models, and the results are generally strong.", "weaknesses": "1. Can the authors include some results on sub 4-bit quantization with the proposed method? It would be good to know the limitations of the method.\n\n2. The authors should include more comparisons in their experimental results. While they compared with MixDQ and ViDiT-Q, some prominent works have been left out, such as Q-Diffusion [1] and QUEST [2].\n\n[1] https://arxiv.org/pdf/2302.04304\n\n[2] https://arxiv.org/pdf/2402.03666v1", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "gulAjHRHeX", "reviewer": "Reviewer_ntPk", "summary": "The paper introduces a post-training quantization method for diffusion models called SVDQuant that successfully quantizes both weights and activations to 4 bits without sacrificing visual quality. By integrating the low-rank branch kernels into the low-bit branch, SVDQuant minimizes overhead, significantly accelerating model inference. Tested on the 12B parameter FLUX.1-schnell model, it reduces memory usage by 3.6 times compared to the BF16 model and achieves a 3.6 times speedup over the NF4 W4A16 baseline on a laptop equipped with an RTX-4090 GPU.", "questions": "1. Could you explain the calculation method for the smoothing factor in more detail?\n2. In the ablation study, what is meant by \"SVD-only\" and \"naive quantization\"? Does \"SVD-only\" indicate that no quantization is applied? And what is the setting for naive quantization?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "1. The paper introduces a full-precision low-rank adapter and Singular Value Decomposition (SVD) to effectively compensate for quantization errors.\n2. The authors have implemented a novel kernel that efficiently fuses low-rank and low-bit branches, minimizing computational overhead.\n3. The extensive experiments conducted on state-of-the-art diffusion models, such as FLUX, provide strong evidence of the method's effectiveness and robustness.", "weaknesses": "1. Adding a W8A8 baseline to the latency comparison would provide valuable insights and a clearer performance reference.\n2. Since the authors use group quantization for weights and activations in the 4-bit setting, it would be beneficial to include methods that also use group quantization, such as Atom [1], as baselines.\n\nReferences: \n[1] Zhao et al. \"Atom: Low-bit Quantization for Efficient and Accurate LLM Serving\" in MLSys'24", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "CER0B0i1wy", "reviewer": "Reviewer_2VPm", "summary": "This work accelerate diffusion models by low-bit quantization and low-rank decomposition. The motivation of this work is the outlier quantization error in activations for recent PTQ methods. The authors propose a multi-branch architecture with both low-bit and low-rank operations. To speed up the inference, a kernel fusion implementation is also proposed for the two branches. In experiments, the DiT and UNet diffusion models are evaluated in 4/8-bit conditions.", "questions": "In the experiment part, this work almost evaluates on 4/8-bit quantization; what is the best trade-off between bit-width and performance for diffusion models? \n\nAlso, adding more best practice of the diffusion model quantization will be fine.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "- It is an interesting topic to combine the low-bit quantization and low-rank decomposition, previous work including Loft-Q (ICLR 24), et al.\n\n- This work proposes a kernel fusion implementation to speed up on-device inference.\n\n- The method is reasoning and the writing is easy to follow.", "weaknesses": "- I don't think splitting low-rank and low-bit branches is a good idea to overcome quantization errors. Different from the QLLM (ICLR 24) et al.,  the computation of the two branches can not merge after training, leading to 5~10% overheads in the paper (line 314).\n\n- Experiments parts are not very solid: the baseline quantization methods are kind-of weak. Recent works including SpinQuant [1], AffineQuant [2], et al. achieve much higher performance than the baseline NF4 in the paper and also without the additional branch.\n\n[1] SpinQuant: LLM quantization with learned rotations\n[2] AffineQuant: Affine Transformation Quantization for Large Language Models", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "r6G20wbdh4", "reviewer": "Reviewer_44PJ", "summary": "The paper proposes an aggressive quantization method (4-bit) for both activations and weight in diffusion models. To mitigate quantization-induced quality losses, the author propose to  migrate outlayers from activations to weights and then to add low-rank branches at high accuracy 16b (as opposed to the traditional smoothing approach which is not sufficient at 4-bit quantization level especially for diffusion models): i.e. the network is modified also for inference by the presence of the branches A baseline implementation of the low-rank branches in inference is slow and negates the speed advantages of the quantized computations, The authors then propose a remedy to this shortcoming which merges the branches, to regain speed (while also retaining memory reduction advantages)", "questions": "Please clarify how the diag lambda is determined\nif possible provide an estimate of the speedup expected by the mixed kernel - possibly using mixed kernel ad higher precision and see how the speedup degrades from \"pure\" low precision kernel.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "The paper presents clearly the flow of ideas: statements are clear and detailed proofs are moved to the appendix, hence they do not distract from the flow f the key messages.\nThe motivation for the work is clearly described as well as the gaps in the SoA, and the claims of novelty. \nThe reader is led step by step to the final solution: this flow helps understanding the technical motivations of the various steps. \nExperiments are clear and the KPI used are well defined.", "weaknesses": "It is not clear ho lambda for the migration of outlayers from activations and weights si computed (it is per-channel - so the array can be quite big). This is in my view a bit of a problem, because lamda migh be impacted by the dataset used for calibration (if it is decided offline) or may be hard to determine efficiently online. \n\nInference time results are missing because the authors have no access to modern gpus (Blackwell) with native 4b support. This is a minor weakness, but it must be noted that it's not fully clear if the merged kernels will benefit from the same speedup claimed for pure 4b kernels. \n\nthe key ideas are not novel per se, but the combination is interesting - the key intuition being the merged low-rank and low-precision kernel to recover speed", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "rU74ctJvTq", "reviewer": "Reviewer_KPne", "summary": "The authors propose SVDQuant to enable 4-bit quantization for diffusion models in terms of both weights and activations, with SVD approach utilized to enable quantization of residual (R=W-L1L2) rather than directly quantizing weight matrices. SVDQuant first use a smoothing technique proposed in previous work to transfer the outliers in activations to weights, then use SVD to enable 16-bit low-rank approximation of the weights and quantize the residual between the two weights, absorbing the weight outliers in the low rank branches L1 and L2.", "questions": "1. Some strong statements need to be properly supported by evidence:\n\n- \"Weight-only quantization *cannot* accelerate diffusion models\" - For modern GPUs, maybe, but this lacks concrete evidence. Also, different hardware platforms have different bottlenecks. \n\n - \"Weights and activations *must* be quantized to the same bit width\" - Can custom hardware support direct mixed precision operations?\n\n - \"they primarily consider weight-only quantization...\" For example, the authors have cited Q-diffusion and EfficientDM which quantize both activations and weights. This statement needs further justification. \n\n2. The authors mention that the \"lower-precision side will be upcast during computation, negating potential performance boosts\". However, as illustrated in Figure 6(b), the XL1L2 branch appears to retain 16-bit full precision before being combined with the quantized residual. This approach seems to be different from the authors' initial claim.\n\n3. It would be very helpful if the authors could elaborate on the similarities and differences between their methods and LoRC (Yao et al.)\n \nMinor:\n\n- How is quantization level defined in Figure 3? And why can it take fractional numbers? I vaguely understand after smoothing, the peak of |X| drops from 10 to 2, but why the peak of |W| also drops? I thought smaller peaks means easier to quantize.\n\n- QKV projection, presumably, is an LLM concept, and it may look abrupt in 4.3 without preliminary discussion.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. I like the idea of translating the weight quantization into residual quantization to eliminate outliers in weights. \n2. The figures are well illustrated, and the math presentations are insightful. \n3. The model with a dinosaur head is hilarious.", "weaknesses": "1. There are a few strong statements in this work with insufficient reasoning. \n2. The method section mainly focuses on justifying the minimization of quantization error but lacking discussion of the computation flow. \n3. The residual quantization approach looks similar to the quantization of error matrix in LoRC.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Spotlight)