DilateQuant: Accurate and Efficient Diffusion Quantization via Weight Dilation

## Abstract

Abstract Diffusion models have shown excellent performance on various image generation tasks, but the substantial computational costs and huge memory footprint hinder their low-latency applications in real-world scenarios.
Quantization is a promising way to compress and accelerate models.
Nevertheless, due to the wide range and time-varying activations in diffusion models, existing methods cannot maintain both accuracy and efficiency simultaneously for low-bit quantization.
To tackle this issue, we propose DilateQuant, a novel quantization framework for diffusion models that offers comparable accuracy and high efficiency.
Specifically, we keenly aware of numerous unsaturated in-channel weights, which can be cleverly exploited to reduce the range of activations without additional computation cost. Based on this insight, we propose Weight Dilation (WD) that maximally dilates the unsaturated in-channel weights to a constrained range through a mathematically equivalent scaling.
WD costlessly absorbs the activation quantization errors into weight quantization. The range of activations decreases, which makes activations quantization easy. The range of weights remains constant, which makes model easy to converge in training stage.
Considering the temporal network leads to time-varying activations, we design a Temporal Parallel Quantizer (TPQ), which sets time-step quantization parameters and supports parallel quantization for different time steps, significantly improving the performance and reducing time cost.
To further enhance performance while preserving efficiency, we introduce a Block-wise Knowledge Distillation (BKD) to align the quantized models with the full-precision models at a block level.
The simultaneous training of time-step quantization parameters and weights minimizes the time required, and the shorter backpropagation paths decreases the memory footprint of the quantization process.
Extensive experiments demonstrate that DilateQuant significantly outperforms existing methods in terms of accuracy and efficiency.

## 1 Introduction

Recently, diffusion models have shown excellent performance on image generation (Li et al., ; Zhang et al., ; ), but the substantial computational costs and huge memory footprint hinder their low-latency applications in real-world scenarios.
Numerous methods (Nichol & Dhariwal, ; Song et al., ; Lu et al., ) have been proposed to find shorter sampling trajectories for the thousand iterations of the denoising process, effectively reducing latency.
However, complex networks with a large number of parameters used in each denoising step are computational and memory intensive, which slow down inference and consume high memory footprint.
For instance, the Stable-Diffusion (Rombach et al., ) with 16GB of running memory still takes over one second to perform one denoising step, even on the high-performance A6000.

Model quantization is one of the most popular compression methods.
By quantizing the weights and activations with low-bit integers, we can reduce memory requirements and accelerate computational operations. The effects become more noticeable as the bit-width decreases.
For example, employing 8-bit models can achieve up to a 4$\times$ memory compression and 2.35$\times$ speedup compared to 32-bit full-precision models on a T4 GPU (Kim et al., ). Adopting 4-bit models can further deliver an additional 2$\times$ compression and 1.59$\times$ speedup compared to 8-bit models.
Thus, quantization is a highly promising way to facilitate the low-latency applications of diffusion models on source-constrained hardware.

Figure: Figure 1: An overview of the cost-vs-performance trade-off across various approaches. Data is collected from DDIM with 4-bit quantization on CIFAR-10.
Refer to caption: x1.png

Typically, existing quantization techniques are implemented through two main approaches: Post-Training Quantization (PTQ) and Quantization-Aware Training (QAT).
As shown in Figure 1, PTQ (Liu et al., ) calibrates the quantization parameter with a small calibration dataset and does not rely on end-to-end retraining, making it data- and time-efficient.
However, it brings severe performance degradation at low bit-width.
In contrast, QAT (Esser et al., ) can maintain performance at lower bit-width, but it requires retraining the whole model, which is time-consuming and resource-intensive.
For instance, when applying both standard approaches to DDIM (Song et al., ) on CIFAR-10, QAT (Esser et al., ) results in a 3.3$\times$ increase in GPU memory footprint (9.97 GB vs. 3.01 GB) and an 14.3$\times$ extension of quantization time (13.89 GPU-hours vs. 0.97 GPU-hours) compared to PTQ (Liu et al., ).
Due to the huge gap in time cost and GPU consumption, PTQ is more preferred despite the fact that QAT outperforms PTQ.

Unfortunately, while previous methods (Xiao et al., ; Li & Gu, ; Xiao et al., ; Li et al., ) of quantization have achieved remarkable success in single-time networks, the wide range and time-varying activations caused by the unique temporal network of diffusion models make them fail.
Specifically, since the diffusion models infer in pixel space or latent space, the absence of layer normalization results in a wide range of activations, complicating activation quantization.
For example, in the same UNet network, the range of activations is almost 2.5$\times$ larger than that of the segmentation models (Ronneberger et al., ), as shown in Figure 2(a).
Equivalent scaling techniques address the wide range of activations by shifting the quantization difficulty from activations to weights.
Some methods (Xiao et al., ; Shao et al., ; Lin et al., ; Zhang et al., ) utilizing equivalent scaling have shown success in large language models (LLMs) by tackling outliers in certain channels, but these methods are not appropriate for diffusion models, where outliers exist in all channels, as shown in Figure 2(b).
Unconstrained scaling of all outlier channels significantly alters the weight range, making it difficult for model to converge in training stage.
In addition, the temporal network induces a highly dynamic distribution of activations that varies across time steps, as shown in Figure 2(c), further diminishing the performance of quantization.
Numerous PTQ methods (Li et al., ; Liu et al., ) have been explored to enhance results based on the properties of diffusion models, none of them break through the 6-bit quantization for activations.
And the QAT methods(Esser et al., ) retrain the whole model separately for each time step using the original datasets, which is not practical due to the significant time and resources.

In this paper, we propose DilateQuant, a novel quantization framework that can achieve QAT-like performance with PTQ-like efficiency.
Specifically, we propose a weight-aware equivalent scaling algorithm, called Weight Dilation (WD), which searches for unsaturated in-channel weights and dilates them to the boundary of the quantized range, using the max-min values of the out-channel weights as constraints.
WD narrows the range of activations while keeping the weights range unchanged, making activation quantization easier and ensuring model convergence during the training stage. This approach effectively alleviates the wide range activations.
To address the difficulty of quantization for time-varying activations, previous methods (He et al., ; Wang et al., ) set multiple activation quantizers for one layer and trains them individually using different time-step calibration sets, which is data- and time-inefficient. On the other hand, we design a Temporal Parallel Quantizer (TPQ), which sets time-step quantization parameters and supports parallel quantization for different time steps, significantly improving performance and training efficiency, as evidenced by a 160$\times$ reduction in calibration and a 2$\times$ reduction in training time compared to the SoTA method (He et al., ) for DDIM on CIFAR-10.
To further enhance performance while preserving efficiency, we introduce a Block-wise Knowledge Distillation (BKD) to avoids data- and time-consuming retraining of the whole model, distilling the full-precision model to its quantized counterpart at block level using a data-free approach.
Additionally, it further minimizes the time and memory footprint required by training the time-step quantization parameters simultaneously and using the shorter backpropagation paths, respectively.

The contributions of our works are summarized as follows.
1) We formulate a novel quantization framework for diffusion models, DilateQuant, which offers comparable accuracy and high efficiency.
2) The WD and TPQ address the wide range and time-varying activations for diffusion models. And the BKD efficiently enhances performance.
3) Through extensive experiments, we demonstrate that DilateQuant outperforms existing methods across lower quantization settings (6-bit, 4-bit), various models (DDPM, LDM-4, LDM-8, Stable-Diffusion), and different datasets (CIFAR-10, LSUN-Bedroom, LSUN-Church, ImageNet, MS-COCO).
The reproduction of DilateQuant is robust and easy as no hyper-parameters are introduced.

Figure: Figure 2: (a) showcases a wider range of activations in diffusion model (DM) compared to segmentation model (Seg). (b) demonstrates the different outlier challenges for DM and LLM. (c) shows the dynamic distribution activations of DM. The activations of DM and Seg are from the first block output of the upsample stage of UNet network. The activations of LLM come from the output of the penultimate layer.
Refer to caption: x2.png

## 2 Related Work

### 2.1 Diffusion model acceleration

While diffusion models have generated high-quality images, the substantial computational costs and huge memory footprint hinder their low-latency applications in real-world scenarios.
To reduce the inference computation, numerous methods have been proposed to find shorter sampling trajectories, efficiently accelerating the denoising process.
For example, (Nichol & Dhariwal, ) shortens the denoising steps by adjusting variance schedule; (Song et al., ) generalizes diffusion process to a non-Markovian process by modifying denoising equations; (Lu et al., ) uses high-order solvers to approximate diffusion generation.
These methods have achieved significant success, obtaining comparable performance with nearly 10% of the denoising steps.
However, they involve expensive retraining and complex computations.
Conversely, we focus on the complex networks of diffusion models, accelerating them at each denoising step with a quantization method, which not only reduces the computational cost but also compresses the model size.

### 2.2 Model quantization

Model quantization, which represents the original floating-point parameters with low-bit values, compresses model size and accelerates inference.
Depending on whether the model’s weights are fine-tuned or not, it generally falls into two categories: Post-Training Quantization (PTQ) and Quantization-Aware Training (QAT).
PTQ calibrates the quantization parameters with a small dataset and does not require fine-tuning the model’s weights, making it data- and time-efficient.
The reconstruction-based PTQ techniques, such as BRECQ (Li et al., ), utilize gradient descent algorithms to optimize quantization parameters, which have yielded remarkable results in conventional models.
Nevertheless, the unique temporal networks of diffusion models cause them to fail.
To address the issues, PTQ4DM (Shang et al., ) and Q-diffusion (Li et al., ) design a specialized calibration dataset, and EDA-DM (Liu et al., ) refines the reconstruction loss.
Although these PTQ methods enhance results based on the properties of diffusion models, none of them break through the 6-bit quantization.
On the other hand, QAT retrains the whole model after the quantization operation, maintaining performance at lower bit-width.
However, the significant training resources (original dataset, training time, and GPU consumption) make it not practical for diffusion models.
For instance, the recent work TDQ (So et al., ) requires 200K training iterations on a 50K original dataset.
To efficiently quantize diffusion models to lower precision, EfficientDM (He et al., ) fine-tunes all of the model’s weights with an additional LoRA module, while QuEST (Wang et al., ) selectively trains some sensitive layers.
Unfortunately, although they achieve 4-bit quantization of the diffusion models, both of them are non-standard (please refer to Appendix LABEL:app:compare for detail).
Hence, the standard quantization of low-bit diffusion models with high accuracy and efficiency is still an open question.

## 3 Preliminaries

### 3.1 Quantization

The uniform quantizer is one of the most hardware-friendly choices, and we use it in our work.
The quantization-dequantization process of it can be defined as:

$$ $\displaystyle Quant$ $\displaystyle:\bm{x}_{int}=clip\left(\left\lfloor\frac{\bm{x}}{\Delta}\right \rceil+z,0,2^{b}-1\right)$ (1) $\displaystyle DeQuant$ $\displaystyle:\hat{\bm{x}}=\Delta\cdot\left(\bm{x}_{int}-z\right)\approx\bm{x}$ (2) $$

where $\bm{x}$ and $\bm{x}_{int}$ are the floating-point and quantized values, respectively, $\left\lfloor\cdot\right\rceil$ represents the rounding function, and the bit-width $b$ determines the range of clipping function $clip(\cdot)$.
In the dequantization process, the dequantized value $\hat{\bm{x}}$ approximately recovers x.
Notably, the upper and lower bounds of $\bm{x}$ determine the quantization parameters: scale factor $\Delta$ and zero-point $z$, as follows:

$$ $\displaystyle\Delta=\frac{max(\bm{x})-min(\bm{x})}{2^{b}-1},\quad z=\left \lfloor\frac{-min(\bm{x})}{\Delta}\right\rceil$ (3) $$

Combining the two processes, we can provide a general definition for the quantization function, $Q(\bm{x})$, as:

$$ $\displaystyle Q(\bm{x})=\Delta\cdot\left(clip\left(\left\lfloor\frac{\bm{x}}{ \Delta}\right\rceil+z,0,2^{b}-1\right)-z\right)$ (4) $$

As can be seen, quantization is the process of introducing errors: $\left\lfloor\cdot\right\rceil$ and $clip(\cdot)$ result in rounding error ($\mathit{E_{round}}$) and clipping error ($\mathit{E_{clip}}$), respectively.
To set the quantization parameters, we commonly use two calibration methods: Max-Min and MSE.
For the former, quantization parameters are calibrated by the max-min values of $\bm{x}$, eliminating the $\mathit{E_{clip}}$, but resulting in the largest $\Delta$; for the latter, quantization parameters are calibrated with appropriate values, but introduce the $\mathit{E_{clip}}$.

### 3.2 Equivalent scaling

Equivalent scaling is a mathematically equivalent per-channel scaling transformation that offline shifts the quantization difficulty from activations to weights.
For a linear layer in diffusion model, the output $\bm{Y}=\bm{X}\bm{W}$, $\bm{Y}\in\mathbb{R}^{N\times C^{o}}$, $\bm{X}\in\mathbb{R}^{N\times C^{i}}$, $\bm{W}\in\mathbb{R}^{C^{i}\times C^{o}}$, where $N$ is the batch-size, $C^{i}$ is the input channel, and $C^{o}$ is the output channel.
The activation $\bm{X}$ divides a per-in-channel scaling factor $\bm{s}\in\mathbb{R}^{C^{i}}$, and weight $\bm{W}$ scales accordingly in the reverse direction to maintain mathematical equivalence:

$$ $\displaystyle\bm{Y}=(\bm{X}/\bm{s})(\bm{s}\cdot\bm{W})$ (5) $$

The formula also suits the conv layer.
By ensuring that $\bm{s}>\bm{1}$, the range of activations can be made smaller and the range of weights larger, thus in transforming the difficulty of quantization from activations to weights.
In addition, given that the $\bm{X}$ is usually produced from previous linear operations, we can easily fuse the scaling factor into previous layers’ parameters offline so as not to introduce additional computational overhead in the inference.
Currently, equivalent scaling is primarily used in the quantization of LLMs to smooth out activation outliers in certain channels.
While some methods (Xiao et al., ; Lin et al., ; Shao et al., ; Zhang et al., ) have achieved success in LLMs, they fail in diffusion models due to different quantization challenges, please see Appendix G for details.

## 4 Method

Figure: Figure 3: An overview of DilateQuant. WD narrows the activations range while maintaining the weights range unchanged. TPQ sets time-step quantization parameters and supports parallel training. BKD aligns the quantized network with the full-precision network at block level.
Refer to caption: x3.png

### 4.1 Weight Dilation

Inspired by the quantization of LLMs, we propose a novel equivalent scaling algorithm, denotes as Weight Dilation (WD), which decreases the range of activations while maintaining the range of weights unchanged (as shown in Figure 3), thereby efficiently alleviating the wide range activations.

Analyzing quantization error.
We start by analyzing the error from weight-activation quantization.
Taking a linear layer with $\bm{X}\in\mathbb{R}^{N\times C^{i}}$ and $\bm{W}\in\mathbb{R}^{C^{i}\times C^{o}}$ as example,
considering that we calibrate the quantization parameters of $\bm{X}$ and $\bm{W}$ with a MSE and Max-Min manner, respectively, the quantization function (Eq. 4) for activations and weights can be briefly written as:

$$ $\displaystyle Q(\bm{X})=\Delta_{x}\cdot clip\left(\left\lfloor\frac{\bm{X}}{ \Delta_{x}}\right\rceil\right),\quad Q(\bm{W})=\Delta_{w}\cdot\left\lfloor \frac{\bm{W}}{\Delta_{w}}\right\rceil$ (6) $$

where $\Delta_{x}$ and $\Delta_{w}$ are scale factors for activations and weights, respectively.
The quantization error of activations $E_{x}$ and weights $E_{w}$ from rounding (denoted as $\mathit{E_{round}}$) and cliping (denoted as $\mathit{E_{clip}}$) function can be represented as:

$$ $\displaystyle E_{x}=\Delta_{x}\cdot(\mathit{E_{round}}+\mathit{E_{clip}}), \quad E_{w}=\Delta_{w}\cdot\mathit{E_{round}}$ (7) $$

Since the rounding function maps a floating-point number to an integer, $\mathit{E_{round}}$ does not vary, as demonstrated in AWQ (Lin et al., ).
We scale the $\bm{X}$ and $\bm{W}$ using a scaling factor $\bm{s}\in\mathbb{R}^{C^{i}}$ to obtain the scaled $\bm{X^{{}^{\prime}}}$ and $\bm{W^{{}^{\prime}}}$.
The quantization functions and errors after scaling are as follows:

$$ $\displaystyle Q(\bm{X^{{}^{\prime}}})$ $\displaystyle=Q(\bm{X}/\bm{s})=\Delta_{x}^{{}^{\prime}}\cdot clip\left(\left \lfloor\frac{\bm{X}/\bm{s}}{\Delta_{x}^{{}^{\prime}}}\right\rceil\right),$ $\displaystyle\quad Q(\bm{W^{{}^{\prime}}})$ $\displaystyle=Q(\bm{s}\cdot\bm{W})=\Delta_{w}^{{}^{\prime}}\cdot\left\lfloor \frac{\bm{s}\cdot\bm{W}}{\Delta_{w}^{{}^{\prime}}}\right\rceil$ (8) $\displaystyle E_{x}^{{}^{\prime}}$ $\displaystyle=\Delta_{x}^{{}^{\prime}}\cdot(\mathit{E_{round}}+\mathit{E_{clip }}^{{}^{\prime}}),$ $\displaystyle\quad E_{w}^{{}^{\prime}}$ $\displaystyle=\Delta_{w}^{{}^{\prime}}\cdot\mathit{E_{round}}$ (9) $$

where $\Delta_{x}^{{}^{\prime}}$ and $\Delta_{w}^{{}^{\prime}}$ are new scale factors, and $\mathit{E_{clip}}^{{}^{\prime}}$ is the new error of cliping function.
By ensuring that $\bm{s}>\bm{1}$, the range of activations becomes smaller, resulting in $\mathit{E_{clip}}^{{}^{\prime}}/\mathit{E_{clip}}<\bm{1}$ and $\Delta_{x}^{{}^{\prime}}/\Delta_{x}<\bm{1}$ (according to Eq. 3).
However, the range of weights becomes larger, resulting in $\Delta_{w}^{{}^{\prime}}/\Delta_{w}>\bm{1}$.
Consequently, while the new error of activations quantization $E_{x}^{{}^{\prime}}$ decreases, the new error of weights quantization $E_{w}^{{}^{\prime}}$ increases, leading to unstable model convergence during training.
Therefore, the perfect scaling we desired is to achieve $\bm{s}>\bm{1}$ while maintaining $\Delta_{w}^{{}^{\prime}}\approx\Delta_{w}$.

Searching channel for scaling.
Given that the dimension of weights quantization is per-out-channel and the dimension of scaling is per-in-channel, in order to maintain $\Delta_{w}^{{}^{\prime}}\approx\Delta_{w}$ we need to ensure the max-min values (denoted as $W_{max}\in\mathbb{R}^{C^{o}}$, $W_{min}\in\mathbb{R}^{C^{o}}$) of each out-channel weights unchanged.
Based on the above demands, we obtain the $W_{max}$ and $W_{min}$, and record their indexes of in-channel to form a set $A$.
For example, the $A$ in Figure 4(a) is {1,4,6,8}.
Iterating through the index of in-channel $k\in\left\{1,\dots,C^{i}\right\}$, if $k\in A$, we set $\bm{s}_{k}=1$, representing no scaling; if $k\notin A$, the $W_{k}$ denotes as unsaturated in-channel weights, and we set $\bm{s}_{k}$ by dilating $W_{k}$ to $W_{max}$ or $W_{min}$:

$$ $\displaystyle\bm{s}_{k1}$ $\displaystyle=min(W_{max}/W_{k}.clamp(min=0))$ (10) $\displaystyle\bm{s}_{k2}$ $\displaystyle=min(W_{min}/W_{k}.clamp(max=0))$ (11) $\displaystyle\bm{s}_{k}$ $\displaystyle=min(\bm{s}_{k1},\bm{s}_{k2})$ (12) $$

where $clamp$ function specifies the range of the $k_{th}$ in-channel of weight $W_{k}$, $\bm{s}_{k1}$ and $\bm{s}_{k2}$ denote the maximum $\bm{s}$ with $W_{max}$ and $W_{min}$ as constraints, respectively.
Consequently, as shown in Figure 4(b), we maximize $\bm{s}>1$ while keeping $W_{max}^{{}^{\prime}}=W_{max}$ and $W_{min}^{{}^{\prime}}=W_{min}$.
The workflow and effects of WD are detailed in Appendix LABEL:app:wd.

Figure: Figure 4: (a) WD searches for unsaturated in-channel weights and determines scaling factor completely dependent on the max-min values of each out-channel of the weights. (b) WD alleviates the wide range activations by dilating unsaturated channels to a constrained range.
Refer to caption: x4.png

### 4.2 Temporal Parallel Quantizer

Previous methods (He et al., ; Wang et al., ) utilize multiple activation quantizers for a layer to quantize activations at different time steps.
However, since each quantizer is independent, these methods optimize each quantizer individually using time-step calibration sets, which is data- and time-inefficient.
For example, EfficientDM uses 819.2K samples for a total of 12.8K iterations for DDIM on CIFAR-10 (Krizhevsky et al., ).

Different from previous methods, as shown in Figure 3, we design a novel quantizer, denotes as Temporal Parallel Quantizer (TPQ), which sets time-step quantization parameters for activations, instead of simply stacking quantizers.
Specifically, it utilizes an indexing approach to call the corresponding quantization parameters for samples at different time steps. This enables support for parallel training of different quantization parameters, significantly reducing the data and time costs of training.
For a model with $T$ time steps, the quantization parameters of TPQ are as follows:

$$ $\displaystyle\Delta_{x}=\left\{\Delta_{x}^{1},\Delta_{x}^{2},\Delta_{x}^{3}, \dots,\Delta_{x}^{T}\right\},\quad z_{x}=\left\{z_{x}^{1},z_{x}^{2},z_{x}^{3}, \dots,z_{x}^{T}\right\}$ (13) $$

We detail TPQ design for the different layers of the diffusion models.
For the conv and linear layers, they take input $\bm{x}\in\mathbb{R}^{|\mathbb{T}|\times C^{i}}$ and $\bm{x}\in\mathbb{R}^{|\mathbb{T}|\times C^{i}\times H\times W}$, respectively, where $\mathbb{T}$ is a set containing different time-step indexes, $\mathbb{T}\subset\left\{1,\dots,T\right\}$, $\left|\cdot\right|$ represents the number of set elements.
The quantization operation of them can be represented as:

$$ $\displaystyle Q(\bm{x})=\Delta_{x}^{\mathbb{T}}\cdot\left(clip\left(\left \lfloor\frac{\bm{x}}{\Delta_{x}^{\mathbb{T}}}\right\rceil+z_{x}^{\mathbb{T}},0 ,2^{b}-1\right)-z_{x}^{\mathbb{T}}\right)$ (14) $$

where $\Delta_{x}^{\mathbb{T}}$ and $z_{x}^{\mathbb{T}}$ denote the quantization parameters corresponding to $\mathbb{T}$, respectively.
For the attention layers, they take input $\bm{x}\in\mathbb{R}^{|\mathbb{T}\ast H|\times C^{i}\times M}$ due to the $H$ heads, where $``\ast"$ represents cat operation and $M$ is the number of tokens.
So the quantization parameters in Eq. 14 are replaced with $\Delta_{x}^{\mathbb{T}\ast H}$ and $z_{x}^{\mathbb{T}\ast H}$, respectively.

### 4.3 Block-wise Knowledge Distillation

QAT significantly alleviates accuracy degradation in low-bit cases, but it has several limitations for diffusion models:
(1) QAT typically requires original training data, which can sometimes be challenging or even impossible to obtain due to privacy or copyright concerns;
(2) QAT involves end-to-end retraining of the whole complex networks, which is training-unstable and time-intensive.

To address these limitations, inspired by the reconstruction method in PTQ (Li et al., ), we propose a novel distillation strategy called Block-wise Knowledge Distillation (BKD).
Assume the target model for quantization has K blocks ($B_{1},\dots,B_{K}$), and the input samples of model are $\bm{x}$, which is generated by the full-precision model. BKD trains the quantized network block-by-block and aligns it with full-precision network at block level. More specifically, assume that block $B_{k}$ is going to be quantized, and its quantized version is $\hat{B_{k}}$.
We update the quantization parameters $(\Delta_{x}^{\mathbb{T}},z_{x}^{\mathbb{T}},\Delta_{w})$ and weights ($\bm{w}$) of $\hat{B_{k}}$ using the mean square loss $\mathcal{L}$:

$$ $\displaystyle\mathcal{L}_{\Delta_{x}^{\mathbb{T}},z_{x}^{\mathbb{T}},\Delta_{w },\bm{w}}=MSE\left(B_{k}\cdot B_{k-1}\cdot B_{k-2}\cdot...\cdot B_{1}(\bm{x})- \hat{B}_{k}\cdot\hat{B}_{k-1}\cdot\hat{B}_{k-2}\cdot...\cdot\hat{B}_{1}(\bm{x} )\right)$ (15) $$

As can be seen,
(1) BKD does not rely on original training data;
(2) BKD shortens the gradient backpropagation path by aligning blocks, which enhances training stability and decreases the memory footprint of the quantization process.
In addition, BKD trains quantization parameters and weights in parallel, which not only further saves training time but adapts the weights to each time step.

**Table 1: Quantization results of unconditional image generation. The “Calib.” presents the number of calibration samples and “Prec. (W/A)” indicates the bit-width of quantization. ^$\star$ denotes our implementation according to open-source codes and ^$\dagger$ represents results directly obtained by papers or rerunning open-source codes.**
| Task | Method | Calib. | Prec. (W/A) | TBops | Size (MB) | FID $\downarrow$ | sFID $\downarrow$ | IS $\uparrow$ |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| CIFAR-10<br>32 $\times$ 32<br>DDPM<br>steps = 100 | FP | - | 32/32 | 6.2 | 143.0 | 4.26 | 4.46 | 9.03 |
| EDA-DM ^$\star$ | 5120 | 6/6 | 0.2 | 27.0 | 26.68 | 14.10 | 9.35 |  |
| EfficientDM ^$\dagger$ | 1.6384M | 6/6 | 0.2 | 27.0 | 17.29 | 9.38 | 8.85 |  |
| DilateQuant | 5120 | 6/6 | 0.2 | 27.0 | 4.46 | 4.64 | 8.92 |  |
| EDA-DM ^$\star$ | 5120 | 4/4 | 0.1 | 18.1 | 120.24 | 36.72 | 4.42 |  |
| EfficientDM ^$\dagger$ | 1.6384M | 4/4 | 0.1 | 18.1 | 81.27 | 30.95 | 6.68 |  |
| DilateQuant | 5120 | 4/4 | 0.1 | 18.1 | 9.13 | 6.92 | 8.56 |  |
| LSUN-Bedroom<br>(Yu et al., )<br>256 $\times$ 256<br>LDM-4<br>steps = 100<br>eta = 1.0 | FP | - | 32/32 | 98.4 | 1317.4 | 3.02 | 7.21 | 2.29 |
| EDA-DM ^$\star$ | 5120 | 6/6 | 3.5 | 247.8 | 10.56 | 16.22 | 2.12 |  |
| EfficientDM ^$\dagger$ | 102.4K | 6/6 | 3.5 | 247.8 | 5.43 | 15.11 | 2.15 |  |
| QuEST ^$\dagger$ | 5120 | 6/6 | 3.5 | 247.8 | 10.1 | 19.57 | 2.20 |  |
| DilateQuant | 5120 | 6/6 | 3.5 | 247.8 | 3.92 | 8.90 | 2.17 |  |
| EDA-DM ^$\star$ | 5120 | 4/4 | 1.6 | 165.5 | N/A | N/A | N/A |  |
| EfficientDM ^$\dagger$ | 102.4K | 4/4 | 1.6 | 165.5 | 15.27 | 19.87 | 2.11 |  |
| QuEST ^$\dagger$ | 5120 | 4/4 | 1.6 | 165.5 | N/A | N/A | N/A |  |
| DilateQuant | 5120 | 4/4 | 1.6 | 165.5 | 8.99 | 14.88 | 2.13 |  |
| LSUN-Church<br>(Yu et al., )<br>256 $\times$ 256<br>LDM-8<br>steps = 100<br>eta = 0.0 | FP | - | 32/32 | 19.1 | 1514.5 | 4.06 | 10.89 | 2.70 |
| EDA-DM ^$\star$ | 5120 | 6/6 | 0.7 | 284.9 | 10.76 | 18.23 | 2.43 |  |
| EfficientDM ^$\dagger$ | 102.4K | 6/6 | 0.7 | 284.9 | 6.92 | 12.84 | 2.65 |  |
| QuEST ^$\dagger$ | 5120 | 6/6 | 0.7 | 284.9 | 6.83 | 11.93 | 2.65 |  |
| DilateQuant | 5120 | 6/6 | 0.7 | 284.9 | 5.33 | 11.61 | 2.66 |  |
| EDA-DM ^$\star$ | 5120 | 4/4 | 0.3 | 190.3 | N/A | N/A | N/A |  |
| EfficientDM ^$\dagger$ | 102.4K | 4/4 | 0.3 | 190.3 | 15.08 | 16.53 | 2.67 |  |
| QuEST ^$\dagger$ | 5120 | 4/4 | 0.3 | 190.3 | 13.03 | 19.50 | 2.63 |  |
| DilateQuant | 5120 | 4/4 | 0.3 | 190.3 | 10.10 | 16.22 | 2.62 |  |

## 5 Experiment

### 5.1 Experimental Setup

#### Models and metrics.

We evaluate DilateQuant for both unconditional and conditional image generation.
The comprehensive experiments include DDPM (Song et al., ), LDM (Rombach et al., ) and Stable-Diffusion on 5 datasets.
The performance of the quantized models is evaluated with Fréchet Inception Distance (FID) (Heusel et al., ), Sliding FID (sFID) (Salimans et al., ), and Inception Score (IS) (Salimans et al., ).
For text-guided generation, we add CLIP score (Hessel et al., ) to evaluate the semantic relevance of the generated images to the text prompts.
Following the common practice, the Stable-Diffusion generates 10,000 images, while all other models generate 50,000 images.
Besides, we also calculate the Bit Operations and Size of models to visualize the effects of model acceleration and compression.

#### Quantization and comparison settings.

We employ DilateQuant with the standard channel-wise quantization for weights and layer-wise quantization for activations.
To highlight the efficiency, DilateQuant selects 5120 samples for calibration and trains for 5K iterations with a batch size of 32, aligning with PTQ-based method (Liu et al., ).
The Adam (Kingma & Ba, ) optimizer is adopted, and the learning rates for quantization parameters and weights are set as 1e-4 and 1e-2, respectively.
For the experimental comparison, we compare DilateQuant with PTQ-based method (Liu et al., ) and variant QAT-based methods (He et al., ; Wang et al., ).
Since these two variant QAT-based methods employ non-standard settings, we modify them to standard settings for a fair comparison.
To further compare with them, we also employ the same non-standard settings on DilateQuant to conduct experiments in the Appendix LABEL:app:compare.
All experiments are performed on one RTX A6000.
The more detailed experimental implementations are showcased in Appendix B.

### 5.2 Main Result

#### Unconditional generation.

We start by comparing the quantization results for unconditional generation, as reported in Table 4.3.
We focus on the performance of low-bit quantization to highlight the advantages of DilateQuant.
In 4-bit quantization, previous works all suffer from non-trivial performance degradation. For instance, EDA-DM and QuEST become infeasible on LSUN-Bedroom, and EfficientDM remains far from practical usability on LSUN-Church.
In sharp contrast, DilateQuant achieves a substantial improvement in quantization performance, with encouraging 6.28 and 4.98 FID improvement over EfficientDM on two LSUN datasets, respectively.
Additionally, in 6-bit quantization, DilateQuant can achieve a fidelity comparable to that of the full-precision baseline.

#### Conditional generation.

The quantization results for conditional generation are reported in Table 4.3.
For text-guided generation with 6-bit precision, DilateQuant improves the FID to 24.69 with 5.3$\times$ Model size compression and 27.9$\times$ Bit Operations reduction, effectively advancing the low-latency applications of Stable-Diffusion in real-world scenarios.
Besides, DilateQuant achieves significant improvements at all bit-width settings on class-guided generation.

We add human preference assessments in Appendix I.

**Table 3: The efficacy of different component proposed in this paper.**
| Method | Prec. (W/A) | Time Cost (hours) | GPU Memory (MB) | FID $\downarrow$ | sFID $\downarrow$ | IS $\uparrow$ |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| WD | TPQ | BKD |  |  |  |  |  |  |
| ✗ | ✗ | ✗ | 4/4 | 0.97 | 3019 | 120.24 | 36.72 | 4.42 |
| ✓ | ✓ | ✗ | 4/4 | 1.08 | 3439 | 16.27 | 11.83 | 8.09 |
| ✗ | ✓ | ✓ | 4/4 | 0.98 | 3278 | 9.63 | 7.08 | 8.45 |
| ✓ | ✓ | ✓ | 4/4 | 1.08 | 3439 | 9.13 | 6.92 | 8.56 |

### 5.3 Ablation Study

The ablation experiments are conducted over DDIM on CIFAR-10 with 4-bit quantization.
We start by analysing the efficacy of each proposed component, as reported in Table 3.
We use the SoTA PTQ-based framework, EDA-DM (Liu et al., ), as the baseline, which fails to maintain performance, yielding an exceedingly high FID score of 120.24. This demonstrates that PTQ-based methods are not suitable for the low-bit quantization.
Since time-varying activations severely degrade performance, we utilize TPQ as the backbone of our approach.
By incorporating WD, we reduce the FID score to 16.27, demonstrating the effectiveness of TPQ and WD.
Introducing BKD further enhances performance while maintaining PTQ-like time costs and GPU memory during the training process.

**Table 4: Efficiency comparisons of various quantization frameworks.**
| Method | Calib. | Training Data | Time Cost (hours) | GPU Memory (MB) | FID $\downarrow$ |
| --- | --- | --- | --- | --- | --- |
| PTQ | 5120 | 0 | 0.97 | 3019 | 120.24 |
| QAT | - | 50K | 13.89 | 9974 | 7.30 |
| V-QAT | 1.6384M | 0 | 2.98 | 9546 | 81.27 |
| Ours | 5120 | 0 | 1.08 | 3439 | 9.13 |

We conduct the efficiency analysis of DilateQuant by comparing it with PTQ (Liu et al., ), QAT (Esser et al., ), and variant QAT (He et al., ) methods.
As reported in Table 4, the PTQ method fails to maintain performance and the QAT method requires significant resources.
In sharp contrast, DilateQuant does not rely on original data and uses a small calibration to achieve QAT-like accuracy (9.13 vs. 7.30) with PTQ-like time cost (1.08 vs. 0.97) and GPU memory (3439 vs. 3019).
The efficiency comparisons on other models are reported in Appendix LABEL:app:efficient.

We also add the ablation experiments of DilateQuant for time steps and samplers in Appendix C.

## 6 Conclusion

In this work, we propose DilateQuant, a novel quantization framework for diffusion models that offers comparable accuracy and high efficiency.
Specifically, we find the unsaturation property of the in-channel weights and exploit it to alleviate the wide range of activations.
By dilating the unsaturated channels to a constrained range, our method costlessly absorbs the activation quantization errors into weight quantization.
Furthermore, we design a flexible quantizer that sets time-step quantization parameters to time-varying activations and supports parallel quantization for training process, significantly improving the performance and reducing time cost.
We also introduce a novel knowledge distillation strategy to enhance performance, which aligns the quantized models with the full-precision models at a block level.
The simultaneous training of parameters and shorter backpropagation paths minimize the time and memory footprint required.
Exhaustive experiments demonstrate that DilateQuant significantly outperforms existing methods in low-bit quantization.

DilateQuant: Supplementary Materials

## Appendix A Supplementary material introduction

In this supplementary material, we present the correlative introductions and some experiments mentioned in the paper. The following items are provided:

- •
Detailed experimental implementations for all experiments in Appendix B.
- •
Robustness of DilateQuant for time steps and samplers in Appendix C.
- •
Efficiency comparisons of various quantization frameworks in Appendix LABEL:app:efficient
- •
Thorough comparison with EfficientDM and QuEST in Appendix LABEL:app:compare.
- •
Workflow and effects of Weight Dilation algorithm in Appendix LABEL:app:wd.
- •
Different equivalent scaling algorithms for diffusion models in Appendix G.
- •
Hardware-Friendly quantization in Appendix H.
- •
Human preference evaluation in Appendix I.

## Appendix B Detailed experimental implementations

In this section, we present detailed experimental implementations, including the pre-training models, qunatization settings, and evaluation.

The DDPM(^1^11[https://github.com/ermongroup/ddim](https://github.com/ermongroup/ddim)) models and LDM(^2^22[https://github.com/CompVis/latent-diffusion](https://github.com/CompVis/latent-diffusion)) models we used for the experiments are obtained from the official websites.
For text-guided generation with Stable-Diffusion, we use the CompVis codebase(^3^33[https://github.com/CompVis/stable-diffusion](https://github.com/CompVis/stable-diffusion)) and its v1.4 checkpoint.
The LDMs consist of a diffusion model and a decoder model.
Following the previous works (Liu et al., ; He et al., ; Wang et al., ), DilateQuant focus only on the diffusion models and does not quantize the decoder models.
We empoly channel-wise asymmetric quantization for weights and layer-wise asymmetric quantization for activations.
The input and output layers of models use a fixed 8-bit quantization, as it is a common practice. The weight and activation quantization ranges are initially determined by minimizing values error, and then optimized by our knowledge distillation strategy to align quantized models with full-precision models at block level.
Since the two compared methods employ non-standard settings, we modify them to standard settings for a fair comparison.
More specifically, we quantize all layers for EfficientDM, including Upsample, Skip_connection, and AttentionBlock’s qkvw, which lack quantization in open-source code(^4^44[https://github.com/ThisisBillhe/EfficientDM](https://github.com/ThisisBillhe/EfficientDM)).
However, when these layers, which are important for quantization, are added, the performance of EfficientDM degrades drastically.
To recover performance, we double the number of training iterations.
QuEST utilizes channel-wise quantization for activations at 4-bit precision in the code(^5^55[https://github.com/hatchetProject/QuEST](https://github.com/hatchetProject/QuEST)), which is not supported by hardware. Therefore, we adjust the quantization setting to layer-wise quantization for activations.
For experimental evaluation, we use open-source tool *pytorch-OpCounter*(^6^66[https://github.com/Lyken17/pytorch-OpCounter](https://github.com/Lyken17/pytorch-OpCounter)) to calculate the Size and Bops of models before and after quantization.
And following the quantization settings, we only calculate the diffusion model part, not the decoder and encoder parts.
We use the ADM’s TensorFlow evaluation suite *guided-diffusion*(^7^77[https://github.com/openai/guided-diffusion](https://github.com/openai/guided-diffusion)) to evaluate FID, sFID, and IS, and use the open-source code *clip-score*(^8^88[https://github.com/Taited/clip-score](https://github.com/Taited/clip-score)) to evaluate CLIP scores.
As the per practice (Liu et al., ; Wang et al., ), we employ the zero-shot approach to evaluate Stable-Diffusion on COCO-val for the text-guided experiments, resizing the generated 512 $\times$ 512 images and validation images in 300 $\times$ 300 with the center cropping to evaluate FID score and using text prompts from COCO-val to evaluate CLIP score.

## Appendix C Robustness of DilateQuant for time steps and samplers

To assess the robustness of DilateQuant for samplers, we conduct experiments over LDM-4 on ImageNet with three distant samplers, including DDIMsampler Song et al. (), PLMSsampler Liu et al. (), and DPMSolversampler Lu et al. ().
Given that time step is the most important hyperparameter for diffusion models, we also evaluate DilateQuant for models with different time steps, including 20 steps and 100 steps.
As shown in Table LABEL:tab:robust, our method showcases excellent robustness across different samplers and time steps, leading to significant performance enhancements compared to previous methods.
Specifically, our method outperforms the full-precision models in terms of FID and sFID at 6-bit quantization, and the advantages of our method are more pronounced compared to existing methods at the lower 4-bit quantization.

## Appendix G Different equivalent scaling algorithms for diffusion models

In this section, we start by analyzing the differences between LLMs and diffusion models in terms of the challenges of activation quantization.
As shown in Figure 2(b), the activation outliers of the diffusion models are present in all channels, unlike in LLMs where the activation outliers only exist in fixed channels.
Additionally, the range of activations for diffusion models is also larger than that of the LLMs.
Therefore, it is essential to scale the number of channels as much as possible for the diffusion models.
Some equivalent scaling algorithms are proposed to smooth out the activation outliers in LLMs, and these methods have achieved success.
SmoothQuant (Xiao et al., ) scales all channels using a hand-designed scaling factor. AWQ (Lin et al., ) only scales a few of channels based on the salient weight. OmniQuant (Shao et al., ) proposes a learnable equivalent transformation to optimize the scaling factors in a differentiable manner. DGQ (Zhang et al., ) devises a percentile scaling scheme to select the scaled channels and calculate the scaling factors.

Unfortunately, when we applied methods similar to these previous equivalent scaling algorithms to diffusion models, we find that none of them work.
Specifically, we employ these four methods for diffusion models as follows: (1) For the method similar to SmoothQuant, we scale all channels before quantization using a smoothing factor $\alpha=0.5$; (2) For the method similar to AWQ, we scale 1% of channels based on the salient weight, setting smoothing factor the same as SmoothQuant; (3) For the method similar to OmniQuant, we modify the scaling factors to be learnable variants and train them block by block with a learning rate of 1e-5; (4) For the method similar to DGQ, we scale the top 0.5% of quantization-sensitive channels, setting scaling factor based on the clipping threshold.
However, as shown in Table 9, all of these methods result in higher FID and sFID scores compared to no scaling.
The reason for this result is that although the range of activations decreases, the range of weights also increases significantly, making it more difficult for the model to converge during the training stage.
In contrast, the Weight Dilation algorithm we proposed scales the number of channels as much as possible. It searches for unsaturated in-channel weights and dilates them to a constrained range based on the max-min values of the out-channel weights.
The algorithm reduces the range of activations while maintaining the weights range unchanged. This effectively makes activation quantization easier and ensures model convergence, reducting the FID and sFID scores to 9.13 and 6.92, respectively.

**Table 9: The results of various equivalent scaling algorithms with W4A4 quantization for DDIM on CIFAR-10.**
| Methods | No scaling | SmoothQuant | OmniQuant | AWQ | DGQ | Ours |
| --- | --- | --- | --- | --- | --- | --- |
| proportion of $s>1$ | 0% | 100% | 100% | 1% | 0.5% | 39.2% |
| FID $\downarrow$ | 9.63 | 9.99 | 9.86 | 10.34 | 9.72 | 9.13 |
| sFID $\downarrow$ | 7.08 | 7.29 | 7.34 | 7.53 | 7.78 | 6.92 |
| IS $\uparrow$ | 8.45 | 8.46 | 8.50 | 8.38 | 8.52 | 8.56 |

## Appendix H Hardware-Friendly quantization

In this section, we investigate the correlation between quantization settings and hardware acceleration.
We start with the principle of quantization to achieve hardware acceleration.
A matrix-vector multiplication, $y=Wx+b$, is calculated by a neural network accelerator, which comprises two fundamental components: the processing elements $C_{n,m}$ and the accumulators $A_{n}$.
The calculation operation of accelerator is as follows: firstly, the bias values $b_{n}$ are loaded into accumulators; secondly, the weight values $W_{n,m}$ and the input values $x_{m}$ are loaded into $C_{n,m}$ and computed in a single cycle; finally, their results are added in the accumulators.
The overall operation is also referred to as Multiply-Accumulate (MAC):

$$ $\displaystyle A_{n}=\sum_{m}W_{n,m}x_{m}+b_{n}$ (16) $$

where $n$ and $m$ represent the out-channel and in-channel of the weights, respectively.
The pre-trained models are commonly trained using FP32 weights and activations.
In addition to MAC calculations, data needs to be transferred from memory to the processing units.
Both of them severely impact the speed of inference.
Quantization transforms floating-point parameters into fixed-point parameters, which not only reduces the amount of data transfer but also the size and energy consumption of the MAC operation.
This is because the cost of digital arithmetic typically scales linearly to quadratically with the number of bits, and fixed-point addition is more efficient than its floating-point counterpart.
Quantization approximates a floating-point tensor $\bm{x}$ as:

$$ $\displaystyle\hat{\bm{x}}=\Delta\cdot\bm{x}_{int}\approx\bm{x}$ (17) $$

Figure: Figure 5: A schematic of matrix-multiply logic in accelerator for quantized inference.
Refer to caption: x5.png

where $\bm{x}_{int}$ and $\hat{\bm{x}}$ are integer tensors and quantized tensors, respectively, and $\Delta$ is scale factor.
Quantization settings have different granularity levels. Figure 5 shows the accelerator operation after the introduction of quantization. If we set both activations and weights to be layer-wise quantization, the new MAC operation can be represented as:

$$ $\displaystyle\hat{A_{n}}$ $\displaystyle=\sum_{m}\hat{W}_{n,m}\hat{x}_{m}+b_{n}$ $\displaystyle=\sum_{m}(\Delta_{w}\hat{W}_{n,m}^{int})(\Delta_{x}\hat{x}_{m}^{ int})+b_{n}$ $\displaystyle=\Delta_{w}\Delta_{x}\sum_{m}\hat{W}_{n,m}^{int}\hat{x}_{m}^{int} +b_{n}$ (18) $$

where $\Delta_{w}$ and $\Delta_{x}$ are scale factors for weights and activations, respectively, $\hat{W}_{n,m}^{int}$ and $\hat{x}_{m}^{int}$ are integer values.
The bias is typically stored in higher bit-width (32-bits), so we ignore bias quantization for now.
As can be seen, this scheme factors out the scale factors from the summation and performs MAC operations in fixed-point format, which accelerates the calculation process.
The activations are quantized back to integer values $\hat{x}_{n}^{int}$ through a requantization step, which reduces data transfer and simplifies the operations of the next layer.

To approximate the operations of quantization to full-precision, channel-wise quantization for weights is widely used, which sets quantization parameters to each out-channel.
With this setting, the MAC operation in Eq. H can be represented as:

$$ $\displaystyle\hat{A_{n}}$ $\displaystyle=\sum_{m}(\Delta_{w_{n}}\hat{W}_{n,m}^{int})(\Delta_{x}\hat{x}_{m }^{int})+b_{n}$ $\displaystyle=\Delta_{w_{n}}\Delta_{x}\sum_{m}\hat{W}_{n,m}^{int}\hat{x}_{m}^{ int}+b_{n}$ (19) $$

where $\Delta_{w_{n}}$ is scale factor for the $n_{th}$ out-channel of weights.
However, the channel-wise quantization for activations sets quantization parameters to each in-channel.
This setting is hardly supported by hardware, as the MAC operation is performed as follows:

$$ $\displaystyle\hat{A_{n}}$ $\displaystyle=\sum_{m}(\Delta_{w}\hat{W}_{n,m}^{int})(\Delta_{x_{m}}\hat{x}_{m }^{int})+b_{n}$ $\displaystyle=\Delta_{w}\sum_{m}\Delta_{x_{m}}\hat{W}_{n,m}^{int}\hat{x}_{m}^{ int}+b_{n}$ (20) $$

where $\Delta_{x_{m}}$ is scale factor for the $m_{th}$ in-channel of activations.
Due to its inability to factor out the different scales from the accumulator summation, it is not hardware-friendly, leading to invalid acceleration.

## Appendix I Human preference evaluation

In this section, we use an open-source *aesthetic predictor*(^9^99[https://github.com/shunk031/simple-aesthetics-predictor](https://github.com/shunk031/simple-aesthetics-predictor)) to evaluate Aesthetic Score $\uparrow$, mimicking human preference assessment of the generated images.
As reported in Table 10, DilateQuant has a better aesthetic representation compared to EfficientDM, which demonstrates that the quantized models with our method are more aesthetically pleasing to humans.
For the large text-to-image model, we use the convincing DrawBench benchmark to evaluate human performance, as shown in Figure 6.
Additionally, we visualize the random samples of quantization results in Figure 7 (LSUN-church), 8 (LSUN-Bedroom), and 9 (ImageNet).
As can be seen, DilateQuant outperforms previous methods in terms of image quality, fidelity, and diversity.

**Table 10: Aesthetic assessment of the different quantized models with 4-bit quantization.**
| Method | LSUN-Bedroom | ImageNet | DrawBench |
| --- | --- | --- | --- |
| FP | 5.91 | 5.32 | 5.80 |
| EfficientDM | 5.47 | 3.51 | 2.84 |
| DilateQuant | 5.72 | 4.85 | 5.23 |

Figure: Figure 6: Random samples of different quantized models on DrawBench with 6-bit quantization.
Refer to caption: x6.png

Figure: Figure 7: Random samples of quantized models with DilateQuant on LSUN-Church.
Refer to caption: x7.png

Figure: Figure 8: Random samples of different quantized models on LSUN-Bedroom with 4-bit quantization.
Refer to caption: x8.png

Figure: Figure 9: Random samples of different quantized models on ImageNet with 4-bit quantization.
Refer to caption: x9.png

## Peer Reviews

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: 1) For the scaling from activation to weight idea, it has been proposed in the paper AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration. It would be good if can compare with this paper. 

2) For the ablation studies in table 3, it lacks of one combination: WD(Yes) TPQ(No) BKD(Yes). Besides, comparing last two rows, seems the module WD doesn't bring significant contributions. This makes me doubt about the contributions of both WD and TPQ to the final performance. I just doubt maybe the BKD contributes most and nearly all to the performance improvement. Can you conduct one more experiment on the combination:  WD(No) TPQ(No) BKD(Yes)? Thanks.

3) For table 4, can you compare with baselines mentioned in Table 2 for Efficiency comparisons? The baselines in Table 4 seems out-dated.
soundness: 3
strengths: The authors have conducted extensive experiments on diffusion model tasks. We can see the proposed methods performs consistently well as compared to SOTA baselines. Besides, they conducted ablation studies to check the impact of each component and do efficiency analysis.
summary: The authors proposed DilateQuant framework to quantize the diffiusion model. First, they propose Weight Dilation (WD) that maximally dilates the unsaturated in-channel weights to a constrained range through a mathematically equivalent scaling.  Then they propose  Temporal Parallel Quantizer (TPQ), sets time-step quantization parameters and supports parallel quantization for different time steps. Finally, they propose Block-wise Knowledge Distillation (BKD) to align the quantized models with the full-precision models at a block level.
weaknesses: 1) Ideas of scaling from activations to weights have been proposed in other papers. 

2) Abalation studies is not convincing, details can be seen in Questions section.

3) I am doubting whether the module of BKD can be applied to large-scale diffusion models without training. 

I will improve my score if the reviewers can address my concern.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: Please refer to the weakness.
soundness: 3
strengths: - The idea of the proposed Weight Dilation is novel and interesting.
- The paper is well-written and easy to understand.
summary: This paper proposes a QAT framework dubbed DilateQuant for diffusion models, which consists of three components, i.e, Weight Dilation (WD), Temporal Parallel Quantizer (TPQ), and Block-wise Knowledge Distillation (BKD). The core contribution of the paper is the WD, which reduces the difficulty of activation quantization via equivalent scaling like SmoothQuant, and constrains the scaling intensity based on the maximum values of the output channels.
weaknesses: - The contributions of TPQ and BKD appear to be modest, especially for BKD, which is well explored in the previous quantization and neural architecture search literature.
- According to the ablation study (Table. 3), most of the improvement owes to knowledge distillation (BKD). Adding the WD to the BKD only improves the FID from 9.63 to 9.13. So is the key component actually the BKD?
- The paper only validates the method on Unet-based models, however, given that DiT-based diffusion models (e.g., SD3, Pixart, FLUX) have predominated, the reviewer believes doing experiments on DiT-based models can enhance the paper. Besides, there are some papers adopting the equivalent scaling for DiT quantization[1][2], the proposed method might need to be compared with them to show the superiority of the proposed equivalent scaling method (WD).
- The reviewer is confused about the motivation of WD. The paper states that outliers exist in all channels, so equivalent scaling methods of LLMs are not suitable for DMs, which is also illustrated by Figure 2b. However, since there are “outliers” in all channels, can we still call this “outlier” an outlier? It seems that the difference in activation values of DMs across input channels is not much less salient than LLMs.

[1] PTQ4DiT: Post-training Quantization for Diffusion Transformers

[2] ViDiT-Q: Efficient and Accurate Quantization of Diffusion Transformers for Image and Video Generation

---

confidence: 4
contribution: 2
initial_rating: 3
presentation: 3
questions: 1. Please add the results of delta'x/delta x in Table 8
2. This work is strongly correlated with other equivalent scaling algorithms. Please add the results of W6A6 in Table 9 to demonstrate the generalization ability.
3. Please explain the difference between TPQ in this article and TLSQ in EfficientDM[1]. It seems that the training process is the only difference that each time step has different activation quantization parameters.
4. Please explain the block splitting method of BKD. In addition, please explain the difference between BKD and the related work BK-SDM[2].
5. If possible, please supplement the comparison results with PTQD[3] and TFMQ-DM[4].

[1] He Y, Liu J, Wu W, et al. Efficientdm: Efficient quantization-aware fine-tuning of low-bit diffusion models[J]. arXiv preprint arXiv:2310.03270, 2023.
[2] Kim B K, Song H K, Castells T, et al. Bk-sdm: A lightweight, fast, and cheap version of stable diffusion[J]. arXiv preprint arXiv:2305.15798, 2023.
[3] He Y, Liu L, Liu J, et al. Ptqd: Accurate post-training quantization for diffusion models[J]. Advances in Neural Information Processing Systems, 2024, 36.
[4] Huang Y, Gong R, Liu J, et al. Tfmq-dm: Temporal feature maintenance quantization for diffusion models[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024: 7362-7371.
soundness: 3
strengths: The most innovative point of this paper is the proposed Weight Dilation (WD) technology, which maximally expands the weight in the unsaturated channel to the restricted range by equivalent scaling mathematically. WD can absorb activation quantization error into weight quantization with no extra cost. The activation range is reduced, making it easy to quantify the activation. The weight range remains the same, which makes it easy for the model to converge during the training phase. DilateQuant is significantly superior to existing methods in terms of accuracy and efficiency.
summary: Quantization is one of the most important method to reduce the model size and improve the computing efficiency. This manuscript proposes a novel quantization framework (DilateQuant ) for diffusion models, which offers comparable accuracy and high efficiency. The main contribution of the manuscript is that it introduces Weight Dilation (WD) that costlessly absorbs the activation quantization errors into weight quantization. In addition, a Temporal Parallel Quantizer (TPQ) and a Block-wise Knowledge Distillation (BKD) are further proposed to improve the quantization quality. However, the main issue is that this paper innovation. It combines a few existing methods to achieve a good quantization effect.
weaknesses: 1. It can be seen from Table 3 that the biggest innovation in this paper, WD, has little improvement on performance.
2. The full text lacks innovation. For example, TPQ and BKD in this paper are not innovative.
3. Setting the scale factor for the out-channel of weights is not a new method, which has been widely used in the channel-wise quantization.
4. The paper lacks the effect of image generation at higher resolution (i.e., 1024x1024).

---

confidence: 4
contribution: 2
initial_rating: 3
presentation: 3
questions: 1. The formulations presented in Eqs. (10) and (11) may encounter issues if the denominator becomes zero, leading to undefined results. A safeguard or modification in these equations to handle zero denominators is necessary to ensure their correctness and stability under all conditions. For instance, adding a small epsilon value to the denominator could prevent such undefined outcomes.

2. The comparison of calibration sample numbers between DilateQuant and EfficientDM in Tables 1 and 2 seems inaccurate. EfficientDM operates as a data-free method that does not depend on traditional training data; instead, it employs random Gaussian noise to generate images for network-wide reconstruction. Therefore, the number of calibration samples for EfficientDM should be indicated as zero. It’s possible the authors intended to reference the number of training steps instead.

3. The ablation studies presented are inadequate. For instance, the performance impact of using Weight Dilation (WD) alone (without TPQ and BKD) is not clearly isolated in Table 3. It would be beneficial to provide specific results demonstrating the effect of implementing WD independently to better understand its individual contribution to the overall performance enhancements.
soundness: 3
strengths: The paper is written with exceptional clarity. The motivation is well-articulated and compelling. The proposed weight dilation introduces a novel technique for adjusting in-channel weights to minimize activation range while avoiding added complexities in weight quantization.
summary: The paper introduces DilateQuant, a quantization framework designed for diffusion models that enhances efficiency while preserving accuracy. It consists of three key components: Weight Dilation (WD), Temporal Parallel Quantizer (TPQ), and Block-wise Knowledge Distillation (BKD). WD scales activations to be more quantization-friendly without introducing additional quantization difficulty to weights. TPQ adapts to time-varying activations by setting specific quantization parameters for different temporal stages. BKD employs a block-level distillation strategy, transferring knowledge from a full-precision model to its quantized version, which reduces the backpropagation path and memory demands. Comprehensive experiments demonstrate that DilateQuant surpasses existing methods in reducing quantization errors and improving efficiency across diverse models and configurations.
weaknesses: 1. The proposed weight dilation method successfully maintains the max-min values of each out-channel weight unchanged, thereby avoiding additional quantization difficulties. However, there are potential limitations to consider. For example, if the index of an activation outlier coincides with the in-channel index corresponding to these max-min values, the method might struggle to effectively manage these outliers. In such scenarios, the proposed technique could encounter challenges in achieving its intended quantization effectiveness. Could the authors provide an analysis of how frequently activation outliers coincide with max-min weight indices in diffusion models?

2. The constraint in the proposed weight dilation method, which aims to maintain the max-min values of each out-channel weight unchanged, might be overly restrictive. A slight adjustment of these max-min values could potentially introduce minimal quantization errors while providing a more flexible approach to manage the issues noted in the previous point. This adjustment could enable the method to handle cases where the activation outlier index coincides with the in-channel index of max-min values. For instance, considering both the activation quantization error and weight error simultaneously could help determine the most effective channel-wise scaling.

3. The proposed weight dilation method calculates channel-wise scaling based solely on the magnitude of weights, without considering activations. This approach might not sufficiently mitigate extreme activation outliers. A more comprehensive strategy that takes into account both the magnitudes of activations and weights could potentially offer a more balanced and effective solution for managing these outliers. Could the authors provide a comparison of the proposed weight dilation with existing methods [1][2], which jointly consider the magnitudes of activations and weights to determine channel-wise scaling?

4. The novelty and contributions of the proposed Temporal Parallel Quantizer (TPQ) and Block-wise Knowledge Distillation (BKD) appear to be limited. Specifically, the TPQ’s methodology of assigning unique quantization parameters for each time step has already been addressed in prior works [3][4]. Furthermore, the use of block-wise reconstruction in BKD, although effective, follows a well-known path in the post-training quantization (PTQ) of diffusion models as in [3], [5], and [6]. The primary distinction lies in BKD’s approach: instead of relying solely on calibration, it updates weight parameters and quantization parameters through gradient-based methods.

Reference:

[1] SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models. ICML 2023.

[2] Outlier Suppression: Pushing the Limit of Low-bit Transformer Language Models. NeurIPS 2022.

[3] TFMQ-DM: Temporal Feature Maintenance Quantization for Diffusion Models. CVPR 2024.

[4] EfficientDM: Efficient Quantization-Aware Fine-Tuning of Low-Bit Diffusion Models. ICLR 2024.

[5] Post-training Quantization on Diffusion Models. CVPR 2023.

[6] Q-Diffusion: Quantizing Diffusion Models. ICCV 2023.

Meta-review: The paper presents DilateQuant, a quantization framework for diffusion models featuring three key components: Weight Dilation (WD), Temporal Parallel Quantizer (TPQ), and Block-wise Knowledge Distillation (BKD). The overall objective is to improve quantization efficiency while maintaining high accuracy. The paper is well-structured and easy to follow. However, there are significant concerns about the novelty, technical contributions, and experimental validations. For example, the WD approach is closely aligned with existing methods like SmoothQuant and channel-wise scaling techniques. TPQ and BKD are also incremental contributions that follow established techniques in time-step-specific quantization and block-wise distillation. For these limitations, I would like to recommend rejecting this paper.

Decision: Reject