Q-DM: An Efficient Low-bit Quantized Diffusion Model
Authors: Yanjing Li
Sheng Xu
Xianbin Cao
Xiao Sun
Baochang Zhang
Venue: NeurIPS 2023 poster (2023)
Keywords: network quantization
diffusion model
image synthesize

Abstract: Denoising diffusion generative models are capable of generating high-quality data, but suffers from the computation-costly generation process, due to a iterative noise estimation using full-precision  networks. As an intuitive solution, quantization can significantly reduce the computational and memory consumption by low-bit parameters and operations. However, low-bit noise estimation networks in diffusion models (DMs) remain unexplored yet and perform much worse than the full-precision counterparts as observed in our experimental studies. In this paper, we first identify that the bottlenecks of low-bit quantized DMs come from a large distribution oscillation  on activations  and accumulated quantization error caused by the multi-step denoising process. To address these issues, we first develop a Timestep-aware Quantization (TaQ) method and a Noise-estimating Mimicking (NeM) scheme for low-bit quantized DMs (Q-DM) to effectively eliminate such oscillation and accumulated error respectively, leading to well-performed low-bit DMs. In this way, we propose an efficient Q-DM to calculate low-bit DMs by considering both training and inference process in the same framework. We evaluate our methods on popular DDPM and DDIM models. Extensive experimental results show that our method achieves a much better performance than the prior arts. For example, the 4-bit Q-DM theoretically accelerates the 1000-step DDPM by 7.8x and achieves a FID score of 5.17, on the unconditional CIFAR-10 dataset.

Q-DM: An Efficient Low-bit Quantized Diffusion Model
Yanjing Li Sheng Xult, Xianbin Cao' Xiao Sun?", Baochang Zhang!
'Beihang University
SShanghai Artificial Intelligence Laboratory
"Zhongguancun Laboratory
Nanchang Institute of Technology
Lyanjingli, shengxu) Obuaa. edu. cn
Abstract
Denoising diffusion generative models are capable of generating high-quality data, but suffers from the computation-con costll generation process, due to iterative noise estimation using full-precision networks. As an intuitive solution, quantization can significantly reduce the computational and memory consumption by low- bit parameters and operations. However, low-bit noise estimation networks in diffusion models (DMs) remain unexplored yet and perform much worse than the full-pre aillprrcissoon counterparts observed in experimental studies. In this paper, we first identify that the bottlenecks low-bit quantized DMs come from a large distribution oscillation activations and accumulated quantization error caused by the multi-step denoising process. To address these issues, we first develop a Timestep-awar Quantization (TaQ) method and Noise-estimating Mimicking (NeM) scheme for low-bit quantized DMs (Q-DM) to effectively eliminate such oscillation and accumulated error respectively, leading
well-performed low-bit DMs. In this way. we propose an efficient Q-DM to calculate low-bit DMs by considering both training and inference process in the same framework. We evaluate our methods on popular DDPM and DDIM models. Extensive experimental results show that our method achieves much better performance than the prior arts. For example, the 4-bit Q-DM theoretically accelerates the 1000-step DDPM by 7.8x and achieves FID score of 5.17, on the unconditional CIFAR-10 dataset.
Introduction
Denoising diffusion models, also known score-based generative models 10
33 35
have recently
shown remarkable success various generative tasks such as images [10
35
22 audio (21),
video (3I] and graphs [23]. These models have also demonstrated flexibility in downstream tasks, making them attractive for tasks such as super-resolution (26 7 and image-to-image translation [29].
Compared to Generative Adversarial Networks (GANs) [8] historically considered state-of-the- art, diffusion models have proven to be superior in terms of quality and diversity in most these tasks and applications. The process of diffusion models involves gradually transforming real data into Gaussian noise, which is then reversed via a denoising process to generate real data [10] However, such denoising process is time-consuming and involves iterating neural network for noise estimation over thousands of timesteps, despite producing a significant amount of images. Therefore, researchers actively working on accelerating this generation process reduce its long iterative process and high inference cost for sample generation. To achieve this, one pipeline is to
Equal contribution. Corresponding author. { This work was done during her internship at Shanghai Artificial Intelligence Laboratory.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Channol
700
Figure 1: Studies on the activation distribution w.t. time-step. Per (output) channel activation ranges of the first attention block in diffusion model on different timestep. The boxplot visualizes key statistical measures for each channel, including the minimum and maximum values, the 2nd and 3rd quartiles, and the median.
focus on sample trajectory learning, develop faster sampling strategies 28 22
While the
other pipeline directly compresses and accelerates the noise estimation networks based on network quantization technology [30], which is particularly suitable for AI chips because of the low-bit parameters and operations. Prior post-training quantization (PTQ) methods [30] 19 17] on diffusion models (DMs) or other neural networks directly compute quantized parameters based on pre-trained full-precision models, which constrains the model performance to sub-optimized level without fine-tuning. Furthermore, quantizing DMs based on PTQ methods to ultra-low bits (e.g., bits or lower) is ineffective and suffers from significant performance reduction.
Differently, quantization-aware training (QAT) 16 [8] methods perform quantization during back propagation and generally achieve less performance drop with a higher compression rate than PTQ. For instance, QAT has been shown to be effective for CNNs 18] and ViT's [16] and BERT 24 However, QAT methods for low-bit quantization of diffusion models remain largely unexplored. Therefore, we first build low-bit quantized DM baseline, straightforward yet effective solution based on common techniques [5]. Our experimental studies reveal that the severe performance drop of low-bit quantized DMs, such PTQ 30 and baseline [5], lies in the activation distribution oscillation and quantization error accumulation caused by the denoising process.
As shown in Fig.D. the output distribution of the noise estimation network each time step can differ significantly, resulting in activation distribution oscillation. Particularly, the distribution of activation in a specifie layer varies significantly across different timesteps during training. We also observe that errors between full-precision activations and quantized activations gradually accumulate across timesteps during the sampling process (inference), making it harder to produce well-performed quantized DMs.
Drawing on the aforementioned insights, propose Timestep-awar Quantization (TaQ) method to address the oscillating distribution issue. By smoothing out these fluctuations and introducing more precise scaling factors into activations, quantized DMs. We further design a new training scheme for quantized DMs, dubbed Noise- estimating Mimicking (NeM), which can reduce the accumulated errors and promote the performance of quantized DMs based on the knowledge of full-precision counterparts. In this way, we achieve a new QAT method for low-bit quantized DM (Q-DM) via incorporating all the explorations (see the overview in Fig. 2 Overall, the contributions of this paper can be summarized as follows:
effectively enhance the performance of the low-bit
To the best of our knowledge, we proposed the first QAT method towards efficient low-bit DMs, dubbed Q-DM, by fully considering both training and inference process in the same framework.
We introduce a Timestep-aware Quantization (TaQ) method to mitigate activation distribution oscillation caused by the random-sampled timestep in the training process. We develop Noise-estimating Mimicking (NeM) scheme to reduce accumulated errors, by which the Q-DMs are able to achieve comparable performance the full-precision counterparts.

---

Q-AttnBlock in Q-DM
Learning Q-DM with Noise-estimating Mimicking
Input activation
U-Net
NoM
Comaz
Connaa
Q Com22
Bot
quary Rg
key au
volue
Timeskop- aware Quantizalion on
Timelles- aware Quantiinaion
aware Quandoation
3,
Sample
Concat
Conv
Q-ResBlock
Attention score
Quardoo
Mett: M.ltpication
Q-UpSampll
Q-DownSample
Fooward paok
0
Superdtion 9066
Figure Overview of the proposed Q-DM framework. We introduce the timestep-aware quantization in an architecture perspective and noise imitation training scheme incorporated in the optimization process. From left to right, we respectively show the detailed architecture of single Q-AttBllock in Q-DM and the training framework of Q-DM.
Extensive experiments on the CIFAR-10 and ImageNet datasets show that our Q-DM outperforms the baseline and 8-bit PTQ method by a large margin, and achieves comparable performances the full-precision counterparts with considerable acceleration rate.
2
Related Work
Quantizing neural networks (QNNs) often possess low-bit 4-bit) weights and activations to accelerate the model inference and save the memory usage. Specifically, ternary weights are introduced to reduce the quantization error TWN [[S]. DoReFa-Net 41 exploits convolution kernels with low bit-width parameters and gradients to accelerate both the training and inference. TTQ [42 uses two full-precision scaling coefficients to quantize the weights to ternary present 2 ~ 4-bit quantization scheme using two-stage approach
Network Quantization.
values. Zhuang et al. 45 to alternately quantize the weights and activations, which provides an optimal trade-off among memory, efficiency, and performance. Jung al. (12) parameterize the quantization intervals and obtain their optimal values by directly minimizing the task loss of the network and also the accuracy degeneration with further bit-width reduction. ZeroQ [2] supports both uniform and mixed-precision quantization by optimizing for a distilled dataset. which is engineered to match the statistics of batch normalization across different layers the network. Xie al. (39) introduces transfer learning into network quantization to obtain an accurate low- wow-precision model by utilizing the Dallback-Leibler (KL) divergence. PWLQ 6 enables accurate approximation for tensor values that have bell-shaped distributions with long tails and finds the entire range by minimizing the quantization error.
Diffusion Model. The high cost of denoising through networks and the long iterative process make it difficult to implement diffusion models widely. To accelerate diffusion probabilistic models (DMs) 10], previous research has focused finding shorter sampling trajectories while maintaining DM performance. Wavegrad (3] introduces grid search, which finds an effective trajectory with only six timesteps, but this approach cannot be generalized for longer trajectories due its exponentially growing time complexity. Watson et al. (38) model the trajectory searching as dynamic programming problem. Song al. objective, but whose reverse process can be much faster to sample from. For DMs with continuous
34 construct Mon-Markovian diffusion processes that lead the same training
timesteps, Song et al. [33 35) have formulated the DM the form of an ordinary differential
equation (ODE) and improved sampling efficiency by using faster ODE solvers. Jolicoeur- JolicoerrMaartneauu al. [II] have introduced an advanced SDE solver to accelerate larger sampling rate. Analytic-dpm [ has estimated variance and KI. divergence using the Monte Carlo method and a pretrained score-based model with derived analytic forms that are simplified from the ccore-functio In addition to those training- free methods, Luhman & Luhman (20] have
reverse process via an adaptively

---

(xi 1 04 EQ,1), compressed the reverse denoising process into single-step model, while San-Roman et al. [28 has dynamically adjusted the trajectory during inference. However, implementing these methods requires additional training after obtaining pretrained DM, which makes them less desirable in most situations. In summary, all these DM acceleration methods can be categorized as finding effective sampling trajectories.
Unlike prior works, we demonstrate that diffusion models can be accelerated by compressing the network in each noise estimating iteration, which is orthogonal with the fast sampling methods mentioned above. To the best of knowledge, this is the first study to explore low-bit quantized diffusion models in a quantization-awar aware training (QAT) manner.
Background and Challenge
3.1 Diffusion Models
Forward process. Let X be sample from the data distribution x, q(x). A forward diffusion process adds Gaussian noise to the sample for T'' times, resulting in sequence of noisy samples XI, XT XT as:
q(x,|x,-1) - 11 1,,,,,8,1,,88 Br1),
where B, E 1) is the variance schedule and controls the strength of the Gaussian noise in each step. The forward diffusion process satisfies the Markov property since each step relies solely on the preceding step. Additionally, as the number of steps increases towards infinity (T + oo), the final state XT converges to an isotropic Gaussian distribution. notable property of the forward process that it admits sampling x, at an arbitrary timestep t in closed form as:
((x:) xo) = (x4; Vo, (1 āt I.).
(2)
Reverse process. To generate sample from Gaussian noise input Xy ~N(0,I) using diffusion models, the forward process is reversed. However, since the actual reverse conditional distribution qxx,11 (xx) is unknown, diffusion models learned conditional distribution P /x,) that ap- proximates the real reverse conditional distribution with a Gaussian distribution. This approximation expressed
pe(x (x),
By using the re-parameterization trick presented in [10] becomes possible to derive the mean (x,) and 3,1 as follows:
(x11 1
Var V1 di
(4)
3. 1 - ái-1 Br,
1 Tee
where a₂ = 1 Br, a. = II a and co a function approximator intended to predict e from X. [10]
Training. At training time, the goal of optimization to minimize the negative log-likelihood, iee,
po (xo). With variational inference, lower bound of it could be found, denoted as LVLB:
q(x1:7 (x)) Pe (Xo.T)
LVLB = Egtxor) [log
2 log pe (xo)-
(5)
It is found in 10 that using a simplified loss function to TVLB often obtains hetter performance:
Liimpl Ex.xo.e. [||e: - co(Va,x + vI a,E,, ]
(6)
Sampling. At inference time, Gaussian noise tensor XT is sampled and is denoised by repeatedly sampling the reverse distribution po (xxx11/xxx xx)) ia,1 (x1) ) is taken as the final generation result, with no noise added in the final denoising step.

---

MSE with full- aull-preciiion aclivalions
(a)
(b)
50
40
30
10
[6] 700
[d) 1000
Sampling Timestep (t)
Figure 3: Input activation distribution of the first Q-AttnBlock in diffusion model differ- ent timestep with a model trained on CIFAR- 10 13 by DDPM.
Figure 4: Distance between the outputs of the full-precision model and different bit-width baseline models trained on CIFAR-10 [33 by DDIM with 100 sampling steps.
3.2 Quantization
In=1 and input feature map set as = {a" NN In= The w" R XC x x and a E RC n x W x H. are the convolutional weight and the input feature map in the n-th layer, where Cm Cuut and K respectively stand for input channel number, output channel number and the kernel size. Also, W and H are the width and height of the feature maps. Then, the convolutional outputs aut can be technically formulated as:
Given an N-layer CNN model, we denote its weight set as W = [w")]N
Lott w
a
where @ represents the convolution operation. Herein, we omit the non-linear function for simplicity. Quantized neural network intends to represent w and a" in a low-bit format such that the float-point convolutional outputs can be approximated
w
Q((w))
[clip( /8"" -220-1,2" 1)1
à 2h Q( "in cclip(a /s "in 2 2-- 1)]
(8)
aous = A & w" in [Q(w") o Q(a
where denotes the channel-wise multiplication, @ denotes the efficient GEMM operations, and gte
{ $2 sw=" E RR Cou is known as the channel-wise scaling factor vector [25] to
mitigate the output gap between Eq. 7 and its approximation of Eq. (8). Meanwhile, we use the layer-wise quantization for input activations and the scaling factor of activations 8tth E R is scalar.
3.3 Challenge Analysis
Here identify two major challenges low-bit DMs, specific to the multi-step inference process and randon-sampled-ster training process of diffusion models. Namely, we investigate on the distribution oscillation of the activations, and the accumulated quantization error resulted from the multi-step denoising process.
Activation distribution oscillation. To understand the distribution change of diffusion models, investigate the activation distribution, w.r.f. timestep in the training process. Theoretically. the distribution changes w.E.f timestep. it would be difficult to implement previous QAT methods. We analyze the overall activation distributions the noise estimation network, as shown in Fig. We can observe that different timesteps, the corresponding activation distributions have large discrepancies, e.g., Fig.B v.s. Fig b)),, which makes previous QAT methods [16] in-applicable for multi multi-timestep models, Le., diffusion models.
Quantization error accumulation. Quantization of noise estimation network introduces distur- bances to the weights and activations, resulting in errors in each layer's output. Previous studies 4
5

---

have found that these errors tend to accumulate across layers, making it more challenging to quantize deeper neural networks. In the case of diffusion models (DMs), each time step t, the input of the model (x--1) is obtained from the model's output at the previous time step 1, i.e, X₁. As depicted Fig. 4 the MSE distance, representing the quantization error of low-bit quantized DMs, exhibits noticeable growth along with the decrease of sampling timestep. This implies that as the denois- ing process moves towards later timestep, the accumulation of quantization errors becomes more prominent.
4
The Proposed Q-DM
4.1 Timestep-aware Quantization
To tackle the distribution oscillation in the training process, we first introduce the quantized attention block that efficiently takes into account the timestep. This structure allows for the numerical analysis of activation ranges across different timesteps and mitigates distribution oscillation of low-bit quantized DMs. We recall the quantization in the attention block based on Eq. (8), which is formulated as:
(x(x,, (0.()) Q(a, t)), ân (x,, (x.(xx)) ((aa((xi,)))
A (x: t) s0ftmax((a 1) an (xe, (xx,))))/V[))
Á t) sA(x)..) Q(A(x), t)),
Aout (xe, A(xx,)) (x,,
where the attention score.
In the i-th mini-batch, the timestep is represented as {4,,-.- 4, J. where b; is the batch size of the i-th batch. We denote E (1..... B), and B is the number of batch. Therefore, we calculate the timestep-aware distribution divergence for the query activation a, as:
B
b,
Nazt
ag (xt,: (3):
i=1
(10)
B
b1
2
ag
(Xt, (T)
b;
1=1
j=1
where Yg and oe are statistical mean and variance of query activation ag. And the calculation of the key activation ag is likewise.
Based such statistical results, the query and key activations in each specific timestep are smoothed as:
(x,, t) = (x,, t) - /q:el/ 00 + 1b á (x., t) (x,() / Voi + tt,
(11)
where y is constant to avoid 0 denominator. With the above timestep-aware smoothing process, we formulate our timestep-awan quantization as:
á,(X,, t) (x..))) TaQ(a, t)), å (x,,,) sox(x)))) TaQ(ax(x;.))) t))
A A(x,,1) 1) = softmax[(
ân (x1, (x), //vdl.
(12)
A(x;, t) ((x()))) TaQ(A(Xx,,))) t)),
aout (xe, t) = A(xx,t)) a (Xc, 8)
in which TaQ(*) [clip([* /s V Vo + v/). -2-11220-1 1)]. The smoothed activations
are less sensitive to the random sampled timestep the trianing process and the timestep-aware quantization, to some extent, dismisses the distribution oscillation phenomenon.

---

Table 1: Evaluating the components of Q-DM based on 50-step DDIM sampler with 32 x2 generating resolution on CI 13 "#Bits" denotes bit-width of weights and activations
Method
#Bits
FID
1st
#Bits
FID 4
ISt
#Bits
FID
1st
Full-precision
32-32
4.67
9.27
PTQ4DM
8-8
18.02
8.87
Baseline (LSQ (51)
4-4
10.22
8,91
3-3
13.24
8.88
2-2
18.74
8.65
+TaQ
4-4
9.25
8.95
3-3
11.19
8.91
2-2
16.83
8.71
+NeM
4-4
8.98
8.92
3-3
11.02
8.90
2-2
16.97
8.79
+TaQ+NeM
4-4
6.89
8.96
3-3
9.07
8.98
2-2
15.26
8.86
(Q-DM)
4.2 Noise-estimating Mimicking
To mitigate the negative impact of quantization error accumulation on the training of a quantized DM 8 a full-precision DM, denoted as gFP, is incorporated into the training process to facil- itate the learning objective. Following [10], with poo (x--1/x; = ( 3,I) and
Porr (xx)) = (x MOPe, (xx), B.I). we can write:
1 = E,, 233
|| papr (x,, 1) Mog (Xr, (xr, +]]
(13)
where C is constant that does not depend 0 BFP. As in Eq. 13) aim compel the quantized model to replicate the noise estimation capability of the full-precisin model. Further, by pe-aammtttrriingg Eq. (2) as x, (x0, E)
Xo v1 diE for (o and following the formulation in 10], which utilizes the formula for the posterior the forward process, we can derive that:
L--1-C C Ex.,
MgFF (x: e), t) paga (x),))))
(14)
2B,
where MaEF (xx E), 6) and par (x, <), t) parameterized as:
MgrF (Xe, 1) = (Xt, 1
B EgPp (x1,()), ()),
[xt v1 GrEger (x:)))
-
(x:
Var
vI az
(15)
1
Br
Mgq (Xt, [Xe,
[x:
V1 Egg (x,)))
(x:
Ego (x11())
04
VI
at
where d and gfP are the noise estimated by the quantized DM and full full-preciison counterpart. In Eq. [15 Eg is a function approximator intended to predict from x,. Therefore, Eq. (14) is simplified to:
3 28, - āe)
L--1 = Ex,
||Corr (opr((aag (Va,Xo + VI āic, f) Cga (oolVoix + vI a.c, 1)
(16)
With the aforementioned derivation and parameterization, we have the final objective of our noise- estimating imitation, which is formulated as:
arg min @
INOM( OFP)
(17)
== E,
(V + vI- -ā,,tt) cga(Va + vI &,,,)
5
Experiments
In this section, we evaluate the proposed Q-DM framework on several popular diffusion models (i.e. DDPM [0] and DDIM (2)) for unconditional image generation. To best of our knowledge, there is no published work done on low-bit quantized diffusion models at this point, we report I.SQ (5) as a baseline. Experiments show our approach can achieve competitive generation quality to the full-precision scenario all experimental settings under low-bit quantization.
7

---

Table 2: Experiment on 2/3/4-bit quantized diffusion models generating CIFAR-10 [S] ImageNet [14] image. "#Bits" denotes the bit-width of weights/activations. "Reso." represents the generating resolution.
image
Dataset
Model & Reso.
Step
Method
#Bits
(in)
(P))
FID L
IS 1
Full-precision
32/32
4.47
390.4
4.67
9.27
PTQ4DM [6]
8/8
1.12
99.5
18.02
8.87
Baseline
4/4
0,56
49.9
10.22
8.91
CIFAR-10 32x32
Q-DM
4/4
0.56
49.9
6.89
8.96
DDIM
Baseline
3/3
0.28
25.I
13.24
8.88
Q-DM
3/3
0.28
25.1
9.07
8.98
Baseline
2/2
0.14
12.6
18.74
8.65
Q-DM
2/2
0.14
12.6
15.26
8.86
Full-precision
32/32
4.47
780.7
4.16
9.32
PTQ4DM 66
8/8
1.12
199.0
14.18
9.31
Baseline
4/4
0.56
99.8
9.02
8.95
CIFAR-10 32x32
Q-DM
4/4
0.56
99.8
5.12
9.21
DDIM
100
Baseline
3/3
0.28
50.1
12.24
8.90
Q-DM
3/3
0.28
50.1
8.12
8.94
Baseline
2/2
0.14
25.2
16.99
8.74
Q-DM
2/2
0.14
25.2
14.31
8.77
Full-precision
32/32
4.47
7807.2
3.17
9.46
PTQ4DM 6
8/8
1.12
1990.0
7.10
9.55
Baseline
4/4
0.56
997.7
9.11
8.96
CIFAR-10 32x32
Q-DM
4/4
0.56
997.7
5.17
9.15
DDPM
1000
Baseline
3/3
0.28
501.0
12.28
8.91
Q-DM
3/3
0.28
501.0
8.14
8.93
Baseline
2/2
0.14
232.0
16.93
8.72
Q-DM
2/2
0.14
252.0
14.35
8.76
Full-precision
32/32
4.47
390.4
20.57
15.72
PTQ4DM 6
8/8
1.12
99.5
25.87
14.99
Baseline
4/4
0.56
49.9
24.78
15.37
ImageNet 64x64
Q-DM
4/4
0.56
49.9
20.02
15.68
DDIM
50
Baseline
3/3
0.28
25.1
26.35
15.24
Q-DM
3/3
0.28
25.1
22.19
15.32
Baseline
2/2
0.14
12.6
32,43
14.66
Q-DM
2/2
0.14
12.6
28.42
15.03
Full-precision
32/32
4.47
780.7
19.70
15.98
FTQ4DM (6]
8/8
1.12
199.0
24.92
15.52
Baseline
4/4
0.56
99.8
24.46
15.51
ImageNet 64x64
Q-DM
4/4
0.56
99.8
19.56
15.92
DDIM
100
Baseline
3/3
0.28
50.1
26.23
15.42
Q-DM
3/3
0.28
50.1
21.97
15.92
Baseline
2/2
0.14
25.2
31.19
14.89
Q-DM
2/2
0.14
25.2
27.94
14.99
Full-precision
32/32
4.47
7807.2
18.98
16.63
PTQ4DM (6]
8/8
1.12
1990.0
22.32
15.31
Baseline
4/4
0.56
997.7
22.91
15.29
ImageNet 64x64
Q-DM
4/4
0.56
997.7
18.52
16.72
DDPM
1000
Baseline
3/3
0.28
501.0
24.75
15.17
Q-DM
3/3
0.28
501.0
20.21
16.17
Baseline
2/2
0.14
252.0
29.33
14.87
Q-DM
2/2
0.14
252.0
25.62
15.48

---

5.1 Datasets and Implementation Details
We evaluate our method on two datasets including 32x32 generating size in CIFAR-I0 and
x4 generating size in ImageNet [14]. For the CIFAR-I0 [133 and ImageNet [14 datasets, use the DDIM [32 sampler with 50/100 sampling timesteps and DDPM [0] with 1000 sampling timesteps. All the training settings the same DDPM [T]] For DDIM sampler, we 7 in
DDIM 132 as 0.5 for the best performance. We evaluate the performance of our method using FID 9 and Inception Score (IS) 2 on both CIFAR-10 [33 and ImageNet [4 datasets. We set the training timestep T = 1000 for all experiments, following 10 We set the forward process variances constants increasing linearly from 3 le- to 3T 0.02. To represent the reverse process, we use a U-Net backbone, following 10 32 Parameters are shared across time, which is specified to the network using the Transformer sinusoidal position embedding (36). We self-attention the 16x 16 feature map resolution 33 37.
5.2 Ablation Study
We give quantitative results of the proposed TaQ and NeM in Tab. As can be seen, the low-bit quantized DM baseline [5] suffers a severe performance drop on image generation task compared with full-precision DMs (5.55, 8.57, and 14.07 performance gap in terms of FID score with 4/t/2-bit, respectively). TaQ and NeM improve the performance of generation when used alone. For example, the 4-bit quantized DM basline with TaQ and NeM introduced separately achieves 0.97 and 1.24 FID score decrease, respectively.
Moreover, the two techniques further boost the performance considerably when combined together. For instance, when combining the TaQ and NeM together, the performance of 4/3/2-bit quantized DMs improvement achieves 3.33, 4.07, and 3.48 respectively. To conclude, the two techniques can promote each other to improve Q-DM and close the performance gap between low-bit quantized DMs and full-precision counterpart.
5.3
Main Results
The experimental results are shown in Tab. 2 We compare our method with 4/3/2-bit baseline based on the same frameworks for the task of unconditional image generation with the CIFAR-10 [3 and ImageNet [4] dataset. We also report the classification performance the 8-bit PTQ method,
5
i.e. PTQ4DM 30) We firstly evaluate the proposed method on CIAFR-10 [33 with DDIM [32 and
DDPM 10. We use the model size and OPs (defined in 18] to evaluate the efficiency of quantized
and full-precision models.
For 50-step DDIM sampler, compared with 8-bit PTQ4DM [30], our 4-bit Q-DM achieves much larger compression ratio than 8-bit PTQ4DM, but with significant performance improvement (6.89 FID 4 vs. 18.02 FID 4). And it is worth noting that the proposed 2-bit model significantly compresses the DDIM by 30.9x x on OPs. The proposed method boosts the performance of 4/3/2-bir bit Baseline by 3.33, 4.17, and 3.48 in terms of FID score with the same architecture and bit- bittwidth, which is significant on the CIFAR-10 [3] dataset with 32x32 generating resolution. For 1000-step DDPM, the performance of the proposed method outperforms the 4/i/2-bit Baseline by 3.94, 4. 14, and 2.58, a large margin. Also note that the proposed 4/3/2-bit model significantly accelerates the generation by 7.8x, 15.6x, and 30.9x on OPs. Compared with 8-bit PTQ4DM, ours achieve significantly higher compression and acceleration rate, while the performance improvement is considerable.
Also, our method generates convincing results on ImageNet 14 dataset. As shown in Tab. 2 the performance of the proposed method with 50-step DDIM significantly outperforms the 4/3/2-bit Baseline method by 4.76. 4.16, and 4.01. Compared with bib PTQ method, our method achieves significantly higher compression rate and acceleration rate, but with better performance. For 1000- step DDPM on ImageNet [14] dataset, the performance of the proposed method outperforms the 4/3/2-bit Baseline by 4.39, 4.54, and 3.71. Also note that our 4-bit Q-DM surpasses the full-precision 50/100-step DDIM and 1000-step DDPM and significantly compresses the noise estimation networks by 7.9x which demonstrates the effectiveness and efficiency of our Q-DM.

---

Conclusion
In this paper, we present Q-DM, an efficient low-bit quantized diffusion model that offers a high compression ratio and competitive performance in image generation task. Initially, we analyze the challenges of the low-bit quantized DM. Our empirical analysis show that distribution oscillation in activation is the one of the cause of the significant drop in DM quantization. Another challenge lies in the accumulated quantization error resulted from the multi-step denoising process during inference. To address these issues, we first develop a timestep-awand quantization (TaQ) method and noise-estimating mimicking (NeM) scheme for low-bit quantized DMs, to effectively address these two challenges. Our work provides comprehensive analysis and effective solutions for the crucial issues low-bit quantized diffusion model, paving the way for the extreme compression and acceleration of diffusion model.
7
Acknowledgement
This work was supported in part by the National Natural Science Foundation of China under Grant 62076016, under Grant 61827901, "One Thousand Plan" projects in Jiangxi Province Asg2003312228 Foundation of China Energy Project GJNY-19-90, the National Key R&D Program of China (N0.2022ZD0160100))
References
[1] Fan Bao, Chongxuan Li, Jun Zhu, and Bo Zhang. Analytic-dpm: analytic estimate of the optimal reverse variance in diffusion probabilistic models. In Proc. of ICLR, pages 1-39, 2022.
[2] Yaohui Cai, Zhewei Yao, Zhen Dong, Amir Gholami, Michael W Mahoney, and Kurt Keutzer. Zeroq: novel zero shot quantization framework. Proc. of CVPR, pages 2020.
[3] Nanxin Chen, Yu Zhang, Heiga Zen, Ron J Weiss, Mohammad Norouzi, and William Chan. Wavegrad: Estimating gradients for waveform generation. arXiv preprint arXiv:2009,0077 2020.
[4] Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer. Llm. int8 O:: 8-bit matrix multiplication for transformers at scale. arXiv preprint arXiv:2208.07339, 2022.
[5] Steven K Esser, Jeffrey L McKinstry, Deepika Bablani, Rathinakumar Appuswamy, and Dhar- mendra S Modha. Learned step size quantization. In Proc. of ICIR, pages 1-12, 2019.
[6] Jun Fang, Ali Shafice, Hamzah Abdel-Aziz, David Thorsley, Georgios Georgiadis, and Joseph H Hassoun. Post-training piecewise linear quantization for deep neural networks. In Proc. of ECCV, pages 69 19-86,77 2020.
[7] Sicheng Gao, Xuhui Liu, Bohan Zeng, Sheng Xu, Yanjing Li, Xiaoyan Luo, Jianzhuang Liu, Xiantong Zhen, and Baochang Zhang. Implicit diffusion models for continuous super-resolution. arXiv preprini arXiv:2303. 16491, 2023.
[8] Ian Goodfellow, Jean Pouget-Abadic Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks. Communications of the ACM, pages 139-144, 2020.
[9] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by two time-scale update rule converge to a local nash equilibrium. In Proc. of NeurlPS, pages 1-12, 2017.
[10] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In Proc. of NeurIPS, pages 6840-6851, 2020.
[11] Alexia Dolicoeur-Martinennn Ke Li, Rémi Piché-Tailleffer Tal Kachman, and loannis Mitliagkas. Gotta go fast when generating data with score-based models. arXiv preprint arXiv:2 105. 14080, 2021.
10

---

[12] Sangil Jung, Changyong Son, Seohyung Lee, Jinwoo Son, Jae-Joon Han, Youngjun Kwak, Sung Ju Hwang, and Changkyu Choi. Learning to quantize deep networks by optimizing quantization intervals with task loss. In Proc. of CVPR, pages 4350-4359 2019.
[13] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. Technical report, Citeseer, 2009.
[14] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. In Proc. of NeurlPS, pages 1097-1105, 2012.
[15] Fengfu Li, Bo Zhang, and Bin Liu. Temnry weight networks. arXiv preprint arXiv:1605.04711, 2016.
[16] Yanjing Li, Sheng Xu, Baochang Zhang, Xianbin Cao. Peng Gao, and Guodong Guo. Q-vit: Accurate and fully quantized low-bit vision transformer. In Proc. of NeurlPS, pages 1-12, 2022.
[17] Yang Lin, Tianyu Zhang, Peiqin Sun, Zheng Li, and Shuchang Zhou. Fit Fully quantized vision transformer without retraining. In Proc. of IJCAI, pages 1179-1799, 2022.
[18] Zechun Liu, Zhiqiang Shen, Marios Savvides, and Kwang- Ting Cheng. Reactnet: Towards precise binary neural network with generalized activation functions. In Proc. of ECCV, pages 143-159, 2020.
[19] Zhenhua Liu, Yunhe Wang, Kai Han, Wei Zhang, Siwei Ma, and Wen Gao. Post-training quantization for vision transformer. Proc. of NeurlPS, pages 1-12, 2021.
[20] Eric Luhman and Troy Luhman. Knowledge distillation in iterative generative models for improved sampling speed. arXiv preprint arXiv:2101.02388, 2021.
[21] Gautam Mittal, Jesse Engel, Curtis Hawthorne, and Ian Simon. Symbolic music generation with diffusion models. arXiv preprint (tti::2003.6699)) 2021.
[22] Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. In Proc. of ICML, pages 8162-8171, 2021.
[23] Chenhao NIu, Yang Song, Jiaming Song, Shengjia Zhao, Aditya Grover, and Stefano Ermon. Permutation invariant graph generation via score-based generative modeling. In Proc. of AISTATS, pages 4474-4484, 2020.
[24] Haotong Qin, Yifu Ding, Mingyuan Zhang, Qinghua Yan, Aishan Liu, Qingqing Dang, Ziwei Liu, and Xianglong Liu. Bibert: Accurate fully binarized bert. In Proc. of ICLR, pages 1-24, 2022.
[25] Mohammad Rastegari, Vicente Ordonez, Joseph Redmon, and Ali Farhadi. Xnor-net: Imagenet classification using binary convolutional neural networks. In Proc. of ECCV, pages 525-542, 2016.
[26] Chitwan Saharia, Jonathan Ho, William Chan, Tim Salimans, David Fleet, and Mohammad Norouzi. Image super-resolution via iterative refinement. TPAMI, 2022.
[27] Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans. In Proc. of NeurIPS, pages 1-9, 2016.
[28] Robin San-Roman, Eliya Nachmani, and Lior Wolf. Noise estimation for generative diffusion models. arXiv preprint arXiv:2104.02600 2021.
[29] Hiroshi Sasaki, Chris G Willcocks, and Toby P Breckon. Unit-ddpm: Unpaired image translation with denoising diffusion probabilistic models. arXiv preprint arXiv:2104.05358, 2021.
[30] Yuzhang Shang, Zhihang Yuan, Bin Xie, Bingzhe Wu, and Yan Yan. Post-training quantization on diffusion models. arXiv preprint 1...iv22211.5555 15736, 2022.
[31] Uriel Singer, Adam Polyak, Thomas Hayes, Xi Yin, Jie An, Songyang Zhang, Qiyuan Hu, Harry Yang, Oron Ashual, Oran Gafni, al. Make-a-video: Text-to-video generation without text-video data. arXiv preprint arXiv:2209, 14792, 2022.

---

[32] Jiaming Song. Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In Proc. of ICLR, pages 1-20, 2020.
[33] Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. In Proc. of NeurIPS, pages 1-13, 2019.
[34] Yang Song and Stefano Ermon. Improved techniques for training score-based generative models. In Proc. of NeurIPS, pages 12438 12438-12448, 2020.
[35] Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In Proc. of ICLR, pages 1-36, 2021.
[36] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, tukasz Kaiser, and Illia Polosukhin. Attention is you need. Proc. of NewrlPS, pages 1-11, 2017.
[37] Xiaolong Wang, Ross Girshick, Abhinav Gupta, and Kaiming He. Non-local neural networks. In Proc. of CVPR, pages 7794-7803, 2018.
[38] Daniel Watson, Jonathan Ho, Mohammad Norouzi, and William Chan. Learning to efficiently sample from diffusion probabilistic models. arXiv preprint arXiv:2106.0388 2021.
[39] Zheng Xie, Zhiquan Wen, Jing Liu, Zhiqiang Liu, Xixian Wu, and Mingkui Tan. Deep transferring quantization. In Proc. of ECCV, pages 625-642, 2020.
[40] Ling Yang. Zhilong Zhang, Yang Song, Shenda Hong, Runsheng Xu, Yue Zhao, Yingxia Shao, Wentao Zhang, Bin Cui, and Ming-Hsuan Yang. Diffusion models: comprehensive survey of methods and applications. arXiv preprint arXiv:220.000796, 2022.
[41] Shuchang Zhou, Yuxin Wu, Zekun Ni, Xinyu Zhou, He Wen, and Yuheng Zou. Dorefa-net Training low bitwidth convolutional neural networks with low bitwidth gradients. arXiv preprins wrXiv:166.006160 2016.
[42] Chenzhuo Zhu, Song Han, Huizi Mao, and William J Dally. Trained ternary quantization. In Proc. of ICLR, pages 1-10, 2017
[43] Bohan Zhuang, Chunhua Shen, Mingkui Tan, Lingqiao Liu, and Ian Reid. Towards effective bow-bitwidth convolutional neural networks, In Proc. of CVPR, pages 7920-7928. 2018
12


Reviews:
[{"review_id": "bYNWFGSzdP", "reviewer": "Reviewer_mdg9", "summary": "The paper identifies two challenges in low-bit diffusion models (DMs): activation distribution oscillation and quantization error accumulation. To tackle these challenges, the paper introduces two novel techniques: Timestep-aware Quantization (TaQ) and Noise-estimating Mimicking (NeM). Experimental results demonstrate superior performance compared to previous approaches on CIFAR-10 and ImageNet 64x64 datasets.", "questions": "How are the model size and OPs calculated? The quantized model's size reported in the paper is precisely 1/bits of the full precision model. However, this is unusual because there must be some parameters that remain in full-precision.", "limitations": "I did not identify any issues related to limitations and potential negative societal impact.", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "The paper is easy to follow and low-bit diffusion model is an interesting idea worth to try. The presented ablation studies demonstrate that the proposed methods have a positive impact on quantized diffusion models.", "weaknesses": "1. The main contribution of this paper involves the utilization of statistical mean and variance to address distribution oscillation. Essentially, this approach is akin to applying a shift and scale operation following quantized operations, which is a common technique employed in low-bit quantization to achieve balanced quantization bins. Similar methods for weight and activation balancing can be found in [1] and [2, 3] respectively. \n\n2. In Equation (12), the application of TaQ to the softmax attention scores appears questionable. Unlike other activations that typically conform to a Gaussian or bell-shaped distribution, softmax attention scores often exhibit a long-tailed distribution, where the sum of the probabilities is 1. Normalizing the softmax attention scores may disrupt the probabilistic interpretation of these scores, which raises concerns about its appropriateness.\n\n3. The proposed Noise-estimating Mimicking technique necessitates fine-tuning the entire model, which may pose practical challenges for large diffusion models trained on extensive datasets like LAION-5B. Efficient fine-tuning method should be considered.\n\n4. The experiments in the paper are limited to CIFAR-10 and ImageNet 64x64 datasets, with a maximum model size of 4.47M. To provide a comprehensive evaluation, it is recommended to conduct additional experiments on larger models, such as LDM[4], and include commonly used datasets like LSUN and ImageNet 256x256.\n\n5. The accuracy comparison experiment in this paper seems to be inadequate, and there are issues with the citations for the compared method PTQ4DM [5]. Moreover, discrepancies exist between the FID and IS results for full-precision (FP) models in this paper and the results reported in PTQ4DM, which could potentially introduce unfairness in the comparisons. As an example, when employing the DDIM sampler for 100 steps on CIFAR-10, the FID for the FP model is reported as 10.05 in PTQ4DM, whereas this paper indicates a result of 4.16.\n\n[1] Qin, Haotong, et al. \"Forward and backward information retention for accurate binary neural networks.\" CVPR 2020.\n[2] Liu, Zechun, et al. \"Reactnet: Towards precise binary neural network with generalized activation functions.\" ECCV 2020.\n[3] Wei, Xiuying, et al. \"Outlier Suppression+: Accurate quantization of large language models by equivalent and optimal shifting and scaling.\" arXiv:2304.09145 (2023).\n[4] Rombach, Robin, et al. \"High-resolution image synthesis with latent diffusion models.\" CVPR 2022.\n[5] Shang, Yuzhang, et al. \"Post-training quantization on diffusion models.\" CVPR 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1**: The main contribution of this paper involves the utilization of statistical mean and variance to address distribution oscillation. Essentially, this approach is akin to applying a shift and scale operation following quantized operations, which is a common technique employed in low-bit quantization to achieve balanced quantization bins. Similar methods for weight and activation balancing can be found in [1] and [2, 3] respectively.\n\n**A1**: To the best of our knowledge, our Q-DM is the first QAT-based work to effectively quantize diffusion models. Our method can well address the activation oscillation problem by introducing Timestep-aware Quantization (TaQ) and Noise-estimating Mimicking (NeM) into the QAT framework, which effectively improve the performance of quantizing DM. Different from quantization of other deep models, our Q-DM involves both training and inference process, in this situation our method is very efficient based on a simple method with the shift and scale operations. You can also refer to Q8 of Reviewer bfDz, where we also compare the performance of Q-DM with other normalization methods.\n\n**Q2**: In Eq. (12), the application of TaQ to the softmax attention scores appears questionable. Unlike other activations that typically conform to a Gaussian or bell-shaped distribution, softmax attention scores often exhibit a long-tailed distribution, where the sum of the probabilities is 1. Normalizing the softmax attention scores may disrupt the probabilistic interpretation of these scores, which raises concerns about its appropriateness.\n\n**A2**: We agree with your point on the Gaussian or bell-shaped distribution,  our proposed TaQ method is exactly applied to $a_q$ , $a_k$  before the softmax operation, rather than directly normalizing the softmax attention score. Details are shown in the line 2 of  Eq. (12). Therefore, Eq. (12) is appropriate and does not disrupt the distribution of attention score. We will visualize the effect of TaQ method on probabilistic interpretation of Q-AttnBlock in the final version for clarity.\n\n**Q3**: The proposed Noise-estimating Mimicking technique necessitates fine-tuning the entire model, which may pose practical challenges for large diffusion models trained on extensive datasets like LAION-5B. Efficient fine-tuning method should be considered.\n\n**A3**: We will add more fine-tuning method experiments in the final version. As shown in the table below, our method also effects on  large diffusion models fine-tuned by LoRA-R8 on extensive datasets like LAION-5B. Our method consistently boosts the performance of baseline method in 2/3/4 bit-width format. We will add the experimental results in the final version.\n\n| Model | Method |\\#Bits| FID$\\downarrow$ |\n| ------ | ------ | ------ | ------ |\n| LDM-4 | Full-precision | 32/32 |20.68|\n| LDM-4 | Baseline (LSQ) | 4/4 |23.42|\n| LDM-4 | Q-DM | 4/4 | 21.56 |\n| LDM-4 | Baseline (LSQ) | 3/3 | 25.81 |\n| LDM-4 | Q-DM | 3/3 | 23.72 |\n| LDM-4 | Baseline (LSQ) | 2/2 | 27.49 |\n| LDM-4 | Q-DM | 2/2 | 25.96 |\n\n**Q4**: The experiments in the paper are limited to CIFAR-10 and ImageNet 64x64 datasets, with a maximum model size of 4.47M. To provide a comprehensive evaluation, it is recommended to conduct additional experiments on larger models, such as LDM [4], and include commonly used datasets like LSUN and ImageNet 256x256.\n\n**A4**: We apply our Q-DM on LDM [4] and test it on the LSUN dataset, and the results are shown in the table below. As can be seen, the proposed Q-DM also show its superiority on the LDM with larger LSUN dataset. For example, Q-DM boosts the baseline models by about 0.3\\%~0.5\\% FID score with the same bit-width, which is significant. We will add these experiments in the final version. \n\n| Model | Method |\\#Bits| FID$\\downarrow$ |\n| ------ | ------ | ------ | ------ |\n| LDM-4 | Full-precision | 32/32 | 2.98 |\n| LDM-4 | Q-Diffusion |8/8 | 3.63 |\n| LDM-4 | Baseline (LSQ) | 4/4 | 3.54 |\n| LDM-4 | Q-DM | 4/4 | 3.01 |\n| LDM-4 | Baseline (LSQ) | 3/3 | 3.68 |\n| LDM-4 | Q-DM | 3/3 | 3.37 |\n| LDM-4 | Baseline (LSQ) | 2/2 | 3.99 |\n| LDM-4 | Q-DM | 2/2 | 3.76 |\n\n**Q5**: The accuracy comparison experiment in this paper seems to be inadequate, and there are issues with the citations for the compared method PTQ4DM [5]. Moreover, discrepancies exist between the FID and IS results for full-precision (FP) models in this paper and the results reported in PTQ4DM, which could potentially introduce unfairness in the comparisons. As an example, when employing the DDIM sampler for 100 steps on CIFAR-10, the FID for the FP model is reported as 10.05 in PTQ4DM, whereas this paper indicates a result of 4.16.\n\n**A5**: We follow the same experiments setting as DDPM [6] and DDIM [7]. The FID score of DDIM sampler for 100 steps on CIFAR-10 is reported as 4.16 in the original paper (Table 1, Page 7 in DDIM [7]), which is the same as our re-implementation.\n\n[1] Forward and Backward Information Retention for Accurate Binary Neural Networks. CVPR'2020.\n\n[2] ReActNet: Towards Precise Binary Neural Network with Generalized Activation Functions. ECCV'2020.\n\n[3] Outlier Suppression+: Accurate quantization of large language models by equivalent and optimal shifting and scaling. ArXiv:2304.09145.\n\n[4] High-Resolution Image Synthesis with Latent Diffusion Models. CVPR'2022.\n\n[5] Post-Training Quantization on Diffusion Models. CVPR'2023.\n\n[6] Denoising Diffusion Probabilistic Models. NeurIPS'2020.\n\n[7] Denoising Diffusion Implicit Models. ICLR'2021."}, {"review_id": "QdJcnHFGzu", "reviewer": "Reviewer_acST", "summary": "The paper proposes two method to mitigate the accuracy degradation caused by quantization of diffusion models: one is time-step aware quantization (different calibration data and range for each time-step of diffusion), the other is using a full-precision network for training time distillation. Experiments on image generation in Cifar-10 and ImageNet datasets show the efficacy of the method.", "questions": "Can visual results be provided for intuitive inspection? In particular, spotty noise may not be well captured by FID/IS metrics but will be blatant when doing visual inspection.\n\nIn Table 2, there are a few pairs of results with similar IS but very different FID, like PTQ4DM CIFAR-10 32×32 50 steps and Baseline CIFAR-10 32×32 3/3 . Any discussions of this variation?", "limitations": "No particular limitations.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "By using QAT, the proposed method manages to bring down the bitwidth of the neural network to lower than 8-bit. The FID and IS metrics support the efficacy of the method.", "weaknesses": "The benefits of per-time-step quantization has been known since earlier works like PTQ4DM and q-diffusion. It is desirable to know the proposed method and prior works. Note that QAT and PTQ do not make much difference here, as the underlying motivation of per-time-step quantization to deal with changing dynamic range change is the same here. Also having per-time-step quantization is almost zero-overhead as the weights need be loaded for computation of each step anyway.\n\nAlso having a full-precision network as teacher for knowledge distillation (including having full precision or higher precision teacher) is quite a standard approach. It's not obvious what specialties of QAT are incorporated here.\n\nq-diffusion: https://arxiv.org/abs/2302.04304", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1**: The benefits of per-time-step quantization has been known since earlier works like PTQ4DM [1] and Q-Diffusion [2]. It is desirable to know the proposed method and prior works. Note that QAT and PTQ do not make much difference here, as the underlying motivation of per-time-step quantization to deal with changing dynamic range change is the same here. Also having per-time-step quantization is almost zero-overhead as the weights need be loaded for computation of each step anyway.\n\n**A1**: QAT is technically different from PTQ, and also QAT can achieve a better performance than PTQ as shown in our experimental parts. Previous PTQ methods for quantized DM, e.g., PTQ4DM [1], partially address the activation oscillation issue by constructing a calibration dataset, which is not applicable in the QAT process (see Fig. 4 in [3]). Differently, our method addressed this problem by introducing TaQ and NeM into the QAT framework, which can well reduce the activations oscillation and effectively improve the performance of Q-DM. We will add more related work to clarify the difference from our method in the final version.\n\n**Q2**: Also having a full-precision network as teacher for knowledge distillation (including having full precision or higher precision teacher) is quite a standard approach. It's not obvious what specialties of QAT are incorporated here.\n\n**A2**: Although using knowledge distillation is a standard approach in network quantization, the KD method for quantized DMs remains largely under-developed. As one of our contributions, we propose a new knowledge distillation method, called Noise-estimating Mimicking (NeM), to enhance the performance of quantized DMs. We provide the motivation and theoretical derivation of NeM in Sec. 4.2. As shown in Tab.1, the proposed NeM significantly improves the performance of the baseline method, which validates the motivation and theoretical derivation.\n\n**Q3**: Can visual results be provided for intuitive inspection? In particular, spotty noise may not be well captured by FID/IS metrics but will be blatant when doing visual inspection.\n\n**A3**: Thanks for the advice, we will provide qualitative results in the final version. \n\n**Q4**: In Table 2, there are a few pairs of results with similar IS but very different FID, like PTQ4DM CIFAR-10 32×32 50 steps and Baseline CIFAR-10 32×32 3/3 . Any discussions of this variation?\n\n**A4**: Generally, the IS metric has similar results as shown in our practice and also other methods [1,4], even they are very different on FID. For example, similar phenomenon can also be observed in PTQ4DM [1] (Table 4) and DDPM [4] (Table 1). We will add this discussion in the final version.\n\n[1] Post-Training Quantization on Diffusion Models. CVPR'2023.\n\n[2] Q-Diffusion: Quantizing Diffusion Models. ArXiv:2302.04304.\n\n[3] A Survey of Quantization Methods for Efficient Neural Network Inference. ArXiv: 2103.13630.\n\n[4] Denoising Diffusion Probabilistic Models. NeurIPS'2020.\n\n"}, {"review_id": "4xmkNUQeYz", "reviewer": "Reviewer_6Qdf", "summary": "In this paper, a novel method called Q-DM is introduced, which enables the creation of low-bit quantized diffusion models. The authors first give extensive analysis about two primary challenges faced by low-bit quantized DMs: significant distribution oscillation on activations and accumulated quantization error arising from the multi-step denoising process. To address these issues, the authors propose two techniques: Timestep-aware Quantization (TaQ) and Noise-estimating Mimicking (NeM). TaQ is designed to mitigate the distribution oscillation problem, while NeM aims to reduce the accumulated quantization error. By incorporating these techniques into Q-DM, the paper demonstrates its ability to overcome these challenges effectively, leading to superior performance compared to existing methods. The experimental results provided in the paper serve as evidence of the high-quality performance achieved by Q-DM.", "questions": "See weaknesses.", "limitations": "Yes", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "1.This paper is easy to follow. The organization of this paper are exemplary, as it effectively presents the proposed Q-DM in a clear and comprehensible manner. The paper provides comprehensive explanations of how Q-DM improved the performance of quantized DMs. \n\n2.This paper presents an novel quantization method known as Q-DM, which addresses several challenges in low-bit quantized diffusion models. The authors propose the Timestep-aware Quantization (TaQ) method to tackle the issue of activation distribution oscillation, which is caused by the random-sampled timestep during training. Additionally, they introduce the Noise-estimating Mimicking (NeM) scheme to effectively minimize accumulated errors. \n\n3.The experimental results are significant. The Q-DMs exhibit performance comparable to that of full-precision models, showcasing the effectiveness of the proposed methods.\n\n", "weaknesses": "1.Is the model quantized during all the training and sampling process? Since these may lead to different inference speed. And are the quantization-related parameter all the same across different timestep?\n\n2.Is the proposed TaQ method only used in Q-AttnBlock module? And how is the conv layer quantized in Q-DM?\n\n3.In Table 2, how is the ‘OPs’ calculated?  It would be more detailed if authors can provide some description about this metric. \n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1**: Is the model quantized during all the training and sampling process? Since these may lead to different inference speed. And are the quantization-related parameter all the same across different timestep?\n\n**A1**: Yes, the model is quantized during both the training (to quantized value in floating-point format [1]) and sampling process. Also, the quantization-related parameter are all the same across different timestep during inference. \n\n**Q2**: Is the proposed TaQ method only used in Q-AttnBlock module? And how is the conv layer quantized in Q-DM?\n\n**A2**: The proposed TaQ method is only deployed in the Q-AttnBlock module (Eq. (12)). The quantization method of convolution layer in Q-DM is same as LSQ [2], i.e., the baseline method. \n\n**Q3**: In Table 2, how is the ‘OPs’ calculated? It would be more detailed if authors can provide some description about this metric.\n\n**A3**: The OPs is calculated through \"the respective number of FLOPs adds {$\\frac {1}{32}$, $\\frac {1}{16}$, $\\frac {1}{8}$} of the number of {$2$, $3$, $4$}-bit multiplications equals the OPs\" following [3].  We will add detailed description in the final version. \n\n[1] A Survey of Quantization Methods for Efficient Neural Network Inference. ArXiv: 2103.13630.\n\n[2] Learned Step Size Quantization. ICLR'2020.\n\n[3] Q-DETR: An Efficient Low-Bit Quantized Detection Transformer. CVPR'2023."}, {"review_id": "he0yDLi4gM", "reviewer": "Reviewer_bfDz", "summary": "The paper proposed a quantization-aware training scheme for diffusion models, based on the well-known method, LSQ. \nIn the paper, they identified the bottleneck come from a large distribution oscillation on activations and accumulated quantization error caused by the denoising process. \nThen, they suggest method to address the issues:\nTimestep-aware quantization (TaQ) – Time Step-aware activation smoothing to handle a large oscillation on activation\nNoise-estimating Mimicking (NeM):  reduceing error accumulation throughout the multi-step denoising process with quantization\n", "questions": "1.\tGenerally, QAT requires more data and training time than PTQ, and U-net inherently has a large number of parameters, so requires a lot of training data and takes a long time to optimize. How much data is used for QAT for DM? And how much time did you take to complete the quantization for U-net?\n2.\tWhen performing inference, computing the statistical mean and variance are needed every time step? If so it imposes the additional cost for H/W. How are these values considered at inference time?\n3.\tAs a naive approach, we may have a different step size of activation for each time step to handle the oscillation on activations. Is there any problem with the naive approach compared to your approach?\n4.\tPTQ4DM and DFQ also handle the dynamic range of activations. Compared with them, what are the advantages of your method?\n5.\tThe problem you observed in quantizing DM may have been dealt with in other literature, even if it is not related to quantization and DM. For example, there are many approaches to normalize the value. The paper would be more robust if the authors provide sufficient related works handling the issues the authors addressed.\n6.\tIn Eq(11) and (12), TaQ quantize normalized activations. Is there no need for compensation for the normalization?\n", "limitations": "See weakness and questions.", "rating": 4, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "They observed and identified the issues when compressing DMs in a quantization-aware training manner.", "weaknesses": "1.\tThe suggested methods to tackle the issues in QAT for DM are not novel. TaQ and NeM seem just normalization and knowledge distillation which can be commonly used in the related literature.\n2.\tThere is much lack of numerical descriptions for the QAT process, such as training time, the number of calibration datasets, and the number of epochs, etc. They have to provide detail about how many resources are needed for reproduction.\n3.\tThe models used in the paper to show their performance are a little bit outdated. The datasets they used are also insufficient to prove the performance. They should provide more thorough experimental results with recent models and datasets.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Due to the character number constraint of rebuttal, we abridge the question in the rebuttal part. All experiments below are conducted on 50-step DDIM sampler with 32×32 generating resolution on CIFAR-10 dataset. \n\n**Q1**: Regarding TaQ and NeM. \n\n**A1**: Our method is proposed based on the observed activation oscillation and quantification error accumulation phenomena. And the QAT method used for the diffusion model is far from being explored. The experimental results also show the advantages of our method. \n\n**Q2**: Regrading numerical descriptions for the QAT process. \n\n**A2**: Sorry for the missing. The training epoch and training time of DDPM and DDIM is 80k step, which needs 6 GPU days. In the QAT of diffusion models, there is no calibration dataset. We will these description in the final version. \n\n**Q3**: Regarding experiments with recent models and datasets.\n\n**A3**: We have conducted experiments of other models and datasets. Please also see the Q3 & Q4 of Reviewer mdg9. \n\n**Q4**: Regarding the data and training cost of Q-DM.\n\n**A4**: We employ the complete set of images from the training dataset, which are same as  other  QAT methods [3-5]. The training cost comparison  between Q-DM and baseline LSQ is shown below. Our Q-DM is about 1.6% lower than baseline method but achieves higher performance. We will add  explanations in the final version for clarity.\n\n|Method|Training time (gpu days)|\n|--|--|\n|Baseline|6.0|\n|Q-DM|6.1|\n\n**Q5**: Regarding computation cost of TaQ.\n\n**A5**: We reply to this question point-to-point.\n- Yes, each step requires computation, but the computational cost is negligible. The OPs value are shown in Tab. 2 of the submission, which include the inference costs of Q-DM  with or without the mean and variance computation. \n\n- We further detailed calculate the OPs of Q-DM and TaQ module, as an example, in the below table. As shown, the extra cost of TaQ module is counted less than 1% in terms of total OPs, which is almost negligible during inference.\n\n|Method|\\#Bits|OPs|TaQ OPs (ratio)|\n|--|--|--|--|\n|Full-precision|32/32|390.4|- (-)|\n|Q-DM|4/4|49.9|0.10 (0.2%)|\n|Q-DM|3/3|25.1|0.10 (0.4%)|\n|Q-DM|2/2|12.6|0.10 (0.8%)|\n\n**Q6**: Regarding different step size\n\n**A6**: Using different step sizes for each weight at every time step will result in increased storage and computational burden. As shown below, our method achieves higher compression and acceleration ratio. \n\n|Method|\\#Bits|Size|OPs|\n|--|--|--|--|\n|Q-DM |4/4|0.56|49.98|\n|Different step size|4/4|0.57|50.01|\n|Q-DM|3/3|0.28|25.12|\n|Different step size|3/3|0.29|25.15|\n|Q-DM|2/2|0.14|12.66|\n|Different step size|2/2|0.15|12.69|\n\n**Q7**: Regarding advantages of Q-DM.\n\n**A7**: We state the difference and advantages of out method in two aspects: in technology and in performance.\n\n- **Regarding technology**: PTQ4DM [6] is a PTQ method designed for the diffusion model, which can partially address the activation oscillation issue by constructing a more suitable calibration dataset (not applicable in the QAT process, see Fig. 4 in [7]). DFQ [8] eliminates this phenomenon through cross-layer equalization for backbones regarding image classification and detection task, not suitable for the QAT process of DMs. (also a particular phenomenon in the PTQ process), which is not suitable for QAT methods. In contrast, our method aims to improve the QAT process for DMs and successfully eliminates the activation oscillation issue, from a perspective of QAT-specific technology. \n\n- **Regarding performance**: Compared with PTQ methods, the proposed Q-DM also benefits from the main advantage of the QAT process, i.e., superior performance with lower bit-width precision. We further evaluate the PTQ4DM and DFQ in 8- and 4-bit bit-width, where the results are shown in below. We have two main observations: 1) the 4-bit Q-DM achieves better performance  that 8-bit PTQ methods while possessing lower-precision weights and activations; 2) the PTQ methods deteriorates in 4-bit format, while Q-DM performs well with lower precision. These observations demonstrate the superiority of the Q-DM method.\n\n |Method|\\#Bits|FID$\\downarrow$|IS$\\uparrow$|\n |--|--|--|--|\n |Full-precision|32/32|4.67|9.27|\n |PTQ4DM |8/8|18.02|8.87|\n |DFQ|8/8|18.96|8.83|\n |PTQ4DM|4/4|19.78|8.76|\n |DFQ |4/4|20.02|8.68|\n |**Q-DM**|4/4|**8.98**|**8.92**|\n\n**Q8**: Difference of Q-DM and other normalization.\n\n**A8**: Different from prior works, our TaQ  numerically analyzes the activation ranges across different timesteps and effectively mitigates distribution oscillation specifically for low-bit quantized DMs. To validate, we show some experimental comparisons with the prior methods which normalize the activation value as below. We will add more references and this comparison in the final version for clarity.\n\n |Method|\\#Bits|FID$\\downarrow$|IS$\\uparrow$|\n |--|--|--|---|\n |Q-ViT|4/4|9.48|8.85|\n |**Q-DM**|4/4|**8.98**|**8.92**|\n\n**Q9**: Regarding compensation for normalization?\n\n**A9**: We conduct experiments regarding normalization and its compensation as below. As shown, our TaQ achieves better performance with less extra parameters introduced. This will be added in the final version. \n\n |Formulation of Eq. (11)|FID$\\downarrow$|IS$\\uparrow$|\n |--|--|--|\n |**a** (Baseline) |8.98|8.92|\n |**Norm(**a**) (TaQ)**|**6.89**|**8.96**|\n |Norm(**a**)*scale |6.92|8.95|\n |Norm(**a**)*scale+bias|6.91|8.95|\n\n[1] Denoising Diffusion Probabilistic Models. NeurIPS'2020.\n\n[2] Denoising Diffusion Implicit Models. ICLR'2021.\n\n[3] Q-ViT: Accurate and Fully Quantized Low-bit Vision Transformer. NeurIPS'2022.\n\n[4] Q-DETR: An Efficient Low-Bit Quantized Detection Transformer. CVPR’2023.\n\n[5] Post-Training Quantization on Diffusion Models. CVPR'2023.\n\n[6] A Survey of Quantization Methods for Efficient Neural Network Inference. ArXiv: 2103.13630.\n\n[7] Data-Free Quantization Through Weight Equalization and Bias Correction. ICCV'2019."}, {"review_id": "G1mJhK2dW4", "reviewer": "Reviewer_3MiP", "summary": "This paper presents a quantization-aware training framework for diffusion models. The authors identified activation distribution oscillation and quantization error accumulation as the main causes of the performance drop. To close the performance gap, the authors developed a timestep-aware quantization (TaQ) method for data normalization and a noise-estimating mimicking (NeM) training scheme. Experiments were conducted to demonstrate the effectiveness of the proposed method. ", "questions": "1. What does \"PTQ4DM\" exactly refer to in Tab. 2? Tab.2 cites [1] while referring PTQ4DM to [2] in Line 243. As far as I am concerned, PTQ4DM refers to a recently published work [2], presenting a method for calibration data collection in diffusion quantization, then what does Line 242 mean by saying \"we also report the **classification performance**...\" ?\n\n2. I have some questions regarding the method details, I would like to ask for some clarifications:\n    a. Did you calculate the statistical mean and variance of each activation in an offline manner, i.e. the statistical values were pre-computed with the full-precision DMs and not updated with the training procedure?\n    b. How did you conduct inference with such the timestep-aware design? Did you conduct activation normalization with the pre-computed values on-the-fly? If so, how would it affect the inference efficiency?\n   c. How is the \"distance\" in Figure 4 calculated? Also, is the activation distribution in Figure 3 layer-wise or channel-wise?\n   d. Is it possible to combine the original training loss in DMs with the NeM loss?\n\n[1] Post-training piecewise linear quantization for deep neural networks\n\n[2] Post-Training Quantization on Diffusion Models", "limitations": "The authors do not include the limitations and potential negative societal impact in the paper.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. The motivation of this paper is valid. The proposed method is reasonable. The shifting of the activation distributions and the quantization error accumulation are quite likely to affect the overall performance. The authors propose Timestep-aware Quantization (TaQ) and Noise-estimating Mimicking (NeM) accordingly to address these two problems.\n2. The paper is well-organized and the writing is fluent. This helps the readers capture the high-level ideas well and the detailed experimental setups make this paper easy to replicate the results.\n3. The experimental results are promising. Q-DMs demonstrate performance on par with FP models.", "weaknesses": "1. Will the TaQ quantization process affect the inference speed? See 3.b in the ``Questions'' section.\n2. Some technical details should be clarified (see the questions below).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1**: What does \"PTQ4DM\" exactly refer to in Tab. 2? Tab.2 cites [1] while referring PTQ4DM to [2] in Line 243. As far as I am concerned, PTQ4DM refers to a recently published work [2], presenting a method for calibration data collection in diffusion quantization, then what does Line 242 mean by saying \"we also report the classification performance...\" ?\n\n**A1**: Sorry for the typo, the citation should be “Post-training quantization on diffusion models [2]”, and the Line 242 should be “We also report the generation performance of the 8-bit PTQ method”. The typos will be revised it in the final version. \n\n**Q2(a)**:  Did you calculate the statistical mean and variance of each activation in an offline manner, i.e. the statistical values were pre-computed with the full-precision DMs and not updated with the training procedure?\n\n**A2(a)**: Only the statistical values in Fig. 3 are calculated by a full-precision diffusion model in an offline manner, to explain our motivation. In our QAT process, the statistical mean and variance of each activation are calculated on-the-fly. We apology for the typo and misleading in the caption. We will polish it in the final version. \n\n**Q2(b)**: How did you conduct inference with such the timestep-aware design? Did you conduct activation normalization with the pre-computed values on-the-fly? If so, how would it affect the inference efficiency?\n\n**A2(b)**: The calculation of statistical mean and variance is performed on-the-fly during inference. As shown in Tab. 2, the increase of flops is negligible. (Copy from Q2 of R\\#bfDz).\n\n**Q2(c)**: How is the \"distance\" in Figure 4 calculated? Also, is the activation distribution in Figure 3 layer-wise or channel-wise?\n\n**A2(c)**: In Fig. 4, we calculate the distance between the same layer of the full-precision DMs and quantized counterparts. In Fig. 3, the activation distribution is about all elements in one specific layer, i.e., layer-wise distribution. We will add these detailed descriptions in the final version. \n\n**Q2(d)**: Is it possible to combine the original training loss in DMs with the NeM loss?\n\n**A2(d)**: We have conducted experiments of combining the original training loss in DMs with the NeM loss. We evaluate the combination of original loss in DDPM [1] and NeM with 4-bit Q-DM based on 50-step DDIM sampler with 32×32 generating resolution on CIFAR-10, in below. As can be seen, this combination affects the performance of Q-DM. The best result (in **bold**) is achieve by singly combining the 4-bit baseline with our NeM model. Hence, we find that the NeM method performs best when singly being applied. These results and analysis will be added in the final version.\n\n\n\n | Method | FID$\\downarrow$ | IS $\\uparrow$|\n | ------ | ----- | ------ |\n | Full-precision | 4.67 | 9.27 |\n | 4-bit Baseline (LSQ) + Original Loss [1] | 10.22 | 8.91 |\n | **4-bit Baseline (LSQ) + NeM** | **8.98** | **8.92** |\n | 4-bit Baseline (LSQ) + Original Loss [1] + NeM | 9.05 | 8.87 |\n\n[1] Post-training Piecewise Linear Quantization for Deep Neural Networks. ECCV'2020.\n\n[2] Post-Training Quantization on Diffusion Models. CVPR'2023.\n\n[3] Denoising Diffusion Probabilistic Models. NeurIPS'2020."}]

Author rebuttal:

Meta-review: This paper introduces a novel method called Q-DM (Quantized Diffusion Models) to address the challenges faced by low-bit quantized diffusion models. The authors identify activation distribution oscillation and quantization error accumulation as primary concerns and propose two techniques, Timestep-aware Quantization (TaQ) and Noise-estimating Mimicking (NeM), to mitigate these issues. Experimental results on CIFAR-10 and ImageNet datasets validate the superior performance of Q-DM compared to previous approaches. 

Reviewers raised concerns about the lack of novelty in the proposed techniques, the need for more detailed numerical descriptions and broader experimental evaluation, and discrepancies in the accuracy comparison and citations. The authors provided a thorough response addressing the concerns raised by the reviewers, leading to a final score of 4, 4, 5, 7, 7. As two reviewers still had doubts about the novelty of the paper, the AC carefully reviewed the paper and all the reviewers' comments. 

Considering that this paper is the first to introduce the QAT technique to diffusion models, the AC recognizes its significant value in the quantization of diffusion models. With an average score of 5.4, the paper is competitive for this year's NeurIPS conference. The AC agrees to accept the paper and strongly recommends incorporating the content from the rebuttal into the final version.
Decision: Accept (poster)