Title: Q-DM: An Efficient Low-bit Quantized Diffusion Model
Venue: NeurIPS 2023
Accepted: yes
Keywords: network quantization

diffusion model

image synthesize
Abstract: Denoising diffusion generative models are capable of generating high-quality data, but suffers from the computation-costly generation process, due to a iterative noise estimation using full-precision  networks. As an intuitive solution, quantization can significantly reduce the computational and memory consumption by low-bit parameters and operations. However, low-bit noise estimation networks in diffusion models (DMs) remain unexplored yet and perform much worse than the full-precision counterparts as observed in our experimental studies. In this paper, we first identify that the bottlenecks of low-bit quantized DMs come from a large distribution oscillation  on activations  and accumulated quantization error caused by the multi-step denoising process. To address these issues, we first develop a Timestep-aware Quantization (TaQ) method and a Noise-estimating Mimicking (NeM) scheme for low-bit quantized DMs (Q-DM) to effectively eliminate such oscillation and accumulated error respectively, leading to well-performed low-bit DMs. In this way, we propose an efficient Q-DM to calculate low-bit DMs by considering both training and inference process in the same framework. We evaluate our methods on popular DDPM and DDIM models. Extensive experimental results show that our method achieves a much better performance than the prior arts. For example, the 4-bit Q-DM theoretically accelerates the 1000-step DDPM by 7.8x and achieves a FID score of 5.17, on the unconditional CIFAR-10 dataset.

Q-DM: An Efficient Low-bit Quantized Diffusion Model

Abstract

Denoising diffusion generative models are capable of generating high-quality data, but suffers from the computation-costly generation process, due to a iterative noise estimation using full-precision networks. As an intuitive solution, quantization can significantly reduce the computational and memory consumption by lowbit parameters and operations. However, low-bit noise estimation networks in diffusion models (DMs) remain unexplored yet and perform much worse than the full-precision counterparts as observed in our experimental studies. In this paper, we first identify that the bottlenecks of low-bit quantized DMs come from a large distribution oscillation on activations and accumulated quantization error caused by the multi-step denoising process. To address these issues, we first develop a Timestep-aware Quantization (TaQ) method and a Noise-estimating Mimicking (NeM) scheme for low-bit quantized DMs (Q-DM) to effectively eliminate such oscillation and accumulated error respectively, leading to well-performed low-bit DMs. In this way, we propose an efficient Q-DM to calculate low-bit DMs by considering both training and inference process in the same framework. We evaluate our methods on popular DDPM and DDIM models. Extensive experimental results show that our method achieves a much better performance than the prior arts. For example, the 4-bit Q-DM theoretically accelerates the 1000-step DDPM by 7.8 × and achieves a FID score of 5.17, on the unconditional CIFAR-10 dataset.

1 Introduction

Denoising diffusion models, also known as score-based generative models [10, 33, 35], have recently shown remarkable success in various generative tasks such as images [10, 35, 22], audio [21], video [31], and graphs [23]. These models have also demonstrated flexibility in downstream tasks, making them attractive for tasks such as super-resolution [26, 7] and image-to-image translation [29]. Compared to Generative Adversarial Networks (GANs) [8], historically considered state-of-theart, diffusion models have proven to be superior in terms of quality and diversity in most of these tasks and applications. The process of diffusion models involves gradually transforming real data into Gaussian noise, which is then reversed via a denoising process to generate real data [10, 40]. However, such denoising process is time-consuming and involves iterating a neural network for noise estimation over thousands of timesteps, despite producing a significant amount of images. Therefore, researchers are actively working on accelerating this generation process to reduce its long iterative process and high inference cost for sample generation. To achieve this, one pipeline is to

Range

Range

Range

Channel_index

Channel_index

Channel_index

(a)

𝑡	 = 	100

(b)

𝑡	 = 400

(c)

𝑡	 = 700

1.5

1.5

1.5

1.0

1.0

1.0

0.5

0.5

0.5

0.0

0'0

0'0

0.5

-0.5

0.5

-1.0

1.0

1.0

1.

456

9 10111213141516171819202122232425262728293031

3 4 5 6 7 8 9 10111213141516171819202122232425262728293031

focus on sample trajectory learning, to develop faster sampling strategies [28, 22, 1]. While the other pipeline directly compresses and accelerates the noise estimation networks based on network quantization technology [30], which is particularly suitable for AI chips because of the low-bit parameters and operations. Prior post-training quantization (PTQ) methods [30, 19, 17] on diffusion models (DMs) or other neural networks directly compute quantized parameters based on pre-trained full-precision models, which constrains the model performance to a sub-optimized level without fine-tuning. Furthermore, quantizing DMs based on PTQ methods to ultra-low bits ( e.g. , 4 bits or lower) is ineffective and suffers from a significant performance reduction.

Differently, quantization-aware training (QAT) [16, 18] methods perform quantization during back propagation and generally achieve a less performance drop with a higher compression rate than PTQ. For instance, QAT has been shown to be effective for CNNs [5, 18] and ViTs [16, 18] and BERT [24]. However, QAT methods for low-bit quantization of diffusion models remain largely unexplored. Therefore, we first build a low-bit quantized DM baseline, a straightforward yet effective solution based on common techniques [5]. Our experimental studies reveal that the severe performance drop of low-bit quantized DMs, such as PTQ [30] and baseline [5], lies in the activation distribution oscillation and quantization error accumulation caused by the denoising process.

As shown in Fig. 1, the output distribution of the noise estimation network at each time step can differ significantly, resulting in activation distribution oscillation. Particularly, the distribution of activation in a specific layer varies significantly across different timesteps during training. We also observe that errors between full-precision activations and quantized activations gradually accumulate across timesteps during the sampling process (inference), making it harder to produce well-performed quantized DMs.

Drawing on the aforementioned insights, we propose a Timestep-aware Quantization (TaQ) method to address the oscillating distribution issue. By smoothing out these fluctuations and introducing more precise scaling factors into activations, we effectively enhance the performance of the low-bit quantized DMs. We further design a new training scheme for quantized DMs, dubbed Noiseestimating Mimicking (NeM), which can reduce the accumulated errors and promote the performance of quantized DMs based on the knowledge of full-precision counterparts. In this way, we achieve a new QAT method for low-bit quantized DM (Q-DM) via incorporating all the explorations (see the overview in Fig. 2). Overall, the contributions of this paper can be summarized as follows:

To the best of our knowledge, we proposed the first QAT method towards efficient low-bit DMs, dubbed Q-DM, by fully considering both training and inference process in the same framework.

Weintroduce a Timestep-aware Quantization (TaQ) method to mitigate activation distribution oscillation caused by the random-sampled timestep in the training process. We develop a Noise-estimating Mimicking (NeM) scheme to reduce accumulated errors, by which the Q-DMs are able to achieve comparable performance as the full-precision counterparts.

query

𝐚

!

key

𝐚

"

value

𝐚

#

𝐚 #

!

Attention score

𝐀

𝑸

$

Q-Conv2d

Q-Conv2d

Q-Conv2d

Quantize

Input activation

Q-AttnBlock in Q-DM

Timestep-

aware

Quantization

𝐚 #

"

𝐚 #

#

softmax(𝐚 *

!

⋅ 𝐚

*

"

⊺

)

Conv

Q-ResBlock

Q-AttnBlock

Q-UpSample

Q-DownSample

𝑡, 𝐱

!

𝝁 %

",!

𝐱

!$%

Sample

𝛜

",!

&

𝛜

",!

'(

ℒ

!"#

Forward pass

Supervision pass

U-Net

Learning Q-DM with Noise-estimating Mimicking

Concat

Timestep-

aware

Quantization

Timestep-

aware

Quantization

Matrix Multiplication

Extensive experiments on the CIFAR-10 and ImageNet datasets show that our Q-DM outperforms the baseline and 8-bit PTQ method by a large margin, and achieves comparable performances as the full-precision counterparts with a considerable acceleration rate.

2 Related Work

Network Quantization . Quantizing neural networks (QNNs) often possess low-bit (1 ∼ 4-bit) weights and activations to accelerate the model inference and save the memory usage. Specifically, ternary weights are introduced to reduce the quantization error in TWN [15]. DoReFa-Net [41] exploits convolution kernels with low bit-width parameters and gradients to accelerate both the training and inference. TTQ [42] uses two full-precision scaling coefficients to quantize the weights to ternary values. Zhuang et al. [43] present a 2 ∼ 4 -bit quantization scheme using a two-stage approach to alternately quantize the weights and activations, which provides an optimal trade-off among memory, efficiency, and performance. Jung et al. [12] parameterize the quantization intervals and obtain their optimal values by directly minimizing the task loss of the network and also the accuracy degeneration with further bit-width reduction. ZeroQ [2] supports both uniform and mixed-precision quantization by optimizing for a distilled dataset, which is engineered to match the statistics of batch normalization across different layers of the network. Xie et al. [39] introduces transfer learning into network quantization to obtain an accurate low-precision model by utilizing the Kullback-Leibler (KL) divergence. PWLQ [6] enables accurate approximation for tensor values that have bell-shaped distributions with long tails and finds the entire range by minimizing the quantization error.

Diffusion Model . The high cost of denoising through networks and the long iterative process make it difficult to implement diffusion models widely. To accelerate diffusion probabilistic models (DMs) [10], previous research has focused on finding shorter sampling trajectories while maintaining DMperformance. Wavegrad [3] introduces grid search, which finds an effective trajectory with only six timesteps, but this approach cannot be generalized for longer trajectories due to its exponentially growing time complexity. Watson et al. [38] model the trajectory searching as a dynamic programming problem. Song et al. [34] construct non-Markovian diffusion processes that lead to the same training objective, but whose reverse process can be much faster to sample from. For DMs with continuous timesteps, Song et al. [33, 35] have formulated the DM in the form of an ordinary differential equation (ODE) and improved sampling efficiency by using faster ODE solvers. Jolicoeur-Martineau et al. [11] have introduced an advanced SDE solver to accelerate the reverse process via an adaptively larger sampling rate. Analytic-dpm [1] has estimated variance and KL divergence using the Monte Carlo method and a pretrained score-based model with derived analytic forms that are simplified from the score-function. In addition to those training-free methods, Luhman & Luhman [20] have

compressed the reverse denoising process into a single-step model, while San-Roman et al. [28] has dynamically adjusted the trajectory during inference. However, implementing these methods requires additional training after obtaining a pretrained DM, which makes them less desirable in most situations. In summary, all these DM acceleration methods can be categorized as finding effective sampling trajectories.

Unlike prior works, we demonstrate that diffusion models can be accelerated by compressing the network in each noise estimating iteration, which is orthogonal with the fast sampling methods mentioned above. To the best of our knowledge, this is the first study to explore low-bit quantized diffusion models in a quantization-aware training (QAT) manner.

3 Background and Challenge

3.1 Diffusion Models

Forward process . Let x 0 be a sample from the data distribution x 0 ∼ q ( x ) . A forward diffusion process adds Gaussian noise to the sample for T times, resulting in a sequence of noisy samples x 1 , · · · , x T as:

where β t ∈ (0 , 1) is the variance schedule and controls the strength of the Gaussian noise in each step. The forward diffusion process satisfies the Markov property since each step relies solely on the preceding step. Additionally, as the number of steps increases towards infinity ( T →∞ ), the final state x T converges to an isotropic Gaussian distribution. A notable property of the forward process is that it admits sampling x t at an arbitrary timestep t in closed form as:

Reverse process . To generate a sample from a Gaussian noise input x T ∼ N ( 0 , I ) using diffusion models, the forward process is reversed. However, since the actual reverse conditional distribution q ( x t -1 | x t ) is unknown, diffusion models use a learned conditional distribution p θ ( x t -1 | x t ) that approximates the real reverse conditional distribution with a Gaussian distribution. This approximation is expressed as:

By using the re-parameterization trick presented in [10], it becomes possible to derive the mean ˜ µ θ,t ( x t ) and ˜ β t I as follows:

where α t = 1 -β t , ¯ α t = ∏ t i =1 α i and ϵ θ is a function approximator intended to predict ϵ from x t [10].

Training . At training time, the goal of optimization is to minimize the negative log-likelihood, i.e. , -log p θ ( x 0 ) . With variational inference, a lower bound of it could be found, denoted as L VLB :

It is found in [10] that using a simplified loss function to L VLB often obtains better performance:

Sampling . At inference time, a Gaussian noise tensor x T is sampled and is denoised by repeatedly sampling the reverse distribution p θ ( x t -1 | x t ) . ˜ µ θ, 1 ( x 1 ) is taken as the final generation result, with no noise added in the final denoising step.

(a)

𝑡 = 100

(b)

𝑡 = 400

(c)

𝑡 = 700

(d)

𝑡 = 1000

0.16

0.16

0.14

0.14

2T'0

0.08

0.08

0.06

0.06

0.04

+0'0

0.02

200

15

10

5

15

20

0.00

20

15

10

5

10

20

0.16

0.16

0.14

0.14

0.12

Density

0.10

Density

0.10

0.06

0.08

0.06

90'0

0.04

+00

200

200

000

-20

-15

10

20

00'0

-15

=10

3.2 Quantization

Given an N -layer CNN model, we denote its weight set as W = { w n } N n =1 and input feature map set as A = { a n in } N n =1 . The w n ∈ R C n out × C n in × K n × K n and a n in ∈ R C n in × W n in × H n in are the convolutional weight and the input feature map in the n -th layer, where C n in , C n out and K n respectively stand for input channel number, output channel number and the kernel size. Also, W n in and H n in are the width and height of the feature maps. Then, the convolutional outputs a n out can be technically formulated as:

where ⊗ represents the convolution operation. Herein, we omit the non-linear function for simplicity. Quantized neural network intends to represent w n and a n in a low-bit format such that the float-point convolutional outputs can be approximated as:

where ◦ denotes the channel-wise multiplication, ⊙ denotes the efficient GEMM operations, and s w n = { s w n 1 , s w n 2 , ..., s w n C n out } ∈ R C n out + is known as the channel-wise scaling factor vector [25] to mitigate the output gap between Eq. (7) and its approximation of Eq. (8). Meanwhile, we use the layer-wise quantization for input activations and the scaling factor of activations s a n in ∈ R + is a scalar.

3.3 Challenge Analysis

Here we identify two major challenges on low-bit DMs, specific to the multi-step inference process and random-sampled-step training process of diffusion models. Namely, we investigate on the distribution oscillation of the activations, and the accumulated quantization error resulted from the multi-step denoising process.

Activation distribution oscillation . To understand the distribution change of diffusion models, we investigate the activation distribution, w.r.t. timestep in the training process. Theoretically, if the distribution changes w.r.t. timestep, it would be difficult to implement previous QAT methods. We analyze the overall activation distributions of the noise estimation network, as shown in Fig. 3. We can observe that at different timesteps, the corresponding activation distributions have large discrepancies, e.g. , Fig. 3(a) v.s. Fig. 3(b), which makes previous QAT methods [16] in-applicable for multi-timestep models, i.e. , diffusion models.

Quantization error accumulation . Quantization of a noise estimation network introduces disturbances to the weights and activations, resulting in errors in each layer's output. Previous studies [4]

Sampling Timestep (

𝑡

)

MSE with full-precision activations

50

40

30

0

20

10

2-bit Mid Q-AttnBlock

2-bit DecoderQ-AttnBlock

3-bit Mid Q-AttnBlock

3-bit DecoderQ-AttnBlock

4-bit Mid Q-AttnBlock

4-bit Decoder Q-AttnBlock

have found that these errors tend to accumulate across layers, making it more challenging to quantize deeper neural networks. In the case of diffusion models (DMs), at each time step t , the input of the model ( x t -1 ) is obtained from the model's output at the previous time step t , i.e. , x t . As depicted in Fig. 4, the MSE distance, representing the quantization error of low-bit quantized DMs, exhibits a noticeable growth along with the decrease of sampling timestep. This implies that as the denoising process moves towards later timestep, the accumulation of quantization errors becomes more prominent.

4 The Proposed Q-DM

4.1 Timestep-aware Quantization

To tackle the distribution oscillation in the training process, we first introduce the quantized attention block that efficiently takes into account the timestep. This structure allows for the numerical analysis of activation ranges across different timesteps and mitigates distribution oscillation of low-bit quantized DMs. We recall the quantization in the attention block based on Eq. (8), which is formulated as:

where A is the attention score.

In the i -th mini-batch, the timestep is represented as { t 1 , · · · , t b i } , where b i is the batch size of the i -th batch. We denote i ∈ { 1 , · · · , B } , and B is the number of batch. Therefore, we calculate the timestep-aware distribution divergence for the query activation a q as:

where γ q ; t and σ 2 q ; t are statistical mean and variance of query activation a q . And the calculation of the key activation a k is likewise.

Based on such statistical results, the query and key activations in each specific timestep are smoothed as:

where ψ is constant to avoid 0 denominator. With the above timestep-aware smoothing process, we formulate our timestep-aware quantization as:

in which TaQ( ∗ ) = ⌊ clip([ ∗-γ ∗ ; t ] / [ s ∗ · √ σ 2 ∗ ; t + ψ ] , -2 b -1 , 2 b -1 -1) ⌉ . The smoothed activations are less sensitive to the random sampled timestep in the trianing process and the timestep-aware quantization, to some extent, dismisses the distribution oscillation phenomenon.

4.2 Noise-estimating Mimicking

To mitigate the negative impact of quantization error accumulation on the training of a quantized DM θ Q , a full-precision DM, denoted as θ FP , is incorporated into the training process to facilitate the learning objective. Following [10], with p θ Q ( x t -1 | x t ) = N ( x t -1 ; ˜ µ θ Q ,t ( x t ) , ˜ β t I ) and p θ FP ( x t -1 | x t ) = N ( x t -1 ; ˜ µ θ FP ,t ( x t ) , ˜ β t I ) , we can write:

where C is a constant that does not depend on θ Q or θ FP . As in Eq. (13), we aim to compel the quantized model to replicate the noise estimation capability of the full-precision model. Further, by re-parameterizing Eq. (2) as x t ( x 0 , ϵ ) = √ ¯ α t x 0 + √ 1 -¯ α t ϵ for ϵ ∼ N ( 0 , I ) and following the formulation in [10], which utilizes the formula for the posterior of the forward process, we can derive that:

where µ θ FP ( x t ( x 0 , ϵ ) , t ) and µ θ Q ( x t ( x 0 , ϵ ) , t ) are parameterized as:

where θ Q and θ fp are the noise estimated by the quantized DM and full-preciison counterpart. In Eq. (15), ϵ θ is a function approximator intended to predict ϵ from x t . Therefore, Eq. (14) is simplified to:

With the aforementioned derivation and parameterization, we have the final objective of our noiseestimating imitation, which is formulated as:

5 Experiments

In this section, we evaluate the proposed Q-DM framework on several popular diffusion models ( i.e. DDPM [10] and DDIM [32]) for unconditional image generation. To the best of our knowledge, there is no published work done on low-bit quantized diffusion models at this point, so we report LSQ [5] as a baseline. Experiments show our approach can achieve competitive generation quality to the full-precision scenario on all experimental settings under low-bit quantization.

5.1 Datasets and Implementation Details

We evaluate our method on two datasets including 32 × 32 generating size in CIFAR-10 [13] and 64 × 64 generating size in ImageNet [14]. For the CIFAR-10 [13] and ImageNet [14] datasets, we use the DDIM [32] sampler with 50/100 sampling timesteps and DDPM [10] with 1000 sampling timesteps. All the training settings are the same as DDPM [10]. For DDIM sampler, we set η in DDIM [32] as 0 . 5 for the best performance. We evaluate the performance of our method using FID [9] and Inception Score (IS) [27] on both CIFAR-10 [13] and ImageNet [14] datasets. We set the training timestep T = 1000 for all experiments, following [10]. We set the forward process variances to constants increasing linearly from β 1 = 1 e -4 to β T = 0 . 02 . To represent the reverse process, we use a U-Net backbone, following [10, 32]. Parameters are shared across time, which is specified to the network using the Transformer sinusoidal position embedding [36]. We use self-attention at the 16 × 16 feature map resolution [36, 37].

5.2 Ablation Study

We give quantitative results of the proposed TaQ and NeM in Tab. 1 As can be seen, the low-bit quantized DM baseline [5] suffers a severe performance drop on image generation task compared with full-precision DMs (5.55, 8.57, and 14.07 performance gap in terms of FID score with 4/3/2-bit, respectively). TaQ and NeM improve the performance of generation when used alone. For example, the 4-bit quantized DM basline with TaQ and NeM introduced separately achieves 0.97 and 1.24 FID score decrease, respectively.

Moreover, the two techniques further boost the performance considerably when combined together. For instance, when combining the TaQ and NeM together, the performance of 4/3/2-bit quantized DMs improvement achieves 3.33, 4.07, and 3.48 respectively. To conclude, the two techniques can promote each other to improve Q-DM and close the performance gap between low-bit quantized DMs and full-precision counterpart.

5.3 Main Results

The experimental results are shown in Tab. 2. We compare our method with 4/3/2-bit baseline [5] based on the same frameworks for the task of unconditional image generation with the CIFAR-10 [13] and ImageNet [14] dataset. We also report the classification performance of the 8-bit PTQ method, i.e. , PTQ4DM [30]. We firstly evaluate the proposed method on CIAFR-10 [13] with DDIM [32] and DDPM [10]. We use the model size and OPs (defined in [18]) to evaluate the efficiency of quantized and full-precision models.

For 50-step DDIM sampler, compared with 8-bit PTQ4DM [30], our 4-bit Q-DM achieves a much larger compression ratio than 8-bit PTQ4DM, but with significant performance improvement (6.89 FID ↓ vs. 18.02 FID ↓ ). And it is worth noting that the proposed 2-bit model significantly compresses the DDIM by 30.9 × on OPs. The proposed method boosts the performance of 4/3/2-bit Baseline by 3.33, 4.17, and 3.48 in terms of FID score with the same architecture and bit-width, which is significant on the CIFAR-10 [13] dataset with 32 × 32 generating resolution. For 1000-step DDPM, the performance of the proposed method outperforms the 4/3/2-bit Baseline by 3.94, 4.14, and 2.58, a large margin. Also note that the proposed 4/3/2-bit model significantly accelerates the generation by 7.8 × , 15.6 × , and 30.9 × on OPs. Compared with 8-bit PTQ4DM, ours achieve significantly higher compression and acceleration rate, while the performance improvement is considerable.

Also, our method generates convincing results on ImageNet [14] dataset. As shown in Tab. 2, the performance of the proposed method with 50-step DDIM significantly outperforms the 4/3/2-bit Baseline method by 4.76 , 4.16, and 4.01. Compared with 8-bit PTQ method, our method achieves significantly higher compression rate and acceleration rate, but with better performance. For 1000step DDPM on ImageNet [14] dataset, the performance of the proposed method outperforms the 4/3/2-bit Baseline by 4.39, 4.54, and 3.71. Also note that our 4-bit Q-DM surpasses the full-precision 50/100-step DDIM and 1000-step DDPM and significantly compresses the noise estimation networks by 7.9 × , which demonstrates the effectiveness and efficiency of our Q-DM.

6 Conclusion

In this paper, we present Q-DM, an efficient low-bit quantized diffusion model that offers a high compression ratio and competitive performance in image generation task. Initially, we analyze the challenges of the low-bit quantized DM. Our empirical analysis show that distribution oscillation in activation is the one of the cause of the significant drop in DM quantization. Another challenge lies in the accumulated quantization error resulted from the multi-step denoising process during inference. To address these issues, we first develop a timestep-aware quantization (TaQ) method and a noise-estimating mimicking (NeM) scheme for low-bit quantized DMs, to effectively address these two challenges. Our work provides a comprehensive analysis and effective solutions for the crucial issues in low-bit quantized diffusion model, paving the way for the extreme compression and acceleration of diffusion model.

References

Fan Bao, Chongxuan Li, Jun Zhu, and Bo Zhang. Analytic-dpm: an analytic estimate of the optimal reverse variance in diffusion probabilistic models. In Proc. of ICLR , pages 1-39, 2022.

Yaohui Cai, Zhewei Yao, Zhen Dong, Amir Gholami, Michael W Mahoney, and Kurt Keutzer. Zeroq: A novel zero shot quantization framework. In Proc. of CVPR , pages 13169-13178, 2020.

Nanxin Chen, Yu Zhang, Heiga Zen, Ron J Weiss, Mohammad Norouzi, and William Chan. Wavegrad: Estimating gradients for waveform generation. arXiv preprint arXiv:2009.00713 , 2020.

Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer. Llm. int8 (): 8-bit matrix multiplication for transformers at scale. arXiv preprint arXiv:2208.07339 , 2022.

Steven K Esser, Jeffrey L McKinstry, Deepika Bablani, Rathinakumar Appuswamy, and Dharmendra S Modha. Learned step size quantization. In Proc. of ICLR , pages 1-12, 2019.

Jun Fang, Ali Shafiee, Hamzah Abdel-Aziz, David Thorsley, Georgios Georgiadis, and Joseph H Hassoun. Post-training piecewise linear quantization for deep neural networks. In Proc. of ECCV , pages 69-86, 2020.

Sicheng Gao, Xuhui Liu, Bohan Zeng, Sheng Xu, Yanjing Li, Xiaoyan Luo, Jianzhuang Liu, Xiantong Zhen, and Baochang Zhang. Implicit diffusion models for continuous super-resolution. arXiv preprint arXiv:2303.16491 , 2023.

Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks. Communications of the ACM , pages 139-144, 2020.

Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. In Proc. of NeurIPS , pages 1-12, 2017.

Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In Proc. of NeurIPS , pages 6840-6851, 2020.

Alexia Jolicoeur-Martineau, Ke Li, Rémi Piché-Taillefer, Tal Kachman, and Ioannis Mitliagkas. Gotta go fast when generating data with score-based models. arXiv preprint arXiv:2105.14080 , 2021.

Sangil Jung, Changyong Son, Seohyung Lee, Jinwoo Son, Jae-Joon Han, Youngjun Kwak, Sung Ju Hwang, and Changkyu Choi. Learning to quantize deep networks by optimizing quantization intervals with task loss. In Proc. of CVPR , pages 4350-4359, 2019.

Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. Technical report, Citeseer, 2009.

Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. In Proc. of NeurIPS , pages 1097-1105, 2012.

Fengfu Li, Bo Zhang, and Bin Liu. Ternary weight networks. arXiv preprint arXiv:1605.04711 , 2016.

Yanjing Li, Sheng Xu, Baochang Zhang, Xianbin Cao, Peng Gao, and Guodong Guo. Q-vit: Accurate and fully quantized low-bit vision transformer. In Proc. of NeurIPS , pages 1-12, 2022.

Yang Lin, Tianyu Zhang, Peiqin Sun, Zheng Li, and Shuchang Zhou. Fq-vit: Fully quantized vision transformer without retraining. In Proc. of IJCAI , pages 1173-1179, 2022.

Zechun Liu, Zhiqiang Shen, Marios Savvides, and Kwang-Ting Cheng. Reactnet: Towards precise binary neural network with generalized activation functions. In Proc. of ECCV , pages 143-159, 2020.

Zhenhua Liu, Yunhe Wang, Kai Han, Wei Zhang, Siwei Ma, and Wen Gao. Post-training quantization for vision transformer. In Proc. of NeurIPS , pages 1-12, 2021.

Eric Luhman and Troy Luhman. Knowledge distillation in iterative generative models for improved sampling speed. arXiv preprint arXiv:2101.02388 , 2021.

Gautam Mittal, Jesse Engel, Curtis Hawthorne, and Ian Simon. Symbolic music generation with diffusion models. arXiv preprint arXiv:2103.16091 , 2021.

Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. In Proc. of ICML , pages 8162-8171, 2021.

Chenhao Niu, Yang Song, Jiaming Song, Shengjia Zhao, Aditya Grover, and Stefano Ermon. Permutation invariant graph generation via score-based generative modeling. In Proc. of AISTATS , pages 4474-4484, 2020.

Haotong Qin, Yifu Ding, Mingyuan Zhang, Qinghua Yan, Aishan Liu, Qingqing Dang, Ziwei Liu, and Xianglong Liu. Bibert: Accurate fully binarized bert. In Proc. of ICLR , pages 1-24, 2022.

Mohammad Rastegari, Vicente Ordonez, Joseph Redmon, and Ali Farhadi. Xnor-net: Imagenet classification using binary convolutional neural networks. In Proc. of ECCV , pages 525-542, 2016.

Chitwan Saharia, Jonathan Ho, William Chan, Tim Salimans, David J Fleet, and Mohammad Norouzi. Image super-resolution via iterative refinement. TPAMI , 2022.

Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans. In Proc. of NeurIPS , pages 1-9, 2016.

Robin San-Roman, Eliya Nachmani, and Lior Wolf. Noise estimation for generative diffusion models. arXiv preprint arXiv:2104.02600 , 2021.

Hiroshi Sasaki, Chris G Willcocks, and Toby P Breckon. Unit-ddpm: Unpaired image translation with denoising diffusion probabilistic models. arXiv preprint arXiv:2104.05358 , 2021.

Yuzhang Shang, Zhihang Yuan, Bin Xie, Bingzhe Wu, and Yan Yan. Post-training quantization on diffusion models. arXiv preprint arXiv:2211.15736 , 2022.

Uriel Singer, Adam Polyak, Thomas Hayes, Xi Yin, Jie An, Songyang Zhang, Qiyuan Hu, Harry Yang, Oron Ashual, Oran Gafni, et al. Make-a-video: Text-to-video generation without text-video data. arXiv preprint arXiv:2209.14792 , 2022.

Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In Proc. of ICLR , pages 1-20, 2020.

Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. In Proc. of NeurIPS , pages 1-13, 2019.

Yang Song and Stefano Ermon. Improved techniques for training score-based generative models. In Proc. of NeurIPS , pages 12438-12448, 2020.

Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In Proc. of ICLR , pages 1-36, 2021.

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Proc. of NeurIPS , pages 1-11, 2017.

Xiaolong Wang, Ross Girshick, Abhinav Gupta, and Kaiming He. Non-local neural networks. In Proc. of CVPR , pages 7794-7803, 2018.

Daniel Watson, Jonathan Ho, Mohammad Norouzi, and William Chan. Learning to efficiently sample from diffusion probabilistic models. arXiv preprint arXiv:2106.03802 , 2021.

Zheng Xie, Zhiquan Wen, Jing Liu, Zhiqiang Liu, Xixian Wu, and Mingkui Tan. Deep transferring quantization. In Proc. of ECCV , pages 625-642, 2020.

Ling Yang, Zhilong Zhang, Yang Song, Shenda Hong, Runsheng Xu, Yue Zhao, Yingxia Shao, Wentao Zhang, Bin Cui, and Ming-Hsuan Yang. Diffusion models: A comprehensive survey of methods and applications. arXiv preprint arXiv:2209.00796 , 2022.

Shuchang Zhou, Yuxin Wu, Zekun Ni, Xinyu Zhou, He Wen, and Yuheng Zou. Dorefa-net: Training low bitwidth convolutional neural networks with low bitwidth gradients. arXiv preprint arXiv:1606.06160 , 2016.

Chenzhuo Zhu, Song Han, Huizi Mao, and William J Dally. Trained ternary quantization. In Proc. of ICLR , pages 1-10, 2017.

Bohan Zhuang, Chunhua Shen, Mingkui Tan, Lingqiao Liu, and Ian Reid. Towards effective low-bitwidth convolutional neural networks. In Proc. of CVPR , pages 7920-7928, 2018.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a quantization-aware training framework for diffusion models. The authors identified activation distribution oscillation and quantization error accumulation as the main causes of the performance drop. To close the performance gap, the authors developed a timestep-aware quantization (TaQ) method for data normalization and a noise-estimating mimicking (NeM) training scheme. Experiments were conducted to demonstrate the effectiveness of the proposed method.\n\nSTRENGTHS:\n1. The motivation of this paper is valid. The proposed method is reasonable. The shifting of the activation distributions and the quantization error accumulation are quite likely to affect the overall performance. The authors propose Timestep-aware Quantization (TaQ) and Noise-estimating Mimicking (NeM) accordingly to address these two problems.\n2. The paper is well-organized and the writing is fluent. This helps the readers capture the high-level ideas well and the detailed experimental setups make this paper easy to replicate the results.\n3. The experimental results are promising. Q-DMs demonstrate performance on par with FP models.\n\nWEAKNESSES:\n1. Will the TaQ quantization process affect the inference speed? See 3.b in the ``Questions'' section.\n2. Some technical details should be clarified (see the questions below).\n\nQUESTIONS:\n1. What does \"PTQ4DM\" exactly refer to in Tab. 2? Tab.2 cites [1] while referring PTQ4DM to [2] in Line 243. As far as I am concerned, PTQ4DM refers to a recently published work [2], presenting a method for calibration data collection in diffusion quantization, then what does Line 242 mean by saying \"we also report the **classification performance**...\" ?\n\n2. I have some questions regarding the method details, I would like to ask for some clarifications:\n    a. Did you calculate the statistical mean and variance of each activation in an offline manner, i.e. the statistical values were pre-computed with the full-precision DMs and not updated with the training procedure?\n    b. How did you conduct inference with such the timestep-aware design? Did you conduct activation normalization with the pre-computed values on-the-fly? If so, how would it affect the inference efficiency?\n   c. How is the \"distance\" in Figure 4 calculated? Also, is the activation distribution in Figure 3 layer-wise or channel-wise?\n   d. Is it possible to combine the original training loss in DMs with the NeM loss?\n\n[1] Post-training piecewise linear quantization for deep neural networks\n\n[2] Post-Training Quantization on Diffusion Models\n\nLIMITATIONS:\nThe authors do not include the limitations and potential negative societal impact in the paper.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposed a quantization-aware training scheme for diffusion models, based on the well-known method, LSQ. \nIn the paper, they identified the bottleneck come from a large distribution oscillation on activations and accumulated quantization error caused by the denoising process. \nThen, they suggest method to address the issues:\nTimestep-aware quantization (TaQ) – Time Step-aware activation smoothing to handle a large oscillation on activation\nNoise-estimating Mimicking (NeM):  reduceing error accumulation throughout the multi-step denoising process with quantization\n\nSTRENGTHS:\nThey observed and identified the issues when compressing DMs in a quantization-aware training manner.\n\nWEAKNESSES:\n1.\tThe suggested methods to tackle the issues in QAT for DM are not novel. TaQ and NeM seem just normalization and knowledge distillation which can be commonly used in the related literature.\n2.\tThere is much lack of numerical descriptions for the QAT process, such as training time, the number of calibration datasets, and the number of epochs, etc. They have to provide detail about how many resources are needed for reproduction.\n3.\tThe models used in the paper to show their performance are a little bit outdated. The datasets they used are also insufficient to prove the performance. They should provide more thorough experimental results with recent models and datasets.\n\nQUESTIONS:\n1.\tGenerally, QAT requires more data and training time than PTQ, and U-net inherently has a large number of parameters, so requires a lot of training data and takes a long time to optimize. How much data is used for QAT for DM? And how much time did you take to complete the quantization for U-net?\n2.\tWhen performing inference, computing the statistical mean and variance are needed every time step? If so it imposes the additional cost for H/W. How are these values considered at inference time?\n3.\tAs a naive approach, we may have a different step size of activation for each time step to handle the oscillation on activations. Is there any problem with the naive approach compared to your approach?\n4.\tPTQ4DM and DFQ also handle the dynamic range of activations. Compared with them, what are the advantages of your method?\n5.\tThe problem you observed in quantizing DM may have been dealt with in other literature, even if it is not related to quantization and DM. For example, there are many approaches to normalize the value. The paper would be more robust if the authors provide sufficient related works handling the issues the authors addressed.\n6.\tIn Eq(11) and (12), TaQ quantize normalized activations. Is there no need for compensation for the normalization?\n\nLIMITATIONS:\nSee weakness and questions.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this paper, a novel method called Q-DM is introduced, which enables the creation of low-bit quantized diffusion models. The authors first give extensive analysis about two primary challenges faced by low-bit quantized DMs: significant distribution oscillation on activations and accumulated quantization error arising from the multi-step denoising process. To address these issues, the authors propose two techniques: Timestep-aware Quantization (TaQ) and Noise-estimating Mimicking (NeM). TaQ is designed to mitigate the distribution oscillation problem, while NeM aims to reduce the accumulated quantization error. By incorporating these techniques into Q-DM, the paper demonstrates its ability to overcome these challenges effectively, leading to superior performance compared to existing methods. The experimental results provided in the paper serve as evidence of the high-quality performance achieved by Q-DM.\n\nSTRENGTHS:\n1.This paper is easy to follow. The organization of this paper are exemplary, as it effectively presents the proposed Q-DM in a clear and comprehensible manner. The paper provides comprehensive explanations of how Q-DM improved the performance of quantized DMs. \n\n2.This paper presents an novel quantization method known as Q-DM, which addresses several challenges in low-bit quantized diffusion models. The authors propose the Timestep-aware Quantization (TaQ) method to tackle the issue of activation distribution oscillation, which is caused by the random-sampled timestep during training. Additionally, they introduce the Noise-estimating Mimicking (NeM) scheme to effectively minimize accumulated errors. \n\n3.The experimental results are significant. The Q-DMs exhibit performance comparable to that of full-precision models, showcasing the effectiveness of the proposed methods.\n\nWEAKNESSES:\n1.Is the model quantized during all the training and sampling process? Since these may lead to different inference speed. And are the quantization-related parameter all the same across different timestep?\n\n2.Is the proposed TaQ method only used in Q-AttnBlock module? And how is the conv layer quantized in Q-DM?\n\n3.In Table 2, how is the ‘OPs’ calculated?  It would be more detailed if authors can provide some description about this metric.\n\nQUESTIONS:\nSee weaknesses.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

comments: SUMMARY:
The paper proposes two method to mitigate the accuracy degradation caused by quantization of diffusion models: one is time-step aware quantization (different calibration data and range for each time-step of diffusion), the other is using a full-precision network for training time distillation. Experiments on image generation in Cifar-10 and ImageNet datasets show the efficacy of the method.

STRENGTHS:
By using QAT, the proposed method manages to bring down the bitwidth of the neural network to lower than 8-bit. The FID and IS metrics support the efficacy of the method.

WEAKNESSES:
The benefits of per-time-step quantization has been known since earlier works like PTQ4DM and q-diffusion. It is desirable to know the proposed method and prior works. Note that QAT and PTQ do not make much difference here, as the underlying motivation of per-time-step quantization to deal with changing dynamic range change is the same here. Also having per-time-step quantization is almost zero-overhead as the weights need be loaded for computation of each step anyway.

Also having a full-precision network as teacher for knowledge distillation (including having full precision or higher precision teacher) is quite a standard approach. It's not obvious what specialties of QAT are incorporated here.

q-diffusion: https://arxiv.org/abs/2302.04304

QUESTIONS:
Can visual results be provided for intuitive inspection? In particular, spotty noise may not be well captured by FID/IS metrics but will be blatant when doing visual inspection.

In Table 2, there are a few pairs of results with similar IS but very different FID, like PTQ4DM CIFAR-10 32×32 50 steps and Baseline CIFAR-10 32×32 3/3 . Any discussions of this variation?

LIMITATIONS:
No particular limitations.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2023-07-06

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper identifies two challenges in low-bit diffusion models (DMs): activation distribution oscillation and quantization error accumulation. To tackle these challenges, the paper introduces two novel techniques: Timestep-aware Quantization (TaQ) and Noise-estimating Mimicking (NeM). Experimental results demonstrate superior performance compared to previous approaches on CIFAR-10 and ImageNet 64x64 datasets.\n\nSTRENGTHS:\nThe paper is easy to follow and low-bit diffusion model is an interesting idea worth to try. The presented ablation studies demonstrate that the proposed methods have a positive impact on quantized diffusion models.\n\nWEAKNESSES:\n1. The main contribution of this paper involves the utilization of statistical mean and variance to address distribution oscillation. Essentially, this approach is akin to applying a shift and scale operation following quantized operations, which is a common technique employed in low-bit quantization to achieve balanced quantization bins. Similar methods for weight and activation balancing can be found in [1] and [2, 3] respectively. \n\n2. In Equation (12), the application of TaQ to the softmax attention scores appears questionable. Unlike other activations that typically conform to a Gaussian or bell-shaped distribution, softmax attention scores often exhibit a long-tailed distribution, where the sum of the probabilities is 1. Normalizing the softmax attention scores may disrupt the probabilistic interpretation of these scores, which raises concerns about its appropriateness.\n\n3. The proposed Noise-estimating Mimicking technique necessitates fine-tuning the entire model, which may pose practical challenges for large diffusion models trained on extensive datasets like LAION-5B. Efficient fine-tuning method should be considered.\n\n4. The experiments in the paper are limited to CIFAR-10 and ImageNet 64x64 datasets, with a maximum model size of 4.47M. To provide a comprehensive evaluation, it is recommended to conduct additional experiments on larger models, such as LDM[4], and include commonly used datasets like LSUN and ImageNet 256x256.\n\n5. The accuracy comparison experiment in this paper seems to be inadequate, and there are issues with the citations for the compared method PTQ4DM [5]. Moreover, discrepancies exist between the FID and IS results for full-precision (FP) models in this paper and the results reported in PTQ4DM, which could potentially introduce unfairness in the comparisons. As an example, when employing the DDIM sampler for 100 steps on CIFAR-10, the FID for the FP model is reported as 10.05 in PTQ4DM, whereas this paper indicates a result of 4.16.\n\n[1] Qin, Haotong, et al. \"Forward and backward information retention for accurate binary neural networks.\" CVPR 2020.\n[2] Liu, Zechun, et al. \"Reactnet: Towards precise binary neural network with generalized activation functions.\" ECCV 2020.\n[3] Wei, Xiuying, et al. \"Outlier Suppression+: Accurate quantization of large language models by equivalent and optimal shifting and scaling.\" arXiv:2304.09145 (2023).\n[4] Rombach, Robin, et al. \"High-resolution image synthesis with latent diffusion models.\" CVPR 2022.\n[5] Shang, Yuzhang, et al. \"Post-training quantization on diffusion models.\" CVPR 2023.\n\nQUESTIONS:\nHow are the model size and OPs calculated? The quantized model's size reported in the paper is precisely 1/bits of the full precision model. However, this is unusual because there must be some parameters that remain in full-precision.\n\nLIMITATIONS:\nI did not identify any issues related to limitations and potential negative societal impact.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-14"}