Title: Temporal Dynamic Quantization for Diffusion Models
Venue: NeurIPS 2023
Accepted: yes
Keywords: deep learning optimization

quantization

diffusion model

generative model
Abstract: Diffusion model has gained popularity in vision applications due to its remarkable generative performance and versatility. However, its high storage and computation demands, resulting from the model size and iterative generation, hinder its use on mobile devices. Existing quantization techniques struggle to maintain performance even in 8-bit precision due to the diffusion model's unique property of temporal variation in activation. We introduce a novel quantization method that dynamically adjusts the quantization interval based on time step information, significantly improving output quality. Unlike conventional dynamic quantization techniques, our approach has no computational overhead during inference and is compatible with both post-training quantization (PTQ) and quantization-aware training (QAT). Our extensive experiments demonstrate substantial improvements in output quality with the quantized model across various configurations.

Temporal Dynamic Quantization for Diffusion Models

Abstract

The diffusion model has gained popularity in vision applications due to its remarkable generative performance and versatility. However, high storage and computation demands, resulting from the model size and iterative generation, hinder its use on mobile devices. Existing quantization techniques struggle to maintain performance even in 8-bit precision due to the diffusion model's unique property of temporal variation in activation. We introduce a novel quantization method that dynamically adjusts the quantization interval based on time step information, significantly improving output quality. Unlike conventional dynamic quantization techniques, our approach has no computational overhead during inference and is compatible with both post-training quantization (PTQ) and quantization-aware training (QAT). Our extensive experiments demonstrate substantial improvements in output quality with the quantized diffusion model across various datasets.

1 Introduction

Generative modeling is crucial in machine learning for applications such as image [1, 2, 3, 4, 5, 6, 7, 8], voice [9, 10], and text synthesis [11, 12, 13]. Diffusion models [3, 2, 4], which progressively refine input images through a denoising process involving hundreds of iterative inferences, have recently gained prominence due to their superior performance compared to alternatives like GANs [14]. However, the high cost of diffusion models presents a significant barrier to their widespread adoption. These models have large sizes, often reaching several gigabytes, and demand enormous computation of iterative inferences for a single image generation. Consequently, executing diffusion models on resource-limited mobile devices is practically infeasible, thus most applications are currently implemented on expensive, high-performance servers.

To fully exploit the potential of diffusion models, several methods to reduce the computational cost and memory requirement of diffusion models while preserving the generative performance have been proposed. For example, J. Song et al. [2] and L. Liu et al. [15] proposed a more efficient sampling scheduler, and T. Salimans & J. Ho proposed to reduce the number of sampling steps using knowledge distillation technique [16, 17]. As a result, high-fidelity images can be generated with fewer sampling steps, making the use of diffusion models more affordable and feasible. Despite these advancements, the denoising process of diffusion models still demands a substantial computational cost, necessitating further performance enhancements and model compression.

While the majority of previous approaches have focused on reducing the number of sampling steps to accelerate the denoising process, it is also important to lighten the individual denoising steps. Since the single denoising step can be regarded as a conventional deep learning model inference, various model compression techniques can be used. Quantization is a widely used compression technique where both weights and activations are mapped to the low-precision domain. While advanced quantization schemes have been extensively studied for conventional Convolution Neural Networks (CNNs) and

𝒙

𝟎

~	𝒑(𝒙)

𝐑𝐞𝐯𝐞𝐫𝐬𝐞	𝐏𝐫𝐨𝐜𝐞𝐬𝐬				𝑝

"

𝑥

#$%

|𝑥

#

= 𝑁(𝑥

#$%

; 𝜇

"

𝑥

#

, 𝑡

, 𝜎

#

&

𝐼)

𝐅𝐨𝐰𝐫𝐚𝐝	𝐏𝐫𝐨𝐜𝐞𝐬𝐬				𝑞 𝑥

#

𝑥

#$%

= 𝑁(𝑥

#

;

1 - 𝛽

#

𝑥

#$%

, 𝛽

#

𝐼)

𝒙

𝑻

~	𝑵(𝟎, 𝑰)

language models, their application to diffusion models has shown significant performance degradation. Due to the unique property of diffusion models, such as the significant changes in the activation distribution throughout the iterative inferences, the output is heavily distorted as the activation bitwidth decreases [18]. Existing quantization techniques, including both quantization-aware training (QAT) [19, 20, 21] and Post-training quantization (PTQ) [22, 23, 24, 25], are designed to address a specific distribution in existing DNNs, therefore cannot deal with time-variant activation distributions in diffusion models.

To tackle the unique challenges of diffusion model quantization, we introduce a novel design called Temporal Dynamic Quantization (TDQ) module. The proposed TDQ module generates a timedependent optimal quantization configuration that minimizes activation quantization errors. The strong benefit of the TDQ module is the seamless integration of the existing QAT and PTQ algorithms, where the TDQ module extends these algorithms to create a time-dependent optimal quantization configuration that minimizes activation quantization errors. Specifically, the module is designed to generate no additional computational overhead during inference, making it compatible with existing acceleration frameworks without requiring modifications. The TDQ module significantly improves quality over traditional quantization schemes by generating optimal quantization parameters at each time step while preserving the advantages of quantization.

2 Backgrounds and Related Works

2.1 Diffusion Model

Diffusion models have been first introduced in 2015 [26] and revolutionized image generation by characterizing it as a sequential denoising process. As shown in Fig. 1, the forward diffusion process gradually transforms the image( x 0 ) into a random data( x T ) which follows standard normal distribution by adding small Gaussian noise at each time step. The reverse diffusion process generates a clean image( x 0 ) from a random data( x T ) by gradually removing noise from the data through iterative denoising steps. Therefore, a diffusion model learns the reverse process which is estimating the noise amount on a given noisy data at each time step( x t ). The forward( q ) and reverse( p θ ) processes can be described as

where β t denotes the magnitude of Gaussian noise.

[3] introduced a reparameterization trick for µ θ and the corresponding loss function, which facilitates the training of diffusion models.

While diffusion models can produce high-quality images, the iterative denoising process makes diffusion models difficult to be used for real-world scenarios. The early works on diffusion models such as DDPM [3] required hundreds to thousands of iterative inferences to generate single image, resulting extremely slow sampling speed. Therefore, numerous studies have been investigating algorithmic enhancements and various optimizations to boost performance, aiming to make diffusion

models more efficient and suitable for real-world applications. DDIM [2] introduced an implicit probabilistic model that reinterprets the Markov process of the DDPM method achieving competitive image quality with one-tenth of denoising stpes. Distillation-based methods [16, 17] proposed to reduce the number of denoising steps using knowledge distillation techniques.

On the other hand, to address the significant computational overhead associated with generating high-resolution images, [4] proposed a Latent Diffusion Model (LDM) where the diffusion model takes latent variables instead of images. Especially, the large-scale diffusion model (e.g., Stable Diffusion [4]) has leveraged LDM and learned from a large-scale dataset (LAION-Dataset [27]), enabling the creation of high-quality, high-resolution images conditioned on textual input.

2.2 Quantization

Quantization is a prominent neural network optimization technique that reduces storage requirements with low-precision representation and performance improvement when the corresponding lowprecision acceleration is available. With b -bit precision, only 2 b quantization levels are accessible, which considerably limits the degree of freedom of the quantized model compared to floating-point representation. Therefore, the final quality significantly varies depending on the tuning of quantization hyperparameters, such as the quantization interval or zero offset. To preserve the quality of output in low-precision, it is crucial to update the quantization parameters as well as the model parameters, taking into account the specific features and requirements of the target task.

2.3 Quantization-aware Training and Post-training Quantization

Quantization algorithms can be broadly categorized into two types: Quantization-Aware Training (QAT) [19, 28, 29, 20, 30] and Post-Training Quantization (PTQ) [23, 24, 25]. QAT applies additional training after introducing quantization operators, allowing the update of network parameters toward minimizing the final loss value considering the effect of the quantization operators. On the other hand, PTQ does not apply end-to-end forward/backward propagation after quantization. Instead, it focuses on reducing block-wise reconstruction errors induced by quantization. QAT typically outperforms PTQ in low-precision scenarios, but it may not always be applicable due to limitations such as the deficiencies of datasets, training pipelines, and resource constraints. Due to its practical usefulness, PTQ has been actively researched recently. In the literature of diffusion models, [18] introduced a dedicated 8-bit post-training quantization method, demonstrating high-fidelity image generation performance.

On the other hand, while most QAT and PTQ algorithms focus on static quantization, recent research has highlighted the benefits of input-dependent dynamic quantization [31, 32, 33, 34]. Dynamic quantization enables the adjustment of quantization intervals based on the varying input-dependent distribution of activations, which has the potential to reduce quantization error. However, implementing these dynamic approaches often incur additional costs for extracting statistical information from activations, making it challenging to achieve performance improvements in practice. For instance, [35] pointed out that input-dependent activation functions can cause significant latency increases despite having relatively little computation.

While a number of previous works proposed different methods to accelerate sampling process of diffusion models, there have been limited works tried to exploit dynamic nature of diffusion models. In this paper, we propose a novel quantization scheme for diffusion models which minimizes activation quantization errors by enabling the generation of suitable quantization interval based on the time step information, all without incurring additional inference costs.

3 Temporal Dynamic Quantization

3.1 Quantization Methods

Before elaborating the proposed method, we define the quantization function used in our work. In this study, we focus on b -bit linear quantization, where the 2 b possible quantization levels are evenly spaced. Linear quantization involves two key hyperparameters: the quantization interval s and the zero offset z . Given the full-precision data x , the quantized data ˆ x can be calculated as follows:

0

10

20

30

40

50

60

70

80

90

T

4

2

0

2

4

conv_in.0

0

10

20

30

40

50

60

70

80

90

T

100

0

100

200

up.1.upsample.conv.0

(a) Step size is calibrated at T=0 (a) The interval is calibrated at T=0 (a) Step size is calibrated at T=0

activation distribution

Truncation error is too high

T=100

T=0

(a) Step size is calibrated at T=0 (a) Step size is calibrated at T=0 (b) The interval is calibrated at T=100

activation distribution

Rounding error is too high

T=100

T=0

activation distribution

Truncation error is too high

T=100

T=0

activation distribution

Rounding error is too high

T=100

T=0

̸

W8A8 W8A8 where n ( p ) is the smallest(largest) quantization index, clip ( · ) is a clipping function and ⌊·⌉ is a rounding function. For practical acceleration purposes, we utilize symmetric quantization for weights, where z = 0 and p = -n = 2 ( b -1) -1 . On the other hand, for activation quantization, we assume asymmetric quantization with z = 0 , n = 0 , and p = 2 b -1 , which is a commonly adopted approach [36].

3.2 Challenges of Diffusion Model Quantization

The biggest challenge of quantizing diffusion models is to find optimal quantization parameters ( s and z ) for activation that minimize the quantization error. As shown in Fig. 2, activation distribution of diffusion models have a unique property due to the iterative denoising process, whose distribution highly varies depending on the time step ( t ) regardless of the layer index.

Therefore, using static values for quantization parameters causes significant quantization error for different time steps, as shown in Fig. 3. Previous studies [37, 18] has also reported the dynamic property of activations in diffusion models and attempted to address it by sampling the calibration dataset across overall time frames. However, despite these efforts, these studies still relied on static parameters, resulting in sub-optimal convergence of minimizing quantization error. To tackle this problem fundamentally, it is crucial to enable the update of quantization parameters considering the dynamics in the input activation distribution.

3.3 Implementation of TDQ Module

To address the rapid changes in input activation, one easy way of think is input-dependent dynamic quantization. Previous studies have demonstrated that incorporating a quantization module that generates the quantization parameters based on input features such as minimum, maximum, and average values can lead to significant improvements in accuracy for specific applications [38, 32]. According to our observation, as will be presented in the Appendix, our own implementation of input-dependent dynamic quantization offers notable quality improvement. However, the process of gathering statistics on the given activation introduces complex implementation and notable overhead [35]. Despite the potential of quality improvement, this approach may not be an attractive solution in practice.

s

Training & inference

min

Layer activation

Quantized Activation

Quantization

𝜇

𝜎

∇

Quantization

s

Layer activation

TDQ

Module

Quantized Activation

T

Frequency Encoding

1

Training Only

10

MLP

(a) Dynamic Quantization

(b) Ours : Temporal Dynamic Quantization

Instead, we propose a novel dynamic quantization that utilizes temporal information rather than input activation. One important observation we made is that when we measured the Pearson correlation between time steps and per-tensor activation variation, 62.1% of the layers exhibited moderate temporal dependence (|r| > 0.5), and 38.8% exhibited strong temporal dependence (|r| > 0.7). This implies that many layers in diffusion models indeed possess strong temporal dependence. Although the activation distribution might change based on the input data, the overall trends remain similar within the same time frame (see Fig 2). Therefore, we can determine the optimal interval based on the differences in time steps, leading to more reliable and robust quantization results

Intuitively, we can employ a scalar quantization interval for each time step. However, this strategy falls short in capturing inter-temporal relationships and cannot be applied to scenarios where different time steps are used between training and inference. Instead, we propose using a learnable small network that predicts quantization interval by taking time step information. This strategy can evaluate activation alterations collectively across multiple timesteps, thereby making the learning process more stable and improving performance, as will be shown in Table 3.

Fig. 4 (b) presents the overview of our idea, called TDQ module. In the TDQ module, the dynamic interval ˜ s is generated based on the provided time step t as follows:

where enc ( · ) represents the encoding function of the time step, which will be described in Section 3.4. Here, I is the encoded feature, and f ( · ) signifies the generator module function.

In our approach, the TDQ module is attached to each quantization operator, and the independent dynamic quantization interval is generated based on the given time step. As illustrated in the Fig. 4, the generator is implemented by simply stacking multiple linear layers with a softplus function to constrain the data range to non-negative value. Please note that all components of the generator are differentiable. As a result, during the PTQ or QAT process, the interval can be updated toward minimizing quantization error using gradient descent.

For instance, when employing a well-known QAT algorithm such as LSQ [29], the static interval can be substituted by the output of the TDQ module. The quantization function from Eq. 5 is then modified as follows:

When we use a straight-through estimator [39], the gradient can propagate through the learnable parameters of the generator. These parameters are then updated via iterative gradient descent, with the goal of minimizing the final task loss. The same pipeline can be applied to PTQ algorithms that utilize local gradient of minimizing reconstruction error [23, 24]. Consequently, the TDQ module is easily applicable to both QAT and PTQ schemes.

However, unlike input-dependent dynamic quantization, the TDQ module enables cost-free inference. After the PTQ or QAT process, the time-dependent interval can be pre-computed offline, and during inference, we can utilize the pre-computed value. In addition, the pre-computed interval provides

a strong advantage by enabling the seamless integration of TDQ module on existing frameworks without modifications.

3.4 Engineering Details

We conducted a series of experiments to enhance the stability and effectiveness of the TDQ module, and figured out that several engineering improvements play a key role in achieving reliable results. This section provides a detailed description of these improvements and their impact on the performance of the TDQ module.

Frequency Encoding of Time Step Feeding the time step directly to the generator results in inferior convergence quality. This is primarily due to the well-known low-frequency inductive bias of neural networks [40, 41]. If the time step is directly input to the generator, it tends to produce an interval that barely changes regardless of the time step. To mitigate this low-frequency bias, we use geometric Fourier encoding for time step [42, 43], as described in Eq. 6.

where t is the current time step, d is the dimension of encoding vector, and I is the encoded vector. This encoding approach allows the TDQ module to accommodate high-frequency dynamics of time step. In this paper, we set t max as 10000, empirically.

Initialization of TDQ Module Proper initialization of quantization interval is crucial, as incorrect initialization can lead to instability in QAT or PTQ processes. Existing quantization techniques only need to initialize the static step value, but we need to initialize the TDQ module's output to the desired value. To achieve this, we utilize He initialization [44] for the weights and set the bias of the TDQ module (MLP)'s last linear layer to the desired value. Given that the input to the MLP (geometric Fourier encoding) can be treated as a random variable with a mean of zero, the output of the He-initialized MLP will also have a mean of zero. Thus, we can control the mean of the MLP output to a desired value via bias adjustment. After extracting 1000 samples for the entire time step, we initialized the quantization interval to minimize the overall error and then conducted an update to allow adaptation to each time step.

4 Experimental Setup

In order to demonstrate the superior performance of TDQ, we conducted tests using two different models: DDIM [2], a pixel space diffusion model, and LDM [4], a latent space diffusion model. For the DDIM experiments, we utilized the CIFAR-10 dataset [45] (32x32), while for the LDM experiments, we employed the LSUN Churches dataset [46] (256x256). This allowed us to showcase the effectiveness of the proposed method in both low and high-resolution image generation scenarios. We applied PTQ and QAT to both models. However, it is important to note that while the latent diffusion model consists of a VAE and a diffusion model, we focused on quantizing the diffusion model and did not perform quantization on the VAE component.

In the absence of prior QAT studies for the diffusion model, we experimented with well-known static quantization methods, i.e., PACT [19], LSQ [29], and NIPQ [30] as baselines. Our idea is integrated on top of LSQ, by replacing the static interval as an output of TDQ module. We also provide additional experiments of TDQ integration with other methods in the Appendix. Per-layer quantization was applied to activations and weights of all convolutional and linear layers, including the activation of the attention layer. The models were trained for 200K iterations on CIFAR-10 and LSUN-churches, with respective batch sizes of 128 and 32. The learning rate schedule was consistent with the full precision model.

In PTQ, we used PTQ4DM [18], a state-of-the-art study, as a baseline for extensive analysis. For a fair comparison, we mirrored the experimental settings of PTQ4DM but modified the activation quantization operator with a TDQ module for dynamic quantization interval generation. Like PTQ4DM, we used per-channel asymmetric quantization for weight and per-tensor asymmetric quantization for activation. The weight quantization range was determined by per-channel minimum/maximum values, and the activation quantization range was trained via gradient descent to minimize blockwise

reconstruction loss, as in methods like BRECQ [24]. Quantization was applied to all layers, but in the LDM PTQ experiment, the activation of the attention matrix was not quantized. Following PTQ4DM's approach, we used a calibration set of 5120 samples for PTQ, consisting of 256 images with 20 random time steps selected per image.

To measure the performance of the models, we used Fréchet Inception Distance (FID) [47] and Inception Score (IS) [48] for CIFAR-10, and FID for LSUN-churches. For evaluation, we generated 50,000 images using QAT with DDIM 200 step sampling, and 50,000 images using PTQ with DDIM 100 steps. In the case of QAT, we selected 5 checkpoints with the lowest validation loss and reported the scores from the best performing models.

All of experiments were conducted on the high performance servers having 4xA100 GPUs and 8xRTX3090 GPUs with PyTorch [49] 2.0 framework. The source code is available at https: //github.com/ECoLab-POSTECH/TDQ_NeurIPS2023 . Besides, we use the notation WxAy to represent x-bit weight & y-bit activation quantization for brevity.

5 Results

5.1 Quality Analysis after QAT and PTQ

Table 1 compares the TDQ module with existing static quantization methods. As the activation bit decreases, all static quantization methods show inferior quality, while our approach shows consistent output. TDQ module gives substantial quality improvement even in 8-bit, and the benefit becomes even large in 4-bit precision, showing negligible quality degradation to full-precision output. TDQ achieves these benefits by efficiently allocating the limited activation quantization levels.

Besides, NIPQ was introduced to address the instability of the straight-through estimator by applying pseudo quantization based on artificial noise. However, the noise of NIPQ is indistinguishable from the input noise, hindering the diffusion model's convergence. Additional efforts are required to exploit the benefit of PQN-based QAT for diffusion models.

Table 2 presents PTQ comparison of TDQ module against existing static PTQ schemes. The MinMax method represents a naive linear quantization approach where the range is determined by the minimum and maximum values of the target tensor. The experiments demonstrate that while all baselines maintain a good level of FID when the activation bit is high, they experience significant performance degradation as the activation bit decreases. In contrast, TDQ exhibits only a slight level of FID degradation, indicating that the TDQ module is a robust methodology that performs well in both QAT and PTQ scenarios.

Fig. 5 to 7 visualize the generated images of quantized diffusion models. Our method consistently outperforms other quantization techniques in producing images with high-fidelity within the same bit-width configuration. Figs. 5 and 6 further illustrate this, as conventional QAT and PTQ produce blurred and unrecognizable images, whereas our method generates realistic images. The integration of temporal information in activation quantization is shown to be highly effective in maintaining the perceptual quality of the output.

10

25

50

100

DDIM Steps

4

8

12

16

20

FID

W8A4

LSQ

Ours

FP

10

25

50

100

DDIM Steps

5

10

15

20

25

FID

W4A4

LSQ

Ours

FP

0

50

100

150

200

0.143

0.153

0.164

0.174

0.184

step size

Input_blocks.4.0.in_layers.2.act_func

0.106

0.110

0.114

0.118

0.122

0

50

100

150

200

0.030

0.046

0.062

0.078

0.094

Input_blocks.1.0.out_layers.3.act_func

0.016

0.019

0.022

0.026

0.029

0

50

100

150

200

Time

0.136

0.140

0.144

0.148

0.152

step size

Input_blocks.10.0.in_layers.2.act_func

0.072

0.074

0.076

0.078

0.080

0

50

100

150

200

Time

0.320

0.334

0.348

0.361

0.375

Middle_block.0.out_layers.3.act_func

0.088

0.109

0.130

0.151

0.173

0

50

100

150

200

Time

0.315

0.330

0.345

0.360

0.375

Middle_block.1.attention.qsym2

0.253

0.266

0.279

0.291

0.304

0

50

100

150

200

0.168

0.177

0.187

0.196

0.205

Output_blocks.0.0.in_layers.2.act_func

0.074

0.076

0.078

0.080

0.082

0

50

100

150

200

0.267

0.283

0.299

0.315

0.331

Output_blocks.0.0.out_layers.3.act_func

0.143

0.155

0.168

0.180

0.193

activation var

0

50

100

150

200

Time

0.909

0.977

1.045

1.113

1.181

Output_blocks.2.1.op.act_func

2.66

11.62

20.58

29.53

38.49

activation var

5.2 Generalization Performance of TDQ Module

This section presents the experimental results on the performance of the TDQ module for fast-forward inference. Training is carried out to fully encompass all time steps from 1 to 1000, however, inference can be executed with fewer time steps (between 50 to 100) for enhanced performance. This changes the distribution of the time steps during training/testing, thus, the TDQ module's generation requires good generalization abilities.

Fig. 8 displays the FID measurement for the DDIM model quantized by the LSQ algorithm for both W8A4 and W4A4 configurations. The time step gradually decreases from 100 to 10 during inference. As depicted, LSQ's performance significantly deteriorates as the number of time step decreases, whereas our method's performance declines similarly to the full-precision baseline. This experiment demonstrates that the TDQ module functions effectively even when the sampling time step varies.

5.3 Ablation Studies of TDQ module

To investigate the output dynamics of the TDQ module, we visualized the updates of the dynamic interval in relation to time steps (Fig. 9). The interval, trained using W4A4 LSQ on DDIM, demonstrates a tendency to change in alignment with activation variations. However, this pattern is not consistently observed across all layers. The inconsistencies could potentially indicate that the TDQ module is attempting to generate an interval that minimizes the final loss value, as LSQ adjusts the quantization interval accordingly.

To offer a more comprehensive analysis of the advantages of the TDQ module, we conducted a comparison with input-dependent dynamic quantization methods and performed an ablation study on the TDQ module. As depicted in Table 4, TDQ achieves superior performance compared to dynamic

quantization, even without directly utilizing the layer's distribution information. We also investigated the effect of the number of layers within the TDQ module on performance. The results demonstrate that the output quality becomes substantially consistent when the number of layers exceeds 4. Based on these findings, we have empirically selected 4 layers of MLP, considering a balance of performance and complexity.

We also conducted experiments to compare the performance of TDQ with a configuration where the per-time quantization intervals are directly learned. As presented in Table 3, while the concept of directly learning the quantization interval for each time step and using a shared quantization interval across multiple time steps can enhance output quality compared to the baseline ( S 1 ), TDQ continues to exhibit superior output quality and offers greater versatility. The success of TDQ lies in its ability to facilitate continuous and stable learning of the quantization interval while taking into account the evolution of the activation distribution over neighboring time steps.

6 Discussion

In this section, we discuss additional intuitions and limitations. While many layers exhibit strong temporal dependency, approximately 30% of the layers still do not show such correlations. These layers are primarily located in the middle block of the U-Net and are heavily influenced by their instance-wise semantic information. Moreover, we observed that the LDM model [4] has fewer temporal dependencies and thus demonstrates fewer performance improvements. However, it's crucial to emphasize that even in these cases, the TDQ module ensures convergence, resulting in consistent outputs across time steps. This alignment guarantees that the output quality remains comparable to that of existing PTQ/QAT algorithms, even in scenarios where temporal dependencies might be less.

7 Conclusion

In this paper, we explore the challenge of activation quantization in the diffusion model, specifically the dynamics of activation across time steps, and show that existing static quantization methods fall short of effectively addressing this issue. We introduce the TDQ module, seamlessly integrated into both QAT and PTQ, while enhancing the output quality significantly based on the dynamic quantization interval generation. Since our approach can be implemented without any overhead during inference, we expect our study to help improve performance while preserving quality in low-bit diffusion models for mobile and edge devices.

References

Minguk Kang, Jun-Yan Zhu, Richard Zhang, Jaesik Park, Eli Shechtman, Sylvain Paris, and Taesung Park. Scaling up gans for text-to-image synthesis, 2023.

Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. International Conference on Learning Representations ICLR, 2021.

Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models, 2020.

Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. Highresolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10684-10695, 2022.

Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. arXiv preprint arXiv:2207.12598, 2022.

Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding. Advances in Neural Information Processing Systems, 35:36479-36494, 2022.

Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. Advances in neural information processing systems, 32, 2019.

Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:2011.13456, 2020.

Jonathan Shen, Ruoming Pang, Ron J Weiss, Mike Schuster, Navdeep Jaitly, Zongheng Yang, Zhifeng Chen, Yu Zhang, Yuxuan Wang, Rj Skerrv-Ryan, et al. Natural tts synthesis by conditioning wavenet on mel spectrogram predictions. In 2018 IEEE international conference on acoustics, speech and signal processing (ICASSP), pages 4779-4783. IEEE, 2018.

Yi Ren, Chenxu Hu, Xu Tan, Tao Qin, Sheng Zhao, Zhou Zhao, and Tie-Yan Liu. Fastspeech 2: Fast and high-quality end-to-end text to speech. In International Conference on Learning Representations, 2021.

Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.

Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.01068, 2022.

Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models, 2023.

Prafulla Dhariwal and Alex Nichol. Diffusion models beat gans on image synthesis, 2021.

Luping Liu, Yi Ren, Zhijie Lin, and Zhou Zhao. Pseudo numerical methods for diffusion models on manifolds. In International Conference on Learning Representations, 2022.

Tim Salimans and Jonathan Ho. Progressive distillation for fast sampling of diffusion models. In International Conference on Learning Representations, 2022.

Chenlin Meng, Ruiqi Gao, Diederik P Kingma, Stefano Ermon, Jonathan Ho, and Tim Salimans. On distillation of guided diffusion models. In NeurIPS 2022 Workshop on Score-Based Methods, 2022.

Yuzhang Shang, Zhihang Yuan, Bin Xie, Bingzhe Wu, and Yan Yan. Post-training quantization on diffusion models, 2023.

Jungwook Choi, Zhuo Wang, Swagath Venkataramani, Pierce I-Jen Chuang, Vijayalakshmi Srinivasan, and Kailash Gopalakrishnan. PACT: parameterized clipping activation for quantized neural networks. CoRR, abs/1805.06085, 2018.

Yash Bhalgat, Jinwon Lee, Markus Nagel, Tijmen Blankevoort, and Nojun Kwak. LSQ+: improving low-bit quantization through learnable offsets and better initialization. Conference on Computer Vision and Pattern Recognition (CVPR), 2020.

Eunhyeok Park and Sungjoo Yoo. PROFIT: A novel training method for sub-4-bit mobilenet models. European Conference on Computer Vision (ECCV), 2020.

Markus Nagel, Mart van Baalen, Tijmen Blankevoort, and Max Welling. Data-free quantization through weight equalization and bias correction. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1325-1334, 2019.

Markus Nagel, Rana Ali Amjad, Mart Van Baalen, Christos Louizos, and Tijmen Blankevoort. Up or down? adaptive rounding for post-training quantization. In International Conference on Machine Learning, pages 7197-7206. PMLR, 2020.

Yuhang Li, Ruihao Gong, Xu Tan, Yang Yang, Peng Hu, Qi Zhang, Fengwei Yu, Wei Wang, and Shi Gu. Brecq: Pushing the limit of post-training quantization by block reconstruction. In International Conference on Learning Representations, 2021.

Xiuying Wei, Ruihao Gong, Yuhang Li, Xianglong Liu, and Fengwei Yu. Qdrop: Randomly dropping quantization for extremely low-bit post-training quantization. In International Conference on Learning Representations, 2022.

Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International Conference on Machine Learning, pages 2256-2265. PMLR, 2015.

Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade W Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. Laion5b: An open large-scale dataset for training next generation image-text models. In Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2022.

Sangil Jung, Changyong Son, Seohyung Lee, JinWoo Son, Jae-Joon Han, Youngjun Kwak, Sung Ju Hwang, and Changkyu Choi. Learning to quantize deep networks by optimizing quantization intervals with task loss. Conference on Computer Vision and Pattern Recognition, (CVPR), 2019.

Steven K. Esser, Jeffrey L. McKinstry, Deepika Bablani, Rathinakumar Appuswamy, and Dharmendra S. Modha. Learned step size quantization. International Conference on Learning Representations (ICLR), 2020.

Juncheol Shin, Junhyuk So, Sein Park, Seungyeop Kang, Sungjoo Yoo, and Eunhyeok Park. Nipq: Noise proxy-based integrated pseudo-quantization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 3852-3861, June 2023.

Yunshan Zhong, Mingbao Lin, Xunchao Li, Ke Li, Yunhang Shen, Fei Chao, Yongjian Wu, and Rongrong Ji. Dynamic dual trainable bounds for ultra-low precision super-resolution networks. arXiv preprint arXiv:2203.03844, 2022.

Changhun Lee, Hyungjun Kim, Eunhyeok Park, and Jae-Joon Kim. Insta-bnn: Binary neural network with instance-aware threshold. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 17325-17334, 2023.

Cheeun Hong, Sungyong Baik, Heewon Kim, Seungjun Nah, and Kyoung Mu Lee. Cadyq: Content-aware dynamic quantization for image super-resolution. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part VII, pages 367-383. Springer, 2022.

Zhenhua Liu, Yunhe Wang, Kai Han, Siwei Ma, and Wen Gao. Instance-aware dynamic neural network quantization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12434-12443, 2022.

Pavan Kumar Anasosalu Vasu, James Gabriel, Jeff Zhu, Oncel Tuzel, and Anurag Ranjan. Mobileone: An improved one millisecond mobile backbone. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7907-7917, 2023.

Andrew Tulloch and Yangqing Jia. Quantization and training of neural networks for efficient integer-arithmetic-only inference. Conference on Computer Vision and Pattern Recognition (CVPR), 2018.

Xiuyu Li, Long Lian, Yijiang Liu, Huanrui Yang, Zhen Dong, Daniel Kang, Shanghang Zhang, and Kurt Keutzer. Q-diffusion: Quantizing diffusion models, 2023.

Zhewei Yao, Reza Yazdani Aminabadi, Minjia Zhang, Xiaoxia Wu, Conglong Li, and Yuxiong He. Zeroquant: Efficient and affordable post-training quantization for large-scale transformers. Advances in Neural Information Processing Systems, 35:27168-27183, 2022.

Yoshua Bengio, Nicholas Léonard, and Aaron Courville. Estimating or propagating gradients through stochastic neurons for conditional computation. arXiv preprint arXiv:1308.3432, 2013.

Nasim Rahaman, Aristide Baratin, Devansh Arpit, Felix Draxler, Min Lin, Fred Hamprecht, Yoshua Bengio, and Aaron Courville. On the spectral bias of neural networks. In International Conference on Machine Learning, pages 5301-5310. PMLR, 2019.

Matthew Tancik, Pratul Srinivasan, Ben Mildenhall, Sara Fridovich-Keil, Nithin Raghavan, Utkarsh Singhal, Ravi Ramamoorthi, Jonathan Barron, and Ren Ng. Fourier features let networks learn high frequency functions in low dimensional domains. Advances in Neural Information Processing Systems, 33:7537-7547, 2020.

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.

Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. Communications of the ACM, 65(1):99-106, 2021.

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In Proceedings of the IEEE international conference on computer vision, pages 1026-1034, 2015.

Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. Citeseer, 2009.

Fisher Yu, Ari Seff, Yinda Zhang, Shuran Song, Thomas Funkhouser, and Jianxiong Xiao. Lsun: Construction of a large-scale image dataset using deep learning with humans in the loop. arXiv preprint arXiv:1506.03365, 2015.

Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. Advances in neural information processing systems, 30, 2017.

Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans. Advances in neural information processing systems, 29, 2016.

Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 32, 2019.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a novel quantization method for diffusion models that dynamically adjusts the quantization interval. To this end, the authors present a unique Temporal Dynamic Quantization (TDQ) module, which utilizes the time step as an input to determine the quantization interval, which significantly improves the generation quality. Extensive experiments on CIFAR-10 and LSUN Churches demonstrate the superior performance of the proposed method.\n\nSTRENGTHS:\nThe main idea is well-written and easy to follow. The proposed method, though simple in design, proves to be remarkably effective, notably diminishing the decline in performance. Its simplicity does not compromise its efficacy, making it a compelling solution to the problem at hand.\n\nWEAKNESSES:\nThe primary weaknesses of the paper include a lack of a vital comparison with Q-diffusion, a key baseline in diffusion models, and a need for more comprehensive ablation studies, such as investigating the impact of the number of layers in the generator module on the model's overall performance.\n\nQUESTIONS:\n1.\tIn Section 3.1, the authors state that uses symmetric quantization for weights and set $p=-n=2^{b-1}-1$. However, compared with LSQ that set $n=-2^{b-1}$, it appears that the proposed method might be under-utilizing one quantization level. This could potentially lead to a slight inefficiency in the model’s performance.\n\n2.\tIn Section 3.3, the authors state that previous studies have demonstrated that incorporating a quantization module … can lead to significant improvements in accuracy for specific applications. However, there appears to be a lack of citation or reference supporting this assertion.\n\n3.\tInstead of using the proposed TDQ module, it is possible to directly learn distinct quantization intervals corresponding to different time steps?\n\n4.\tThe paper lacks a comparison in its experimental section. Specifically, the study has not considered Q-diffusion (as detailed in reference [36]) as a baseline. Since Q-diffusion also applies post-training quantization to diffusion models, it's critical for the authors to include a comparative analysis of their proposed method against Q-diffusion. This would help to further validate the efficacy of their approach and enrich the overall discussion.\n\n5.\tThe paper could benefit from additional ablation studies, which would provide a more comprehensive understanding of the proposed method. One specific point of interest mentioned is the influence of the number of layers in the generator module on the overall performance. By investigating this aspect, it would add depth to the analysis and potential insights into the efficiency of the proposed model.\n\nLIMITATIONS:\nThe authors do not provide any discussions on limitations.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nWhile diffusion models provide outstanding quality and diversity of generated data, they demand high storage and computation, limiting its applications on mobile devices. Further, the paper points out that current existing works on quantization are not suitable for diffusion models as diffusion models have unique temporal properties due to iterative sampling process. As such, the paper introduuces a new quantization technique for diffusion models, where quantization interval is dynamically adjusted. As a result, the proposed framework prvoides strong performance.\n\nSTRENGTHS:\n- The proposed method can be easily integrated with the existing QAT and PTQ algorithms.\n- Clear visualizations which help the understanding of the paper and motivations. \n- Simple yet effective quantization method without additional inference costs.\n\nWEAKNESSES:\n- Don't input-dependent quantization methods [31,32,33,34] provide computational efficiency benefits even after considering extra overhead? Is there a proof for supporting the following claim \"dynamic approaches often incurs additional costs for extracting statistical information from activations, making it challenging to achieve performance improvements in practice.\"? It would be good to refer to the results or references for supporting this claim or tone down the claim a bit.\n.\n- Missing experimental comparisons against [36], which is another work on quantization of diffusion model.\n\n- The paper claims that the activation distribution is similar across inputs with Figure 2. But, Figure 2 shows the activations of an earlier convolutional layer, which is known to have low-level features that are common across diverse layers. It would be good to show that similar trends can be observed across different layers. Indeed, less clear and less time dependent trends can be observed in deeper layers (mid.attn_1.proj_out.0, down.1.downsample.conv.0, down.1.attn.0.k.0, mid.block_1.conv1.0) in Figure 7 of the supplementary material. I suspect that deeper layers will exhibit even more complex trends that may not look like time-dependent. More detailed discussions are needed. \n\n- The paper claims that the proposed method can be seamlessly integrated into existing quantization algorithms, but the paper only shows the benefit of doing that with LSQ in the case of QAT.\n\n- Non-exhaustive list of typos (a thorough proof-reading is needed):\n   - can decipts in Line 148\n   - there have been limited works tried to in Line 109\n\nQUESTIONS:\nPlease refer to the weakness section\n\nLIMITATIONS:\nNo limitations mentioned in the paper. The potential limitation of the work is not taking into account for activation distribution variation that exists among different inputs.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes Temporal Dynamic Quantization (TDQ), a new quantization approach for diffusion models. This method utilizes a learned module to adjust the quantization interval with respect to timestep to deal with activation quantization errors. Unlike conventional dynamic quantization techniques, the TDQ approach has no computational overhead during inference and is compatible with both post-training quantization (PTQ) and quantization-aware training (QAT). Experiments have been performed using DDIM on the CIFAR-10 dataset and LDM on the LSUN-Churches dataset with TDQ and various traditional QAT and PTQ approaches.\n\nSTRENGTHS:\n- The paper is well organized and written with clear motivation; the idea of addressing the rapid changes in input activation to reduce the truncation & rounding errors with a time-dependent quantization configuration is natural and interesting.\n- The proposed TDQ framework can be applied to both QAT and PTQ cases.\n- Experimental results demonstrate that the proposed framework can outperform several previous QAT and PTQ methods. Ablation studies have been conducted to show that TDQ can generalize to scenarios with fewer denoising steps and analyze output dynamics across layers.\n\nWEAKNESSES:\nMy concerns are mainly about the evaluation in this paper:\n- The proposed TDQ framework primarily aims to deal with the activation quantization, but it is not specified whether the act-to-act matrix multiplications (attn and v) in the attention layers are quantized. By examining the code attached in the supplementary materials, it seems like those computations still remain in FP. However, they can take a significant portion of the compute & memory overheads and are fully quantized in closely related work [1], while past literature [1, 2] has pointed out that the post-softmax attention scores are usually hard to quantize (e.g. PTQ4DM suffers from performance degradation after quantizing them according to Appendix E in [1]). It is unclear if TDQ can deal with this performance degradation without providing experimental results with act-to-act matmuls in attention layers quantized.\n- It is not accurate to call PTQ4DM the 'state-of-the-art study'—Q-Diffusion [1] was proposed around the same time as PTQ4DM and contained results that went down to 4-bit weights on both CIFAR and LSUN datasets, and achieved superior results on CIFAR. Its code (with quantized checkpoints) was released in April. I would highly recommend the authors include and compare Q-Diffusion results for CIFAR and LSUN-Churches experiments as it is a highly related work that natively covers many experimental settings in this work that are not in PTQ4DM.\n\n[1] Li, X., Liu, Y., Lian, L., Yang, H., Dong, Z., Kang, D., ... & Keutzer, K. (2023). Q-diffusion: Quantizing diffusion models. arXiv preprint arXiv:2302.04304.  \n[2] Yuan, Z., Xue, C., Chen, Y., Wu, Q., & Sun, G. (2021). PTQ4ViT: Post-Training Quantization Framework for Vision Transformers with Twin Uniform Quantization. arXiv preprint arXiv:2111.12293.\n\nQUESTIONS:\n- Can you further explain the output dynamics in Figure 9 and Appendix Figure 6? E.g. what do blue and red curves stand for and how to interpret them?\n- Looking at Appendix Table 1, I am curious if some simple baselines following the same principle can be helpful, such as using multiple quantizers at different time steps under the LSQ setting e.g. using a separate activation quantizer every 20 steps.\n- How sensitive is the TDQ module (MLP) to the hyperparameters? Are there different design choices for the training process that cause different results? For instance, it is mentioned in the paper that initialization is crucial – are there experiments showing that a different initialization (e.g. gaussian, Xavier, zero…) can make results worse?\n\nLIMITATIONS:\nThe authors adequately addressed all limitations.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors present and empirically evaluate a novel quantization scheme for diffusion models. Static quantization of the evolving activation distribution in diffusion models leads to performance degradation. By using a quantization range which is a function of the timestep, the authors improve performance.\n\nSTRENGTHS:\nThe paper is very well structured and easy to follow and provides clear and concise descriptions of both introductory quantization literature, as well as diffusion model literature. The problem definition and proposed solution are also very well described, especially in Figures 2 and 3. Empirically, evaluation under multiple metrics (FID and ID) is always appreciated. The experimental setup is well described.\n\nWEAKNESSES:\n- The form of the TDQ module (network size, parameters, structure) isn't provided.\n- Not a particularly major issue but there are a few spelling/grammatical mistakes. For brevity I won't list all the errors but online tools like Grammarly are free to use and would help in this regard. \n- For other weaknesses, please see my questions.\n\nQUESTIONS:\n- This paper seems slightly contradictory. In section 3.3, the authors assume that the time evolution of the activation distribution is constant, which permits the use of their TDQ module. The authors also make note of the high cost of inference (clearly outweighing the training costs). So why is the TDQ network module neccessary? Why not just learn the the activation distribution directly for each timestep. Learning this function through a network (rather than simply per timestep) would impose a smoothness contraint on the function. Is this the only benefit?\n- Have the authors provided an ablation study where the activation variations, as in Figure 9, are used rather than the TDQ module? Again, why not just compute these beforehand? Does using this pre-computed activation variance perform better? Perhaps some more detailed ablations studies are required?\n- How robust is this TDQ module to training? No error bars are provided so it's unclear if there are any failure cases, or if the predicted intervals remain quite consistent across trials. Given the assumption in section 3.3, they should always converge to the same interval values. Do the authors observe this?\n\nLIMITATIONS:\n- The limitations of this work aren't explicitly outlined. For example, whilst the authors show a couple of plots illustrating the time evolution of the activation distribution, the assumption doesn't appear to be investigated. What are the effects of this assumption?\n- In addition, it doesn't appear that the authors have outlined any limitations wrt to how text conditioning (v. common for diffusion models in practice) could break this assumption. Perhaps useful for future work, but the authors could make a comment for the sake of practitioners looking to employ this method?\n- I see no wider negative societal implications of this work.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}