Temporal Dynamic Quantization for Diffusion Models
Authors: Junhyuk So
Jungwon Lee
Daehyun Ahn
Hyungjun Kim
Eunhyeok Park
Venue: NeurIPS 2023 poster (2023)
Keywords: deep learning optimization
quantization
diffusion model
generative model

Abstract: Diffusion model has gained popularity in vision applications due to its remarkable generative performance and versatility. However, its high storage and computation demands, resulting from the model size and iterative generation, hinder its use on mobile devices. Existing quantization techniques struggle to maintain performance even in 8-bit precision due to the diffusion model's unique property of temporal variation in activation. We introduce a novel quantization method that dynamically adjusts the quantization interval based on time step information, significantly improving output quality. Unlike conventional dynamic quantization techniques, our approach has no computational overhead during inference and is compatible with both post-training quantization (PTQ) and quantization-aware training (QAT). Our extensive experiments demonstrate substantial improvements in output quality with the quantized model across various configurations.

Temporal Dynamic Quantization for Diffusion Models
Junhyuk So!
Jungwon Lee
Daehyun Ahn Hyungjun Kim Eunhyeok Park
Department of Computer Science and Engineering, POSTECH Graduate School of Artificial and Intelligence, POSTECH SqueezeBits. Inc
Abstract
The diffusion model has gained popularity in vision applications due to its remark- able generative performance and versatility. However, high storage and computation demands. resulting from the model size and iterative generation. hinder its use on mobile devices. Existing quantization techniques struggle to maintain performance even in 8-bit precision due to the diffusion model's ' unique property of temporal variation in activation. We introduce novel quantization method that dynamically adjusts the quantization interval based on time step information, significantly im- proving output quality. Unlike conventional dynamic quantization techniques, our approach has computational overhead during inference and is compatible with both post-training quantization (PTQ) and quantization- training (QAT). Our extensive experiments demonstrate substantial improvements in output quality with the quantized diffusion model across various datasets.
Introduction
Generative modeling is crucial machine learning for applications such as image |2 6
voice [9 10]. and text synthesis [1I 12 13). Diffusion models [3 2 4 which progressively refine input images through a denoising process involving hundreds of iterative inferences, have recently gained prominence due to their superior performance compared alternatives like GANs (14). However, the high cost diffusion models presents significant barrier to their widespread adoption. These models have large sizes, often reaching several gigabytes, and demand enormous computation of iterative inferences for single image generation. Consequently, executing diffusion models on resource-limited mobile devices is practically infeasible, thus most applications are currently implemented on expensive, high-perffrmance servers.
To fully exploit the potential of diffusion models, several methods to reduce the computational cost and memory requirement diffusion models while preserving the generative performance have been proposed. For example. J. Song et al. [2] and L. Liu et al. [IS] proposed more efficient sampling scheduler, and T. Salimans & Ho proposed to reduce the number of sampling steps using knowledge distillation technique [16] [7]. As result, high-fidelity images can be generated with fewer sampling steps, making the use of diffusion models more affordable and feasible. Despite these advancements, the denoising process of diffusion models still demands substantial computational cost, necessitating further performance enhancements and model compression.
While the majority of previous approaches have focused reducing the number of sampling steps to accelerate the denoising process, it also important to lighten the individual denoising steps. Since single denoising step can be regarded compression techniques can be used. Quantization a widely used compression technique where both weights and activations mapped to the low-precision domain. While advanced quantization schemes have been extensively studied for conventional Convolution Neural Networks (CNNs) and
conventional deep learning model inference, various model
"qqual Contribution
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Fowrad Process q(x,xx-1) = N(xi ( -1xx/11--x----
xo ~ (x)
~ N(.1)
Reverse Process Pa (xc-1 (x He (Xp, E), aft)
Figure 1: The forward and reverse processes of diffusion models.
language models, their application to diffusion models has shown significant performance degradation. Due to the unique property of diffusion models, such the significant changes in the activation distribution throughout the iterative inferences, the output is heavily distorted as the activation bit- width decreases (18), Existing quantization techniques, including both quantization-awe -waare training (QAT) [19 20 21 and Post-tran fost-training quantization (PTQ) [22] (23 24] are designed to address specific distribution in existing DNNs, therefore cannot deal with time-variant activation distributions in diffusion models.
To tackle the unique challenges of diffusion model quantization, introduce novel design called Temporal Dynamic Quantization (TDQ) module. The proposed TDQ module generates a time- dependent optimal quantization configuration that minimizes activation quantization errors. The strong benefit of the TDQ module the seamless integration of the existing QAT and PTQ algorithms, where the TDQ module extends these algorithms to create time-dependent optimal quantization configuration that minimizes activation quantization errors. Specifically, the module is designed to generate no additional computational overhead during inference, making it compatible with existing acceleration frameworks without requiring modifications. The TDQ module significantly improves quality over traditional quantization schemes by generating optimal quantization parameters each time step while preserving the advantages of quantization.
2 Backgrounds and Related Works
2.1 Diffusion Model
Diffusion models have been first introduced in 2015 [26] and revolutionized image generation by characterizing it sequential denoising process. As shown in Fig. [ the forward diffusion process gradually transforms the image(zp) into a random data(zr) which follows standard normal distribution by adding small Gaussian noise each time step. The reverse diffusion process generates clean image(zo) from random data(ær) by gradually removing noise from the data through iterative denoising steps. Therefore, diffusion model learns the reverse process which is estimating the noise amount on given noisy data at each time step(r;)) The forward(g) and reverse(pa) processes can be described as
((xe(1=11) = N(v;; 1- -
Po(xI-1(z1) = (1)
t),
where B, denotes the magnitude of Gaussian noise.
3 introduced a reparameterization trick for and the corresponding loss function, which facilitates the training of diffusion models.
1 Vau
3e
no (Rt, 1)
6 (Iz, 1)) Lyimple Er, |||c ca (x+, t]]
(xt
(3)
v1 -
(4)
While diffusion models can produce high-quality images, the iterative denoising process makes diffusion models difficult be used for real-world scenarios. The early works on diffusion models such as DDPM 3 required hundreds to thousands of iterative inferences generate single image, resulting extremely slow sampling speed. Therefore, numerous studies have been investigating algorithmic enhancements and various optimizations to boost performance, aiming make diffusion

---

models more efficient and suitable for real-world applications. DDIM 22] introduced an implicit probabilistic model that reinterprets the Markov process of the DDPM method achieving competitive image quality with one-tenth of denoising sipes. Distillation-base methods [16 proposed to reduce the number of denoising steps using knowledge distillation techniques.
On the other hand, to address the significant computational overhead associated with generating high-resolution images, 44) proposed Latent Diffusion Model (LDM) where the diffusion model takes latent variables instead of images. Especially, the large-scale diffusion model (e.g.. Stable Diffusion 4 has leveraged LDM and learned from large-scale dataset [LAION-Datanl (27). enabling the creation of high-quality, high-resolution images conditioned on textual input.
2.2 Quantization
Quantization is a prominent neural network optimization technique that reduces storage requirements with low-pre precision representation and performance improvement when the corresponding low- precision acceleration is available. With b-bit precision, only 2 quantization levels are accessible, which considerably limits the degree of freedom of the quantized model compared to floating-point representation. Therefore, the final quality significantly varies depending on the tuning of quantization hyperparameters, such as the quantization interval or zero offset. To preserve the quality of output in low-precision, it is crucial to update the quantization parameters well the model parameters, taking into account the specific features and requirements of the target task.
2.3
Quantization-awarr Training and Post-training Quantization
Quantization algorithms can be broadly categorized into two types: Quantization-Awar Training (QAT) [19 28 29
20 30 and Post-Training Quantization (PTQ) 24 25 QAT applies additional training after introducing quantization operators, allowing the update of network parameters toward minimizing the final loss value considering the effect of the quantization operators. On the other hand, PTQ does not apply end-to-end forward/backwart propagation after quantization. Instead, it focuses on reducing block-wise reconstruction errors induced by quantization. QAT typically outperforms PTQ in low-precision scenarios, but it may not always be applicable due to limitations such as the deficiencies of datasets, training pipelines, and resource constraints. Due to its practical usefulness, PTQ has been actively researched recently. In the literature of diffusion models, [88] introduced a dedicated 8-bit post-training quantization method, demonstrating high-fidelity image generation performance.
On the other hand, while most QAT and PTQ algorithms focus on static quantization, recent research has highlighted the benefits of input-dependent dynamic quantization [3]
32 33 34). Dynamic
quantization enables the adjustment of quantization intervals based on the varying input-dependent distribution of activations, which has the potential to reduce quantization error. However, implement- ing these dynamic approaches often incur additional costs for extracting statistical information from activations, making it challenging to achieve performance improvements in practice. For instance, (35) pointed out that input-dependent activation functions can cause significant latency increases despite having relatively little computation.
While a number of previous works proposed different methods to accelerate sampling process of diffusion models, there have been limited works tried to exploit dynamic nature of diffusion models. In this paper, we propose novel quantization scheme for diffusion models which minimizes activation quantization errors by enabling the generation of suitable quantization interval based on the time step information, all without incurring additional inference costs.
Temporal Dynamic Quantization
3.1 Quantization Methods
Before elaborating the proposed method, we define the quantization function used in our work. In this study, we focus on b-bit linear quantization, where the 2 possible quantization levels are evenly spaced. Linear quantization involves two key hyperparameters: the quantization interval s and the zero offset z. Given the full-precision data x, the quantized data å can be calculated as follows:
ã clip([z/s) + 2,
f = (i 2),,
(5)
3

---

cony In.0
upsamplee.oom.
200
100
0
-100
10
20
30
40
50
60
70
80
90
Figure 2: Evolution of Activation Distribution of Diffusion Model over Time Steps. Boxplot depicting the maximum and minimum values of activation for the DDIM model trained the CIFAR-10 dataset across different time steps.
activation distribution
activation distribution
T=0
T=100
T=0
T=100
Truncation error is too high
Rounding error is too high
(a) The interval is calibrated at T=0
(b) The interval is calibrated at T=100
Figure 3: Limitation of Static Quantization for Diffusion Models. Assume that the activation distribution enlarges as time step progresses. (a) Large truncation error due to small quantization interval and (b) Large rounding error due large quantization interval.
where n(p) is the smallest(largestt quantization index, clip(-) is clipping function and L-1 is rounding function. For practical acceleration purposes, we utilize symmetric quantization for weights, where 2 = 0 and p = -n ((b- 1) 1. On the other hand, for activation quantization, we assume asymmetric quantization with K 0, n 0, and p 2 1, which is commonly adopted approach (36].
3.2 Challenges of Diffusion Model Quantization
The biggest challenge of quantizing diffusion models is to find optimal quantization parameters (s and 2) for activation that minimize the quantization error. As shown in Fig. [ activation distribution of diffusion models have unique property due to the iterative denoising process, whose distribution highly varies depending on the time step (/) regardless of the layer index.
Therefore, using static values for quantization parameters causes significant quantization enror for different time steps, as shown in Fig. 3 Previous studies [37 [8] has also reported the dynamic property of activations in diffusion models and attempted address it by sampling the calibration dataset across overall time frames. However, despite these efforts, these studies still relied on static parameters, resulting in sub-optimal convergence of minimizing quantization problem fundamentally, it crucial to enable the update of quantization parameters considering the dynamics in the input activation distribution.
To tackle this
3.3 Implementation of TDQ Module
To address the rapid changes in input activation, one easy way of think is input-dependent dynamic quantization. Previous studies have demonstrated that incorporating quantization module that generates the quantization parameters based input features such as minimum, maximum, and average values can lead significant improvements in accuracy for specific applications [38 32 According to our observation, will be presented in the Appendix. our own implementation of input-dependent dynamic quantization offers notable quality improvement. However, the process of gathering statistics the given activation introduces complex implementation and notable overhead
(35. Despite the potential of quality improvement, this approach may not be an attractive solution in practice.

---

Dynamic Module
Frequency Encoding
TDQ Module
Quantization
Layer activation Training & inference Quantized Activation
Layer activation
Training Only
Quantized Activation
(a) Dynamic Quantization
(b) Ours! Temporal Dynamic Quantization
Figure 4: Overview of TDQ module. Comparison between (a) Input-dependent dynamic quantization, which requires activation statistics during both training and inference, and (b) the proposed TDQ module that enables cost-free inference based pre-computed intervals.
Instead, we propose novel dynamic quantization that utilizes temporal information rather than input activation. One important observation we made that when we measured the Pearson correlation between time steps and per-tensor activation variation, 62.1% of the layers exhibited moderate temporal dependence (Irl 0.5), and 38.8% exhibited strong temporal dependence (Irl > 0.7). This implies that many layers in diffusion models indeed possess strong temporal dependence. Although the activation distribution might change based on the input data, the overall trends remain similar determine the optimal interval based on
within the same time frame (see Fig Therefore, the differences in time steps, leading to more reliable and robust quantization results
Intuitively, we can employ scalar quantization interval for each time step. However, this strategy falls short in capturing inter-temporal relationships and cannot be applied to scenarios where different time steps are used between training and inference. Instead, we propose using learnable small network that predicts quantization interval by taking time step information. This strategy can evaluate activation alterations collectively across multiple timesteps. thereby making the learning process more stable and improving performance, as will be shown in Table
Fig. 4 (b) presents the overview of our idea, called TDQ module. In the TDQ module, the dynamic interval generated based on the provided time step follows:
enc(t), f(),
where enc(.) represents the encoding function of the time step, which will be described in Section 3.4. Here, I the encoded feature, and f(-) signifies the generator module function.
In our approach, the TDQ module is attached to each quantization operator, and the independent dynamic quantization interval is generated based on the given time step. As illustrated in the Fig. the generator is implemented by simply stacking multiple linear layers with softplus function to constrain the data range to non-negative value. Please note that components of the generator are differentiable. As result, during the PTQ or QAT process, the interval can be updated toward minimizing quantization error using gradient descent.
For instance, when employing a well-known QAT algorithm such as LSQ (29 the static interval can be substituted by the output of the TDQ module. The quantization function from Eq. iis then modified as follows:
T = clip(1<188 2, 5= (i =))-
When we use a straight-throuse estimator [39], the gradient can propagate through the learnable parameters of the generator. These parameters are then updated via iterative gradient descent, with the goal of minimizing the final task loss. The same pipeline can be applied to PTQ algorithms that utilize local gradient of minimizing reconstruction error 23 24 Consequently, the TDQ module is easily applicable to both QAT and PTQ schemes.
However, unlike input-dependent dynamic quantization, the TDQ module enables cost-free inference. After the PTQ or QAT process, the time-dependent interval can be pre-computed offline, and during inference, we can utilize the pre-computed value. In addition, the pre-computed interval provides
5

---

strong advantage by enabling the seamless integration of TDQ module on existing frameworks without modifications.
3.4 Engineering Details
We conducted a series of experiments to enhance the stability and effectiveness of the TDQ mod- ule, and figured out that several engineering improvements play key role in achieving reliable results. This section provides a detailed description of these improvements and their impact on the performance of the TDQ module.
Frequency Encoding of Time Step Feeding the time step directly to the generator results in inferior convergence quality. This is primarily due to the well-known low-frequency inductive bias of neural networks 40 41] If the time step is directly input to the generator, it tends to produce an interval that barely changes regardless of the time step. To mitigate this owwfrreuenccy bias, we use geometric Fourier encoding for time step [42 43
as described in Eq.
L
= enc(t) (sin(
ofd COS(
sin(
2/d ) cOS
sin
COS
)), (8)
0/d
2/d
4jd Eman
d/d bmaz
Fman
Eyinx
Emin
tmar
where is the current time step, d is the dimension of encoding vector, and / is the encoded vector. This encoding approach allows the TDQ module to accommodate tighffrequency dynamics of time step. In this paper, we set maa as 10000, empirically.
Initialization of TDQ Module Proper initialization of quantization interval is crucial, as incorrect initialization can lead to instability in QAT or PTQ processes. Existing quantization techniques only need to initialize the static step value, but we need to initialize the TDQ module's output to the desired value. To achieve this, we utilize He initialization [44] for the weights and set the bias of the TDQ module (MLP)'s last linear layer to the desired value. Given that the input the MLP (geometric Fourier encoding) can be treated as a random variable with a mean of zero, the output of the He-initialized MLP will also have mean of zero. Thus, we can control the mean of the MLP output to desired value via bias adjustment. After extracting 1000 samples for the entire time step, we initialized the quantization interval to minimize the overall error and then conducted an update to allow adaptation to each time step.
4 Experimental Setup
In order to demonstrate the superior performance of TDQ, we conducted tests using two different models: DDIM 2, a pixel space diffusion model, and LDM [4, latent space diffusion model. For the DDIM experiments, we utilized the CIFAR-10 dataset (45] (32x32), while for the LDM experiments, employed the LSUN Churches dataset [6) (256x256). This allowed us to showcase the effectiveness of the proposed method in both low and high-resolution image generation scenarios. We applied PTQ and QAT to both models. However, it is important to note that while the latent diffusion model consists of a VAE and a diffusion model, we focused on quantizing the diffusion model and did not perform quantization on the VAE component.
In the absence of prior QAT studies for the diffusion model, we experimented with well-known static quantization methods, i.e., PACT [19. LSQ (29), and NIPQ (30 as baselines. Our idea is integrated on top of LSQ, by replacing the static interval as an output of TDQ module. We also provide additional experiments of TDQ integration with other methods in the Appendix. Per-layer quantization was applicd to activations and weights of all convolutional and linear layers, including the activation of the attention layer. The models were trained for 200K iterations on CIFAR- 10 and LSUN- -huhchursses, with respective batch sizes of 128 and 32. The learning rate schedule was consistent with the full precision model.
In PTQ, we used PTQ4DM [8], tfute-of-the-ar art study, as a baseline for extensive analysis. For a fair comparison, we mirrored the experimental settings of PTQ4DM but modified the activation quan- tization operator with a TDQ module for dynamic quantization interval generation. Like PTQ4DM, we used per-channel asymmetric quantization for weight and per-tensor asymmetric quantization for activation. The weight quantization range was determined by per-channel minimun/maxiimm values, and the activation guantization range was trained via gradient descent to minimize blockwise

---

Table 1: QAT results of diffusion models. N/A represents the failure of convergence.
FID L7IST
(CIFAR-10)
Methods
W8A8
W4A8
W8A4
W4A4
W3A3
PACT 19
18.43 7.62
22.09 / 7.55
68.58 5.93
51.92/ 6.41
N/A / N/A
LSQ 29
3,87 /9.62
4.53 /9.38
6.2 6.56
7.3/9.45
7.63 9.38
DDIM
NIPQ 30
3.91 /9.49
13.08 99.7
6.36 9.59
30.73/8.99
N/A / N/A
Ours
3.77 99.58
4.13/9.59
4,56 9.64
4.48 /9.76
6,48 / 9.26
FID L
(Churches)
Methods
W8A8
W4A8
W8A4
W4A4
W3A3
PACT 19
9.20
9.94
8.59
10.35
12.95
L.QQ 29
N/A
4.92
5.08
5.06
7.21
LDM 4
NIPQ 30
4.12
7.22
4.68
9.13
N/A
Ours
3.87
4.04
4.86
4.64
6.57
Table 2: PTQ results of diffusion models. FP represents the output of the full full-precision checkpoint.
FID t/ISt
(CIFAR-10)
Methods Man-MMx PTQ4DM (18]
W8A8
W8A7
W8A6
8.73
34,61 / 8,28
332.15 1.47
DDIM 2
6.15/8884
6,2
22.43 8.77
5.59 8.94
Ours
/
5.8/
5.71 8.84
FID
(Churches)
Methods
W8A8
W8A6
W8A5
FP
Min- Max
4.34
103.15
269.05
LDM
PTQ4DM I8
3.97
4.26
7.06
4.04
Ours
3.89
4.24
4.85
reconstruction loss, in methods like BRECQ 24 Quantization applied to all layers, but in the LDM PTQ experiment, the activation the attention matrix was not quantized. Following PTQ4DM's approach, used calibration of 5120 samples for PTQ, consisting of 256 images with 20 random time steps selected per image.
To measure the performance of the models, used Fréchet Inception Distance (FID) 47 and Inception Score (IS) 48] for CIFAR-10, and FID for LSUN-churches. For evaluation, we generated 50,000 images using QAT with DDIM 200 step sampling, and 50,000 images using PTQ with DDIM 100 steps. In the case QAT, we selected checkpoints with the lowest validation loss and reported the scores from the best performing models.
of experiments were conducted on the high performance servers having 4xA 100 GPUs and 8xRTX3090 GPUs with PyTorrhh [49] 2.0 framework. The source code is available at httpps Betpe/TTQQN_eeuuueeeeee
beggthhu.ccmm/eeee.e //github. www.cComallaaaaaeeo PitooaaaPPSTECCHAAA /TQQ BDQQ/DQQ__MM_ Neur IPS2033 Besides, we use the notation WxAy to rep-
resent x-bit weight & b-t activation quantization for brevity.
Figure 5: Visualization of W4A4 QAT results, DDIM on CIFAR-10 10 dataset, (up) Ours (Down) LSQ
7

---

Figure 6: Visualization of W8A6 PTQ results, DDIM CIFAR-10 10 dataset, (up) Ours (Down) PTQ4DM
Figure 7: Visualization of W8A5 PTQ results, LDM on LSUN-churches dataset, (up) Ours (Down) PTQ4DM
5
Results
5.1 Quality Analysis after QAT and PTQ
Table I compares the TDQ module with existing static quantization methods. As the activation bit decreases, all static quantization methods show inferior quality, while our approach shows consistent output. TDQ module gives substantial quality improvement even in 8-bit, and the benefit becomes even large in 4-bit precision, showing negligible quality degradation full-precision output. TDQ achieves these benefits by efficiently allocating the limited activation quantization levels.
Besides, NIPQ introduced to address the instability of the straight-through estimator by applying pseudo quantization based on artificial noise. However, the noise of NIPQ is indistinguishable from the input noise, hindering the diffusion model's convergence. Additional efforts required to exploit the benefit of PQN-based QAT for diffusion models.
Table 2 presents PTQ comparison of TDQ module against existing static PTQ schemes. The Min- Max method represents a naive linear quantization approach where the range is determined by the minimum and maximum values of the target tensor. The experiments demonstrate that while all baselines maintain a good level of FID when the activation bit is high, they experience significant performance degradation the activation bit decreases. In contrast, TDQ exhibits only a slight level of FID degradation, indicating that the TDQ module is a robust methodology that performs well in both QAT and PTQ scenarios.
Fig. 5 to 1 visualize the generated images of quantized diffusion models. Our method consistently outperforms other quantization techniques in producing images with high-fidelity within the same bit-width configuration. Figs. [nddthee illustrate this, as conventional QAT and PTQ produce blurred and unrecognizable images, whereas our method generates realistic images. The integration of temporal information in activation quantization is shown be highly effective in maintaining the perceptual quality of the output.
8

---

W8A4
W4A4
20
25
LSQ
LSQ
Ours
Ours
16
20
FP
12
15
10
25
50
100
10
25
50
100
DDIM Steps
DDIM Steps
Figure 8: Number of sampling step FID. Comparison between the proposed method and conven- tional QAT when reducing the inference time step from 100 to 10.
Input blocks 4n layers.2.cc act func
Irpuc L.ocks.L...uut layers Juctt func
Ousput blecks.0.ddn layers 2-act func Ousput blecks 0.0 layers
func
0.204
Mlle
3322 2354
3.000
6285
1.00 3.30
0.59
226
0300
6
1.02 9:877
8.076
0.3
233
100
200
200
220
126
200
3.50
200
300
150
Irput blcks. 10...in byeer 2.ct, 2.ct Func
Midste Nockk-Oout out byers, actt func
Middic block wwwonton.ccom coym2
Outout Blocks 2.1op.att func
200
10.40
2300 5166
0
2.6 2242
6:45
2279 0i5
Tiree
Timee
Time
Time
Figure 9: Output dynamics of TDQ module. Visualization of the generated interval versus the variation of input activation.
5.2 Generalization Performance of TDQ Module
This section presents the experimental results the performance of the TDQ module for fast-forward inference. Training is carried out to fully encompass time steps from I to 1000, however, inference can be executed with fewer time steps (between 50 to 100) for enhanced performance. This changes the distribution of the time steps during trainingttesting, thus, the TDQ module's 's generation requires good generalization abilities.
Fig. X displays the FID measurement the DDIM model quantized by the LSQ algorithm for both W8A4 and W444 configurations. The time step gradually decreases from 100 to 10 during inference. As depicted, LSQ's performance significantly deteriorates as the number of time step decreases, whereas our method's performance declines similarly to the full full-precision baseline. This experiment demonstrates that the TDQ module functions effectively even when the sampling time step varies.
5.3 Ablation Studies of TDQ module
To investigate the output dynamics of the TDQ module, visualized the updates of the dynamic interval in relation to time steps (Fig. D.. The interval, trained using W4A4 LSQ on DDIM, demon- strates tendency to change in alignment with activation variations. However, this pattern is not consistently observed across all layers. The inconsistencies could potentially indicate that the TDQ module is attempting to generate interval that minimizes the final loss value, LSQ adjusts the quantization interval accordingly.
To offer a more comprehensive analysis of the advantages of the TDQ module, we conducted comparison with input-dependent dynamic quantization methods and performed an ablation study on the TDQ module. As depicted in Table TDQ achieves superior performance compared to dynamic

---

Table Learning quantization interval directly for each time step. The notation SN means that there are total of N learnable quantization interval that cover 1000 / time steps uniformly.
FID +
(QTT)
Bit-width
Ours
LSQ(S1)
S10
S50
S100
S1000
W4A4
4.48
7.30
4.75
5.17
4.88
5.53
2 -IFARROO
W3A3
6.48
7.63
8.89
6.92
6.82
9.94
W4A4
4.64
5,06
5.17
5.07
4.78
5,10
LDM
Churches
W3A3
6.57
7.21
6.70
6.47
6.64
6.86
quantization, even without directly utilizing the layer's distribution information. We also investigated the effect of the number of layers within the TDQ module performance. The results demonstrate that the output quality becomes substantially consistent when the number of layers exceeds Based on these findings, we have empirically selected 4 layers of MLP, considering balance of performance and complexity.
We also conducted experiments to compare the performance of TDQ with configuration where the per-time quantization intervals are directly learned. As presented in Table 3 while the concept directly learning the quantization interval for each time step and using shared quantization interval across multiple time steps can enhance output quality compared to the baseline (S1), TDQ continues to exhibit superior output quality and offers greater versatility. The success of TDQ lies in its ability facilitate continuous and stable learning of the quantization interval while taking into account the evolution of the activation distribution neighboring time steps.
Table 4: Ablation TDQ module
method
FID.
FP32
3.56
Static Quantization(L
7.3
Dynamic Quantization (max)
6.30
TDQ (2 layer)
5.18
TDQ (4 layer)
4.48
TDQ (8 layer)
4.88
Discussion
In this section, we discuss additional intuitions and limitations. While many layers exhibit strong temporal dependency, approximately 30% of the layers still do not show such correlations. These layers are primarily located in the middle block of the U-Net and are heavily influenced by their instance-wise semantic information. Moreover, we observed that the LDM model 4 has fewer temporal dependencies and thus demonstrates fewer performance improvements. However, it's crucial emphasize that even these cases, the TDQ module ensures convergence, resulting in consistent outputs across time steps. This alignment guarantees that the output quality remains comparable to that of existing PTQ/QAT algorithms, even in scenarios where temporal dependencies might be less.
7 Conclusion
In this paper, we explore the challenge of activation quantization in the diffusion model, specifically the dynamics of activation across time steps, and show that existing static quantization methods fall short of effectively addressing this issue. We introduce the TDQ module, seamlessly integrated into both QAT and PTQ, while enhaneing the output quality significantly based on the dynamic quantization interval generation. Since our approach can be implemented without any overhead during inference, we expect our study help improve performance while preserving quality in low-bit diffusion models for mobile and edge devices.
Acknowledgements
This work was supported by ITTP grant funded by the Korea government (MSIT, No. 20.2099-0-01906. No. No.2021-0-00105 and No.2021-0-00310),
10

---

References
[1] Minguk Kang, Jun- Yan Zhu, Richard Zhang, Jaesik Park, Eli Shechtman, Sylvain Paris, and Taesung Park. Scaling up gans for text-to-image synthesis, 2023.
[2] Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. International Conference on Learning Representations ICLR, 2021.
[3] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models, 2020.
[4] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10684-10695, 2022.
[5] Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. arXiv:2207.12598, 2022.
arviv preprint
[6] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding. Advances in Neural Information Processing Systems, 35:36479-36494 2022.
[7] Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. Advances in neural information processing systems, 32, 2019.
[8] Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. arXiv preprint 1. 2020.
[9] Jonathan Shen, Ruoming Pang, Ron Weiss, Mike Schuster, Navdeep Jaitly, Zongheng Yang, Zhifeng Chen, Yu Zhang, Yuxuan Wang, Rj Skerrv-Ryan, et al. Natural tts synthesis by conditioning wavenet on mel spectrogram predictions. In 2018 8 IEEE international conference on acoustics, speech and signal processing (ICASSP), pages 4779-4783, IEEE, 2018.
[10] Yi Ren, Chenxu Hu, Xu Tan, Tao Qin, Sheng Zhao, Zhou Zhao, and Tie-Yan Liu. Fastspeech 2: Fast and high-quality end-to-end text to speech. In International Conference on Learning Representations, 2021.
[11] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
[12] Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christo- pher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, al. Opt: Open pre-trained transformer language models. arXiv preprint ar 2200000000000088,, 2022.
[13] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Tim- othée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models, 2023.
[14] Prafulla Dhariwal and Alex Nichol. Diffusion models beat gans on image synthesis, 2021.
[15] Luping Liu, Ren, Zhijie Lin, and Zhou Zhao. Pseudo numerical methods for diffusion models on manifolds. In International Conference on Learning Representations, 2022.
[16] Tim Salimans and Jonathan Ho. Progressive distillation for fast sampling of diffusion models. In International Conference on Learning Representations, 2022.
[17] Chenlin Meng, Ruiqi Gao, Diederik P Kingma, Stefano Ermon, Jonathan Ho, and Tim Sali- mans. On distillation of guided diffusion models. In NeurlPS 2022 Workshop on Score-Based Methods, 2022.
[18] Yuzhang Shang, Zhihang Yuan, Bin Xie, Bingzhe Wu, and Yan Yan, Post-training quantization on diffusion models, 2023.

---

[19] Jungwook Choi, Zhuo Wang, Swagath Venkataramani, Pierce I-Jen Chuang, Vijayalakshmi Srinivasan, and Kailash Gopalakrishnan. PACT: parameterized clipping activation for quantized neural networks. CoRR, abs/1 805.06085, 2018.
[20] Yash Bhalgat, Jinwon Lee, Markus Nagel. Tijmen Blankevoort, and Nojun Kwak. LSQ+: improving low-bit quantization through learnable offsets and better initialization. Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
[21] Eunhyeok Park and Sungjoo Yoo. PROFIT: A novel training method for sub-4-bit mobilenet models. European Conference on Computer Vision (ECCV), 2020.
[22] Markus Nagel, Mart van Baalen, Tijmen Blankevoort, and Max Welling. Data-free quantization through weight equalization and bias correction. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1325-1334, 2019.
[23] Markus Nagel, Rana Ali Amjad, Mart Van Baalen, Christos Louizos, and Tijmen Blankevoort. Up or down? adaptive rounding for post-training quantization. In International Conference on Machine Learning, pages 7197-7206. PMLR, 2020.
[24] Yuhang Li, Ruihao Gong, Xu Tan, Yang Yang, Peng Hu, Qi Zhang. Fengwei Yu, Wei Wang. and Shi Gu. Brecq: Pushing the limit of post-training quantization by block reconstruction. In International Conference on Learning Representations, 2021.
[25] Xiuying Wei, Ruihao Gong, Yuhang Li, Xianglong Liu, and Fengwei Yu. Qdrop: Randomly drop- ping quantization for extremely low-bit post-training quantization. In International Conference on Learning Representations, 2022.
[26] Jascha Sohl- Dickkstii,,, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsuper- vised learning using nonequilibrium thermodynamics. In International Conference Machine Learning, pages 2256-22666 PMLR, 2015.
[27] Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade W Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. Laion- 5b: An open large-scale dataset for training next generation image-text models. In Thitty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2022.
[28] Sangil Jung, Changyong Son, Seohyung Lee, Woo Son, Jac-Joon Han, Youngjun Kwak, Sung Ju Hwang, and Changkyu Choi. Learning to quantize deep networks by optimizing quantization intervals with task loss. Conference Computer Vision and Pattern Recognition, (CVPR), 2019.
[29] Steven K. Esser, Jeffrey L. McKinstry, Deepika Bablani, Rathinakumar Appuswamy, and Dharmendra S. Modha. Learned step size quantization. International Conference on Learning Representations (ICLR), 2020.
[30] Juncheol Shin, Junhyuk So, Sein Park, Seungyeop Kang, Sungjoo Yoo, and Eunhyeok Park. Nipq: Noise proxy-based integrated pseudo-quantization In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 3852-3861. June 2023.
[31] Yunshan Zhong, Mingbao Lin, Xunchao Li, Ke Li, Yunhang Shen, Fei Chao, Yongjian Wu, and Rongrong Ji. Dynamic dual trainable bounds for ultra-low precision super-resolution networks. arXiv preprint
2022.
[32] Changhun Lee, Hyungjun Kim, Eunhyeok Park, and Jae-Joon Kim. Insta-bnn: Binary neural net- work with instance-aware threshold. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 17325-17334, 2023.
[33] Cheeun Hong, Sungyong Baik, Heewon Kim, Seungjun Nah, and Kyoung Mu Lee. Cadyq: Content-aware dynamic quantization for image super-resolution. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part VII, pages 367-383. Springer, 2022.
12

---

[34] Zhenhua Liu, Yunhe Wang, Kai Han, Siwei Ma, and Wen Gao, Instance-award dynamic neural network quantization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12434-12443, 2022.
[35] Pavan Kumar Anasosalu Vasu, James Gabriel, Jeff Zhu, Oncel Tuzel, and Anurag Ranjan. Mobileone: An improved one millisecond mobile backbone. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 7907-7917, 2023.
[36] Andrew Tulloch and Yangqing Jia. Quantization and training of neural networks for efficient integer-aritmmeticoonly inference. Conference on Computer Vision and Pattern Recognition (CVPR), 2018.
[37] Xiuyu Li, Long Lian, Yijiang Liu, Huanrui Yang. Zhen Dong, Daniel Kang, Shanghang Zhang, and Kurt Keutzer. Q-diffusion: Quantizing diffusion models, 2023.
(38) Zhewei Yao, Reza Yazdani Aminabadi, Minjia Zhang, Xiaoxia Wu, Conglong Li, and Yuxiong He. Zeroquant: Efficient and affordable post-training quantization for large-scale transformers. Advances in Neural Information Processing Systems, 35:27168-271883 2022.
[39] Yoshua Bengio, Nicholas Léonard, and Aaron Courville. Estimating or propagating gradients through stochastic neurons for conditional computation. arXiv preprint arXivv1308.344 2013.
[40] Nasim Rahaman, Aristide Baratin, Devansh Arpit, Felix Draxler, Min Lin, Fred Hamprecht, Yoshua Bengio, and Aaron Courville. On the spectral bias of neural networks. In International Conference on Machine Learning, pages 5301-5310. PMLR, 2019.
[41] Matthew Tancik, Pratul Srinivasan, Ben Mildenhall, Sara Fridovich- Keil,, Nithin Raghavan, Utkarsh Singhal, Ravi Ramamoorthi, Jonathan Barron, and Ren Ng. Fourier features let networks learn high frequency functions in low dimensional domains. Advances in Neural Information Processing Systems, 33:7537-7547, 2020.
[42] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
[43] Ben Mildenhall, Pratul Srinivasan, Matthew Tancik, Jonathan Barron, Ravi Ramamoor- thi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. Communications of the ACM, 65(1):99-106, 2021.
[44] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. Proceedings of the IEEE international conference on computer vision, pages 1026-1033 2015.
[45] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. Citeseer, 2009.
[46] Fisher Yu, Ari Seff, Yinda Zhang, Shuran Song. Thomas Funkhouser, and Jianxiong Xiao. Lsun: Construction of large-scale image dataset using deep learning with humans in the loop. arXiv preprint arXiv: 506.03365, 2015.
[47] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to local nash equilibrium. Advances in neural information processing systems, 30, 2017.
[48] Tim Salimans, Ian Goodfellow, Wojeiech Zaremba, Vicki Cheung. Alec Radford, and Xi Chen. Improved techniques for training gans. Advances neural information processing systems, 29, 2016.
[49] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, perforreformanc deep learning library. Advances neural information processing systems, 32, 2019.
13


Reviews:
[{"review_id": "T2QyQiapD9", "reviewer": "Reviewer_qrrn", "summary": "The authors present and empirically evaluate a novel quantization scheme for diffusion models. Static quantization of the evolving activation distribution in diffusion models leads to performance degradation. By using a quantization range which is a function of the timestep, the authors improve performance.", "questions": "- This paper seems slightly contradictory. In section 3.3, the authors assume that the time evolution of the activation distribution is constant, which permits the use of their TDQ module. The authors also make note of the high cost of inference (clearly outweighing the training costs). So why is the TDQ network module neccessary? Why not just learn the the activation distribution directly for each timestep. Learning this function through a network (rather than simply per timestep) would impose a smoothness contraint on the function. Is this the only benefit?\n- Have the authors provided an ablation study where the activation variations, as in Figure 9, are used rather than the TDQ module? Again, why not just compute these beforehand? Does using this pre-computed activation variance perform better? Perhaps some more detailed ablations studies are required?\n- How robust is this TDQ module to training? No error bars are provided so it's unclear if there are any failure cases, or if the predicted intervals remain quite consistent across trials. Given the assumption in section 3.3, they should always converge to the same interval values. Do the authors observe this?", "limitations": "- The limitations of this work aren't explicitly outlined. For example, whilst the authors show a couple of plots illustrating the time evolution of the activation distribution, the assumption doesn't appear to be investigated. What are the effects of this assumption?\n- In addition, it doesn't appear that the authors have outlined any limitations wrt to how text conditioning (v. common for diffusion models in practice) could break this assumption. Perhaps useful for future work, but the authors could make a comment for the sake of practitioners looking to employ this method?\n- I see no wider negative societal implications of this work. ", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "The paper is very well structured and easy to follow and provides clear and concise descriptions of both introductory quantization literature, as well as diffusion model literature. The problem definition and proposed solution are also very well described, especially in Figures 2 and 3. Empirically, evaluation under multiple metrics (FID and ID) is always appreciated. The experimental setup is well described. ", "weaknesses": "- The form of the TDQ module (network size, parameters, structure) isn't provided.\n- Not a particularly major issue but there are a few spelling/grammatical mistakes. For brevity I won't list all the errors but online tools like Grammarly are free to use and would help in this regard. \n- For other weaknesses, please see my questions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable feedback and constructive reviews. \n\n---\n\n### Weakness\n\n**W1: TDQ Module Specifications**\n\nIn this manuscript, we present the TDQ module, a carefully designed 4-layer MLP with ReLU activation. The module takes 128 input elements and has a hidden layer with 64 elements. We will add this details in camera-ready iteration.\n\n**W2: Addressing Spelling and Grammar Mistakes**  \n\nWe appreciate your thorough examination. We will review the grammar and wording once more, utilizing professional tools.\n\n---\n\n### Questions\n\n**Q1: The necessacity of TDQ module** \n\nDuring inference, the TDQ Module's computational cost matches that of learning activation distribution directly for each timestep. The learnable parameters of the TDQ Module undergo training during the PTQ/QAT process, and once this process concludes, these parameter values become static. Because its input depends solely on the time step, the quantization interval to each timestep can be pre-calculated offline. Thus, the TDQ Module exhibits no shortcomings when compared to the direct learning of activation distribution for each timestep. \n\nMoreover, the TDQ Module presents clear advantages over the method of per-timestep learning. Firstly, by evaluating activation alterations collectively across multiple timesteps, the learning process tends to be more stabilized, a point which the reviewer rightly highlighted. Furthermore, even if there are discrepancies in the length of the timestep during training and inference (e.g., SDE[Song et al.] uses 1000 timestep during training but employs 3000 for sampling), the TDQ Module can determine the apt quantization interval for the generation step, referencing the parameters it has learned. \n\nTo address the reviewer's feedback and further validate our assertions, we've carried out additional experiments. Detailed results can be found in the global response 1. We kindly request you to refer to that section for more insights. \n\n**Q2: Ablation Study on the Use of Activation Variation Versus the TDQ Module**\n\nPrior work in QAT (e.g., PACT, QIL, LSQ) and PTQ (e.g., AdaRound, BRECQ) has emphasized the importance of determining the ideal quantization interval. This is crucial in balancing between truncation and rounding errors. These methods typically treat the quantization interval as a learnable parameter, and refine it through gradient descent to minimize objective function like task loss or reconstruction error. Inspired by these principles, the TDQ module is designed to identify optimal quantization ranges for each timestep using gradient descent based on established PTQ/QAT techniques.\n\nTo verify our method's efficiency, we contrasted performance of TDQ-learned temporal intervals with pre-computed maximum intervals. In CIFAR-10 DDIM W4A4 QAT, TDQ yielded an FID score of **4.80**, while the pre-computed maximum's score of 6.30. This highlights TDQ's superior output quality over precomputed statistics based method. A comprehensive ablation study will be elucidated in our finalized version.\n\n**Q3: Robustness of the TDQ Module**\n\nTo evaluate the robustness of the TDQ module, we perform three trials with distinct seeds for both DDIM CIFAR10 QAT and LDM churches QAT. Please check *Table 1* at our Global Response PDF.\n\nAs shown in Table, TDQ consistently outperform of conventional QAT methods, indicating its enhanced stability over LSQ.  The limited adaptability of LSQ—relying on a single quantization interval across all time steps—can occasionally lead to suboptimal performance. In contrast, TDQ dynamically updates the interval at every time step, taking into account adjacent distributions. We will refresh our paper's tables, incorporating the mean variance observed across multiple trials.\n\nFurthermore, an intriguing observation across various trials is the convergence of quantization intervals. In disparate trials, these intervals tended to approach nearly identical values at each time step. This consistency further underlines TDQ's robustness. For a visual representation, please kindly refer to the  *Figure 2* at our Global Response PDF.\n\n---\n\n## Limitations\n\n**L1: The Time Evolution of the Activation Distribution**\n\nA limitation of our assumption resides in the potential non-marginal time-dependent divergence of the activation distribution in specific models or layers.\n\nFor example, the LDM model manifests a more subtle temporal variation compared to a pixel-wise diffusion model, resulting in only a modest performance enhancement when employing the TDQ module. Similarly, even within the DDIM models, the deeper layers may exhibit less temporal dependency, a trait that might appear counter intuitive.\n\nHowever, it is essential to recognize that even in such instances, the TDQ module guarantees convergence, yielding consistent output across time steps. This alignment ensures that the output quality is compatible with that of existing PTQ/QAT algorithms, thus preserving the module's value even where the time-related divergence might be less significant.\n\n**L2: Effect of TDQ Module for Text Conditioning**\n\nThank you for raising an important issue concerning the effect of text conditioning. We extracted 12 images for three distinct texts from Stable-Diffusion and depicted their activation distributions in *Figure 1* of the Global Response PDF.\n\nAs revealed by the figure, input distributions display minor variations depending on text conditions. However, the feature map's overall trends seem to be more reliant on time than the specific text condition. Thus, we have found that our method can be applied to text-conditioned scenarios such as Stable-Diffusion without hindrance.\n\nThe extension of the TDQ Module to cater to text conditioning presents an intriguing aspect and points to a promising avenue for future research. To enhance understanding and offer a visual perspective, we'll include the mentioned figure and a fitting explanation in the supplementary material."}, {"review_id": "zIhj0nPuZZ", "reviewer": "Reviewer_7wR5", "summary": "This paper proposes Temporal Dynamic Quantization (TDQ), a new quantization approach for diffusion models. This method utilizes a learned module to adjust the quantization interval with respect to timestep to deal with activation quantization errors. Unlike conventional dynamic quantization techniques, the TDQ approach has no computational overhead during inference and is compatible with both post-training quantization (PTQ) and quantization-aware training (QAT). Experiments have been performed using DDIM on the CIFAR-10 dataset and LDM on the LSUN-Churches dataset with TDQ and various traditional QAT and PTQ approaches.", "questions": "- Can you further explain the output dynamics in Figure 9 and Appendix Figure 6? E.g. what do blue and red curves stand for and how to interpret them?\n- Looking at Appendix Table 1, I am curious if some simple baselines following the same principle can be helpful, such as using multiple quantizers at different time steps under the LSQ setting e.g. using a separate activation quantizer every 20 steps.\n- How sensitive is the TDQ module (MLP) to the hyperparameters? Are there different design choices for the training process that cause different results? For instance, it is mentioned in the paper that initialization is crucial – are there experiments showing that a different initialization (e.g. gaussian, Xavier, zero…) can make results worse?", "limitations": "The authors adequately addressed all limitations.", "rating": 5, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- The paper is well organized and written with clear motivation; the idea of addressing the rapid changes in input activation to reduce the truncation & rounding errors with a time-dependent quantization configuration is natural and interesting.\n- The proposed TDQ framework can be applied to both QAT and PTQ cases.\n- Experimental results demonstrate that the proposed framework can outperform several previous QAT and PTQ methods. Ablation studies have been conducted to show that TDQ can generalize to scenarios with fewer denoising steps and analyze output dynamics across layers.", "weaknesses": "My concerns are mainly about the evaluation in this paper:\n- The proposed TDQ framework primarily aims to deal with the activation quantization, but it is not specified whether the act-to-act matrix multiplications (attn and v) in the attention layers are quantized. By examining the code attached in the supplementary materials, it seems like those computations still remain in FP. However, they can take a significant portion of the compute & memory overheads and are fully quantized in closely related work [1], while past literature [1, 2] has pointed out that the post-softmax attention scores are usually hard to quantize (e.g. PTQ4DM suffers from performance degradation after quantizing them according to Appendix E in [1]). It is unclear if TDQ can deal with this performance degradation without providing experimental results with act-to-act matmuls in attention layers quantized.\n- It is not accurate to call PTQ4DM the 'state-of-the-art study'—Q-Diffusion [1] was proposed around the same time as PTQ4DM and contained results that went down to 4-bit weights on both CIFAR and LSUN datasets, and achieved superior results on CIFAR. Its code (with quantized checkpoints) was released in April. I would highly recommend the authors include and compare Q-Diffusion results for CIFAR and LSUN-Churches experiments as it is a highly related work that natively covers many experimental settings in this work that are not in PTQ4DM.\n\n[1] Li, X., Liu, Y., Lian, L., Yang, H., Dong, Z., Kang, D., ... & Keutzer, K. (2023). Q-diffusion: Quantizing diffusion models. arXiv preprint arXiv:2302.04304.  \n[2] Yuan, Z., Xue, C., Chen, Y., Wu, Q., & Sun, G. (2021). PTQ4ViT: Post-Training Quantization Framework for Vision Transformers with Twin Uniform Quantization. arXiv preprint arXiv:2111.12293.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "\nThank you for bringing up important points with productive feedback.\n\n---\n\n### Weaknesses\n\n**W1: Quantization Configuration Details for Different Layers**\n\nThank you for raising the pivotal aspect of activation quantization for attention multiplicands. To clarify, our implementation does indeed account for the quantization of the Query, Key, and Value activations. In scenarios involving matrix multiplication between the Query and Key, the full benefits of low-precision multiplication are harnessed.\n\nHowever, you rightly identified a nuance: the attention scores following softmax were not subjected to quantization. While Q-Diffusion only experimented with reducing the activation bit width to 8 bits, our experiments reduced it down to 3 bits. In this low-bit configuration, we found that excessive quantization of the attention scores led to a substantial decline in output quality.\n\nTo manage this, we refrained from applying quantization to the attention score, considering its high sensitivity to the quantization process. It's noteworthy to mention that this approach was consistently applied; none of the baselines in our experiment incorporated quantization in this segment either. Although specialized quantization techniques exist for post-softmax scores, we did not employ them, as this aspect was not the primary focus of our experiments.\n\nIn response to concerns regarding the application of quantization to the softmax score, we conducted a supplementary experiment. We wanted to validate whether TDQ performs well, even in scenarios where the output of attention score is quantized. The experiment was designed with four distinct scenarios in mind, and the LSUN-church 256 dataset with QAT was utilized.\n\nThe results are as follows:\n\n1. **Without softmax quantization:**\n    - LSQ: FID score of 5.06\n    - TDQ: FID score of **4.64**\n2. **With softmax Quantization:**\n    - LSQ: FID score of 5.21\n    - TDQ: FID score of **4.99**\n\nThese findings quantitatively affirm that TDQ consistently achieves superior quality results, even when attention quantization is introduced. In the forthcoming camera-ready version, we will provide more extensive details and additional results related to the quantization of the attention score.\n\n**W2: Comparsion with Q-Diffusion**\n\nWe have thoroughly prepared a response to address your concern. Kindly refer to the global response section where we detail the comparison with Q-Diffusion.\n\n---\n\n### Questions\n\n**Q1:  Interpretation of Fig. 9 in main paper and Fig 6 in Appendix .**\n\nIn our visualizations, the blue curve represents the quantization interval for each time step, while the red curve illustrates the fluctuations in the magnitudes of the activations over each time step. As expected, the ideal behavior is for the quantization interval (blue curve) to expand as the variance (depicted by the red curve) amplifies. This expansion ensures that quantization errors are minimized.\n\nFor many layers, this desired behavior is evident. For instance, the layer `Input_blocks.1.0.emb_layers.1.act_func`, as showcased in Appendix Figure 6, clearly demonstrates this pattern. However, certain layers, such as `Middle_block.0.out_layers.3.act_func` depicted in the same figure, don't exhibit a pronounced effect from TDQ. This discrepancy arises primarily due to the combination of significant variations in data and the layer's low temporal dependency. In scenarios like these, the output from TDQ tends to stabilize, converging to a specific value. This behavior mirrors that of LSQ, where the quantization interval becomes essentially constant.\n\n**Q2. Learning quantization interval directly for each time step**\n\nWe have thoroughly prepared a response to address your concern. Kindly refer to the global response section where we detail the comparison with the direct interval learning for each time step.\n\n**Q3. Robustness of TDQ module**\n\nIn our paper, when we refer to the sensitivity of initialization, we are specifically addressing the initial value of the quantization interval. It's essential that the quantization interval align with the data distribution right from random initialization. Otherwise, a too-small interval might truncate valuable information, while a too-large one could increase discretization errors significantly. To ensure stable convergence, we initialize the bias of the last layer of the TDQ module following the initialization method described in the LSQ paper.\n\nAdditionally, the TDQ module seems relatively insensitive to other design choices. For example, it exhibits comparable accuracy across different random initialization methods, as evidenced by the table below (except for Gaussian initialization). In some instances, TDQ fails to converge (though it's worth noting that TDQ enhances convergence stability, and existing quantization schemes like LSQ are more prone to failure). However, once trained properly, the output quality is consistently good.\n\n- **TDQ Weight Initialization ablation**\n    \n    \n    | (FID↓) | He (Default) | Gaussian | Zero | Xavier |\n    | --- | --- | --- | --- | --- |\n    | CIFAR10 W4A4 | **4.80** | Fail | 4.98 | 4.90 |\n\nThe effect of varying the number of layers was also examined, and the results remained consistent across different configurations. We empirically selected four layers as our default number. These findings and a more detailed explanation will be included in the supplementary material of the camera-ready paper.\n\n\n- **TDQ Number of layer ablation**\n    \n    \n    | (FID↓) | 2 Layer  | 4 Layer (Default) | 8 Layer  |\n    | --- | --- | --- | --- |\n    | CIFAR10  W4A4 | 5.18 | **4.80** | 4.88 |"}, {"review_id": "Dc6kT7Wvgh", "reviewer": "Reviewer_xsTz", "summary": "While diffusion models provide outstanding quality and diversity of generated data, they demand high storage and computation, limiting its applications on mobile devices. Further, the paper points out that current existing works on quantization are not suitable for diffusion models as diffusion models have unique temporal properties due to iterative sampling process. As such, the paper introduuces a new quantization technique for diffusion models, where quantization interval is dynamically adjusted. As a result, the proposed framework prvoides strong performance.", "questions": "Please refer to the weakness section", "limitations": "No limitations mentioned in the paper. The potential limitation of the work is not taking into account for activation distribution variation that exists among different inputs.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The proposed method can be easily integrated with the existing QAT and PTQ algorithms.\n- Clear visualizations which help the understanding of the paper and motivations. \n- Simple yet effective quantization method without additional inference costs.", "weaknesses": "- Don't input-dependent quantization methods [31,32,33,34] provide computational efficiency benefits even after considering extra overhead? Is there a proof for supporting the following claim \"dynamic approaches often incurs additional costs for extracting statistical information from activations, making it challenging to achieve performance improvements in practice.\"? It would be good to refer to the results or references for supporting this claim or tone down the claim a bit.\n.\n- Missing experimental comparisons against [36], which is another work on quantization of diffusion model.\n\n- The paper claims that the activation distribution is similar across inputs with Figure 2. But, Figure 2 shows the activations of an earlier convolutional layer, which is known to have low-level features that are common across diverse layers. It would be good to show that similar trends can be observed across different layers. Indeed, less clear and less time dependent trends can be observed in deeper layers (mid.attn_1.proj_out.0, down.1.downsample.conv.0, down.1.attn.0.k.0, mid.block_1.conv1.0) in Figure 7 of the supplementary material. I suspect that deeper layers will exhibit even more complex trends that may not look like time-dependent. More detailed discussions are needed. \n\n- The paper claims that the proposed method can be seamlessly integrated into existing quantization algorithms, but the paper only shows the benefit of doing that with LSQ in the case of QAT.\n\n- Non-exhaustive list of typos (a thorough proof-reading is needed):\n   - can decipts in Line 148\n   - there have been limited works tried to in Line 109", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "\nThank you for your positive comment and constructive feedbacks\n\n---\n\n### Weaknesses\n\n**W1: Extra overhead of input-dependent quantization method**\n\nWe appreciate the feedback and understand the concerns raised. We would like to direct the reviewer's attention to the recently published paper, \"MobileOne.\"[1]. This paper highlights that input-dependent activation functions, such as Dynamic Shift-Max and Dynamic ReLU, experience latency that is more than 50 times slower compared to conventional activation functions like ReLU. Despite the relatively small number of operations these functions involve, the computation of input-dependent statistics greatly intensifies memory access, resulting in a significant increase in latency. Given that input-dependent quantization operators also leverage similar input-dependent statistics, latency becomes a predominant bottleneck. Nevertheless, we acknowledge the need for clarity and will revise our statement to be more measured and nuanced.\n\n**W2: Comparison with Q-Diffusion**\n\nWe have thoroughly prepared a response to address your concern. Kindly refer to the global response section 2 where we detail the comparison with Q-Diffusion. \n\n**W3: Time dependency in deeper Layers**\n\nWe acknowledge the feedback and appreciate the depth of your observation. For many layers, the expected behavior, where the activation distribution changes over time, is indeed prominent. As an illustrative example, the layer **`Input_blocks.1.0.emb_layers.1.act_func`**, depicted in Appendix Figure 6, vividly demonstrates the time-evolving nature of the activation distribution.\n\nHowever, as rightly pointed out by you, certain intermediate layers, notably **`mid.attn_1.proj_out.0`** (as referenced in Figure 7 of the supplementary material), do not exhibit a similar trend. This lack of temporal dependency in such layers can be attributed mainly to a combination of significant data variation and the inherent low temporal dependency of the layer itself. In situations of this kind, the TDQ output tends to converge, effectively stabilizing to produce a consistent value. This phenomenon is reminiscent of the behavior observed with LSQ, wherein the quantization interval largely remains static.\n\n**W4: Integration of TDQ with various QAT schemes**\n\nTo investigate the general applicability of the TDQ approach across different Quantization-Aware Training (QAT) methods, we conducted further experiments with four QAT schemes: PACT, NIPQ, DuQ, and QIL. The training was conducted on CIFAR-10 DDIM W4A4 QAT. The results, as summarized in the table below, reveal that NIPQ, DuQ, and QIL demonstrate marked improvements when integrated with the TDQ scheme, underscoring the adaptability of TDQ with various gradient-based QAT methodologies.\n\n- **Integration with other quantization scheme**\n\n  | (FID↓) | PACT | NIPQ | DuQ | QIL |\n  | --- | --- | --- | --- | --- |\n  | baseline | **51.92** | 30.73 | 7.88 | 9.85 |\n  | +TDQ | 74.56 | **24.27** | **4.46** | **7.51** |\n\nHowever, an exception was observed with PACT, where the integration with TDQ led to a degradation in quality. This anomaly can be attributed to the specific nature of PACT, where the quantization interval is updated based on artificial L2 regularization, rather than through the minimization of quantization error. This divergence from the typical quantization error minimization objective renders the TDQ scheme less effective in this particular context.\n\nThese findings collectively affirm that TDQ exhibits broad compatibility with a variety of QAT methods, thereby reinforcing its potential as a versatile tool in the quantization domain. We will include these supplementary results and an analysis of the observations in the camera-ready version of our paper.\n\n**W5: thorough proof reading**\n\nThank you for the constructive feedback. we will conduct sincere proof reading and refine the writing. \n\n---\n\n### Limitations\n\n**L1: Input-dependent quantization**\n\nAs detailed in our response to weakness 1, extracting instance-wise dynamic information incurs significant latency costs. Furthermore, the instance-wise quantization interval poses challenges by preventing batched references. Nonetheless, in specific hardware environments where the batch size is 1, or when the costs of extracting statistical information can be amortized, utilizing such information alongside time data may lead to performance improvements. We see this integration as an exciting avenue for future research, as it could offer additional enhancements in efficiency and effectiveness.\n\n---\n### References\n[1] Vasu, Pavan Kumar Anasosalu, et al. \"MobileOne: An Improved One Millisecond Mobile Backbone.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023."}, {"review_id": "eOGf913UAy", "reviewer": "Reviewer_ijsE", "summary": "This paper presents a novel quantization method for diffusion models that dynamically adjusts the quantization interval. To this end, the authors present a unique Temporal Dynamic Quantization (TDQ) module, which utilizes the time step as an input to determine the quantization interval, which significantly improves the generation quality. Extensive experiments on CIFAR-10 and LSUN Churches demonstrate the superior performance of the proposed method.", "questions": "1.\tIn Section 3.1, the authors state that uses symmetric quantization for weights and set $p=-n=2^{b-1}-1$. However, compared with LSQ that set $n=-2^{b-1}$, it appears that the proposed method might be under-utilizing one quantization level. This could potentially lead to a slight inefficiency in the model’s performance.\n\n2.\tIn Section 3.3, the authors state that previous studies have demonstrated that incorporating a quantization module … can lead to significant improvements in accuracy for specific applications. However, there appears to be a lack of citation or reference supporting this assertion.\n\n3.\tInstead of using the proposed TDQ module, it is possible to directly learn distinct quantization intervals corresponding to different time steps?\n\n4.\tThe paper lacks a comparison in its experimental section. Specifically, the study has not considered Q-diffusion (as detailed in reference [36]) as a baseline. Since Q-diffusion also applies post-training quantization to diffusion models, it's critical for the authors to include a comparative analysis of their proposed method against Q-diffusion. This would help to further validate the efficacy of their approach and enrich the overall discussion.\n\n5.\tThe paper could benefit from additional ablation studies, which would provide a more comprehensive understanding of the proposed method. One specific point of interest mentioned is the influence of the number of layers in the generator module on the overall performance. By investigating this aspect, it would add depth to the analysis and potential insights into the efficiency of the proposed model.\n", "limitations": "The authors do not provide any discussions on limitations.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The main idea is well-written and easy to follow. The proposed method, though simple in design, proves to be remarkably effective, notably diminishing the decline in performance. Its simplicity does not compromise its efficacy, making it a compelling solution to the problem at hand.", "weaknesses": "The primary weaknesses of the paper include a lack of a vital comparison with Q-diffusion, a key baseline in diffusion models, and a need for more comprehensive ablation studies, such as investigating the impact of the number of layers in the generator module on the model's overall performance.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your positive comment and insightful feedback.\n\n---\n\n### Weaknesses\n\n**W1: Comparsion with Q-Diffusion**\n\nWe have thoroughly prepared a response to address your concern. Kindly refer to the global response section 2 where we detail the comparison with Q-Diffusion.\n\n### Questions:\n\n**Q1: Rationale behind using \\( $p = -n = 2^{b-1} - 1$ \\)**\n\nThank you for the keen observation. Indeed, in a pure mathematical context, the smallest integer level could be represented as $-2^{b-1}$ . However, it's essential to highlight that many prominent acceleration libraries, such as XNNPack in TensorFlow Lite, deliberately exclude the smallest integer value to maintain the symmetry of the data distribution. Our approach adheres to this widely accepted convention seen in quantization acceleration frameworks. Besides, please note that we apply the same restriction for LSQ algorithm where $n $ is set to $-2^{b-1} + 1 $ for fair comparison. \n\n**Q2. Citing dynamic quantization schemes for quality enhancement**\n\nThank you for drawing attention to this aspect. As highlighted in reference [1], utilizing a dynamic computation of the token-wise Min-Max range has led to significant improvements in accuracy. Similarly, reference [2] underscores the enhancement in output quality, achieved by factoring in the Minimum, Variance, and Skewness of individual image instances during the quantization process. We will duly incorporate these references into our paper for a more comprehensive perspective.\n\n**Q3. Learning quantization interval directly for each time step**\n\nWe have thoroughly prepared a response to address your concern. Kindly refer to the global response section where we detail the comparison with the direct interval learning for each time step \n\n**Q4. Comparison with Q-Diffusion**\n\nWe have thoroughly prepared a response to address your concern. Kindly refer to the global response section where we detail the comparison with Q-Diffusion. \n\n****Q5. Ablation study for the number of layers in the TDQ module****\n\nWe appreciate your interest in understanding the effect of the number of layers within the TDQ module on the performance. In response to this, we conducted an ablation study to examine the impact of changing the number of layers within the TDQ module.\n\nThe results, as detailed in the table below, illustrate that the output quality becomes substantially consistent when the number of layers exceeds 4. Based on these experimental findings, we have empirically selected 4 layers of MLP for the TDQ module. This decision was driven by a balanced consideration of performance and complexity.\n\nWe recognize the importance of this analysis in enhancing the understanding of the TDQ module's design, and we will include the complete details and additional ablations in the camera-ready version of our paper.\n\n- **TDQ Number of layer ablation**\n  | (FID↓) | 2 Layer  | 4 Layer (Default) | 8 Layer |\n  | --- | --- | --- | --- |\n  | CIFAR10 W4A4 | 5.18 | **4.80** | 4.88 |\n\n### Limitations\n\n**L1: Discussion on the limitations of our study**\n\nTo elucidate the findings and potential limitations of our research, we have outlined key points below:\n\n1. **Temporal dependency across layers**: The majority of layers in our study manifest the anticipated behavior where the activation distribution exhibits a temporal evolution. A salient representation of this is the layer **`Input_blocks.1.0.emb_layers.1.act_func`**, as highlighted in Appendix Figure 6, which lucidly captures the dynamics of the activation distribution over time. Conversely, specific intermediate layers, most notably **`mid.attn_1.proj_out.0`** (referenced in Figure 7 of the supplementary material), deviate from this behavior. The absence of marked temporal dependency in such layers stems chiefly from pronounced data variations and the intrinsically weak temporal relationship of the layer. Under these circumstances, the output from the TDQ module tends to stabilize, yielding a largely uniform value. This behavior parallels the tendencies exhibited by LSQ, where the quantization interval predominantly retains a constant magnitude.\n2. **Issues of overfitting with the TDQ module during PTQ**: Subsequent to the review process, we observed that the default TDQ module is prone to overfitting during Post-training Quantization (PTQ), leading to a diminished output quality. This observation is further elaborated in the section titled \"Learning quantization interval directly for each time step\" in the global response. Our analysis suggests that employing a thin TDQ module, achieved by trimming the layer count to 3 and curtailing the hidden dimension to 16, can substantially enhance PTQ performance. We aim to incorporate this important observation in the forthcoming camera-ready iteration of our paper.\n\n### References\n\n[1] Yao, Zhewei, et al. \"Zeroquant: Efficient and affordable post-training quantization for large-scale transformers.\" Advances in Neural Information Processing Systems 35 (2022): 27168-27183.\n\n[2] Lee, Changhun, et al. \"INSTA-BNN: Binary neural network with instance-aware threshold.\" *arXiv preprint arXiv:2204.07439*(2022)."}]

Author rebuttal:
# Global response for reviewers
We appreciate all the reviewers for taking time to review our work and giving valuable comments and feedbacks that help to improve the completion of our paper. 

---
## 1. Learning quantization interval directly for each time step

We highlighted that as the activation distribution evolves over time, the quantization interval should adapt correspondingly. Consequently, we introduced the TDQ Module. However, several reviewers inquired if a more direct approach might be viable: namely, could the quantization interval be trained using scalar variables assigned to each time step, sidestepping the need for more intricate structures like the TDQ Module? To address this, we carried out experiments contrasting the performance of the TDQ Module with a configuration where the quantization interval parameter is directly learned. Furthermore, we introduced an alternative concept: employing a shared quantization interval across multiple time steps. In the table we provide, the notation \( S_{N} \) signifies that there are a total of \( N \) learnable quantization interval parameters that cover 1000 / N time steps uniformly. 

- **Table 1. Quantization-Aware Training (QAT) Results**
    
    
    | (FID↓) | TDQ | S_1 (LSQ) | S_10 | S_50 | S_100 | S_1000 |
    | --- | --- | --- | --- | --- | --- | --- |
    | CIFAR10-DDIM W4A4 | **4.48** | 7.3 | 4.75 | 5.17 | 4.88 | 5.53 |
    | CIFAR10-DDIM W3A3 | **6.48** | 7.63 | 8.89 | 6.92 | 6.82 | 9.94 |
    | Church-LDM W4A4 | **4.64** | 5.06 | 5.17 | 5.07 | 4.78 | 5.10 |
    | Church-LDM W3A3 | 6.57 | 7.21 | 6.70 | **6.47** | 6.64 | 6.86 |
- **Table 2. Post-Training Quantization (PTQ) Results**
    
    | (FID↓) | TDQ | TDQ_Thin | S_1(LSQ) | S_2 | S_4 | S_10 | S_20 |
    | --- | --- | --- | --- | --- | --- | --- | --- |
    | Church-LDM W4A8 | 44.88 | **28.74** | 76.36  | N/A | 33.19 | 37.63 | 29.87 |
    | Church-LDM W4A6 | 120.53 | 55.27 | 158.07 | 193.11 | 185.19 | 72.37 | **47.39** |
    | ImageNet-LDM W4A6 | 41.23 | **16.96** | 47.26 | - | 85.11 | 21.71 | 17.38 |

In our QAT experiments, TDQ exhibited outstanding or near-optimal performance, while configurations with \( S_{50} \) or \( S_{100} \) also demonstrated significant effectiveness. The TDQ module's success lies in its ability to enable continuous and stable learning of the quantization interval, taking into consideration the evolution of the activation distribution over neighboring time steps. This results in generator output that is more apt for generation.

Simultaneously, the concept of a shared quantization interval across multiple time steps can partially capture the benefits of TDQ, significantly enhancing output quality over the \( S_{1} \) configuration. However, TDQ still achieves superior output quality and offers greater versatility, particularly in scenarios such as employing different timesteps during training and inference.

In our PTQ experiments, while the standard TDQ module yielded a notable performance boost over \( S_1 \), it lagged behind configurations such as \( S_{20} \). Our investigations revealed that the default TDQ module was prone to overfitting due to two primary reasons: 1. Training the MLP with a limited calibration dataset (containing only 256 samples) proves challenging. 2. The relatively brief training iteration, typically 5000 in BRCEQ, finds it hard to filter out the high-frequency component present in frequency-encoded input features. To mitigate these constraints, we introduced a streamlined version of TDQ—referred to as **TDQ Thin**. This refined module uses a 3-layer MLP with a mere 16 hidden dimensions and omits the frequency encoding for time steps. Notably, this adaptation showcased results that either surpassed or matched the best performances observed in the \( S_{N} \) experiments.

We deeply appreciate the constructive feedback. It has provided insights enabling us to refine TDQ to be more PTQ-friendly. We are committed to incorporating these insights judiciously, ensuring they feature prominently in the finalized version of the paper.

---

## 2. Comparison with Q-Diffusion

It's important to highlight that at the time of our paper's composition, Q-Diffusion was not a peer-reviewed paper and was only available on Arxiv. Although it is currently marked as an ICCV2023 accepted paper on their GitHub, no official accepted list has been published. Moreover, Q-Diffusion only provided inference code, lacking the essential training (calibration) code. Our attempts to reproduce their results during the submission period were met with challenges. Fortunately, the recently released paper PTQD[1] offered a codebase that reimplemented Q-Diffusion. We utilized this codebase to conduct our experiments. However, it's worth noting that PTQD's implementation differs slightly from the original Q-Diffusion paper, such as the absence of shortcut-splitting quantization and its use of symmetric per-tensor weight quantization instead of asymmetric per-channel weight quantization found in Q-Diffusion.
The comparative results are tabled below:

- **Table 3. Performance comparison with Q-Diffusion**
    | (FID↓) | Church-LDM W4A8 | Church-LDM W4A6 | ImageNet-LDM W4A6 |
    | --- | --- | --- | --- |
    | QDiffusion  | 76.36 | 158.07 | 47.26 |
    | + TDQ | **28.74** | **55.27** | **16.96** |


As demonstrated in the table, our TDQ method shows significant improvement compared to Q-Diffusion. These results underscore the importance of utilizing different quantization intervals for different time steps when quantizing diffusion models. Because TDQ and QDiffusion are orthogonal methods, the integration of TDQ over QDiffusion offsers notable performance improvement.  We plan to include these results (or an updated version if the official code becomes available) in the camera-ready version of our work.


---
### References

[1] He, Yefei, et al. "PTQD: Accurate Post-Training Quantization for Diffusion Models." arXiv. (2023).

Meta-review: Most reviewers are positive about the paper. The reviewer with borderline rejection has some concerns, but the rebuttal has provided an explanation. This work has presented an interesting quantization method that dynamically adjusts quantization intervals, and the proposed method is compatible with existing QAT and PTQ algorithms. The experiments are thorough and convincing. The authors may also add the comparison to Q-Diffusion in this camera-ready version. After discussing this paper with SAC, I would recommend acceptance.
Decision: Accept (poster)