SlowGuess/NeurIPS_2023_OCR huggingface dataset:
To measure the performance of the models, we used Fréchet Inception Distance (FID) [47] and Inception Score (IS) [48] for CIFAR-10, and FID for LSUN-churches. For evaluation, we generated 50,000 images using QAT with DDIM 200 step sampling, and 50,000 images using PTQ with DDIM 100 steps. In the case of QAT, we selected 5 checkpoints with the lowest validation loss and reported the scores from the best performing models.
All of experiments were conducted on the high performance servers having 4xA100 GPUs and 8xRTX3090 GPUs with PyTorch [49] 2.0 framework. The source code is available at https://github.com/ECoLab-POSTECH/TDQ_NeurIPS2023. Besides, we use the notation WxAy to represent x-bit weight & y-bit activation quantization for brevity.
Figure 5: Visualization of W4A4 QAT results, DDIM on CIFAR-10 dataset, (up) Ours (Down) LSQ
Figure 6: Visualization of W8A6 PTQ results, DDIM on CIFAR-10 dataset, (up) Ours (Down) PTQ4DM
Figure 7: Visualization of W8A5 PTQ results, LDM on LSUN-churches dataset, (up) Ours (Down) PTQ4DM
5.1 Quality Analysis after QAT and PTQ
Table 1 compares the TDQ module with existing static quantization methods. As the activation bit decreases, all static quantization methods show inferior quality, while our approach shows consistent output. TDQ module gives substantial quality improvement even in 8-bit, and the benefit becomes even large in 4-bit precision, showing negligible quality degradation to full-precision output. TDQ achieves these benefits by efficiently allocating the limited activation quantization levels.
Besides, NIPQ was introduced to address the instability of the straight-through estimator by applying pseudo quantization based on artificial noise. However, the noise of NIPQ is indistinguishable from the input noise, hindering the diffusion model's convergence. Additional efforts are required to exploit the benefit of PQN-based QAT for diffusion models.
Table 2 presents PTQ comparison of TDQ module against existing static PTQ schemes. The MinMax method represents a naive linear quantization approach where the range is determined by the minimum and maximum values of the target tensor. The experiments demonstrate that while all baselines maintain a good level of FID when the activation bit is high, they experience significant performance degradation as the activation bit decreases. In contrast, TDQ exhibits only a slight level of FID degradation, indicating that the TDQ module is a robust methodology that performs well in both QAT and PTQ scenarios.
Fig. 5 to 7 visualize the generated images of quantized diffusion models. Our method consistently outperforms other quantization techniques in producing images with high-fidelity within the same bit-width configuration. Figs. 5 and 6 further illustrate this, as conventional QAT and PTQ produce blurred and unrecognizable images, whereas our method generates realistic images. The integration of temporal information in activation quantization is shown to be highly effective in maintaining the perceptual quality of the output.
Figure 8: Number of sampling step vs FID. Comparison between the proposed method and conventional QAT when reducing the inference time step from 100 to 10.
Figure 9: Output dynamics of TDQ module. Visualization of the generated interval versus the variation of input activation.
5.2 Generalization Performance of TDQ Module
This section presents the experimental results on the performance of the TDQ module for fast-forward inference. Training is carried out to fully encompass all time steps from 1 to 1000, however, inference can be executed with fewer time steps (between 50 to 100) for enhanced performance. This changes the distribution of the time steps during training/testing, thus, the TDQ module's generation requires good generalization abilities.
Fig. 8 displays the FID measurement for the DDIM model quantized by the LSQ algorithm for both W8A4 and W4A4 configurations. The time step gradually decreases from 100 to 10 during inference. As depicted, LSQ's performance significantly deteriorates as the number of time step decreases, whereas our method's performance declines similarly to the full-precision baseline. This experiment demonstrates that the TDQ module functions effectively even when the sampling time step varies.
5.3 Ablation Studies of TDQ module
To investigate the output dynamics of the TDQ module, we visualized the updates of the dynamic interval in relation to time steps (Fig. 9). The interval, trained using W4A4 LSQ on DDIM, demonstrates a tendency to change in alignment with activation variations. However, this pattern is not consistently observed across all layers. The inconsistencies could potentially indicate that the TDQ module is attempting to generate an interval that minimizes the final loss value, as LSQ adjusts the quantization interval accordingly.
To offer a more comprehensive analysis of the advantages of the TDQ module, we conducted a comparison with input-dependent dynamic quantization methods and performed an ablation study on the TDQ module. As depicted in Table 4, TDQ achieves superior performance compared to dynamic
Table 3: Learning quantization interval directly for each time step. The notation  $S_N$  means that there are total of  $N$  learnable quantization intervals that cover  $1000 / N$  time steps uniformly.
(QAT)	Bit-width	FID ↓
Ours	LSQ(S1)	S10	S50	S100	S1000
DDIM[2]-CIFAR10	W4A4	4.48	7.30	4.75	5.17	4.88	5.53
W3A3	6.48	7.63	8.89	6.92	6.82	9.94
LDM[4]-Churches	W4A4	4.64	5.06	5.17	5.07	4.78	5.10
W3A3	6.57	7.21	6.70	6.47	6.64	6.86
quantization, even without directly utilizing the layer's distribution information. We also investigated the effect of the number of layers within the TDQ module on performance. The results demonstrate that the output quality becomes substantially consistent when the number of layers exceeds 4. Based on these findings, we have empirically selected 4 layers of MLP, considering a balance of performance and complexity.
We also conducted experiments to compare the performance of TDQ with a configuration where the per-time quantization intervals are directly learned. As presented in Table 3, while the concept of directly learning the quantization interval for each time step and using a shared quantization interval across multiple time steps can enhance output quality compared to the baseline  $(S_{1})$ , TDQ continues to exhibit superior output quality and offers greater versatility. The success of TDQ lies in its ability to facilitate continuous and stable learning of the quantization interval while taking into account the evolution of the activation distribution over neighboring time steps.
Table 4: Ablation on TDQ module
method	FID↓
FP32	3.56
Static Quantization(LSQ)	7.3
Dynamic Quantization (max)	6.30
TDQ (2 layer)	5.18
TDQ (4 layer)	4.48
TDQ (8 layer)	4.88
In this section, we discuss additional intuitions and limitations. While many layers exhibit strong temporal dependency, approximately  $30\%$  of the layers still do not show such correlations. These layers are primarily located in the middle block of the U-Net and are heavily influenced by their instance-wise semantic information. Moreover, we observed that the LDM model [4] has fewer temporal dependencies and thus demonstrates fewer performance improvements. However, it's crucial to emphasize that even in these cases, the TDQ module ensures convergence, resulting in consistent outputs across time steps. This alignment guarantees that the output quality remains comparable to that of existing PTQ/QAT algorithms, even in scenarios where temporal dependencies might be less.
In this paper, we explore the challenge of activation quantization in the diffusion model, specifically the dynamics of activation across time steps, and show that existing static quantization methods fall short of effectively addressing this issue. We introduce the TDQ module, seamlessly integrated into both QAT and PTQ, while enhancing the output quality significantly based on the dynamic quantization interval generation. Since our approach can be implemented without any overhead during inference, we expect our study to help improve performance while preserving quality in low-bit diffusion models for mobile and edge devices.
This work was supported by IITP grant funded by the Korea government (MSIT, No.2019-0-01906, No.2021-0-00105, and No.2021-0-00310).