SlowGuess/NeurIPS_2023_OCR huggingface dataset:
The proposed robustness-aware quantization (RAQ) is presented in Algorithm 1. It begins with pre-quantizing the weights of the denoising network, utilizing 4 bits for weight quantization, as weights are relatively easier to quantize. The optimization of activation bits starts from diffusion step  $T$  and proceeds sequentially for each step. At each step, the dictionary  $D_{act}$  is updated to establish the correspondence between the given activation bit  $Bit_{act}$  and the diffusion step  $t$ . This dictionary is used by the denoising network to adjust activation bits. FP32 activations are employed for diffusion steps that have not yet been updated in the dictionary. Subsequently, using the updated bit information, the diffusion model generates  $n$  small subset images, and their FID is evaluated. If the FID exceeds the desired threshold, the activation bit  $Bit_{act}$  is increased by 1, and the image quality is re-evaluated. Note that RAQ employs the FID of diffusion models without activation quantization as the FID threshold to preserve the quality of image sampling after the quantization. Nonetheless, there exists the flexibility to modify the FID threshold according to the user-defined level of FID tolerance, enabling greater reduction in activation bitwidths. The iterative process continues until the desired FID is achieved. Once the FID meets the desired criterion, the optimization proceeds to the next diffusion step. Notably, the optimization starts from the previously adopted activation bit and does not explore lower bit values. This strategy acknowledges the increasing difficulty of lowering activation bits as diffusion steps progress. As a result, the time complexity for optimizing activation bits at each step is reduced from  $O(m^T)$  to  $O(m + T)$ , where  $m$  represents the number of bit candidates for activation.
Table 1: Unconditional image generation results of the baselines and the proposed RAQ. (W: weight bidwidth, A: effective activation bitwidth, TBOPs: Tera Bit Operations)
Baseline
LSUN-Churches (256x256)	LSUN-Bedrooms (256x256)
Model	W/A	FID↓	TBOPs	Model	W/A	FID↓	TBOPs
LDM-8 [8]	32/32	4.09	4285.2	LDM-4 [8]	32/32	2.96	20725.8
Q-diffusion [16]	4/32	4.46	769.2	Q-diffusion [16]	4/32	4.13	3120.7
4/8	4.45	148.6	4/8	4.17	681.2
Proposed RAQ (w/ 4-bit weights)
LSUN-Churches (256x256)	LSUN-Bedrooms (256x256)
A	Dact	FID↓	TBOPs	A	Dact	FID↓	TBOPs
6.00	entire range - 6b	5.31	108.8	6.00	entire range - 6b	5.58	504.8
6.00	[1.00T:0.80T] - 4b[0.80T:0.60T] - 5b[0.60T:0.50T] - 6b[0.50T:0.10T] - 7b[0.10T:0.00T] - 8b	4.64	108.8	6.00	[1.00T:0.90T] - 4b[0.90T:0.60T] - 5b[0.60T:0.40T] - 6b[0.40T:0.10T] - 7b[0.10T:0.00T] - 8b	3.99	504.8
5.60	[1.00T:0.75T] - 4b[0.75T:0.60T] - 5b[0.60T:0.25T] - 6b[0.25T:0.00T] - 7b	5.12	101.3	5.50	[1.00T:0.90T] - 4b[0.90T:0.60T] - 5b[0.60T:0.00T] - 6b	4.94	461.6
5.35	[1.00T:0.75T] - 4b[0.75T:0.60T] - 5b[0.60T:0.00T] - 6b	5.51	96.4	5.45	[1.00T:0.85T] - 4b[0.85T:0.60T] - 5b[0.60T:0.00T] - 6b	5.32	457.3
5.1 Unconditional Image Generation with Proposed Robustness-Aware Quantization
We first evaluate the effectiveness of the proposed method for unconditional image generation. We use Latent Diffusion Model (LDM) [8] with DDIM sampling [6] for  $256 \times 256$  image generations, with a total of 200 diffusion steps. The lower bound of the activation bit  $a$  is set to 4, as we observed that using 3 bits significantly degrades the quality of the generated images even with the proposed RAQ. As we find that consecutive timesteps within a 0.05T range exhibit similar sensitivity to noise injection, we set the granularity of  $Bit_{act}$  update as  $0.05T$ . We sample 5,000 images for the optimization of the quantization bits, but we use a larger sample size of 50,000 images for the final evaluation. We evaluate the performance of our approach using two datasets: 256x256 LSUN Bedrooms and LSUN Churches. Note that the generation of LSUN Bedrooms use a diffusion model LDM-4 and LSUN Churches use a diffusion model LDM-8 [8]. The quality of the generated images is evaluated using the FID metric. To estimate the efficiency improvement of RAQ, we introduce the concept of Bit Operations (BOPs), where BOPs are calculated as the product of OPs (Operations) with weight bitwidth and activation bitwidth.
The evaluation results of the baseline method, which does not employ the step-wise bitwidth optimization, are presented in the top of Table 1. Despite optimizing the activation bitwidth with a  $0.05\mathrm{T}$  granularity, certain consecutive intervals exhibit similar sensitivity to quantization, resulting in the RAQ results being composed of only 5 or fewer intervals. As demonstrated in Q-diffusion [16], utilizing 4-bit weights and 8-bit activations preserves FID at a similar level to that of floating-point models. However, it is observed that FID substantially increases when 6-bit activation values are used in the entire range of the steps. In contrast, when applying our proposed RAQ method, we observe FID scores comparable to those obtained with 8-bit activations even when the effective bit count of the activation is 6 (Table 1). Thus, for LSUN-Churches/Bedrooms, the proposed RAQ
Full Precision
W4A6
W4A8 (Q-Diffusion)
W4A6/8 (Proposed)
Figure 7: Text-guided  $512 \times 512$  image generation results with Stable Diffusion. The denoising model with full-precision, 8-bit (Q-diffusion), and 6-bit activations, and scalable-bit activations with the proposed RAQ method are evaluated for the comparison. Here, W4 denotes adopting 4-bit weights, An denotes  $n$ -bit activations, and An/m indicates adopting  $n$ -bit activations in the early stage of the diffusion stages, while the later stage adopts  $m$ -bit activations
achieves 39.4/41.1 times smaller BOPs compared to the full-precision baseline, while Q-diffusion only achieves 28.8/30.4 times smaller BOPs compared to the full-precision baseline. Furthermore, by slightly compromising FID, it becomes possible to further reduce the effective activation bits. These experimental results clearly demonstrate that leveraging the early-stage robustness of the diffusion model contributes to maintaining the quality of image generation even with low-bit activations.
5.2 Conditional Image Generation with Proposed Robustness-Aware Quantization
In this section, we assess the effectiveness of the proposed method for conditional image generation using Stable Diffusion [8]. We employ the PLMS sampler [28] with 50 diffusion steps to generate  $512 \times 512$  images. Since there is no established metric to quantitatively measure the image generation quality in this case, we rely on qualitative comparison for each image. As presented in Figure 7, even when the proposed RAQ method is not applied, 8-bit activations produce high-quality images that are comparable with full-precision models. However, there is a significant degradation in image quality
when the activation bits are reduced to 6 bits. On the other hand, by applying 6-bit activations only to the  $[1.0T:0.8T]$  diffusion steps and retaining 8-bit activations for the remaining steps, the diffusion models are able to generate high-quality images. This result underscores the importance of leveraging the early-stage robustness of the diffusion process.
Meanwhile, unlike the unconditional image generation cases in Section 5.1, we had difficulties in applying 4-bit activations to the early-stage diffusion process of Stable Diffusion. This challenge arises from the large dynamic range of the activation values in Stable Diffusion. Hence, we plan to explore more sophisticated techniques to address this issue in the future work.
An important limitation of the proposed RAQ is that current results do not enhance running time efficiency on GPUs. This is primarily due to the substantial presence of irregular activation bitwidths, such as 6 bits (Table 1), and accelerating diffusion models with irregular bitwidths on a GPU does not lead to performance improvement due to the lack of corresponding arithmetic units in GPU.
However, we would like to emphasize that our main contribution is to show a direction that we can reduce bit resolution for a part of parameters, and our approach is not fundamentally limited to a certain bit resolution. Hence, if other works that can further reduce the overall bit resolution of the network are developed independent of our scheme, then it can be combined with our proposed scheme, and there is a chance that portion of the 4-bit parameters in our scheme can be increased to see realistic performance benefits on GPU. For example, our current approach involves a basic min/max-based quantization mechanism for activation quantization, leading to an 8-bit quantization for the baseline fixed bitwidth case. Then, the allocation of activation bits using the RAQ method is distributed as (4b, 5b, 6b, 7b, 8b) = (20%, 20%, 10%, 40%, 10%) for LSUN-churches (Table 1). However, in the scenario where an advanced quantization mechanism enables 6-bit activation quantization even for the baseline fixed bitwidth case, there exists the possibility of increasing the proportion of 4-bit activations with the proposed RAQ, such as (4b, 5b, 6b) = (50%, 30%, 20%), which can then be effectively accelerated on GPUs.
Moreover, the utilization of specialized hardware, like bit-scalable accelerators, could offer a promising solution for processing these models [29]. These accelerators are purpose-built to harness the benefits of quantization on a bit-by-bit basis, resulting in a nearly linear improvement in computing efficiency, encompassing both latency and energy consumption, with decreasing bit width. To show potential benefit from such bit-scalable hardware, we analyze the BOPs in Table 1. This metric allows us to estimate the performance gain achievable through the RAQ method. In the case of LSUN-Churches, the full-precision baseline and Q-diffusion necessitates 4285.2 and 148.6 TBOPs for a single image generation respectively, while the proposed RAQ method only requires 108.8 TBOPs. This indicates that with the utilization of specialized accelerators, the implementation of the RAQ approach could potentially lead to more than 39.4 times speedup and energy savings compared to the full-precision baseline, while Q-diffusion can achieve 28.8 times improvement compared to the full-precision baseline.
In this paper, we present a comprehensive analysis of the reverse diffusion process and introduce a strategy to optimize diffusion models using quantization techniques. Leveraging the observed robustness of the early stages in the diffusion process, we successfully optimize the activation bitwidth for each diffusion step, leading to reduced effective activation bit precision without compromising the fidelity and diversity of the generated images. Experimental results show that our proposed method (RAQ) effectively reduces the effective bitwidth of activations to 6 bits while maintaining image quality comparable to conventional approaches that employ 8-bit activations. Additionally, we also evaluate the effectiveness of the proposed method in the context of conditional image generation using Stable Diffusion, a state-of-the-art diffusion model.
This work was supported in part by Institute of Information & communications Technology Planning & Evaluation (IITP) grant funded by the Korea government (MSIT) (No. 2021-0-00105: Development of model compression framework for scalable on-device AI computing on Edge applications, IITP-2023-RS-2023-00256081: artificial intelligence semiconductor support program to nurture the best talents, No. 2021-0-01343: Artificial Intelligence Graduate School Program (Seoul National University), No.2021-0-02068: Artificial Intelligence Innovation Hub), Basic Science Research Program through the National Research Foundation of Korea (NRF) funded by the Ministry of Education (2022R1A6A3A01087416), BK21 FOUR program and Inter-university Semiconductor Research Center at Seoul National University. (Corresponding Author: Jae-Joon Kim)
[1] Jascha Sohl-Dickstein, Eric A. Weiss, Niru Maheswaranathan, Surya Ganguli, "Deep Unsupervised Learning using Nonequilibrium Thermodynamics", International Conference on Machine Learning (ICML), 2015.
[2] Yang Song and Stefano Ermon, "Generative modeling by estimating gradients of the data distribution", Advances in Neural Information Processing Systems, (NeurIPS), 2019.
[3] Jonathan Ho, Ajay Jain, Pieter Abbeel, “Denoising Diffusion Probabilistic Models”, Advances in Neural Information Processing Systems, (NeurIPS), 2020.
[4] Alex Nichol, Prafulla Dhariwal, "Improved Denoising Diffusion Probabilistic Models", International Conference on Machine Learning (ICML), 2021.
[5] Yang Song, Jascha Sohl-Dickstein, Diederik P. Kingma, Abhishek Kumar, Stefano Ermon, Ben Poole, "Score-Based Generative Modeling through Stochastic Differential Equations", International Conference on Learning Representations (ICLR), 2021.
[6] Jiaming Song, Chenlin Meng, and Stefano Ermon, “Denoising diffusion implicit models”, International Conference on Learning Representations (ICLR), 2021.
[7] Prafulla Dhariwal, Alex Nichol, "Diffusion Models Beat GANs on Image Synthesis", Advances in Neural Information Processing Systems, (NeurIPS), 2021.
[8] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, Björn Ommer, "High-Resolution Image Synthesis With Latent Diffusion Models", IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
[9] Olaf Ronneberger, Philipp Fischer, Thomas Brox, Tijmen Blankevoort, "U-Net: Convolutional Networks for Biomedical Image Segmentation", Medical Image Computing and Computer-Assisted Intervention (MICCAI), 2015.
[10] Markus Nagel, Rana Ali Amjad, Mart van Baalen, Christos Louizos, Tijmen Blankevoort, "Up or Down? Adaptive Rounding for Post-Training Quantization", International Conference on Machine Learning (ICML), 2020.
[11] Steven K. Esser, Jeffrey L. McKinstry, Deepika Bablani, Rathinakumar Appuswamy, Dharmendra S. Modha, "Learned Step Size Quantization", International Conference on Learning Representations (ICLR), 2020.
[12] Xiuying Wei, Ruihao Gong, Yuhang Li, Xianglong Liu, Fengwei Yu, "QDrop: Randomly Dropping Quantization for Extremely Low-bit Post-Training Quantization", International Conference on Learning Representations (ICLR), 2022.
[13] Sein Park, Junhyuk So, Juncheol Shin, Eunhyeok Park, "Profit: A novel training method for sub-4-bit mobilenet models", European Conference on Computer Vision (ECCV), 2020.
[14] Kuan Wang, Zhijian Liu, Yujun Lin, Ji Lin, Song Han, "HAQ: Hardware-Aware Automated Quantization with Mixed Precision", IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
[15] Yuzhang Shang and Zhihang Yuan and Bin Xie and Bingzhe Wu and Yan Yan, "Post-training Quantization on Diffusion Models", IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
[16] Xiuyu Li, Long Lian, Yijiang Liu, Huanrui Yang, Zhen Dong, Daniel Kang, Shanghang Zhang, Kurt Keutzer, “Q-Diffusion: Quantizing Diffusion Models”, arXiv preprint arXiv:2302.04304, 2023.
[17] Yury Nahshan, Brian Chmiel, Chaim Baskin, Evgenii Zheltonozhskii, Ron Banner, Alex M. Bronstein, Avi Mendelson, "Loss Aware Post-training Quantization", arXiv preprint arXiv:1911.07190, 2019.
[18] Yuhang Li, Ruihao Gong, Xu Tan, Yang Yang, Peng Hu, Qi Zhang, Fengwei Yu, Wei Wang, Shi Gu, "BRECQ: Pushing the Limit of Post-Training Quantization by Block Reconstruction", International Conference on Learning Representations (ICLR), 2021.
[19] Yash Bhalgat, Jinwon Lee, Markus Nagel, Tijmen Blankevoort, Nojun Kwak, "LSQ+: Improving low-bit quantization through learnable offsets and better initialization", IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), 2020.
[20] Jungwook Choi, Zhuo Wang, Swagath Venkataramani, Pierce I-Jen Chuang, Vijayalakshmi Srinivasan, Kailash Gopalakrishnan, "PACT: Parameterized Clipping Activation for Quantized Neural Networks", arXiv preprint arXiv:1805.06085, 2018.
[21] Sangil Jung, Changyong Son, Seohyung Lee, Jinwoo Son, Youngjun Kwak, Jae-Joon Han, Sung Ju Hwang, Changkyu Choi, "Learning to Quantize Deep Networks by Optimizing Quantization Intervals with Task Loss", IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
[22] Ruihao Gong, Xianglong Liu, Shenghu Jiang, Tianxiang Li, Peng Hu, Jiazhen Lin, Fengwei Yu, Junjie Yan, "Differentiable Soft Quantization: Bridging Full-Precision and Low-Bit Neural Networks", IEEE International Conference on Computer Vision (ICCV), 2019.
[23] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, Sepp Hochreiter, "GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium", Advances in Neural Information Processing Systems, (NeurIPS), 2017.
[24] Krizhevsky, Alex, and Geoffrey Hinton, "Learning Multiple Layers of Features from Tiny Images", 2009.
[25] Fisher Yu, Ari Seff, Yinda Zhang, Shuran Song, Thomas Funkhouser, Jianxiong Xiao, "LSUN: Construction of a Large-scale Image Dataset using Deep Learning with Humans in the Loop", arXiv preprint arXiv:1506.03365, 2015.
[26] Tero Karras, Timo Aila, Samuli Laine, Jaakko Lehtinen, "Progressive Growing of GANs for Improved Quality, Stability, and Variation", International Conference on Learning Representations (ICLR), 2018.
[27] Tero Karras, Samuli Laine, Timo Aila, "A Style-Based Generator Architecture for Generative Adversarial Networks", IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
[28] Luping Liu, Yi Ren, Zhijie Lin, Zhou Zhao, "Pseudo Numerical Methods for Diffusion Models on Manifolds", International Conference on Learning Representations (ICLR), 2022.
[29] Yonggan Fu, Yang Zhao, Qixuan Yu, Chaojian Li, Yingyan Lin, "2-in-1 accelerator: Enabling random precision switch for winning both adversarial robustness and efficiency", 54th Annual IEEE/ACM International Symposium on Microarchitecture (MICRO), 2021.
A.1 Details of Diffusion Models
In order to conduct a comprehensive analysis of the early-stage robustness in diffusion models, we employ five different diffusion models to generate various datasets, including CIFAR10 [24], FFHQ [27], CelebA-HQ [26], LSUN-bedrooms, and LSUN-churches [25]. To measure the Fréchet Inception Distance (FID), we utilize the torch-fidelity library  $^{1}$ , following the methodology established in previous works [6, 8]. Subsequently, we apply the proposed RAQ method to perform high-resolution image generation tasks, encompassing both unconditional image generation for LSUN-bedrooms/LSUN-churches and conditional image generation using Stable Diffusion [8]. The details of the diffusion models utilized in these experiments are thoroughly presented in Table 2. Note that the calculation of  $\sigma_{t}$  in Eq. 2 is performed using  $\eta$  in Table 2 as specified in the following equation [6]:
$$
\sigma_ {t} = \eta \cdot \sqrt {\left(1 - \alpha_ {t - 1}\right) / \left(1 - \alpha_ {t}\right)} \sqrt {1 - \alpha_ {t} / \alpha_ {t - 1}} \tag {5}
$$
Table 2: Implementation specifications of the diffusion models
CIFAR-10	FFHQ/CelebA-HQ	LSUN-Bedrooms	LSUN-Churches	Conditional Generation
Image Size	32×32	256×256	256×256	256×256	512×512
Architecture	DDIM2	LDM-43	LDM-43	LDM-83	Stable Diffusion v1.44
Sampler	DDIM [6]	DDIM	DDIM	DDIM	PLMS [28]
Step Count	100	200	200	200	50
η	0	0/1	1	0	0
A.2 Details of Entropy Analysis
In Section 3.1, we calculate the entropy of the latent variables  $x_{t}$  for each diffusion step. To facilitate this calculation, we transform the values of  $x_{t}$  into histogram bins. Specifically, we map  $x_{t}$  to a histogram bin using the following equation:
$$
h \left(x _ {t}\right) = \operatorname {c l a m p} \left(\left\lfloor \frac {x _ {t}}{2 5 6} \right\rfloor , - 3, + 3\right) \tag {6}
$$
This equation ensures that the values of the histogram bins are constrained within the range of  $-3$  to  $+3$ , allowing us to effectively create a histogram with 256 bins. Once we have the histogram representation of  $x_{t}$ , we can calculate the entropy using the equation:
$$
H (X) = \sum_ {x} - p (X) \log p (X) \tag {7}
$$
A.3 Intermediate Image Prediction during Reverse Diffusion Process
In Fig. 1(a) and Fig. 2, we showcase the intermediate image prediction results during the reverse diffusion process, aiming to illustrate the characteristics of diffusion models. To visualize the prediction results, we utilize the following  $x_0$  prediction of each diffusion step as stated in [6]:
$$
x _ {0} = \frac {x _ {t} - \sqrt {1 - \alpha_ {t}} \epsilon_ {\theta} (x _ {t} , t)}{\sqrt {\alpha_ {t}}} \tag {8}
$$
<sup>1</sup>https://github.com/toshas/torch-fidelity
$^{2}$ https://github.com/ermongroup/ddim
<sup>4</sup>https://github.com/CompVis/stable-diffusion
5Claude Elwood Shannon, "A Mathematical Theory of Communication", Bell system technical journal, 1948
A.4 Details of Activation Quantization
During the activation quantization process, we observed that the skip connections of ResBlocks, the first convolutional layer responsible for transforming the latent variable into the input of the denoising network, and the last convolutional layer responsible for transforming the output of the denoising network (Fig. 1(b)) had a significant impact on the quality of the final image. However, these components constitute a negligible fraction of the overall computation. Therefore, in order to balance computational efficiency and image quality, the proposed RAQ adjust the activation bits of the diffusion models to the desired bit precision while keeping the activation bits of these three components fixed at 8 bits.
B.1 Activation Quantization across Diffusion Steps
Fig. 9 complements the image generation results presented in Fig. 6. Fig. 9 showcases image generation with 4-bit activation quantization at different diffusion steps, alongside the image generation with floating-point activations. The results of the activation quantization demonstrate a consistent trend with the noise injection test (4). When 4-bit activation quantization is applied to the early stages, the resulting images closely resemble those generated using floating-point activations, showcasing high quality with minor shape variations. However, applying 4-bit activation quantization to the entire diffusion process leads to a significant compromise in the generated image quality. This is primarily due to the degradation in image quality caused by the quantization applied to the later diffusion steps.
Figure 8: Examples of  $256 \times 256$  LSUN-Churches generation with FP32 activation or activation quantization across different diffusion steps. Example images with activation quantization are generated by applying 4-bit activation quantization in the target diffusion steps.
B.2 Explanation on FID Improvement with Proposed RAQ on LSUN-Bedrooms
In Table 1 of Section 5.1, we observe that the LSUN-Bedrooms images generated with the proposed RAQ exhibit slightly better FID scores compared to Q-diffusion with W4A32 and W4A8. To investigate the reason behind this FID improvement, we conduct a detailed comparison of the images generated using full-precision activations and 4-bit activations in the early stage of the diffusion
(a) FP32 Activation
(b) 4-bit Activation for  $[1.0T:0.8T]$
Figure 9: Examples of  $256 \times 256$  LSUN-Bedrooms generation with different activation precision.
process. We find that the models with full-precision activations sometimes generate images with complex structures that are not easily recognizable as bedrooms. However, when 4-bit activation quantization is applied to the early stage, it simplifies the complex structures and results in images that more closely resemble bedrooms. This observation suggests that the step-wise activation quantization strategy employed in the proposed RAQ method helps refine the generated images, leading to improved quality and better alignment with the target LSUN-Bedrooms dataset.
B.3 Activation Quantization of Stable Diffusion
For the conditional image generation with Stable Diffusion, we utilize the prompt examples that are publicly available online  $^{67}$ . The prompts used in Section 5.2 are as follows: