SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Table 1: Evaluating the components of Q-DM based on 50-step DDIM sampler with  $32 \times 32$  generating resolution on CIFAR-10 [13]. “#Bits” denotes bit-width of weights and activations
Method	#Bits	FID↓	IS↑	#Bits	FID ↓	IS↑	#Bits	FID ↓	IS↑
Full-precision	32-32	4.67	9.27	-	-	-	-	-	-
PTQ4DM	8-8	18.02	8.87	-	-	-	-	-	-
Baseline (LSQ [5])	4-4	10.22	8.91	3-3	13.24	8.88	2-2	18.74	8.65
+TaQ	4-4	9.25	8.95	3-3	11.19	8.91	2-2	16.83	8.71
+NeM	4-4	8.98	8.92	3-3	11.02	8.90	2-2	16.97	8.79
+TaQ+NeM(Q-DM)	4-4	6.89	8.96	3-3	9.07	8.98	2-2	15.26	8.86
4.2 Noise-estimating Mimicking
To mitigate the negative impact of quantization error accumulation on the training of a quantized DM  $\theta^{\mathrm{Q}}$ , a full-precision DM, denoted as  $\theta^{\mathrm{FP}}$ , is incorporated into the training process to facilitate the learning objective. Following [10], with  $p_{\theta^{\mathrm{Q}}}(\mathbf{x}_{t - 1}|\mathbf{x}_t) = \mathcal{N}(\mathbf{x}_{t - 1};\tilde{\boldsymbol{\mu}}_{\theta^{\mathrm{Q}},t}(\mathbf{x}_t),\tilde{\beta}_t\mathbf{I})$  and  $p_{\theta^{\mathrm{FP}}}(\mathbf{x}_{t - 1}|\mathbf{x}_t) = \mathcal{N}(\mathbf{x}_{t - 1};\tilde{\boldsymbol{\mu}}_{\theta^{\mathrm{FP}},t}(\mathbf{x}_t),\tilde{\beta}_t\mathbf{I})$ , we can write:
$$
L _ {t - 1} = \mathbb {E} _ {q} \left[ \frac {1}{2 \tilde {\beta} _ {t}} \| \boldsymbol {\mu} _ {\theta^ {\mathrm {F P}}} (\mathbf {x} _ {t}, t) - \boldsymbol {\mu} _ {\theta^ {\mathrm {Q}}} (\mathbf {x} _ {t}, t) \| ^ {2} \right] + C, \tag {13}
$$
where  $C$  is a constant that does not depend on  $\theta^{\mathrm{Q}}$  or  $\theta^{\mathrm{FP}}$ . As in Eq. (13), we aim to compel the quantized model to replicate the noise estimation capability of the full-precision model. Further, by re-parameterizing Eq. (2) as  $\mathbf{x}_t(\mathbf{x}_0,\epsilon) = \sqrt{\bar{\alpha}_t}\mathbf{x}_0 + \sqrt{1 - \bar{\alpha}_t}\epsilon$  for  $\epsilon \sim \mathcal{N}(\mathbf{0},\mathbf{I})$  and following the formulation in [10], which utilizes the formula for the posterior of the forward process, we can derive that:
$$
L _ {t - 1} - C = \mathbb {E} _ {\mathbf {x} _ {0}, \epsilon} \left[ \frac {1}{2 \tilde {\beta} _ {t}} \| \boldsymbol {\mu} _ {\theta^ {\mathrm {F P}}} (\mathbf {x} _ {t} (\mathbf {x} _ {0}, \epsilon), t) - \boldsymbol {\mu} _ {\theta^ {\mathrm {Q}}} (\mathbf {x} _ {t} (\mathbf {x} _ {0}, \epsilon), t) \| ^ {2} \right], \tag {14}
$$
where  $\pmb{\mu}_{\theta^{\mathrm{FP}}}\left(\mathbf{x}_t(\mathbf{x}_0,\pmb {\epsilon}),t\right)$  and  $\pmb{\mu}_{\theta^{\mathrm{Q}}}\left(\mathbf{x}_t(\mathbf{x}_0,\pmb {\epsilon}),t\right)$  are parameterized as:
$$
\boldsymbol {\mu} _ {\theta^ {\mathrm {F P}}} (\mathbf {x} _ {t}, t) = \tilde {\boldsymbol {\mu}} _ {t} \left(\mathbf {x} _ {t}, \frac {1}{\sqrt {\bar {\alpha} _ {t}}} \left[ \mathbf {x} _ {t} - \sqrt {1 - \bar {\alpha} _ {t}} \boldsymbol {\epsilon} _ {\theta^ {\mathrm {F P}}} (\mathbf {x} _ {t}) \right]\right) = \frac {1}{\alpha_ {t}} \left(\mathbf {x} _ {t} - \frac {\beta_ {t}}{\sqrt {1 - \bar {\alpha} _ {t}}} \boldsymbol {\epsilon} _ {\theta^ {\mathrm {F P}}} (\mathbf {x} _ {t}, t)\right), \tag {15}
$$
$$
\pmb {\mu} _ {\theta^ {\mathbb {Q}}} (\mathbf {x} _ {t}, t) = \tilde {\pmb {\mu}} _ {t} (\mathbf {x} _ {t}, \frac {1}{\sqrt {\bar {\alpha} _ {t}}} [ \mathbf {x} _ {t} - \sqrt {1 - \bar {\alpha} _ {t}} \pmb {\epsilon} _ {\theta^ {\mathbb {Q}}} (\mathbf {x} _ {t}) ]) = \frac {1}{\alpha_ {t}} (\mathbf {x} _ {t} - \frac {\beta_ {t}}{\sqrt {1 - \bar {\alpha} _ {t}}} \pmb {\epsilon} _ {\theta^ {\mathbb {Q}}} (\mathbf {x} _ {t}, t)),
$$
where  $\theta^{\mathrm{Q}}$  and  $\theta^{\mathrm{fp}}$  are the noise estimated by the quantized DM and full-precision counterpart. In Eq. (15),  $\epsilon_{\theta}$  is a function approximator intended to predict  $\epsilon$  from  $\mathbf{x}_t$ . Therefore, Eq. (14) is simplified to:
$$
L _ {t - 1} = \mathbb {E} _ {\mathbf {x} _ {0}, \boldsymbol {\epsilon}} \left[ \frac {\beta_ {t} ^ {2}}{2 \tilde {\beta} _ {t} \alpha_ {t} (1 - \bar {\alpha} _ {t})} \| \boldsymbol {\epsilon} _ {\theta^ {\mathrm {F P}}} \left(\sqrt {\bar {\alpha} _ {t}} \mathbf {x} _ {0} + \sqrt {1 - \bar {\alpha} _ {t}} \boldsymbol {\epsilon}, t\right) - \boldsymbol {\epsilon} _ {\theta^ {\mathrm {Q}}} \left(\sqrt {\bar {\alpha} _ {t}} \mathbf {x} _ {0} + \sqrt {1 - \bar {\alpha} _ {t}} \boldsymbol {\epsilon}, t\right) \| ^ {2} \right]. \tag {16}
$$
With the aforementioned derivation and parameterization, we have the final objective of our noise-estimating imitation, which is formulated as:
$$
\begin{array}{l} \arg \min  _ {\theta^ {Q}} L _ {\mathrm {N e M}} \left(\theta^ {Q}, \theta^ {\mathrm {F P}}\right) \tag {17} \\ := \mathbb {E} _ {t, \mathbf {x} _ {0}, \epsilon} \left[ \| \boldsymbol {\epsilon} _ {\theta^ {\mathrm {F P}}} \left(\sqrt {\bar {\alpha} _ {t}} \mathbf {x} _ {0} + \sqrt {1 - \bar {\alpha} _ {t}} \boldsymbol {\epsilon}, t\right) - \boldsymbol {\epsilon} _ {\theta^ {\mathrm {Q}}} \left(\sqrt {\bar {\alpha} _ {t}} \mathbf {x} _ {0} + \sqrt {1 - \bar {\alpha} _ {t}} \boldsymbol {\epsilon}, t\right) \| ^ {2} \right], \\ \end{array}
$$
In this section, we evaluate the proposed Q-DM framework on several popular diffusion models (i.e. DDPM [10] and DDIM [32]) for unconditional image generation. To the best of our knowledge, there is no published work done on low-bit quantized diffusion models at this point, so we report LSQ [5] as a baseline. Experiments show our approach can achieve competitive generation quality to the full-precision scenario on all experimental settings under low-bit quantization.
Table 2: Experiment on 2/3/4-bit quantized diffusion models generating CIFAR-10 [13] image or ImageNet [14] image. "Bits" denotes the bit-width of weights/activations. "Reso." represents the generating resolution.
Model	Dataset & Reso.	Step	Method	#Bits	Size(MB)	OPs(G)	FID ↓	IS ↑
DDIM	CIFAR-10 32×32	50	Full-precision	32/32	4.47	390.4	4.67	9.27
PTQ4DM [6]	8/8	1.12	99.5	18.02	8.87
Baseline	4/4	0.56	49.9	10.22	8.91
Q-DM	4/4	0.56	49.9	6.89	8.96
Baseline	3/3	0.28	25.1	13.24	8.88
Q-DM	3/3	0.28	25.1	9.07	8.98
Baseline	2/2	0.14	12.6	18.74	8.65
Q-DM	2/2	0.14	12.6	15.26	8.86
DDIM	CIFAR-10 32×32	100	Full-precision	32/32	4.47	780.7	4.16	9.32
PTQ4DM [6]	8/8	1.12	199.0	14.18	9.31
Baseline	4/4	0.56	99.8	9.02	8.95
Q-DM	4/4	0.56	99.8	5.12	9.21
Baseline	3/3	0.28	50.1	12.24	8.90
Q-DM	3/3	0.28	50.1	8.12	8.94
Baseline	2/2	0.14	25.2	16.99	8.74
Q-DM	2/2	0.14	25.2	14.31	8.77
DDPM	CIFAR-10 32×32	1000	Full-precision	32/32	4.47	7807.2	3.17	9.46
PTQ4DM [6]	8/8	1.12	1990.0	7.10	9.55
Baseline	4/4	0.56	997.7	9.11	8.96
Q-DM	4/4	0.56	997.7	5.17	9.15
Baseline	3/3	0.28	501.0	12.28	8.91
Q-DM	3/3	0.28	501.0	8.14	8.93
Baseline	2/2	0.14	252.0	16.93	8.72
Q-DM	2/2	0.14	252.0	14.35	8.76
DDIM	ImageNet 64×64	50	Full-precision	32/32	4.47	390.4	20.57	15.72
PTQ4DM [6]	8/8	1.12	99.5	25.87	14.99
Baseline	4/4	0.56	49.9	24.78	15.37
Q-DM	4/4	0.56	49.9	20.02	15.68
Baseline	3/3	0.28	25.1	26.35	15.24
Q-DM	3/3	0.28	25.1	22.19	15.32
Baseline	2/2	0.14	12.6	32.43	14.66
Q-DM	2/2	0.14	12.6	28.42	15.03
DDIM	ImageNet 64×64	100	Full-precision	32/32	4.47	780.7	19.70	15.98
PTQ4DM [6]	8/8	1.12	199.0	24.92	15.52
Baseline	4/4	0.56	99.8	24.46	15.51
Q-DM	4/4	0.56	99.8	19.56	15.92
Baseline	3/3	0.28	50.1	26.23	15.42
Q-DM	3/3	0.28	50.1	21.97	15.92
Baseline	2/2	0.14	25.2	31.19	14.89
Q-DM	2/2	0.14	25.2	27.94	14.99
DDPM	ImageNet 64×64	1000	Full-precision	32/32	4.47	7807.2	18.98	16.63
PTQ4DM [6]	8/8	1.12	1990.0	22.32	15.31
Baseline	4/4	0.56	997.7	22.91	15.29
Q-DM	4/4	0.56	997.7	18.52	16.72
Baseline	3/3	0.28	501.0	24.75	15.11
Q-DM	3/3	0.28	501.0	20.21	16.17
Baseline	2/2	0.14	252.0	29.33	14.87
Q-DM	2/2	0.14	252.0	25.62	15.48
5.1 Datasets and Implementation Details
We evaluate our method on two datasets including  $32 \times 32$  generating size in CIFAR-10 [13] and  $64 \times 64$  generating size in ImageNet [14]. For the CIFAR-10 [13] and ImageNet [14] datasets, we use the DDIM [32] sampler with 50/100 sampling timesteps and DDPM [10] with 1000 sampling timesteps. All the training settings are the same as DDPM [10]. For DDIM sampler, we set  $\eta$  in DDIM [32] as 0.5 for the best performance. We evaluate the performance of our method using FID [9] and Inception Score (IS) [27] on both CIFAR-10 [13] and ImageNet [14] datasets. We set the training timestep  $T = 1000$  for all experiments, following [10]. We set the forward process variances to constants increasing linearly from  $\beta_{1} = 1e - 4$  to  $\beta_{T} = 0.02$ . To represent the reverse process, we use a U-Net backbone, following [10, 32]. Parameters are shared across time, which is specified to the network using the Transformer sinusoidal position embedding [36]. We use self-attention at the  $16 \times 16$  feature map resolution [36, 37].
We give quantitative results of the proposed TaQ and NeM in Tab. 1 As can be seen, the low-bit quantized DM baseline [5] suffers a severe performance drop on image generation task compared with full-precision DMs (5.55, 8.57, and 14.07 performance gap in terms of FID score with 4/3/2-bit, respectively). TaQ and NeM improve the performance of generation when used alone. For example, the 4-bit quantized DM baseline with TaQ and NeM introduced separately achieves 0.97 and 1.24 FID score decrease, respectively.
Moreover, the two techniques further boost the performance considerably when combined together. For instance, when combining the TaQ and NeM together, the performance of 4/3/2-bit quantized DMs improvement achieves 3.33, 4.07, and 3.48 respectively. To conclude, the two techniques can promote each other to improve Q-DM and close the performance gap between low-bit quantized DMs and full-precision counterpart.
The experimental results are shown in Tab. 2. We compare our method with 4/3/2-bit baseline [5] based on the same frameworks for the task of unconditional image generation with the CIFAR-10 [13] and ImageNet [14] dataset. We also report the classification performance of the 8-bit PTQ method, i.e., PTQ4DM [30]. We firstly evaluate the proposed method on CIAFR-10 [13] with DDIM [32] and DDPM [10]. We use the model size and OPs (defined in [18]) to evaluate the efficiency of quantized and full-precision models.
For 50-step DDIM sampler, compared with 8-bit PTQ4DM [30], our 4-bit Q-DM achieves a much larger compression ratio than 8-bit PTQ4DM, but with significant performance improvement (6.89 FID  $\downarrow$  vs. 18.02 FID  $\downarrow$ ). And it is worth noting that the proposed 2-bit model significantly compresses the DDIM by  $30.9\times$  on OPs. The proposed method boosts the performance of  $4/3/2$ -bit Baseline by 3.33, 4.17, and 3.48 in terms of FID score with the same architecture and bit-width, which is significant on the CIFAR-10 [13] dataset with  $32\times 32$  generating resolution. For 1000-step DDPM, the performance of the proposed method outperforms the  $4/3/2$ -bit Baseline by 3.94, 4.14, and 2.58, a large margin. Also note that the proposed  $4/3/2$ -bit model significantly accelerates the generation by  $7.8\times$ ,  $15.6\times$ , and  $30.9\times$  on OPs. Compared with 8-bit PTQ4DM, ours achieve significantly higher compression and acceleration rate, while the performance improvement is considerable.
Also, our method generates convincing results on ImageNet [14] dataset. As shown in Tab. 2, the performance of the proposed method with 50-step DDIM significantly outperforms the  $4/3/2$ -bit Baseline method by 4.76, 4.16, and 4.01. Compared with 8-bit PTQ method, our method achieves significantly higher compression rate and acceleration rate, but with better performance. For 1000-step DDPM on ImageNet [14] dataset, the performance of the proposed method outperforms the  $4/3/2$ -bit Baseline by 4.39, 4.54, and 3.71. Also note that our 4-bit Q-DM surpasses the full-precision 50/100-step DDIM and 1000-step DDPM and significantly compresses the noise estimation networks by  $7.9 \times$  , which demonstrates the effectiveness and efficiency of our Q-DM.
In this paper, we present Q-DM, an efficient low-bit quantized diffusion model that offers a high compression ratio and competitive performance in image generation task. Initially, we analyze the challenges of the low-bit quantized DM. Our empirical analysis shows that distribution oscillation in activation is the one of the cause of the significant drop in DM quantization. Another challenge lies in the accumulated quantization error resulted from the multi-step denoising process during inference. To address these issues, we first develop a timestep-aware quantization (TaQ) method and a noise-estimating mimicking (NeM) scheme for low-bit quantized DMs, to effectively address these two challenges. Our work provides a comprehensive analysis and effective solutions for the crucial issues in low-bit quantized diffusion model, paving the way for the extreme compression and acceleration of diffusion model.
This work was supported in part by the National Natural Science Foundation of China under Grant 62076016, under Grant 61827901, "One Thousand Plan" projects in Jiangxi Province Jxsg2023102268, Foundation of China Energy Project GJNY-19-90, the National Key R&D Program of China (NO.2022ZD0160100).
[1] Fan Bao, Chongxuan Li, Jun Zhu, and Bo Zhang. Analytic-dpm: an analytic estimate of the optimal reverse variance in diffusion probabilistic models. In Proc. of ICLR, pages 1-39, 2022.
[2] Yaohui Cai, Zhewei Yao, Zhen Dong, Amir Gholami, Michael W Mahoney, and Kurt Keutzer. Zeroq: A novel zero shot quantization framework. In Proc. of CVPR, pages 13169-13178, 2020.
[3] Nanxin Chen, Yu Zhang, Heiga Zen, Ron J Weiss, Mohammad Norouzi, and William Chan. Wavegrad: Estimating gradients for waveform generation. arXiv preprint arXiv:2009.00713, 2020.
[4] Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer. Llm. int8(): 8-bit matrix multiplication for transformers at scale. arXiv preprint arXiv:2208.07339, 2022.
[5] Steven K Esser, Jeffrey L McKinstry, Deepika Bablani, Rathinakumar Appuswamy, and Dharmendra S Modha. Learned step size quantization. In Proc. of ICLR, pages 1-12, 2019.
[6] Jun Fang, Ali Shafiee, Hamzah Abdel-Aziz, David Thorsley, Georgios Georgiadis, and Joseph H Hassoun. Post-training piecewise linear quantization for deep neural networks. In Proc. of ECCV, pages 69-86, 2020.
[7] Sicheng Gao, Xuhui Liu, Bohan Zeng, Sheng Xu, Yanjing Li, Xiaoyan Luo, Jianzhuang Liu, Xiantong Zhen, and Baochang Zhang. Implicit diffusion models for continuous super-resolution. arXiv preprint arXiv:2303.16491, 2023.
[8] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks. Communications of the ACM, pages 139–144, 2020.
[9] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. In Proc. of NeurIPS, pages 1-12, 2017.
[10] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In Proc. of NeurIPS, pages 6840-6851, 2020.
[11] Alexia Jolicoeur-Martineau, Ke Li, Rémi Piche-Taillefer, Tal Kachman, and Ioannis Mitlagkas. Gotta go fast when generating data with score-based models. arXiv preprint arXiv:2105.14080, 2021.
[32] Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In Proc. of ICLR, pages 1-20, 2020.
[33] Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. In Proc. of NeurIPS, pages 1-13, 2019.
[34] Yang Song and Stefano Ermon. Improved techniques for training score-based generative models. In Proc. of NeurIPS, pages 12438-12448, 2020.
[35] Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In Proc. of ICLR, pages 1-36, 2021.
[36] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Proc. of NeurIPS, pages 1-11, 2017.
[37] Xiaolong Wang, Ross Girshick, Abhinav Gupta, and Kaiming He. Non-local neural networks. In Proc. of CVPR, pages 7794-7803, 2018.
[38] Daniel Watson, Jonathan Ho, Mohammad Norouzi, and William Chan. Learning to efficiently sample from diffusion probabilistic models. arXiv preprint arXiv:2106.03802, 2021.
[39] Zheng Xie, Zhiquan Wen, Jing Liu, Zhiqiang Liu, Xixian Wu, and Mingkui Tan. Deep transferring quantization. In Proc. of ECCV, pages 625-642, 2020.
[40] Ling Yang, Zhilong Zhang, Yang Song, Shenda Hong, Runsheng Xu, Yue Zhao, Yingxia Shao, Wentao Zhang, Bin Cui, and Ming-Hsuan Yang. Diffusion models: A comprehensive survey of methods and applications. arXiv preprint arXiv:2209.00796, 2022.
[41] Shuchang Zhou, Yuxin Wu, Zekun Ni, Xinyu Zhou, He Wen, and Yuheng Zou. Dorefa-net: Training low bitwidth convolutional neural networks with low bitwidth gradients. arXiv preprint arXiv:1606.06160, 2016.
[42] Chenzhuo Zhu, Song Han, Huizi Mao, and William J Dally. Trained ternary quantization. In Proc. of ICLR, pages 1-10, 2017.
[43] Bohan Zhuang, Chunhua Shen, Mingkui Tan, Lingqiao Liu, and Ian Reid. Towards effective low-bitwidth convolutional neural networks. In Proc. of CVPR, pages 7920-7928, 2018.