SlowGuess/NeurIPS_2023_OCR huggingface dataset:
4) Correct the correlated part of the quantization noise by dividing the output of the noise prediction network by  $1 + k$ ;
5) Calibrate the variance schedule by Eq. (12) and subtract the channel-wise biases from the output of the quantized noise prediction network.
Although the proposed method can correct the mean deviation and the numerical value of the variance for each step, generating satisfactory samples with low-bit diffusion models remains challenging due to the low signal-to-noise ratio (SNR) of the quantized noise prediction network. In the next section, we will analyze this issue in detail.
Figure 3: The distribution of uncorrelated quantization noise collected from W4A8 LDM-4 on LSUN-Bedrooms  $256 \times 256$  dataset, where the x-axis represents the range of values and the y-axis is the frequency of values.
Figure 4: Comparison of the signal-to-noise ratio (SNR) in each step of LDM-4 on LSUN-Bedrooms across various bitwidths.
4.3 Step-aware Mixed Precision
Given the output of the full-precision noise prediction network  $\epsilon_{\theta}(\mathbf{x}_t,t)$  and corresponding quantization noise  $\Delta_{\epsilon_{\theta}(\mathbf{x}_t,t)}$ , we define the  $\mathrm{SNR}^{\mathrm{Q}}$  of quantized noise prediction network by:
$$
\mathrm {S N R} ^ {\mathrm {Q}} (\mathrm {t}) = \frac {\left\| \boldsymbol {\epsilon} _ {\theta} \left(\mathbf {x} _ {\mathrm {t}} , \mathrm {t}\right) \right\| _ {2}}{\left\| \Delta_ {\epsilon_ {\theta} \left(\mathbf {x} _ {\mathrm {t}} , \mathrm {t}\right)} \right\| _ {2}}. \tag {13}
$$
Figure 4 depicts the  $\mathrm{SNR}^{\mathrm{Q}}$  with various bitwidths and correction methods. The figure reveals several insights: 1)  $\mathrm{SNR}^{\mathrm{Q}}$  drops drastically as step  $t$  decreases; 2) models with higher bitwidth exhibit larger  $\mathrm{SNR}^{\mathrm{Q}}$ ; 3) the proposed correction methods yield clear  $\mathrm{SNR}^{\mathrm{Q}}$  improvements, especially for large steps. The first observation highlights the challenge of generating high-quality samples using low-bit diffusion models. In particular, as  $t$  approaches zero, the  $\mathrm{SNR}^{\mathrm{Q}}$  of W4A4 diffusion models diminishes and approaches unity, implying that the magnitude of quantization noise is even comparable to the original result of the noise prediction network. To enable low-bit diffusion models while maintaining good generation performance, we propose a novel approach called Step-aware Mixed Precision, which involves setting different bitwidths for synonymous steps to keep  $\mathrm{SNR}^{\mathrm{Q}}$  within a reasonable range across all steps.
Specifically, the bitwidth of weights is fixed and shared across different denoising steps, which eliminates the need to store and reload multiple model state files during the sampling process. As a result, we only adjust the bitwidth of activations. Formally, we predefine a set of bitwidths  $B = \{b_{1}, b_{2}, \ldots, b_{n}\}$  for activations and evaluate the  $\mathrm{SNR}^{\mathrm{Q}}$  under each bitwidth. To establish a benchmark for  $\mathrm{SNR}^{\mathrm{Q}}$ , we follow prior studies [36, 26] and introduce  $\mathrm{SNR}^{\mathrm{F}}$  based on the forward process, which denotes the degree of data noise at each step:
$$
\mathrm {S N R} ^ {\mathrm {F}} (t) = \alpha_ {t} ^ {2} / \sigma_ {t} ^ {2}. \tag {14}
$$
Figure 4 illustrates  $\mathrm{SNR}^{\mathrm{F}}(\mathrm{t})$ , which decreases strictly with respect to steps  $t$ . To determine the optimal bitwidth for each step  $t$ , we compare the  $\mathrm{SNR}^{\mathrm{Q}}$  of each bitwidth with  $\mathrm{SNR}^{\mathrm{F}}$ , and select the minimum bitwidth  $b_{\mathrm{min}}$  that satisfies:
$$
\mathrm {S N R} _ {\mathrm {b} _ {\min }} ^ {\mathrm {Q}} (\mathrm {t}) > \mathrm {S N R} ^ {\mathrm {F}} (\mathrm {t}). \tag {15}
$$
If none of the bitwidths satisfies this condition, we utilize the maximum bitwidth in  $B$  to achieve a higher SNR. In practice, models with different bitwidths are calibrated separately, with the calibration set collected from the corresponding steps.
Datasets and quantization settings. We conduct image synthesis experiments using latent diffusion models (LDM) [45] on three standard benchmarks: ImageNet[9], LSUN-Bedrooms, and LSUN-Churches [60], each with a resolution of  $256 \times 256$ . All experimental configurations, including the number of steps, variance schedule (denoted by eta in the following), and classifier-free guidance scale, follow the official implementation [45]. For low-bit quantization, we use the PTQ method proposed in BRECQ [32] and AdaRound [40], which is congruent with Q-Diffusion [31]. For 8-bit quantization on ImageNet, we only use a naive PTQ method proposed by TensorRT [44], which is simple and fast. The input and output layers in the model are fixed to 8-bit, while all other convolutional and linear layers are quantized to the target bitwidth. In mixed precision experiments, we fix the weights to 4-bit and use Eq. (15) to determine the bitwidth of activations over uncorrected quantized diffusion models with a bitwidth set of  $\{4,8\}$ . Details of bitwidth allocation can be found in the supplementary material.
Evaluation metrics. For each experiment, we report the widely adopted Frechet Inception Distance (FID) [16] and sFID [42] to evaluate the performance. For ImageNet experiments, we additionally report Inception Score (IS) [46] for reference. To ensure consistency in the reported outcomes, including those of the baseline methods, all results are obtained by our implementation. We sample 50,000 images and evaluate them with ADM's TensorFlow evaluation suite [10]. To quantify the computational efficiency, we measure Bit Operations (BOPs) for a single forward pass of the diffusion model using the equation  $\mathrm{BOPs} = \mathrm{MACs}\cdot \mathrm{b_w}\cdot \mathrm{b_a}$ , where MACs denotes Multiply-And-Accumulate operations, and  $\mathrm{b_w}$  and  $\mathrm{b_a}$  represent the bitwidth of weights and activations, respectively, following [54].
Statistics collection. Before implementing our method, three statistics need to be collected: the correlation coefficient, denoted as  $k$  in Eq. (7), and the mean and variance of the uncorrelated quantization noise, as depicted in Eq. (10). To obtain these statistics, we generate 1024 samples using both quantized and full-precision diffusion models, store the quantization noise at each step, and then calculate the required statistics.
As shown in Table 1, we conduct ablation experiments on ImageNet  $256 \times 256$  dataset over LDM-4 model, to demonstrate the effectiveness of the proposed techniques. These techniques include Correlated Noise Correction (CNC) for addressing the correlated quantization noise, as well as Bias Correction (BC) and Variance Schedule Calibration (VSC) for correcting the residual uncorrelated quantization noise. By employing Correlated Noise Correction, we achieved a 0.48 reduction in FID and a 6.55 decrease in sFID. The considerable reduction in sFID suggests that the generated images possess more intricate spatial details than those generated using the baseline method, and that the correlated portions significantly contribute to the quantization noise. With the proposed Variance Schedule Calibration, the additional variance of uncorrelated quantization noise can be absorbed, achieving a reduction of 0.2 in FID and 0.11 in sFID. By further introducing Bias Correction that effectively corrects the mean deviation caused by quantization noise, our proposed PTQD achieved an FID of 6.44 and an sFID of 8.43, with only a 1.33 increase in sFID under the W4A4/W4A8 mixed precision setting. These results demonstrate the efficacy of the proposed techniques in achieving accurate post-training quantization of diffusion models.
Additional ablation experiments can be found in the supplementary material.
5.3.1 Class-conditional Generation
In this section, we evaluate the performance of class-conditional image generation on  $256 \times 256$  ImageNet dataset, as presented in Table 2. By utilizing the Naive PTQ method [44] and quantizing to 8-bit, diffusion models can achieve a notable  $12.39 \times$  reduction in bit operations, while experiencing minimal increases in FID/sFID. With the aggressive W4A8 bitwidth setting, our method effectively narrows the FID gap to a mere 0.06 with 250 generation steps. In this setting, the model size is
Table 1: The effect of different components proposed in the paper. Here, MP denotes the proposed step-aware mixed precision scheme.
Models	Method	Bitwidth (W/A)	FID↓	sFID↓
LDM-4 (steps = 250 eta = 1.0 scale = 1.5)	Q-Diffusion	MP	9.97	18.23
+ CNC	MP	9.49	11.68
+ CNC + VSC	MP	9.29	11.57
PTQD (CNC + VSC + BC)	MP	6.44	8.43
FP	32/32	5.05	7.10
compressed by  $6.83\times$  and the bit operations can be reduced by a remarkable  $19.96\times$ . In experiments utilizing mixed precision with W4A4 and W4A8 bitwidths, previous methods encounter difficulties in mitigating substantial quantization noises caused by low-bit quantization. For instance, in the first set of experiments with a 20-step generation process, Q-Diffusion [31] obtains FID and sFID scores as high as 116.61 and 172.99, respectively, indicating difficulties in handling low-bit diffusion models with fewer generation steps. While our method cannot calibrate the variance schedule due to a zero value for the hyperparameter eta, it still achieves an exceptionally low FID score of 7.75, demonstrating effective rectification of the correlated quantization noise and mean deviation. The second set of mixed precision experiments also yielded similar results, with our method reducing FID and sFID scores by 3.53 and 9.80, respectively.
Table 2: Performance comparisons of class-conditional image generation on ImageNet  ${256} \times  {256}$  .
Model	Method	Bitwidth (W/A)	Model Size (MB)	BOPs (T)	BOP comp. ratio	IS↑	FID↓	sFID↓
LDM-4 (steps = 20 eta = 0.0 scale = 3.0)	FP	32/32	1603.35	102.21	-	225.16	12.45	7.85
Naive PTQ	8/8	430.06	8.25	12.39×	152.91	12.14	8.43
Ours	8/8	430.06	8.25	12.39×	153.92	11.94	8.03
Q-Diffusion	4/8	234.51	5.12	19.96×	212.52	10.63	14.80
Ours	4/8	234.51	5.12	19.96×	214.73	10.40	12.68
Q-Diffusion	MP	234.51	4.73	21.61×	7.86	116.61	172.99
Ours	MP	234.51	4.73	21.61×	175.19	7.75	18.78
LDM-4 (steps = 250 eta = 1.0 scale = 1.5)	FP	32/32	1603.35	102.21	-	185.04	5.05	7.10
Naive PTQ	8/8	430.06	8.25	12.39×	180.56	4.06	5.91
Ours	8/8	430.06	8.25	12.39×	180.83	4.02	5.81
Q-Diffusion	4/8	234.51	5.12	19.96×	148.74	5.37	9.56
Ours	4/8	234.51	5.12	19.96×	149.74	5.11	8.49
Q-Diffusion	MP	234.51	4.81	21.25×	121.10	9.97	18.23
Ours	MP	234.51	4.81	21.25×	126.26	6.44	8.43
5.4 Unconditional Generation
In this section, we present a comprehensive evaluation of our approach on LSUN-Bedrooms and LSUN-Churches [60] datasets for unconditional image generation. As shown in Table 3, our method consistently narrows the performance gap between quantized and full-precision diffusion models. Notably, our proposed method allows for compression of diffusion models to 8-bit with minimal performance degradation, resulting in a mere 0.1 increase in FID on the LSUN-Churches dataset. With the W4A8 bitwidth setting, our method reduces FID and sFID by notably 0.78 and 3.61 compared with Q-Diffusion [31] on LSUN-Bedrooms. Furthermore, Q-Diffusion fails to effectively denoise samples under the mixed precision setting on LSUN-Churches due to its low SNR. In this case, the hyperparameter eta is set to zero, which prevents the use of Variance Schedule Calibration. Despite relying solely on Correlated Noise Correction and Bias Correction, our approach remarkably enhances the quality of the generated images, as demonstrated by a substantial reduction in the FID score from 218.59 to 17.99. This notable improvement highlights the significant impact of the correlated part of quantization noise on the overall image quality, which can be effectively rectified by our method.
Additional evaluation results on CelebA-HQ dataset can be found in the supplementary material.
Table 3: Performance comparisons of unconditional image generation.
LSUN-Bedrooms 256 × 256
LDM-4 (steps = 200, eta = 1.0)	LSUN-Churches 256 × 256
LDM-8 (steps = 200, eta = 0.0)
Method	Bitwidth
(W/A)	FID↓	sFID↓	Method	Bitwidth
(W/A)	FID↓	sFID↓
Full precision	32/32	3.00	7.13	Full precision	32/32	6.30	18.24
Q-Diffusion	8/8	3.80	9.95	Q-Diffusion	8/8	6.94	18.93
Ours	8/8	3.75	9.89	Ours	8/8	6.40	18.34
Q-Diffusion	4/8	6.72	18.80	Q-Diffusion	4/8	7.80	19.97
Ours	4/8	5.94	15.16	Ours	4/8	7.33	19.40
Q-Diffusion	MP	5.75	12.79	Q-Diffusion	MP	218.59	312.86
Ours	MP	5.49	12.04	Ours	MP	17.99	37.34
5.5 Deployment Efficiency
We have measured the latency of matrix multiplication and convolution operations in quantized and full-precision diffusion models using an RTX3090 GPU, as shown in Table 4. Both floating-point and quantized operations are implemented with CUTLASS [24]. When both weights and activations are quantized to 8-bit, we observe a  $2.03 \times$  reduction in latency compared to its full-precision counterpart over LDM-4. Moreover, when weights and activations are quantized to 4-bit, the speedup further increases to  $3.34 \times$ . The mixed-precision settings explored in our experiments strike a good balance between latency and model performance.
Table 4: Comparisons of time cost across various bitwidth configurations on ImageNet  $256 \times 256$ . Due to the current lack of a fast implementation for W4A8, we implement MP scheme with W8A8 and W4A4 kernels.
Model	Bitwidth (W/A)	Model Size (MB)	FID↓	sFID↓	Time (s)
LDM-4	32/32	1603.35	5.05	7.10	5.46
(steps=250	8/8	430.06	4.02	5.81	2.68
eta=1.0	MP	234.51	6.44	8.43	2.45
scale=1.5)	4/4	234.51	-	-	1.63
In this paper, we have proposed PTQD, a novel post-training quantization framework for diffusion models that unifies the formulation of quantization noise and diffusion perturbed noise. To start with, we have disentangled the quantization noise into correlated and residual uncorrelated parts relative to its full-precision counterpart. To reduce mean deviations and additional variance in each step, the correlated part can be easily corrected by estimating the correlation coefficient. For the uncorrelated part, we have proposed Variance Schedule Calibration to absorb its additional variance and Bias Correction to correct the mean deviations. Moreover, we have introduced Step-aware Mixed Precision to adaptively select the optimal bitwidth for each denoising step. By incorporating these techniques, our PTQD has achieved significant performance improvement over existing state-of-the-art post-training quantized diffusion models, with only a 0.06 FID increase compared to the full-precision LDM-4 on ImageNet  $256 \times 256$  while saving  $19.9 \times$  bit-operations. In the future, we can further quantize other components within diffusion models, such as the text encoder and image decoder, to achieve higher compression ratios and accelerated performance. We may also extend PTQD to a wider range of generative tasks to assess its efficacy and generalizability.
Limitations and Broader Impacts. The proposed PTQD framework stands out for its high efficiency and energy-saving properties, which carry significant implications in reducing the carbon emissions attributed to the widespread deployment of diffusion models. However, similar to other deep generative models, PTQD has the potential to be utilized for producing counterfeit images and videos for malicious purposes.
Acknowledgement This work was supported by National Key Research and Development Program of China (2022YFC3602601).
[58] W. Wu, Y. Zhao, M. Z. Shou, H. Zhou, and C. Shen. Diffumask: Synthesizing images with pixel-level annotations for semantic segmentation using diffusion models. arXiv preprint arXiv:2303.11681, 2023.
[59] H. Yang, L. Duan, Y. Chen, and H. Li. BSQ: exploring bit-level sparsity for mixed-precision neural network quantization. In ICLR, 2021.
[60] F. Yu, A. Seff, Y. Zhang, S. Song, T. Funkhouser, and J. Xiao. Lsun: Construction of a large-scale image dataset using deep learning with humans in the loop. arXiv preprint arXiv:1506.03365, 2015.
[61] L. Zhang, Y. He, Z. Lou, X. Ye, Y. Wang, and H. Zhou. Root quantization: a self-adaptive supplement ste. Applied Intelligence, 53(6):6266-6275, 2023.
[62] Q. Zhang and Y. Chen. Fast sampling of diffusion models with exponential integrator. arXiv preprint arXiv:2204.13902, 2022.
[63] Q. Zhang, M. Tao, and Y. Chen. gddim: Generalized denoising diffusion implicit models. In ICLR, 2023.
[64] H. Zheng, P. He, W. Chen, and M. Zhou. Truncated diffusion probabilistic models. stat, 1050:7, 2022.
[65] B. Zhuang, C. Shen, M. Tan, L. Liu, and I. Reid. Towards effective low-bitwidth convolutional neural networks. In CVPR, 2018.