PTQD: Accurate Post-Training Quantization for Diffusion Models
Authors: Yefei He
Luping Liu
Jing Liu
Weijia Wu
Hong Zhou
Bohan Zhuang
Venue: NeurIPS 2023 poster (2023)
Keywords: Diffusion models
Post-training quantization
Mixed precision
TL;DR: We introduce a novel method to disentangle and correct quantization noise in post-training quantized diffusion models, resulting in superior performance and significant bit operation savings.

Abstract: Diffusion models have recently dominated image synthesis and other related generative tasks. However, the iterative denoising process is expensive in computations at inference time, making diffusion models less practical for low-latency and scalable real-world applications. 
Post-training quantization of diffusion models can significantly reduce the model size and accelerate the sampling process without requiring any re-training. Nonetheless, applying existing post-training quantization methods directly to low-bit diffusion models can significantly impair the quality of generated samples. Specifically, for each denoising step, quantization noise leads to deviations in the estimated mean and mismatches with the predetermined variance schedule. Moreover, as the sampling process proceeds, the quantization noise may accumulate, resulting in a low signal-to-noise ratio (SNR) during the later denoising steps. To address these challenges, we propose a unified formulation for the quantization noise and diffusion perturbed noise in the quantized denoising process. 
Specifically, we first disentangle the quantization noise into its correlated and residual uncorrelated parts regarding its full-precision counterpart. The correlated part can be easily corrected by estimating the correlation coefficient. For the uncorrelated part, we subtract the bias from the quantized results to correct the mean deviation and calibrate the denoising variance schedule to absorb the excess variance resulting from quantization. Moreover, we introduce a mixed-precision scheme for selecting the optimal bitwidth for each denoising step, which prioritizes lower bitwidths to expedite early denoising steps, while ensuring that higher bitwidths maintain a high signal-to-noise ratio (SNR) in the later steps. Extensive experiments demonstrate that our method outperforms previous post-training quantized diffusion models in generating high-quality samples, with only a $0.06$ increase in FID score compared to full-precision LDM-4 on ImageNet $256\times256$, while saving $19.9\times$ bit operations. Code is available at [https://github.com/ziplab/PTQD](https://github.com/ziplab/PTQD).

PTQD: Accurate Post-Training Quantization for Diffusion Models
Yefei He!
Luping Liu'
Jing Liu?
Weijia Wu'
Hong Zhou't
Bohan Zhuang21
!Zeejiang University, China
ZZIP Lab, Monash University, Australia
Abstract
Diffusion models have recently dominated image synthesis and other related gener- ative tasks. However, the iterative denoising process is expensive in computations at inference time, making diffusion models less practical for low-latency and scal- able real-world applications. Post-training quantization of diffusion models can significantly reduce the model size and accelerate the sampling process without requiring any re-training. Nonetheless, applying existing post-training quantization methods directly to low-bit diffusion models can significantly impair the quality generated samples. Specifically, for each denoising step, quantization noise leads to deviations in the estimated mean and mismatches with the predetermined variance schedule. Moreover, the sampling process proceeds, the quantization noise may accumulate, resulting in a low signal-to- to-ise ratio (SNR) during the later denoising steps. To address these challenges, propose unified formulation for the quantization noise and diffusion perturbed noise the quantized denoising first disentangle the quantization noise into its correlated and residual uncorrelated parts regarding its full-precision counterpart. The cor- related part can be easily corrected by estimating the correlation coefficient. For the uncorrelated part, we subtract the bias from the quantized results to correct the mean deviation and calibrate the denoising variance schedule absorb the excess variance resulting from quantization. Moreover, we introduce a mixed-precision scheme for selecting the optimal bitwidth for each denoising step, which priori- tizes lower bitwidths to expedite early denoising steps, while ensuring that higher bitwidths maintain high signal-to-ne experiments demonstrate that our method outperforms previous post-training quan- tized diffusion models in generating in FID score compared to full-precision LDM-4 on ImageNet 256 x 256, while saving 19.9x x bit operations. Code available wttpggiiuuuooooiilaaa///000
process. Specifically,
ratio (SNR) the later steps. Extensive
samples, with only a 0.06 increase
Introduction
Diffusion models have demonstrated remarkable ability in generating high-quality samples in multiple fields [10, 5, 58, 18, 38, 30, 13, 52, 8, 49]. Compared to generative adversarial networks (GANs) [15] and variational autoencoders (VAEs) [27], diffusion models do not face the issue of mode collapse and posterior collapse, thus training is more stable. Nonetheless, the application of diffusion models limited by two major bottlenecks. Firstly, diffusion models typically require hundreds of denoising steps to generate high-quality samples, making the process significantly slower than that of GANs. To address this, many studies [50, 36, 2, 28, 34] have proposed advanced training-free sampler to reduce the number of denoising iterations. Among them, a recent study DPM-solver [36] curtails the denoising process to ten steps by analytically computing the linear part of the diffusion ordinary
Corresponding author. Email: zhouh@mail bme wju.edu.ua.... cn, bohan aahaan.hhuangee wawnaang@maii.coom
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Q D-Diffusion (W4A8)
PTQD (W4A8)
Full Precision
Figure 1:
The comparisons of samples generated by Q-Diffusion [31]. PTQD and full-precision LDM-4 [45] on CelebA-HQ 256 256 dataset. Here, WrAy indicates the weights are quantized to x-bit while the activations are quantized to y-bit.
differential equations (ODEs). Nevertheless, diffusion models with these fast samplers are not yet ready for real-time applications. For instance, even when executed on a high-performance platform such as the RTX 3090, Stable Diffusion [45] with the DPM-Solver [36] sampler still takes over second to generate a 512 x 512 image. Second, the application diffusion models on various devices is constrained by the massive parameters and computational complexity. To illustrate, executing Stable Diffusion [45] requires 16GB of running memory and GPUs with over 10GB of VRAM, which is infeasible for most consumer-grade PCs, not to mention resource-constraaned edge devices.
Model quantization, which employs lower numerical bitwidth to represent weights and activations, has been widely studied to reduce memory footprint and computational complexity. For instance, employing 8-bit models can result in a significant speed-up of 2.2x compared to floating -ponnt models on ARM CPUs [21]. Adopting 4-bit quantization can further deliver a throughput increase of up to 59% compared 8-bit quantization [3]. To facilitate the quantization process without the need for re-training, post-training quantization (PTQ) has emerged as widely used technique, which highly practical and easy to implement. While PTQ on traditional models have been widely studied [40, 32, 20, 33, 57]. its application on diffusion models incurs two new challenges at the fundamental level. First, with the noise prediction network quantized, its quantization noise inevitably introduces bias in the estimated mean and brings additional variance that collides with the predetermined variance schedule in each denoising step. Additionally, the quantization noise accumulates the iterative sampling process progresses, leading to significant drop in the signal- to-noise ratio (SNR) of the noise prediction network in the later denoising steps. This diminished SNR severely impedes the denoising capability, resulting in noticeable degradation in the quality of the generated images.
To tackle the aforementioned challenges, we present PTQD, a novel post-training quantization framework for diffusion models. To address the mean deviation and additional variance in each denoising step, model the quantization noise by disentangling it into its correlated and residual uncorrelated parts regarding its full-precision counterpart, and designs separate correction methods for them. By estimating the correlation coefficient, the correlated part can be easily rectified. For the residual uncorrelated part, we subtract the bias from the estimated mean and propose variance schedule calibration, which absorbs the additional variance into the diffusion perturbed noise. To overcome the issue of low SNR that diminishes denoising capability in later denoising steps, we introduce step-awant mixed precision scheme, which adaptively allocates different bitwidths for synonymous steps to maintain a high SNR for the denoising process.
In summary, our contributions are as follows:
We present PTQD, novel post-training quantization framework for diffusion models, which provides a unified formulation for quantization noise and diffusion perturbed noise.
We disentangle the quantization noise into correlated and uncorrelated parts regarding its full- precision counterpart. Then we correct the correlated part by estimating the correlation coefficient, and propose variance schedule calibration to rectify the residual uncorrelated part.
We introduce a step-aware mixed precision scheme, which dynamically selects the appropriate bitwidths for synonymous steps, preserving SNR throughout the denoising process.
Our extensive experiments demonstrate that our method reaches new state-of-the-art performance for post-training quantization of diffusion models.

---

2 Related Work
Efficient diffusion models. While diffusion models can produce high-quality samples, their slow generation speed hinders their large-scale applications downstream tasks. To explore efficient diffusion models, many methods have been proposed to expedite the sampling process. These methods can be classified into two categories: methods that necessitate re-training and advanced samplers for pre-trained models that do not require training. The first category of methods comprises knowledge distillation [37, 47]. diffusion scheme learning [7. 12. 64, 39], noise scale learning [26, 43], and sample trajectory learning [55, 29]. Although these methods can accelerate sampling, re-training diffusion model can be resource-intensee
and time-consuming. On the other hand, the second category of methods designs advanced samplers directly on pre-trained diffusion models, eliminating the need for re-training. The primary methods in this category are implicit sampler [50, 28, 63, 56], analytical trajectory estimation [2, IJ. and differential equation (DE) solvers such as customized SDE [51, 23, 25J and ODE [36, 34, 62]. Although these methods can reduce the sampling iterations, the diffusion model's 's massive parameters and computational complexity restrict their use to high- performance platforms. Conversely, our proposed low-bit diffusion model can significantly reduce the model's computational complexity while speeding up the sampling and reducing the demand for hardware computing resources in a training-fii free manner.
Model quantization. Quantization is dominant technique to save memory costs and speed up computation. can be divided into two categories: quantization-awa -ware training (QAT) [14, 35, 22, 65, 61] and post-training quantization (PTQ) [32, 40, 20, 57, 33]. QAT involves simulating quantization during training to achieve good performance with lower precision, but it requires substantial time, computational resources, and access to the original dataset. In contrast, PTQ does not require fine-tuning and only needs a small amount of unlabeled data to calibrate. Recent studies have pushed the limits of PTQ to 4-bit traditional models by using rounding strategies [40], layer-wise calibration [20, 53], and sccond-order statistics [32, 57]. Additionally, mixed precision (MP) [19, 11, 4, 59, 6] allows part of the model to be represented by lower bitwidths to accelerate inference. Common criteria for determining quantization bitwidths include Hessian spectrum [11, 6] or Pareto frontier [4]. In contrast, we propose novel mixed- precision scheme for diffusion models that adapts different bitwidths for synonymous denoising steps.
Until now, there have been few studies specifically focusing on quantizing pre-trained diffusion model without re-training. PTQ4DM [48] is the first attempt to quantize diffusion models to 8-bit, but its experiments are limited to small datasets and low resolution. Q-Diffusion [31] applies advanced PTQ techniques proposed by BRECQ [32] to improve performance and evaluate it on a wider range of datasets. Our paper aims to analyze systematically the quantization effect diffusion models and establish a unified framework for accurate post-tree pest-training diffusion quantization.
3 Preliminaries
3.1
Diffusion Models
Diffusion models [50, 17] gradually apply Gaussian noise real data Xo in the forward process and learn reverse process to denoise and generate high-quality images. For DDPMs [17], the forward process is Markov chain, which can be formulated as:
T
(x))
II (xxxxx-1) t
= vaxx B,1)
(I)
where aee 3, are hyperparameteres and Br =1 11 dz..
In the reverse process, since directly estimating the real distribution of q(x4-1/x;) is intractable, diffusion models approximate it via variational inference by learning Gaussian distribution po(X(-1(x,) = (x (x,, t), Ex(,,, and reparameterize its mean by a noise prediction
network cx(, ():
1 Vae
B. v1 āi
(x(x,, =
X;
EB Ex(x
(2)
3

---

where 0r [T.=1 Og. The variance Ex( can either be reparameterized or fixed to a constant schedule or. When uses constant schedule, the sampling of x, can be formulated as:
1 vat
Br ae
X
X
e)(
+ o, where ~ (O.,I).
(3)
Our method focuses on post-training quantization of diffusion models without the need of training. Instead, we pre-trained diffusion models and inherit their hyperparameters and variance schedules for inference. Although the derivations presented in this paper are based on DDPM, they can be readily extended other fast sampling methods, such DDIM [50]. Additional information be found in the supplementary material.
3.2 Model Quantization
We use uniform quantization in our study and all the experiments, For uniform quantization, given floating- vector x, the target bitwidth the quantization process can be defined as:
A
(clip((
X A + 2,0, 2 1)
(4)
where [.] is the round operation, A max(x)-min(xx and Z = min(x)
20-1
To ensure clarity and consistency, we introduce notation to define the variables used in the paper. Let X be tensor (weights activations) in the full-precision model, the result after normalization layers is denoted X. The corresponding tensor of the quantized model is represented X. The quantization noise is depicted by Ax, which is the difference between X and X.
4 Method
Model quantization discretizes the weights and activations, which will inevitably introduce quantiza- tion noise into the result. As per Eq. (3), during the reverse process of the quantized diffusion model, the sampling of x can be expressed as:
3
Xp-1
X2
(o(x),))
ozz
(5)
Vai
a
3r
(EO(X), Aco(x..))) ozZ.
X;
VI
Here, és(X;, is the output of the quantized noise prediction network and Aeo(x..) .t) refers the quantization noise. The additional quantization noise will inevitably alter the mean and variance of Xr-1, decreasing the signal-to-noise ratio (SNR) and adversely affecting the quality of the generated samples. Therefore, to mitigate the impact of quantization, it is necessary to correct the mean and variance restore the SNR at each step of the reverse process.
4.1 Correlation Disentanglement
We begin by making an assumption that a correlation exists between the quantization noise and the result of the full-precision noise prediction network. While other factors, such nonlinear operations, may contribute to this correlation, Proposition demonstrates that normalization layers are responsible for part of it.
Proposition 1. Given and Ý as inputs to a normalization layer in a fulll-preciie full-preccssion model and its quantized version, where the quantization noise Dy = = - Y initially uncorrelated with Y, correlation between the quantization noise and the output of the full full-precision model after the normalization layer will exist.
The proof is based on the fact that the mean and variance of Y will differ from that Y (depending on the specific quantization scheme). Therefore, the quantization noise after normalization layer can be expressed as
fl
Y Y py
Ar (oy - or)Y + gypu - My
A
(6)
0Y
σy
Opor

---

t=199
t=100
t=0
0.5
R2=0.98
R²=0,72
0²=0.85
0.5
1
Aujxal)
0.0
0.0
-0.5
-0.
-1
22.5
0.0 6 (x, t)
2.5
-2.5
0.0 e (x t)
2.5
0
1
66 (xx f)
Figure 2: The correlation between the quantization noise (Y-axis) and the output the full-precision noise prediction network (X-axis). Each data point the plot corresponds to specific entries within these vectors. Data were collected by generating samples with 4-bit LDM-8 [45] for 200 steps on LSUN-Churches [60].
Here, omit the affine transform parameters in normalization layers for simplicity. It can be observed from Eq. (6) that the second term in the numerator related to Y, while the other three terms are uncorrelated. Therefore, after normalization layers, the quantization noise Ay will be correlated with Y.
The empirical observation illustrated in Figure confirms strong correlation between the quantiza- tion noise and the output of the frel-precision noise prediction network, which further verifies our assumption. Based on the assumption and observation, the quantization noise of the quantized noise prediction network can disentangled into two parts:
Dco(x.)) kes(X,, t) A ev(x.)))
The first part, denoted by keg(Xe, (), linearly related to (x(x;, The second part, expressed by A d)) represents the residual component the quantization noise, and is assumed to be uncorrelated with ea (x), Here, & is the correlation coefficient, which can be estimated by applying linear regression on the quantization noise (e,(x,))) and the original value eg (Xt, L). Details can be found in Section 5.1.
With the disentanglement presented in Eq. (7), the sampling of x can be further expressed as:
3 (Eo(x:,t)
Des(x..))) + oz2
(8)
Vai
a
1
Br (11 (1+ h1) Eo A Ses(x..t)) O.2.
Val
Consequently, the bias and additional variance arise from both the correlated and uncorrelated parts of quantization noise. In the following section, will provide detailed explanation of how these two parts quantization noise can be separately corrected.
Quantization Noise Correction
4.2.1 Correlated Noise Correction
Based on Eq. (8), the correlated part of the quantization noise can be rectified by dividing the output of the quantized noise prediction network éø(Xx. t) by + k. resulting in:
1 Vot:
3e - ai c0(x..0)
3,
--1
X;
VI
0.2
Vol a.(1 K) A Aoo(x..))
Consequently, only the uncorrelated quantization noise remains. Moreover, for values of & 2 can be deduced that the mean and variance the uncorrelated quantization noise are diminished by I+E. In practice, enforce the non nogaatiivv of k, and reset it to zero if it is negative. In the following, will explain how handle the uncorrelated quantization noise that persists in Eq. (9).
5

---

4.2.2 Uncorrelated Noise Correction
The presence uncorrelated quantization noise introduces additional variance at each step, resulting in total variance that exceeds the scheduled value of. To address this, we propose to calibrate the variance schedule for quantized diffusion models, which is denoted as of² and smaller than the original schedule o.. To estimate o²2, we further make an assumption and model the uncorrelated quantization noise as a Gaussian distribution with a mean of My and a variance of og.
(eo() (Hg: o²)).
(10)
To verify this assumption, we conduct statistical tests (refer to the supplementary material) and present the distribution of the uncorrelated quantization noise in Figure 3. The values of mean and variance can be estimated by generating samples with both quantized and full-precision diffusion models and collecting the statistics of the uncorrelated quantization noise. Following prior work [41], the mean deviation can be rectified through Bias Correction (BC), where we collect the channel-wise means of uncorrelated quantization noise and subtract them from the output of quantized noise prediction network. For the variance of the uncorrelated quantization noise, we propose Variance Schedule Calibration (VSC), where the uncorrelated quantization noise be absorbed into Gaussian diffusion noise with the above assumption. By substituting the calibrated variance schedule of² into Eq. (9) while keeping the variance of each step unaltered, we can solve for the optimal variance schedule using the following approach:
33 at
2
o/ oū
(11)
(1) + 1)22
2 o ae gr σt 0,
of ae ae 11+b)
(12)
otherwise.
can be observed that if the additional variance of quantization noise is smaller than the noise hyperparameter of, the increase variance caused by quantization can be eliminated. According to Eq. (12), the coefficient for the variance which is generally small enough to ensure that the quantization noise can be fully absorbed, except for cases of deterministic sampling where o, is zero. In this case, there is analytical solution for o/2, and we use the optimal solution that is o²² 0. Overall, the process quantization noise correction is summarized in Algorithm 1.
8
quantization noise can calculated as a. (1++)))
Algorithm 1: Quantization noise correction.
Statisties collection before sampling:
1) Quantize diffusion models with BRECQ [32] (or other PTQ methods);
2) Generate samples with both quantized and FP models and collect quantization noise;
3) Calculate the correlated coefficient based on Eq. (7), and the mean and variance of the uncorrelated quantization noise per Eq. (10);
Noise correction for each sampling step:
4) Correct the correlated part of the quantization noise by dividing the output of the noise prediction network by 1 1+ k;
5) Calibrate the variance schedule by Eq. (12) and subtract the channel-wise biases from the output of the quantized noise prediction network.
Although the proposed method can correct the mean deviation and the numerical value of the variance for each step, generating satisfactory samples with low-bit diffusion models remains challenging due to the low signal-to-noise ratio (SNR) of the quantized noise prediction network, In the next section, we will analyze this issue in detail.

---

ShiR IWAMA corrected)
IW4AA, uncorrected)
Stid IWAAl, corrected)
IW4AB, uncorrected) SNRR
5
400
200
1
0
00.06 00.04 -0.02 0.00 0.02 0.04 0.06
25
75
1000
125
150
175
200
Stop
Figure 3: The distribution of uncorrelated quan- tization noise collected from W4A8 LDM-4 on LSUN-Bedrooms 256 x 256 dataset, where the x-axis represents the range of values and the y-
Figure 4: Comparison of the signal-to-noise
ratio (SNR) in each step of LDM-4 on LSUN- Bedrooms across various bitwidths.
axis the frequency of values.
4.3 Step-aware Mixed Precision
Given the output of the full-precision noise prediction network co(x,, and corresponding quantiza- tion noise (x),,.)) we define the SNR of quantized noise prediction network by:
( = |x t)
(13)
De (x)
Figure 4 depicts the S with various bitwidths and correction methods. The figure reveals several insights: 1) SNR drops drastically as step decreases; 2) models with higher bitwidth exhibit larger S 3) the proposed correction methods yield clear SNR improvements, especially for large steps. The first observation highlights the challenge of generating high-quality samples using low-bit diffusion models. In particular, as approaches zero, the SNR of W4A4 diffusion models diminishes and approaches unity, implying that the magnitude of quantization noise is even comparable to the original result of the noise prediction network. To enable low-bit diffusion models while maintaining good generation performance,
propose novel approach called Step-aware Mixed Precision, which involves setting different bitwidths for synonymous steps to keep SNR Q within a reasonable range across steps.
Specifically, the bitwidth of weights is fixed and shared across different denoising steps, which eliminates the need to store and reload multiple model state files during the sampling process. As result, we only adjust the bitwidth of activations. Formally, we predefine a set of bitwidths B = [b1, bg...., bnr for activations and evaluate the SNR under each bitwidth. To establish benchmark for S we follow prior studies [36, 26] and introduce SNR based on the forward process, which denotes the degree of data noise each step:
SNR (t) = az
(14)
Figure 4 illustrates SNR" SNRF(t), which decreases strictly with respect to steps t. To determine the optimal bitwidth for each step compare the S of each bitwidth with SNR and select the minimum bitwidth bmin that satisfies:
SNR (t) SNR (t).
(15)
If none of the bitwidths satisfies this condition, we utilize the maximum bitwidth in B to achieve higher SNR. In practice, models with different bitwidths are calibrated separately, with the calibration set collected from the corresponding steps.
7

---

5 Experiments
5.1 Implementation Details
Datasets and quantization settings. We conduct image synthesis experiments using latent diffusion models (LDM) [45] on three standard benchmarks: ImageNet(9),
and LSUN- Churches [60], each with resolution of 256 x 256. All experimental configurations, including the number of steps, variance schedule (denoted by eta in the following), and classifier-free guidance scale, follow the official implementation [45]. For low-bit quantization, we use the PTQ method proposed in BRECQ [32] and AdaRound [40], which is congruent with O-Diffusion [31]. For 8-bit quantization ImageNet,
naive PTQ method proposed by TensorRT [44], which is simple and fast. The input and output layers in the model are fixed to 8-bit, while all other convolutional and linear layers are quantized to the target bitwidth. In mixed precision experiments, we fix the weights to 4-bit and use Eq. (15) to determine the bitwidth of activations over uncorrected quantized diffusion models with bitwidth set of {4,8). Details of bitwidth allocation can be found in the supplementary material.
only
Evaluation metrics. For each experiment, we report the widely adopted Frechet Inception Distance (FID) [16] and sFID [42] to evaluate the performance. For ImageNet experiments, we additionally report Inception Score (IS) [46] for reference. To ensure consistency in the reported outcomes, including those of the baseline methods, all results are obtained by our implementation. We sample 50,000 images and evaluate them with ADM's TensorFlow evaluation suite [10]. To quantify the computational efficiency, we measure Bit Operations (BOPs) for single forward pass of the diffusion model using the equation BOPs MACs bw ba, where MACs denotes Multiply-And- Accumulate operations, and by and b2 represent the bitwidth of weights and activations, respectively, following [54].
Statisties collection. Before implementing our method, three statistics need to be collected: the correlation coefficient, denoted as A in Eq. (7), and the mean and variance of the uncorrelated quantization noise, as depicted in Eq. (10). To obtain these statistics, we generate 1024 samples using both quantized and full- precision diffusion models, store the quantization noise each step, and then calculate the required statistics.
5.2 Ablation Study
As shown in Table conduct ablation experiments on ImageNet 256 x 256 dataset over LDM- 4 model, to demonstrate the effectiveness of the proposed techniques. These techniques include Correlated Noise Correction (CNC) for addressing the correlated quantization noise, as well Bias Correction (BC) and Variance Schedule Calibration (VSC) for correcting the residual uncorrelated quantization noise. By employing Correlated Noise Correction, we achieved a 0.48 reduction in FID and 6.55 decrease in sFID. The considerable reduction in sFID suggests that the generated images possess more intricate spatial details than those generated using the baseline method, and that the correlated portions significantly contribute to the quantization noise, With the proposed Variance Schedule Calibration, the additional variance of uncorrelated quantization noise can be absorbed, achieving a reduction of 0.2 in FID and 0.11 in sFID. By further introducing Bias Correction that effectively corrects the mean deviation caused by quantization noise, our proposed PTQD achieved an FID of 6.44 and an sFID of 8.43, with only a 1.33 increase in sFID under the W4A4/W4A8 mixed precision setting. These results demonstrate the efficacy of the proposed techniques achieving accurate post-training quantization of diffusion models.
Additional ablation experiments can be found in the supplementary material.
5.3
Main Results
5.3.1
Class-conditional Generation
In this section, we evaluate the performance of class-conditional image generation on 256 x 256 ImageNet dataset, as presented Table By utilizing the Naive PTQ method [44] and quantizing to 8-bit, diffusion models can achieve notable 12.39x reduction in bit operations, while experiencing minimal increases in FID/sFID. With the aggressive W4A8 bitwidth setting, our method effectively narrows the FID gap to mere 0.06 with 250 generation steps. In this setting, the model size is
8

---

Table 1: The effect of different components proposed in the paper. Here, MP denotes the proposed step-aware mixed precision scheme.
Bitwidth
Models
Method
FID
sFID
(W/A)
Q-Diffusion
MP
9.97
18.23
LDM-4 (steps 250
CNC
MP
9.49
11.68
+ CNC VSC
MP
9.29
11.57
1.0
PTQD (CNC VSC BC)
MP
6.44
8.43
scale 1.5)
FP
32/32
7.10
compressed by 6 and the bit operations can be reduced by remarkable 19.96x. In experiments utilizing mixed precision with W4A4 and W4A8 bitwidths, previous methods encounter difficulties in mitigating substantial quantization noises caused by low-bit quantization. For instance, in the first set of experiments with 20-step generation process, Q-Diffusion [31] obtains FID and sFID scores high 116.61 and 172.99, respectively, indicating difficulties in handling low-bit diffusion models with fewer generation steps. While our method cannot calibrate the variance schedule due to zero value for the hyperparameter eta, it still achieves exceptionally low FID score of 7. 7.75, demonstrating effective rectification of the correlated quantization noise and mean deviation. The second set of mixed precision experiments also yielded similar results, with method reducing FID and sFID scores by 3.53 and 9.80, respectively.
Table 2: Performance comparisons of class-condition image generation ImageNet 256
256.
Bitwidth
Model Size
BOPs
BОP comp.
Model
Method
1St
FIDL
sFID
(W/A)
(MB)
(T)
ratio
FP
32/32
1603.35
102.21
225.16
12.45
7.85
Naive PTQ
8/8
430.06
8.25
x2.39x
152.91
12.14
8.43
LDM-4
Ours
8/8
430.06
8.25
12.39x
153.92
11.94
8.03
(steps 20
Q-Diffusion
4/8
234.51
5.12
19.96x
212.52
10.63
14,80
ela
Ours
4/8
234.51
5.12
19.96x
214.73
10.40
12.68
scale 3.0)
Q-Diffusion
MP
234.51
4.73
21.61
7.86
116.61
172.99
Ours
MP
234.51
4.73
21.61x
175.19
7.75
18.78
FP
32/32
1603.35
102.21
185.04
5.05
7.10
Naive PTQ
8/8
430.06
8.25
x2.39
180.56
4.06
5.91
LDM-4
Ours
8/8
430.06
8.25
12.39x
180.83
4.02
5.81
(steps 250
Q-Diffusion
4/8
234.51
5.12
19,96x
148.74
5.37
9.56
cta = 1.0
Ours
4/8
234.51
5.12
19.96x
149.74
5.11
8.49
scale 1.5)
Q-Diffusion
MP
234.51
21.25x
121.10
9.97
18.23
Ours
MP
234.51
4.81
21,25x
126.26
6.44
8,43
5.4
Unconditional Generation
In this section, we present comprehensive evaluation of our approach on L.SUN-Bedroom and [60] datasets for unconditional image generation. As shown in Table 3, our method diffusion models.
consistently narrows the performance gap between quantized and full-precisin Notably, our proposed method allows for compression of diffusion models 8-bit with minimal dataset. performance degradation, resulting in mere 0.1 increase in FID the LSUN-Chureeh With the W4A8 bitwidth setting, our method reduces FID and sFID by notably 0.78 and 3.61 compared with Q-Diffusion [31] denoise samples under the mixed precision setting on LSUN-Chine Churches due to its low SNR. this case, the hyperparameter eta is set to zero, which prevents the use of Variance Schedule Calibration. Despite relying solely on Correlated Noise Correction and Bias Correction, our approach remarkably enhances the quality of the generated images, demonstrated by substantial reduction in the FID score from 218.56 59 to 17.99. This notable improvement highlights the significant impact of the correlated part of quantization noise the overall image quality, which be effectively rectified by our method.
Furthermore, Q-Diffusion fails to effectively
Additional evaluation results CelebA-HQ dataset be found in the supplementary material.

---

Table 3: Performance comparisons of unconditional image generation.
LSUN- Bedrooms 256 x 256
LSUN-Churcher 256 x 256
LDM-4 (steps 200, cta 1.0)
LDM-8 (steps 200, cta = 0.0)
Bitwidth (W/A)
Bitwidth
Method
FID
sFID
Method
FID
sFID
(W/A)
precision
32/32
3.00
7.13
precision
32/32
6.30
18.24
Q-Diffusion
8/8
3.80
9.95
Q Q-Diffusion
8/8
6.94
18.93
Ours
8/8
3.75
9.89
Ours
8/8
6,40
18.34
Q-Diflusion
4/8
6.72
18.80
Q-Diffusion
4/8
7.80
19.97
Ours
4/8
5.94
15.16
Ours
4/8
7.33
19.40
Q-Diffusion
MP
5.75
12.79
Q-Diffusion
MP
218.59
312.86
Ours
MP
5.49
12,04
Ours
MP
17.99
37.34
5.5
Deployment Efficiency
We have measured the latency of matrix multiplication and convolution operations in quantized and full cull-precision diffusion models using an RTX3090 GPU, as shown in Table 4. Both Roating-poin and quantized operations are implemented with CUTLASS [24]. When both weights and activations quantized to 8-bit, we observe a 2.03x reduction in latency compared to its full-precision counterpart over LDM-4. Moreover, when weights and activations are quantized to 4-bit, the speedup further increases to 3.34x. The mexxed-precisioe settings explored in our experiments strike a good balance between latency and model performance.
Table 4: Comparisons of time cost across various bitwidth configurations ImageNet 256 x 256. Due to the current lack of a fast implementation for W4A8, we implement MP scheme with W8A8 and W4A4 kernels.
Bitwidth
Model Size
Time
Model
FID
sFID
(W/A)
(MB)
(s)
LDM-4
32/32
1603.35
5.05
7.10
5.46
(5teps=250
8/8
430.06
4.02
5.81
2.68
eta=1.0
MP
234.51
6.44
8.43
2.45
scale=1.5)
4/4
234.51
1.63
6 Conclusion and Future Work
In this paper, we have proposed PTQD, novel post-training quantization framework for diffusion models that unifies the formulation of quantization noise and diffusion perturbed noise. start with, we have disentangled the quantization noise into correlated and residual uncorrelated parts relative to its full-precision counterpart. To reduce mean deviations and additional variance in each step, the correlated part can be easily corrected by estimating the correlation coefficient. For the uncorrelated part, we have proposed Variance Schedule Calibration to absorb its additional variance and Bias Correction to correct the mean deviations. Moreover, have introduced Step-aware Mixed Precision to adaptively select the optimal bitwidth for each denoising step. By incorporating these techniques, PTQD has achieved significant performance improvement over existing sfate-of-the-att post- training quantized diffusion models, with only a 0.06 FID increase compared to the full-precision further
LDM-4 on ImageNet 256 256 while saving 19.9 bit-operations. In the future, quantize other components within diffusion models, such as the text encoder and image decoder, to achieve higher compression ratios and accelerated performance. We may also extend PTQD wider range of generative tasks to assess its efficacy and generalizability.
Limitations and Broader Impacts. The proposed PTQD framework stands out for its high efficiency and energy-saving properties, which carry significant implications in reducing the carbon emissions attributed the widespread deployment of diffusion models. However, similar other deep generative models, PTQD has the potential to be utilized for producing counterfeit images and videos for malicious purposes.
Acknowledgement This work was supported by National Key Research and Development Program of China (2022YFC360260))
10

---

References
[1] F. Bao. C. Li, J. Sun, J. Zhu, and B. Zhang. Estimating the optimal covariance with imperfect mean in diffusion probabilistic models. In ICMI, 2022.
[2] F. Bao, C. Li. J. Zhu, and B. Zhang. Analytic-dpm: an analytic estimate of the optimal reverse variance in diffusion probabilistic models. In ICLR, 2022.
N.
Int4 for inference. int4 1at4-for-ad -oor-aa wwt---aininfeeee////oeeo May 2021. Accessed on: May 1, 2023.
Blog.
https://developer: www.pi...iiii................... www.ia...o/bblooogoeeeteeooeeeee
[4] Y. Cai, Z. Yao, Z. Dong, A. Gholami, M. W. Mahoney, and K. Keutzer. Zeroq: A novel zero shot quantization framework. CVPR, 2020.
[5] N. Chen, Y. Zhang, H. Zen, R. J. Weiss, M. Norouzi, and W. Chan. Wavegrad: Estimating gradients for waveform generation. In ICLR, 2021.
[6] W. Chen, P. Wang. and J. Cheng. Towards mixed-precision quantization of neural networks via constrained optimization. In ICCV, 2021.
[7] H. Chung, B. Sim, and C. Ye, Come-clnser-diffuee
Accelerating conditional diffusion models for inverse problems through stochastic contraction. In CVPR, 2022.
[8] H. Chung and J. C. Ye. Score-based diffusion models for accelcrated mri. Medical Image Analysis, 80: 80:102479, 2022.
[9] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei. Imagenet: A large-scale hierarchical image database. In CVPR, 2009.
[10] P. Dhariwal and A. Q. Nichol. Diffusion models beat gans on image synthesis. In NeurlPS, pages 8780-8794, 2021.
[11] Z. Dong, Yao, A. Gholami, M. W. Mahoney, and K. Keutzer. Hawq: Hessian aware quantization of neural networks with mixed-precision. In /CCV, 2019.
[12] G. Franzese, S. Rossi, Yang. Finamore, D. Rossi, M. Filippone, and P. Michiardi. How much enough? study on diffusion times in score-based generative models. arXiv preprint arXiv:2206.05173, 2022.
[13] G. Giannone, Nielsen, and Winther. Few-shot diffusion models. arXiv preprint arXiv:2205.15463. 2022.
[14] R. Gong, X. Liu, S. Jiang, T. Li, P. Hu, J. Lin, F. Yu, and J. Yan. Differentiable soft quantization: Bridging full-precision and low-bit neural networks, ICCV, 2019.
[15] 1. Goodfellow, J.
M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio. Generative adversarial networks. Communications the ACM, 63(11):139 144, 2020.
[16] M. Heusel, H. Ramsauer. T. Unterthiner, B. Nessler, and S. Hochreiter. Gans trained by a two time-scale update rule converge to local nash equilibrium. In NewrIPS, volume 30, 2017.
[17] Ho, A. Jain, and P. Abbeel. Denoising diffusion probabilistic models. In NeurlPS, 2020.
[18] J. Ho, T. Salimans, A. A. Gritsenko, W. Chan, M. Norouzi, and D. J. Fleet. Video diffusion models. In NeurIPS, 2022.
[19] X. Huang, 7. Shen, S. Li, 7, Liu, H. Xianghong, J. Wicaksana, E. Xing, and K.-T. Cheng, Sdq: Stochastic differentiable quantization with mixed precision. In ICMI, 2022.
[20] Hubara, Nahshan, Y. Hanani, R. Banner, and D. Soudry. Improving post training neural quantization:
Layer-wise calibration and integer programming. arXiv preprint arXiv:2006. 10518, 2020.
[21] B. Jacob, Kligys, Chen, Zhu, Tang, A. Howard, Adam, and D. Kalenichenko. Quantization and training of neural networks for efficient integer-arihhmeti-onll inference. CVPR, 2018
[22] B. Jacob, S. Kligys, B. Chen, M. Zhu, M. Tang, A. Howard, H. Adam, and D. Kalenichenko. Quantization and training of neural networks for efficient integer-aritmmeciconnly inference. CVPR, 2018.
[23] A. generating data with score-based models. arXiv preprint arXiv:2105. 100.000000. 2021.
K. Li, R. Piché-Taillefer. T. Kachman, and I. Mitliagkas. Gotta fast when
[24] A. Kerr, D. Merrill, J. Demouth, and J. Tran. Cutlass: Fast linear algebra in cuda c++. NVIDIA Developer Blog. 2017.
[25] B. Kim and C. Ye. Denoising meme accelerating diffusion- based generative models. ICML, 2023.
[26] D. Kingma, T. Salimans, B. Poole, and J. Ho. Variational diffusion models. In NewrIPS,
[27] D. Kingma and M. Welling. Auto-encoding variational bayes. ICLR, 2014.
[28] Z. Kong and W. Ping. On fast sampling of diffusion probabilistic models. In ICLR, 2023.

---

[29] M. W. Y. Lam, J. Wang, D. Su, and D. Yu. BDDM: bilateral denoising diffusion models for fast and high-quality speech synthesis. In ICLR, 2022.
[30] A. Levkovitch, E. Nachmani, and L. Wolf. Zero-shot voice conditioning for denoising diffusion TTS models. In Interspeech, pages 1983-29877, 2022.
[31] X. Li. Y. Liu. L. Lian, H. Yang, Z. Dong, D. Kang. S. Zhang, and Keutzer. Q-diffusion: Quantizing diffusion models. ICCV, 2023.
[32] Y. Li, R. Gong, X. Tan, Y. Yang, P. Hu, Q. Zhang, F. Yu, W. Wang, and S. Gu. BRECQ: pushing the limit of post-training quantization by block reconstruction. In ICLR, 2021.
[33] Y. Lin, T. Zhang, P. Sun, Z. Li, and S. Zhou. Fq-vit: Post-training quantization for fully quantized vision transformer. In IJCAI, 2022.
[34] L. Liu, Y. Ren, Lin, and Zhao. Pseudo numerical methods for diffusion models on manifolds. In ICLR, 2022.
[35] C. Louizos, M. Reisser, T. Blankevoort, E, Gavves, and M. Welling. Relaxed quantization for discretized neural networks. ICLR, 2019.
[36] C. Lu, Y. Zhou, F. Bao. J. Chen, C. Li, and J. Zhu, Dpm-solver: A fast ODE solver for diffusion probabilistic model sampling in around 10 steps. In NewrIPS, 2022.
[37] E. Luhman and Luhman. Knowledge distillation in iterative generative models for improved sampling speed. arXiv preprint arXiv:2101.0224 2021.
[38] S. Luo and W. Hu. Diffusion probabilistic models for 3d point cloud generation. In CVPR, 2021.
[39] 7. Lyu, X. Xu, C. Yang, D. Lin, and B. Dai. Accelcrating diffusion models via early stop of the diffusion /2524, process. arXiv preprint arXiv:2205. 2222222222.222 2022.
[40] Nagel, R. A. Amjad, M. van Baalen, C. Louizos, and T. Blankevoort. Up or down? adaptive rounding for post-training quantization. In ICML, volume 119, pages 7197-7206, 2020.
[41] M. Nagel, M. van Baalen, T. Blankevoort, and M. Welling. Data-free quantization through weight equalization and bias correction. In ICCV, pages 1325-1334, 2019.
[42] C. Nash, J. Menick, S. Dieleman, and P. W. Battaglia. Generating images with sparse representations. In ICML, volume 139, pages 7958-7968, 2021.
[43] A. Q Nichol and P. Dhariwal. Improved denoising diffusion probabilistic models. In ICML, 2021.
[44]
NVIDIA Corporation. TensorRT. https: htdee://deeeeeeeee nvidia com/tensorrt, 2019. [Online; ac- cessed 30 April 2023].
[45] R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer. High-resolution image synthesis with latent diffusion models. In CVPR, pages 10674-10685, 2022.
[46] T. Salimans, I. Goodfellow, W. Zaremba, Cheung, A. Radford, and X. Chen. Improved techniques for training gans. In NeurlPS, volume 29, 2016.
[47] T. Salimans and J. Ho. Progressive distillation for fast sampling of diffusion models. In ICT.R, 2022.
[48] Y. Shang, Z. Yuan, B. Xie, B. Wu, and Y. Yan. Post-training quantization on diffusion models. arXiv preprint arXiv:2211.1573: 2022.
[49] C. Shi, S. Luo, M. Xu, and J. Tang. Learning gradient fields for molecular conformation generation. In ICML, 2021.
[50] J. Song, C. Meng, and S. Ermon. Denoising diffusion implicit models. In ICLR, 2021.
[51] Y. Song. J. Sohl- Dicksiiss,,,,, D. P. Kingma, A. Kumar, S. Ermon, and B. Poole. Score-based generative modeling through stochastic differential equations. In /CLR, 2021.
[52] V. Voleti, A. Jolicoeur MaaneeurrMattiee Martineau, and Pal. MCVD masked conditional video diffusion for prediction. generation, and interpolation. In NeurlPS, 2022.
[53] P. Wang, Q. Chen, X. He, and J. Cheng. Towards accurate post-training network quantization via bit-split and stitching. In JCML, 2020.
[54] Y. Wang, Y. Lu, and T. Blankevoort, Differentiable joint pruning and quantization for hardware efficiency. In ECCV, 2020.
[55] D. Watson, W. Chan, J. Ho, and M. Norouzi. Learning fast samplers for diff'usion models by differentiating through sample quality. In ICLR, 2022.
[56] D. Watson, Ho. M. Norouzi, and W. Chan. Learning efficiently sample from diffusion probabilistic models. arxiv preprint arXiv:2106. 03802, 2021.
[57] x. Wei, R. Gong. Li, X. Liu, and F. Yu. Qdrop: Randomly dropping quantization for extremely low-bit post-training quantization. In ICLR, 2022.
12

---

[58] W. Wu. Y. Zhao, M. Z. Shou, H. Zhou, and C. Shen. Diffumask: Synthesizing images with pixel-level annotations for semantic segmentation using diffusion models. arXiv preprini arXiv:2303. 1681, 2023.
[59] H. Yang, L. Duan, Chen, and H. Li. BSQ: exploring bit-level sparsity for mixed-precision neural network quantization. In ICLR, 2021.
[60] F. Yu, Seff, Zhang, Song. T. Funkhouser. and Xiao. Lsun: Construction of large-scale image dataset using deep learning with bumans in the loop. arXiv preprint arXiv:1506.0336. 2015.
[61] L. Zhang, Y. He, Z. Lou, X. Ye, Y. Wang, and H. Zhou. Root quantization: self-adaptive supplement ste. Applied Intelligence, 53(6):226 6275, 2023.
[62] Q. Zhang and Y. Chen. Fast sampling of diffusion models with exponential integrator. arXiv preprint arXiv:2204. 13902, 2022.
[63] Q. Zhang, M. Tao, and Y. Chen. gddim: Generalized denoising diffusion implicit models. ICLR, 2023.
[64] H. Zheng, P. Hc, W. Chen, and M. Zhou. Truncated diffusion probabilistic models. star, 1050:7, 2022.
[65] B. Zhuang, C. Shen, M. Tan, L. Liu, and I. Reid. Towards effective low- www-bitwdthh convolutional neural networks. In CVPR, 2018.
13


Reviews:
[{"review_id": "RuvoXxsFzE", "reviewer": "Reviewer_i5nq", "summary": "This paper introduces PTQD, a novel method designed to tackle issues arising when applying existing post-training quantization techniques directly to low-bit diffusion models. The proposed approach disentangles quantization noise into its correlated and residual uncorrelated components regarding its full-precision counterpart, enabling separate correction for each part . Moreover, the authors present Step-aware Mixed Precision, a scheme that dynamically selects optimal bitwidths for individual denoising steps. Extensive experiments on three image datasets demonstrate significant improvements in image quality compared to the baseline.", "questions": "See weakness.", "limitations": "None.", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. PTQD presents a unique method that disentangles quantization noise and addresses it separately, while Step-aware Mixed Precision dynamically optimizes bitwidths for synonymous steps, demonstrating a comprehensive approach to quantization in diffusion models.\n2. The experimental results provide strong evidence of PTQD's effectiveness in enhancing image quality, validating its practical value.\n3. The paper is well-organized and clearly presented, with the supplementary file extending PTQD to DDIM and including a statistical analysis of residual quantization noise, which bolsters the credibility of their work.", "weaknesses": "1. To further validate PTQD's performance, it would be beneficial to include comparisons with more competitive methods, such as recent DDPM variants, in the experiments.\n2. Evaluating PTQD with other post-training quantization methods on diverse image datasets would enhance its applicability and demonstrate its effectiveness across various tasks, making the findings more robust.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks to the reviewer for the valuable comments.\n\n**Q1: Include comparisons with recent DDPM variants and evaluating PTQD with other post-training quantization methods on diverse image datasets in the experiments.** Table D in the rebuttal PDF presents the results on a **new dataset CelebA-HQ** over recent **DDPM variants PLMS [i]**, demonstrating the strong performance of PTQD under this configuration. Notably, the proposed PTQD reduces the FID and sFID by a considerable margin of $3.23$ and $4.73$ in comparison to Q-Diffusion, respectively. Additionally, we include a comparison with the PTQ method PTQ4DM [ii] on the LSUN-Bedrooms dataset, as shown in Table E in the rebuttal PDF. Remarkably, our proposed approach outperforms PTQ4DM in both W4A8 and W3A8 bitwidth scenarios. The full results will be included in the revised version.\n\n[i] Liu, Luping, et al. \"Pseudo numerical methods for diffusion models on manifolds.\" ICLR 2022.\n\n[ii] Shang, Yuzhang, et al. \"Post-training quantization on diffusion models.\" CVPR 2023."}, {"review_id": "0QpBWL7cJD", "reviewer": "Reviewer_LoUy", "summary": "The paper proposed the quantization scheme for Diffusion models, where they disentangled the quantization noise into the correlated and uncorrelated parts;\nThen, they incorporated the correlated part into diffusion-perturbed noise and calibrated the denoising variance schedule to absorb additional variance into diffusion noise.\nThey also suggested a mixed-precision quantization scheme for handling the difference in activations variance for each time step.", "questions": "1.\tIt would be helpful if you the match the technique the author proposed with the description of the method sections. The terms of CNC, VSC and BC was suddenly advent in the result section.\n2.\tIt would make it easy to follow your work if you provide your algorithm in the manuscript.\n3.\tcould you provide how many bit-width is allocated for each step for a specific model. \n4.\tcould you provide the result when setting k as arbitrarily?\n5.\tWhy did you do the ablation study with mixed precision?\n6.\tIn your result, the mixed-precision scheme seems not well worked out for Q-diffusion. could you provide the reasons?\n7.\tIn Section 4.1, did you want to explain that quantization noise is divided into the correlated part and unrelated part due to normalization layers? Or is it just an example?\n8.\tIn line 180, what are the bias and additional variance? Please notate them with symbols for clarity", "limitations": "There seems to have still room for improving the readability and clarity in the manuscript.", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "It is a novel point of view in incorporating quantization noise into diffusion noise. \nIt also sounds good to me to utilize the bias correction proposed in DFQ to handle the uncorrelated part.", "weaknesses": "It does not seem to show noticeable improvement compared to existing works, Q-diffusion.\nIt seems to require more experiments to show their performance. The ablation study was performed only when quantizing with mixed precision.\nThe mixed-precision scheme has any special things; the effect seems to be insignificant according to table 2.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks to the reviewer for the valuable comments.\n\n**Q1: It does not seem to show noticeable improvement compared to existing works.** 1) It is essential to consider that the absolute performance **improvement is closely related to the precision** of the model. When higher bitwidths are employed, the absolute performance gains may appear relatively small, because the model's performance is already in close proximity to the full-precision counterpart. **As the bitwidth decreases, the efficacy of our approach becomes more noticeable**, particularly in the scenarios where W4A4 bitwidth is utilized, as referred to the results of mixed precision in Tables 1-3 in the paper. Notably, our method substantially reduces the FID score from 218.59 to 17.99 on the LSUN-Churches dataset. In addition, we conducted experiments using lower bitwidth W3A8 to demonstrate the extent of our improvement, as shown in Table A in the attached rebuttal PDF. The experimental results show that our method can bring greater improvement at W3A8 bitwidth on LSUN-Bedrooms dataset, resulting in a noticeable reduction of $1.85$ and $4.02$ in FID and sFID, respectively.\n\n2\\) It is worth noting that while FID provides an informative metric, it might not **holistically capture the improved image quality**. In the supplementary material, we have provided visualizations in Figures H-J that convincingly showcase the superiority of results produced by PTQD. These visualizations underscore higher image quality and a closer resemblance to samples generated by the full-precision model, in stark contrast to Q-Diffusion outputs.\n\n**Q2: Match the proposed techniques with the description of the method sections.** Thanks for the valuable comments. Correlated Noise Correction (CNC) is proposed in section 4.2.1 of the paper, which corrects the correlated part of the quantization noise. Both Bias Correction (BC) and Variance Schedule Calibration (VSC) are proposed in section 4.2.2 to correct the uncorrelated quantization noise, as referred to lines 194-201 of the paper. We will incorporate your suggestions in the revised version.\n\n**Q3: Provide the algorithm in the manuscript.** The algorithm is briefly summarized below and will be included in the revised version.\n\n**Before sampling:**\n| Algorithm Step | Description                                                                                         |\n|------|-----------------------------------------------------------------------------------------------------|\n| 1    | Quantize diffusion models with BRECQ [i] (or other PTQ method).                                   |\n| 2    | Generate samples with both quantized and FP model and collect quantization noise.                 |\n| 3    | Calculate the correlated coefficient $k$ based on Eq. (7), and the mean and variance of the residual quantization noise as per Eq. (10). |\n\n**For each sampling step:**\n| Algorithm Step | Description                                                                                         |\n|------|-----------------------------------------------------------------------------------------------------|\n| 4    | Correct the correlated part of the quantization noise by dividing the output of the noise prediction network by $1+k$. |\n| 5    | Calibrate the variance schedule by Eq. (12) and subtract the channel-wise biases from the output of quantized noise prediction network. |\n\n**Q4: How many bit-width is allocated for each step for a specific model?** As referred to lines 229-234 in the paper, the bitwidth for each step is determined by comparing $\\rm{SNR}^Q$ with $\\rm{SNR}^F$ as per Eq. (15). The results of bitwidth allocation for each dataset are presented below and will be included in the final version.\n\n| Dataset               | W4A4 Step Range | W4A8 Step Range |\n|-----------------------|-----------------|-----------------|\n| ImageNet (250 steps)  | 249 to 202     | 201 to 0        |\n| ImageNet (20 steps)   | 19 to 15       | 14 to 0         |\n| LSUN-Bedrooms         | 199 to 155     | 154 to 0        |\n| LSUN-Churches         | 199 to 146     | 145 to 0        |\n\n\n**Q5: Provide the result when setting $k$ arbitrarily.** As shown in the table below, setting $k$ arbitrarily can greatly impair the quality of generated samples. As referred to lines 168-171 and Eqs. (7)-(9) of the paper, if $k$ is set arbitrarily, the correction of the correlated quantization noise can be inaccurate and there will still be correlation between the remaining quantization noise and the output of the noise prediction network.\n\n| Model      | Method      | FID    | sFID   |\n|------------|-------------|--------|--------|\n| LDM-4      | Q-Diffusion | 6.72   | 18.80  |\n|            | Random $k$  | 18.98  | 46.52  |\n|            | Ours        | **5.94** | **15.16** |\n\n**Q6: Why ablation study is conducted with mixed precision?** Please refer to Q2 in the general response.\n\n**Q7: Why the mixed-precision scheme did not work well for Q-Diffusion?** This can be attributed to the substantial quantization noise inherent in the W4A4 bitwidth. In the absence of our correction method, the excessive noise becomes a hindrance, preventing Q-Diffusion from generating samples of desirable quality.\n\n**Q8: In Section 4.1, is the quantization noise divided into the correlated part and unrelated part due to normalization layers?** As referred to lines 149-163 of the paper, we prove that at least a portion of the correlation comes from the normalization layer. It is plausible that other nonlinear layers within the model could also contribute to this correlation.\n\n**Q9: In line 180, what are the bias and additional variance?** The bias and additional variance refer to the mean and variance of the residual quantization noise, which are denoted in Eq. (10) of the paper. We will make the descriptions consistent in the revised version.\n\n[i] Li, Yuhang, et al. \"Brecq: Pushing the limit of post-training quantization by block reconstruction.\" ICLR 2021.\n"}, {"review_id": "hesl5DnByU", "reviewer": "Reviewer_oS4i", "summary": "The authors propose a new post-training quantization method for diffusion models titled PTQD that disentangles the quantization noise into correlated and uncorrelated parts regarding its full-precision counterpart, and demonstrate that PTQD generates as much high-quality samples as its full-precision counterpart.", "questions": "N/A", "limitations": "N/A", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "(1) The idea of disentangling the quantization noise into correlated and uncorrelated parts regarding its full precision counterpart seems to be novel.\n\n(2) PTQD shows better performance than previous methods in both class-conditional image generation and unconditional image generation.", "weaknesses": "(1) Although PTQD reduces BOPs greatly, it is doubtful whether the speed of PTQD generating samples is really faster than that of its full-precision counterpart in real time.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks to the reviewer for the valuable comments.\n\n**Q1: The speed of PTQD in real time.** We have measured the latency of matrix multiplication and convolution operations in quantized and full-precision diffusion models using an RTX3090 GPU, as presented below. Both floating-point and quantized operations are implemented with CUTLASS. When both weights and activations are quantized to 8-bit, we observe a **2.03$\\times$** reduction in latency compared to its full-precision counterpart over LDM-4. Moreover, when weights and activations are quantized to 4-bit, the speedup further increases to **3.34$\\times$**. The mixed-precision settings explored in our experiments strike a good balance between latency and model performance.\n\n**Comparisons of time cost across various bitwidth configurations on ImageNet 256$\\times$256.** Due to the current lack of a fast implementation for W4A8, we implement MP scheme with W8A8 and W4A4 kernels.\n\n| Model              | Bitwidth (W/A) | Model Size (MB) | FID   | sFID  | Time (s) |\n|--------------------|----------------|------------------|-------|-------|----------|\n| LDM-4 (steps=250, eta=1.0, scale=1.5) | 32/32 | 1603.35 | 5.05 | 7.10 | 5.46     |\n|                    | 8/8            | 430.06           | 4.02 | 5.81 | 2.68     |\n|                    | MP             | 234.51           | 6.44 | 8.43 | 2.45     |\n|                    | 4/4            | 234.51           | -    | -    | 1.63     |\n\n\n"}, {"review_id": "gZOVhONkmS", "reviewer": "Reviewer_MVe5", "summary": "The paper suggests a method for post-training quantization of diffusion models. The method consists of a factorization of the quantization noise into a correlated and an uncorrelated part, and then addressing each component separately, either by linearly regressing for the correlation coefficient, or incorporating the quantization noise into the diffusion noise level. The authors conduct extensive experiments on multiple datasets, and obtain generation quality comparable to that of the full-precision model in most cases.", "questions": "- What does Figure 2 plot? Axis titles represent vectors. If the numbers plotted are entries in these vectors, it should be noted in the figure caption or in the text.\n- Equation 12 and its following text gloss over the case \"otherwise\". Mathematically, $\\sigma_t^2 = 0$ is *not* a solution for Eq. 11 in this case, and should not be presented as such. When quantization noise becomes larger than $\\sigma_t^2$, especially in the mentioned deterministic case, the proposed method cannot deal with the noise. If it does, a thorough explanation is needed here.", "limitations": "Potential negative impact: Yes.\nLimitations: No.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 1, "contribution": 2, "strengths": "- The characterization of the quantization noise, and incorporating its uncorrelated part into the diffusion noise is a nice idea.\n- The obtained results are impressive.", "weaknesses": "- Related work lacks a specific description of methodical/result-based differences between this work and Q-Diffusion. \"Analyze the quantization effect\" and \"unified framework\" are very broad terms that give little to no context.\n- Lines 170-171: When does this linear regression happen? At training time or test time? What data do you use for this? Details lacking. Edit: Data is provided at the end of page 7, just please mention that details will be presented later from lines 170-171.\n- What is $SNR^F$? What is the motivation behind it? Why is it defined this way? What is its purpose? These details should not be delegated to a citation.\n- The reason not to compare with PTQ4DM sounds unreasonable. We can still compare results even if they do not specify the PTQ method used. Moreover, it seems like PTQ4DM has a public git repo with their code. The PTQ method can be inferred from that. Moreover, Q-Diffusion seems to have a curious failure mode for multi-precision. When considering a single precision level, it seems like the new method is not very different from Q-Diffusion in terms of results.\n- No wall clock time comparison is given, even though slow runtime of diffusion models is touted as one of the main drawbacks of these models.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks to the reviewer for the valuable comments.\n\n**Q1: Methodical/result-based differences between this work and Q-Diffusion.** In terms of methodology, Q-Diffusion designed a calibration data collection method and applied the PTQ method BRECQ [i] to the diffusion model. In sharp contrast, we introduce a unified formulation for quantization noise and diffusion perturbed noise, as referred to Eq. (5) in the paper. We argue that quantization noise alters the mean and variance of the predicted noise in each sampling step, resulting in poor sample quality of the quantized diffusion model. Our approach corrects both correlated and residual quantization noises at every step to mitigate these adverse effects. It can be seamlessly integrated with Q-Diffusion or **any other PTQ method** to consistently enhance their performance.\n\n**Q2: Mention the details of linear regression in lines 170-171.** Linear regression is performed before test time. As referred to lines 258-262 of the paper, we generate 1024 samples using both quantized and full-precision diffusion models to collect the data (the quantization noise and the output of the full-precision noise prediction network) for performing linear regression. We will add the details in front in the revised version.\n\n**Q3: Details of $\\rm{SNR}^F$ should not be delegated to a citation.** $\\rm{SNR}$ is a widely adopted notation of signal-to-noise ratio. As referred to lines 226-229 in the paper and Eqs. (1)-(2) in [i], $\\rm{SNR}^F$ is defined by the hyperparameters of the diffusion forward process, where $\\alpha$ is the coefficient for data and $\\sigma$ is the coefficient for noise. It is first introduced by [i] to note the degree of noise of data at each step. Moreover, as referred to lines 212-213 in the paper, $\\rm{SNR}^Q$ is defined as the SNR of the quantized noise prediction network. We compare these two metrics to select the optimal bitwidth that satisfies the $\\rm{SNR}$ requirement for effective denoising.\n\n**Q4: The reason not to compare with PTQ4DM sounds unreasonable.** Our initial attention was directed towards PTQ4DM [ii] upon its initial publication on arXiv, which did not release its code. The CVPR camera-ready paper was made public after the NeurIPS submission deadline. Additionally, PTQ4DM's experimental scope was confined to lower-resolution datasets, whereas our study encompasses datasets with higher resolutions. Nonetheless, we evaluate PTQ4DM on LSUN-Bedrooms dataset as shown in Table E in the rebuttal PDF. Our method outperforms it under both W4A8 and W3A8 bitwidths. Full results will be included in the revised version.\n\n**Q5: Q-Diffusion seems to have a curious failure mode for multi-precision.** This can be attributed to the substantial quantization noise inherent in the W4A4 bitwidth. In the absence of our correction method, the excessive noise becomes a hindrance, preventing Q-Diffusion from generating samples of desirable quality.\n\n**Q6: Method is not very different from Q-Diffusion in terms of results.** 1) It is essential to consider that the absolute performance **improvement is closely related to the precision** of the model. When higher bitwidths are employed, the absolute performance gains may appear relatively small, because the model's performance is already in close proximity to the full-precision counterpart. **As the bitwidth decreases, the efficacy of our approach becomes more noticeable**, particularly in the scenarios where W4A4 bitwidth is utilized, as referred to the results of mixed precision in Tables 1-3 in the paper. Notably, our method substantially reduces the FID score from 218.59 to 17.99 on the LSUN-Churches dataset. In addition, we conducted experiments using lower bitwidth W3A8 to demonstrate the extent of our improvement, as shown in Table A in the attached rebuttal PDF. The experimental results show that our method can bring greater improvement at W3A8 bitwidth on LSUN-Bedrooms dataset, resulting in a noticeable reduction of $1.85$ and $4.02$ in FID and sFID, respectively.\n\n2\\) It is worth noting that while FID provides an informative metric, it might not **holistically capture the improved image quality**. In the supplementary material, we have provided visualizations in Figures H-J that convincingly showcase the superiority of results produced by PTQD. These visualizations underscore higher image quality and a closer resemblance to samples generated by the full-precision model, in stark contrast to Q-Diffusion outputs.\n\n**Q7: No wall clock time comparison is given.** Please refer to Q3 in the general response.\n\n**Q8: What does Figure 2 plot?** Figure 2 illustrates the correlation between the quantization noise (Y-axis) and the output of the full-precision noise prediction network (X-axis). Each data point on the plot corresponds to specific entries within these vectors. We will clarify this point in the figure caption of the revised version.\n\n**Q9: The proposed method cannot deal with the noise in the deterministic case.** Please refer to Q4 in the general response.\n\n**Q10: Mathematically, Equation 12 is not a solution for Equation 11.** When quantization noise becomes larger than ${\\sigma_{t}^{2}}$, ${\\sigma_{t}^{'2}}=0$ is not an analytical solution of Eq. (11), but an optimal solution in this case. We will revise it in the revised version.\n\n[i] Kingma, Diederik, et al. \"Variational diffusion models.\" NeurIPS 2021.\n\n[ii] Shang, Yuzhang, et al. \"Post-training quantization on diffusion models.\" CVPR 2023.\n"}, {"review_id": "opLGVG9H85", "reviewer": "Reviewer_5dgo", "summary": "The paper introduces PTQD, a Post-Training Quantization framework for Diffusion models.\nPTQD analyzes the influence of quantization noise on diffusion noise. \nThe method suggests separating the quantization noise into noise correlated and uncorrelated with the full-precision reverse diffusion.\nThe correlated part is easily fixed by estimating the correlation coefficient.\nThe residual noise is corrected by modifying the stochastic variance of DM in reverse diffusion.\nFinally, a mixed precision which is time-step aware is proposed based on gathered statistics.", "questions": "1) As far as I know most DDPMs make no use of the stochastic reverse diffusion ($\\sigma_t = 0$). In that case, the contribution and impact are certainly reduced.\n\n2) A visualization of the generated images should be provided in order to ass the impact on both the generation and reconstruction quality.\n\n3) Comparison with regular PTQ should be performed at different bit ranges (not only naive TensorRT's uniform quantization but others such as MMSE etc.)\n\n4) Ablations: The ablation is good but should also be performed at a constant precision (not MP) in order to assess the real impact of each suggestion, independently of the mixed precision correction.", "limitations": "Limitations are ok.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper is clearly written.\\\nThe analysis of DM quantization challenges is interesting, well-described, and well-mathematically defined and analyzed.\\\nThe framework while being very simple (mainly based on gathered statistics) still can outperform previous work.", "weaknesses": "The main weaknesses are as follows:\n\n1) The practical ability of the method to deal with the residual noise. (see question 1))\n\n2) The weak comparisons. (see questions 3-4))\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks to the reviewer for the valuable comments.\n\n**Q1: Contribution and impact in deterministic sampling.** 1) We acknowledge that the efficacy of our contributions encounters constraints for deterministic sampling, as pointed out in line 205 of the paper. However, in the deterministic case, we can still **correct the correlated quantization noise** and the **biases stemming from uncorrelated components**. This corrective capability assumes particular significance in instances involving low bit quantization, as referred to the experimental result of LSUN-Churches in Table 3 in the paper and Figure J in supplementary material. 2) While deterministic sampling has gained widespread adoption, it tends to **result in lower output quality** compared to stochastic sampling [i, ii]. This proposition is also substantiated by empirical observations, as referred to Table C in the rebuttal PDF. Specifically, when generating samples on FFHQ dataset with deterministic DDIM sampler, introducing stochastic perturbations lower both the FID and sFID metrics. For experiments on ImageNet dataset, it greatly improves the IS with little increase in FID and sFID. In the case of stochastic sampling, our method can achieve better performance by calibrating the variance schedule.\n\n**Q2: A visualization of the generated images should be provided.** As referred to Figures H, I, J in the supplementary material, we have provided visualization results on three datasets to substantiate the effects of our proposed method.\n\n**Q3: Comparison with regular PTQ should be performed at different bit ranges.** As referred to Tables A and E in the rebuttal PDF, we have conducted experiments on more bitwidth settings and compared the results with PTQ4DM. Due to the limited time slot of the rebuttal, we will conduct experiments on more PTQ methods and bitwidths and add the results to the revised version.\n\n**Q4: The ablation should also be performed at a constant precision.** As referred to lines 219-220 in the paper, the proposed mixed-precision (MP) scheme allows the utilization of low-bit diffusion models during the sampling process, resulting in a greater speedup in generation. Specifically, we introduce W4A4 in the MP experiment, a **more intricate task** in comparison to the fixed W4A8 quantization due to the larger quantization noise. Moreover, we conduct additional ablation experiments with constant precision, which are outlined in Table A of the attached rebuttal PDF. The experimental results consistently demonstrate performance improvements brought by each component of our method under constant precision settings. Notably, our method exhibits more significant improvements at lower bitwidths (W3A8) due to the inherent presence of greater quantization noise at these levels.\n\n[i] Karras, Tero, et al. \"Elucidating the design space of diffusion-based generative models.\" NeurIPS 2022.\n\n[ii] Song, Yang, et al. \"Score-based generative modeling through stochastic differential equations.\" ICLR 2021."}]

Author rebuttal:
We thank all reviewers for their valuable feedback. Overall, our work has been well recognized as it "is well-organized and clearly presented" (Reviewer i5nq), presents a novel idea" (Reviewer oS4i) and "obtains impressive results" (Reviewer MVe5). We have summarized and addressed the main concerns as follows:

**Q1: Improvement is not noticeable compared to Q-Diffusion.** 1) It is essential to consider that the absolute performance **improvement is closely related to the precision** of the model. When higher bitwidths are employed, the absolute performance gains may appear relatively small, because the model's performance is already in close proximity to the full-precision counterpart. **As the bitwidth decreases, the efficacy of our approach becomes more noticeable**, particularly in the scenarios where W4A4 bitwidth is utilized, as referred to the results of mixed precision in Tables 1-3 in the paper. Notably, our method substantially reduces the FID score from 218.59 to 17.99 on the LSUN-Churches dataset. In addition, we conducted experiments using lower bitwidth W3A8 to demonstrate the extent of our improvement, as shown in Table A in the attached rebuttal PDF. The experimental results show that our method can bring greater improvement at W3A8 bitwidth on LSUN-Bedrooms dataset, resulting in a noticeable reduction of $1.85$ and $4.02$ in FID and sFID, respectively.

2\) It is worth noting that while FID provides an informative metric, it might not **holistically capture the improved image quality**. In the supplementary material, we have provided visualizations in Figures H-J that convincingly showcase the superiority of results produced by PTQD. These visualizations underscore higher image quality and a closer resemblance to samples generated by the full-precision model, in stark contrast to Q-Diffusion outputs.

**Q2: Why conduct ablation study with mixed precision?** As referred to lines 219-220 in the paper, the proposed mixed-precision (MP) scheme allows the utilization of low-bit diffusion models during the sampling process, resulting in a greater speedup in generation. Specifically, we introduce W4A4 in the MP experiment, a **more intricate task** in comparison to the fixed W4A8 quantization due to the larger quantization noise. Moreover, we conduct additional ablation experiments with constant precision, which are outlined in Table A of the attached rebuttal PDF. The experimental results consistently demonstrate performance improvements brought by each component of our method under constant precision settings. Notably, our method exhibits more significant improvements at lower bitwidths (W3A8) due to the inherent presence of greater quantization noise at these levels.

**Q3: Real time speed up of PTQD.** We have measured the latency of matrix multiplication and convolution operations in quantized and full-precision diffusion models using an RTX3090 GPU, as shown in Table B in the rebuttal PDF. Both floating-point and quantized operations are implemented with CUTLASS. When both weights and activations are quantized to 8-bit, we observe a **$2.03\times$** reduction in latency compared to its full-precision counterpart over LDM-4. Moreover, when weights and activations are quantized to 4-bit, the speedup further increases to **$3.34\times$**. The mixed-precision settings explored in our experiments strike a good balance between latency and model performance.

**Q4: The contributions are reduced for deterministic sampling ($\sigma_t=0$).** 1) We acknowledge that the efficacy of our contributions encounters constraints for deterministic sampling, as pointed out in line 205 of the paper. However, in the deterministic case, we can still **correct the correlated quantization noise** and the **biases stemming from uncorrelated components**. This corrective capability assumes particular significance in instances involving low bit quantization, as referred to the experimental result of LSUN-Churches in Table 3 in the paper and Figure J in supplementary material. 2) While deterministic sampling has gained widespread adoption, it tends to **result in lower output quality** compared to stochastic sampling [i, ii]. This proposition is also substantiated by empirical observations, as referred to Table C in the rebuttal PDF. Specifically, when generating samples on FFHQ dataset with deterministic DDIM sampler, introducing stochastic perturbations lower both the FID and sFID metrics. For experiments on ImageNet dataset, it greatly improves the IS with little increase in FID and sFID. In the case of stochastic sampling, our method can achieve better performance by calibrating the variance schedule.

[i] Karras, Tero, et al. "Elucidating the design space of diffusion-based generative models." NeurIPS 2022.

[ii] Song, Yang, et al. "Score-based generative modeling through stochastic differential equations." ICLR 2021.


Meta-review: The paper proposes an interesting concept for diffusion models. The authors successfully answered the concerns of the reviewers and now all think the paper should be accepted. Therefore, I recommend accepting the paper. Clearly, the authors should improve the paper in the revision using the reviewers' comments.
Decision: Accept (poster)