StepbaQ: Stepping backward as Correction for Quantized Diffusion Models
Authors: Yi-Chung Chen
Zhi-Kai Huang
Jing-Ren Chen
Venue: NeurIPS 2024 poster (2024)
Keywords: Diffusion Model; Model Quantization
Primary area: diffusion_based_models
TL;DR: This work introduces a novel perspective that considers quantization error as a stepback in the denoising process; through a sampling step correction mechanism it improves the performance of quantized diffusion models.

Abstract: Quantization of diffusion models has attracted considerable attention due to its potential to enable various applications on resource-constrained mobile devices. However, given the cumulative nature of quantization errors in quantized diffusion models, overall performance may still decline even with efforts to minimize quantization error at each sampling step.
Recent studies have proposed several methods to address accumulated quantization error, yet these solutions often suffer from limited applicability due to their underlying assumptions or only partially resolve the issue due to an incomplete understanding.
In this work, we introduce a novel perspective by conceptualizing quantization error as a "stepback" in the denoising process. We investigate how the accumulation of quantization error can distort the sampling trajectory, resulting in a notable decrease in model performance. To address this challenge, we introduce StepbaQ, a method that calibrates the sampling trajectory and counteracts the adverse effects of accumulated quantization error through a sampling step correction mechanism. Notably, StepbaQ relies solely on statistics of quantization error derived from a small calibration dataset, highlighting its strong applicability.
Our experimental results demonstrate that StepbaQ can serve as a plug-and-play technique to enhance the performance of diffusion models quantized by off-the-shelf tools without modifying the quantization settings. For example, StepbaQ significantly improves the performance of the quantized SD v1.5 model by 7.30 in terms of FID on SDprompts dataset under the common W8A8 setting, and it enhances the performance of the quantized SDXL-Turbo model by 17.31 in terms of FID on SDprompts dataset under the challenging W4A8 setting.

StepbaQ: Stepping backward as Correction for Quantized Diffusion Models
Yi-Chung Chen" MediaTek, Purdue University chen5262 chen5262@purdue.cc edu
Zhi-Kai Huang
MediaTek
brent5481 0114488@@mmal...om
Jing-Ren Chen
MediaTekk
j iinreen... www.eeeeiattt.ccommm
Abstract
Quantization diffusion models has attracted considerable attention due its potential to enable various applications on resource-constrained mobile devices. However, given the cumulative nature of quantization errors in quantized diffusion models, overall performance may still decline even with efforts to minimize quanti- zation error at each sampling step. Recent studies have proposed several methods to address accumulated quantization errrr, yet these solutions often suffer from limited applicability due to their underlying assumptions or only partially resolve the issue due to incomplete understanding. In this work, introduce novel perspective by conceptualizing quantization error as a "stepback" the denoising process. We investigate how the accumulation of quantization error distort the sampling trajectory, resulting in a notable decrease in model performance. To address this challenge, introduce StepbaQ, a method that calibrates the sampling trajectory and counteracts the adverse effects of accumulated quantization error through sampling step correction mechanism. Notably, StepbaQ relies solely on statistics of quantization eтror derived from a small calibration dataset, highlighting strong applicability. Our experimental results demonstrate that StepbaQ can serve as plug-and-play technique to enhance the performance of diffusion models quantized by off-the-shelf tools without modifying the quantization settings. For example, StepbaQ significantly improves the performance of the quantized SD v1.5 model by 7.30 in terms of FID on SDprompts dataset under the common W8A8 setting, and it enhances the performance of the quantized SDXL-Turbe model by 17.31 in terms of FIID on SDprompts dataset under the challenging W4A8 setting.
Introduction
Diffusion models wide variety of applications including super resolution [53 58) image enhancement 33
23 49
have demonstrated its power of generating high quality samples in
54] image
inpainting [56], image editing (28) and image-to-imant translation (32. In contrast to GAN [8]
and VAE 30 which are prone issues such as mode and posterior collapse, diffusion models consistently produce diverse, high-quality samples, thus emerging as the predominant technique in the field of image generation. However, the deployment diffusion models on computationally constrained devices, such smartphones, is impeded by their extensive computational demands, which stem from the complex network structures and the multitude of denoising steps required during the sampling phase. To enhance the computational efficiency of diffusion models, body of research
"This work was conducted during Yi-Chung's time MediaTek.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

has concentrated on reducing the number of sampling steps [4 39 41 46]. Concurrently, another strand of investigation has pursued model compression strategies, such as quantization 47 33 48
19
20 52 and pruning [14], to diminish the computational resources necessitated. In this paper, we focus on improving the performance of model quantization.
57
24
Model quantization [31 [6] stands as one of the most popular model compression techniques. This technique transits model parameters and activations from a high bit-width floating-poin format to a more compact low bit-width representation, thereby facilitating a substantial acceleration of model inference with a tolerable drop in performance. However, quantizing diffusion models leads to greater performance degradation compared to standard deep learning models due to two challenges: Activation range varies across timesteps. Diffusion models generate samples from a normal distribution through a multi-step denoising process, wherein noise levels are gradually reduced following predefined schedule. This mechanism results in significant disparity in the activation range across different timesteps, as noted in studies by Shang et al. 477 and Li et al. [3). Such variability poses a challenge in establishing an appropriate quantization range. Opting for a broader interval may mitigate clipping errors at the expense of increased rounding errors and vice versa. Prior research by Shang et al. [47] and Li et al. (33] have proposed sampling strategies to select representative calibration samples over multiple timesteps. To further reduce quantization errors, So et al. [48]. He et al. [9], and Yang et al. (57] relax the quantization constraints, adopting techniques such cenestep-specifeee quantization parameters and mixed precision quantization. However, since these methods do not consider the accumulation of quantization errors, they only achieve mediocre performance. ii) Quantization error accumulates over time. The iterative nature of the denoising process in diffusion models means that quantization errors from each step can accumulate. Although emrors individual sampling steps may appear negligible, the cumulative effect can significantly impair the final sample quality. To mitigate this error propagation, Huang et al. 24 have underscored the significance of temporal features and introduced temporal information-awe method to prevent deviations from the intended sampling trajectory caused by temporal information mismatch. Conversely, He et al. 200 have presented PTQD, which employs variance schedule calibration to integrate the quantization error at each step into the Gaussian diffusion noise of stochastic samplers. thereby addressing the accumulation issue. However, the approach proposed by Huang et al. [24] only resolves the disturbance of temporal features without considering the distribution shift in the latent space. On the other hand, the strategy of error absorption introduced
reconstruction
by He et al. (20) is limited to stochastic samplers, thereby excluding its application to deterministic samplers, such as DDIM 44
In this work, our goal is to devise a general approach capable of addressing the issue of error accumulation and enhancing the performance of the quantized diffusion model. We investigate the mechanisms by which accumulated errors can detrimentally impact overall model performance. Our research reveals that quantization error can alter the distribution of the latent variables. This shift in the distribution can lead to divergence from the original sampling trajectory, ultimately causing a decline in model performance. Intriguingly. we show that with the assumption of quantization error following Gaussian distribution, the distribution shift in the latent space caused by quantization error can be interpreted as "stepback" in the denoising process.
Building upon this insight, we introduce StepbaQ, a novel method that uses a sampling step correction technique to minimize the deviation from the original sampling trajectory. StepbaQ analyzes quanti- zation errors to measure the extent of the distribution shifts and subsequently determine the "corrected sampling steps." We demonstrate that with appropriate modifications based on the corrected sampling steps, StepbaQ can effectively correct the sampling trajectory and mitigate the accumulation of quan- tization error, thereby significantly enhancing the quality of the generated results. It is worth noting that StepbaQ only necessitates the statistics of quantization error derived from a small calibration dataset. Consequently, it can be seamlessly integrated with existing quantization frameworks as a plug-and-pla solution, enhancing the performance of diffusion models quantized by off-the-shel tools without modifying their quantization settings.
The contributions of this paper are summarized as follows:
We introduce a novel perspective that interprets the distribution shift in the latent space, caused by quantization error, as a "stepback" in the denoising process. We demonstrate that such temporal shift can alter the sampling trajectory and adversely affect the generated results.

---

We propose StepbaQ. general strategy designed to enhance the performance of quantized diffusion models. This method employs a sampling step correction technique to realign the sampling trajectory and eliminate the accumulation of quantization error.
Extensive experiments show that StepbaQ can serve as a plug-and-play technique, significantly improving the performance of diffusion models quantized by off-the-shelf tools, and achieve the state-of-the-ard performance for quantized diffusion models.
2 Related Works
2.1 Efficient Diffusion Models
Although diffusion models have powerful generative capabilities, slow inference speeds severely limit their practical application. Various approaches have been developed to improve the efficiency of diffusion models. Rombach et al. [45] speeds up each denoising step by transferring the denoising process to the latent space; Song et al. 49 modifies the original diffusion process with a non- Markovian formulation, enabling fewer sampling steps and deterministic sampling. Lu et al. [39] further reduce the required number of steps by introducing an exact solution formulation for diffusion ODEs. Luo et al. [1 applies consistency model to the latent space, enabling fast sampling with just a few steps or potentially a single step. Sauer et al. [46] leverages adversarial training on the diffusion model, enabling the network to generate images in one step, just like GAN [18]. While the methods above can effectively speed up the inference process of diffusion models, deploying these models on devices with limited computational resources necessitates reduced computation and memory usage, which can be achieved via model quantization.
2.2 Diffusion Model Quantization
Model quantization technique that effectively reduces memory and computation costs required for deep learning models. It can be divided into two categories: Quantization-Awee Aware Training (QAT)
26 which mitigates performance drop after quantization through model fine- tuning at the ex expense of substantial computational cost, and Post- Dost-Training Quantization (PTQ) [42]
13 17
34
+ I0, which requires only a small calibration dataset and therefore is ideal for large models like diffusion models. While prior works have achieved considerable progress, directly applying conventional quantization methods on diffusion models does not yield good results due to challenges such as varying activation range across steps and error accumulation. Shang et al. [47] and Li et al. 33 propose sampling strategies for selecting representative calibration samples across steps to alleviate the issue of varying activation ranges. Some methods relax the quantization constraints and use finer granularity to enhance quantization results. For example, Yang et al. [57) leverage SQNR as metric to find sensitive blocks and quantize them with higher precision. Li et al. 355, So et al. 48). and He et al. [9] employ distinct sets of quantization parameters at various timesteps to address the drastic changes in activation ranges. While effective, these strategies incur additional computational overhead, which reduces their practicality. Moreover, even when efforts are made minimize quantization error at each sampling step, the overall performance still suffers due to error accumulation. To address the issue of error accumulation, Huang et al. (24) propose temporal information aware reconstruction, which alleviates temporal feature disturbance, prevent deviation from the original sampling trajectory. On the other hand, Tang et al. (50) introduces a progressive calibration strategy minimize accumulation eиror. Aside from the approaches above, the method PTQD, proposed by He et al. [20]. is closely related to our work. Both methods utilize the statistics of quantization error to enhance the performance of quantized diffusion models. They can be considered complementary techniques integrated with existing quantization approaches. PTQD decomposes the quantization error into correlated and uncorrelated parts. The correlated part eliminated by simple division, while the uncorrelated part is absorbed into the scheduled Gaussian noise. Though this strategy effective, its practicality is restricted. The design of the error absorption mechanism is only compatible with stochastic samplers, such as DDPM 23), which narrows its applicability. In contrast, our proposed StepbaQ, utilizing a sampling step correction technique, applicable to both deterministic and stochastic samplers.
25
55 15
3

---

fx
(a) Model
Deook ng Sempling
1xx 6. Tw
(b) Model w/o Stepbact
Cuonitztion (Septadd
Tea
[c] Q Model w/ StapheQ
Figure 1: Overview of the denoising process of StepbaQ and existing methods. Figure (a) shows the original denoising process. Figure (b) demonstrates the negative impact of quantization error without changing the step size, leading to significant accumulation error. Figure (c), on the other hand, illustrates how StepbaQ treats the quantization error as a stepback in the denoising process and adopts corrected steps with a larger step size to eliminate cumulative quantization error.
Preliminary
Diffusion models 23 49 are latent variable models characterized by forward process, wherein Gaussian noise of a predetermined magnitude is incrementally applied to the original data zo. The goal is to learn a reverse process that gradually removes noise from latent to generate high-quality images. The equations of the posterior and Gaussian transitions are defined follows:
((11:)) II (()((111),
11-1, 1, (1
I)
0t-1
0:-1
Here, ..o..... aM. represents descending sequence of hyperparameters that schedule the noise level, as described in [49] Through the reparameterization technique [40], we can express 2t as:
Varzo + vI a'e,
E~ c0,1
(2)
The aim of diffusion model to learn a parameterized distribution pe (xp) that approximates ((zo) = f pe (Zo. ]d and is easy to sample from. The parameters 0 are trained by optimizing variational lower bound to accurately predict the added noise, outlined in the following objective:
||c colVarra + vI -l-- t)(,
While the forward process has M steps, the denoising process can employ a condensed sampling path by choosing a sub-sequence length N from the set [1...., .M] to expedite the sampling process. DDIM [49] presents a generalized formula to generate a sample - from sample 2t via:
Et
vI (Ic t)
11 a- of of oees, 4 ~~~ ~(0 (4)
Tt-1
ar
Varying the choices of parameter or yields different denoising processes. Specifically, setting
1-0--1 I-as results in DDPM (3 whereas a, of corresponds to DDIM 49
0t
In previous literature, the notation E [1...., N] has been conventionally used to index the sampling steps in the denoising process. To distinguish between the original sampling steps and the that StepbaQ has modified, we employ the notation |41,-.,44 to represent the original sampling steps, while [T.,..., TN] is used to denote the steps post-correction.
4 Method
In this section, we first present novel perspective that interprets the distribution shift in the latent space, resulting from quantization errors, as a "stepback" along the sampling trajectory. We then explain how such temporal information affects the sampling trajectory. Finally, we introduce StepbaQ.

---

method that calibrates the sampling trajectory and counteracts the negative effects of accumulated quantization error through sampling step correction mechanism. We demonstrate that StepbaQ can effectively mitigate the accumulation of quantization errors, thereby diminishing the total erro. Note that while the derivations discussed herein are based on the denoising process DDIM [49] as in the proposed StepbaQ is adaptable to other samplers with appropriate modifications.
4.1 Quantization Error as Stepback
Following PTQD
we assume the quantization error follows Gaussian distribution. Then the conditional probability of the quantized latent variable Ttz can be formulated as follows:
20
p(a,, [x,)) (z,, of), Lt 24, + A A, ~ (N,,
(5)
where A; denotes the quantization error with H4; and of representing its mean and variance. As the error mean can readily rectified by subtracting the error mean as described (00) here we assume
M4 O. Recalling the Gaussian transition from step 4; to step with 2 4 as mentioned in Eq. T
((1T, -
am sa (1
On
On )
(6)
Ot:
at;
Scaling the latent variable 2ú, by /ay, we obtain:
Ot Tr (m(ss)
-N
Ctt
1)1)
It is observed that the scaled latent variable
Di. Tr, shares the same mean as the quantized latent
variable dt. as shown in Eq. By selecting a step T; such that St - of or equally or, - T+o? ar, we align the variance of the scaled latent variable with the variance of a.. This alignment allows us state that the scaled latent variable is statistically equivalent the quantized latent variable. Consequently, we can consider & as being sampled from the distribution at 7,-th step and scaled by
a factor In practice, given that discrete samplers operate with a finite set of sequence a, T, can be approximated by solving the following equation (See Appendix B for further discussion):
C't I+ of
Ti arg minlla;
ll. E [L...., M]
Note that 7; 2 due to the descending property of This novel perspective treating the quantized latent as if sampled from the distribution at ry-th step implies that a temporal shift, or "stepback," occurs in the latent space. This insight enables to further explore the impact of quantization errors.
4.2 Error Arising from Temporal Shift
Based on the finding that a temporal shift occurs in the latent space, we will now demonstrate how such shift can contribute to error accumulation, resulting in decline in overall performance.
Inaccurate Noise Prediction. Throughout the sampling process, the noise prediction network relies on the temporal feature at the 4:-th step as a conditional input, which informs the network about the noise level of the latent variable. Given that the noise level of the quantized latent c., is increased by quantization error, utilizing the temporal feature at step 4 noise prediction network regarding the actual noise level. This discrepancy has the potential to yield inaccurate noise predictions. An additional concern arises from the distribution shift in the latent space. The noise prediction network trained with the scheduled latent distributions, as described in Eq. 2 However, quantization error changes the latent variable's 's mean and variance. Without appropriate adjustment, the distribution of the latent variable &t, deviates from the expected value range. This deviation in the input distribution can also lead to inaccurate noise predictions, ultimately reducing overall performance.
condition could misguide the
Trajectory Deviation. For DDIM, the first term within Eq. indicates the predicted xp- The expression 2t, vT- - an,co( (4) can be interpreted as advancing from the latent variable 2t, toward the predicted latent variable zo along the direction -co (Z tt), with a step size of /I ar,.
5

---

Algorithm StepbaQ
Require: Noise prediction networks Eo, ég, sampling steps [ti,...,xx] hyperparameters [oo,.... avll
1: # Initialization
2: TN LN. Fiy TeN:
TeN N(O.I)
3: for ==
do
$4 V -a,, (0(8,, (7))
4: 814-1 V0i
a4 Eo (
Quant sampling
yar
24 c0
5:
F4,-1 V
/I ar, (o
# Float sampling
yar
6:
A,- 8t;-1 24--1
# Error calculation
7:
14-1, oi (eeam(ii1)) (ar(Aii))
8:
T-- arg min ||ay -4--
Step correction
9:
# Correct latent if error is large enough
10:
if T--1 > 1/-1 then
11:
- (ft;-1 Hi-1)
Latent adjustment
It,-1 = -/ J1--1
Latent synchronization
13:
else
14:
1i-1 0
15:
return
[T....., TNJ. (N
As the quantized latent variable êt, possesses higher noise level than Tt,, implying that the gap between Ty and é,, is more pronounced, a larger step required to obtain the expected result. However, previous studies ignore this discrepancy and continue utilizing a, during the denoising process. Fig. (b) illustrates that adhering to the original step size, without accounting for the quantization error, can lead significant accumulation of overall error.
4.3 StepbaQ
To tackle the challenges mentioned above, we introduce StephaQ. By employing the corrected step as obtained by Eq. 8 we demonstrate that StepbaQ can significantly reduce the overall accumulated quantization error through a sequence of simple yet effective modifications.
Temporal Information Alignment. The quantized latent variable ft. is characterized by an in- creased noise level in comparison to its original counterpart 2;,- Therefore, the temporal information input must precisely reflect the increased noise levels to aid the noise prediction network make accurate predictions. Replacing the original input t; with the corrected sampling step т; gives the network more accurate noise level indication.
Latent Adjustment. To approximate the quantized latent कt, as though it were sampled from the distribution at the 7--th step, adjustments are necessary to align their distributions. We subtract the channel-wise mean of quantization error from the quantized latent variable, suggested in
PTQD 20 and rescale it by factor jor These adjustments effectively bridge the distribution ae;
shift and improve the noise prediction results.
Step Size Adaptation. Nevertheless, improving the noise prediction results is not enough. As depicted in 2 the quantization errors increase the variance of quantized latent variables and decrease the signal-to-noise ratio (SNR), defined as SNR فيz 44), which enlarges the distance between ét; and Zo. Therefore, the step size should be extended to consider the increased distance. This extension can be realized by substituting the original step size vI- at, with /I ay which the requisite step size for progressing towards Zo from the latent variable of T,-th step. The adjusted step size is longer than equal to the original one, owing to the relationship a,, I ar,, Fig. (c) illustrates that the cumulative error can be markedly diminished by adapting the step size.
Accounting for Error Accumulation. Since discrete samplers only involve finite sampling steps, the modifications mentioned above only take effect when quantization errors are substantial enough to result in corrected step larger than 4. No correction will be made if the quantization error does not meet this threshold. Nonetheless, even minor quantization errors can accumulate over multiple steps, imperceptibly altering the latent distribution and degrading the quality of the generated output.

---

To address this issue. instead of measuring quantization errors at each step independently, StepbaQ measures quantization eπrors across multiple steps to obtain more accurate statistics for correction.
Algo. Algo.[shows the overall correction process of StepbaQ. Given floating-point and quantized noise prediction networks co and ég, Stepba calculates the corrected step based statistics of quantization error. If the error is not substantial enough, no correction is made, allowing the eπror to accumulate until it becomes sufficiently large for correction. For simplicity, we ignore the iterations over the whole calibration set during the measurement of quantization error. The process yields a sequence of corrected steps [Ti...., TN. and corresponding sequence of error means (NN-)...IMNII) which are subsequently applied during the inference phase. The inference phase is identical to Algo. except that lines 5-8 are skipped. As StepbaQ precludes the accumulation of error, the final output is influenced only by the quantization error present in the last sampling step. This error is significantly less than the original accumulated error without corrections.
Experiment
We demonstrate the effectiveness of the proposed StepbaQ in enhancing the performance of quantized diffusion models for text-guided image generation. Experiments utilizing an off-the-shelf quantization tool show that StepbaQ improve the performance of existing quantized diffusion model without altering its quantization configuration. We also provide comparative results to emphasize StepbaQ's superiority over existing approaches. For qualitative results, please refer to Appendix
SNR Logarithmic Scale
(HR(
65.
Megnitude of Stepback 14-44
StepbaQ StepbaQ w/o ACC
Proposed shre
2
Original stop
Sampling direction
-4
Conntiation cecresss S4R
2-
10
11
12
13
15
Figure 2: SNR curve of diffusion pro- cess. The quantization error decreases the SNR, which could be regarded as a stepback. To address this issue, Step- baQ takes larger step to reach the scheduled SNR.
Figure 3: Magnitude of stepback for SD v1.5 on the SD- prompts dataset under W8A8 setting. Most sampling step corrections occur at the last few steps of sampling, showing the importance of these steps. Since StepbaQ considers the error accumulation, it performs corrections more frequently than StepbaQ w/o ACC.
Datasets and Evaluation Metrics. Our experiments are conducted on two datasets: MS-COCO (36 and the Frechet Inception Distance (22 (FID) our evaluation metric to assess the quality of images generated by the quantized diffusion models. Notably, as our objective to minimize the performance discrepancy between the floating-pott and utilize the images produced by the floating-por monting--pintttnn model as the reference for FID evaluation. This approach offers more precise indication of the performance gap resulting from quantization. In addition, to evaluate the alignment between text and generated images, we also present the CLIP score 21). employing ViT-L/14 12 as the backbone.
Prommpts 7 (SDprompts), each with 5,000 samples for evaluation. We employ
model and the quantized model, we follow the setting in (50)
Implementation Details. In implementing StepbaQ. we utilize a compact dataset of 128 samples for correction. Empirical observations indicate that this sample size is adequate for gathering the necessary statistics of quantization error, while expanding the sample size 1024 does not yield a discernible enhancement in performance. The correction can be fast since only a small calibration dataset required. For example, The correction process for the 20-step model can be completed in approximately 20 minutes on single A6000 GPU.
v1.5 9
7

---

Table 1: Quantization results SD v1.5 and SDXL-Turbo on MS-COCO and SDprompts.
SD
v1.5
SDXL-Turbo
Method
bit (W/A)
MS-COCO
SDprompts
MS-COCO
SDprompts
FID4
CLIP
FID4
CLIPT
FID4
CLIPT
FID.
CLIP
Naive PTQ
8/8
16.69
26.93
23.66
28.16
10.48
26.93
10.12
28.05
PTQD
8/8
16.36
26.93
23.50
28,49
9.58
27.20
10.67
28.20
StepbaQ
8/8
12.34
27.01
16.36
28.69
9.53
27.08
9.72
28.64
Naive PTQ
4/8
95.51
23.27
134.91
19.95
44.76
26.19
45.45
28.12
PTQD
4/8
94.55
23.47
134.62
19.77
27.77
26.70
32.38
27.98
StepbaQ
4/8
69.43
24.12
101.31
21.71
23.92
26.74
28.14
28.46
5.1
Improvement upon Off-the-Shelf Tool
To facilitate efficient on-device inference of deep learning models, the industry has proposed toolkits such NeuroPilot 3. OpenVINO (2) SNPE [6), and TensorRT 4 These tools are tailored to different platforms and support various quantization settings and algorithms. In this section, we utilize the post-training quantization tool NeuroPilot and demonstrate that StepbaQ significantly improves the performance of the given quantized diffusion model without modifying the original quantization settings. This underscores StepbaQ's s effectiveness in seamlessly enhancing model performance within current quantization frameworks. Considering that PTQD depends solely on the statistics of quantization error, similar to our approach, we also include comparison with PTQD.
Quantization Settings. Following previous works, we focus on quantizing the noise prediction network since is the primary computation workload. Other parts of the diffusion pipelines remain at a floating-point precision. Notation Wz indicates the weights are quantized to x-bit while the activations are quantized y-bit. We test two bit-width settings, W8A8 and W4A8, using the default symmetric/usymmeetr settings for weights and activations, respectively. The calibration set comprises 270 samples, which are uniformly selected from each step. NeuroPilot facilitates quantization of most operators within the noise prediction network, e.g., convolution, linear layer, batch matrix multiplication, addition, multiplication, softmax, and SiLU. This scenario presents a more difficult challenge than the one adopted by Q-Diffusion, which only accounts for the quantization of selected operators such convolution and linear layer. Hence, our setting more rigorously tests StepbaQ's s robustness under comprehensive and demanding conditions. Note that for wide activation value ranges, the input prompt embedding is quantized to 16-bit instead to prevent loss of conceptual information. The consuming linear layers take 16-bit prompt embedding and output 8-bit activations; the weights are still quantized to 4-bit/8-bit as described.
due to its
v1.5, utilizing a DDIM sampler with 20 sampling steps. We opt for DDIM over the default PNDM[38) sampler our empirical findings suggest that DDIM yields better results in 20-step sampling setting. The results are provided in Table T. which indicate that our proposed correction technique enhances performance in terms of FID and CLIP-score compared to simple post-training quantization. Specifically, for the W8A8 setting, StepbaQ markedly improves FID by 4.35 on the MS-COCU dataset and 7.30 on the SDprompts dataset. Meanwhile, PTQD, constrained by its limited applicability, achieves only modest improvements of 0.33 and 0.16 FID on the MS-COSS COCO and SDprompts datasets, respectively. For the W4A8 setting, the impact of quantization error is profoundly detrimental, as evidenced by the significant deterioration in both FID and CLIP-score. In this scenario, StepbaQ dramatically reduces the FID by 26.08 and 33.60 the MS-COCO and SDprompts datasets, respectively. These results underscore the efficacy of StepbaQ. demonstrating its capability to substantially mitigate the adverse effects of quantization error, even under challenging scenarios.
Stable-Diffusion v1.5 99. We perform experiments on
SDXL-Turbo [5]. To explore whether StepbaQ is compatible with few-step approaches, we conduct experiments on SDXL-Turbo with 4 sampling steps, employing the default FulerAncestralDis. creteScheduler (Euler-a) [ (See Appendix [orr implementation details). Experimental results are presented in Table I Interestingly, SDXL- exhibits greater resilience to quantization errors than SD v1.5. As Fuler-a a stochastic sampler, the error absorption strategy proposed by PTQD applicable in this setting. The results show that both PTQD and StepbaQ significantly improve the
8

---

quality of the generated images under the challenging W4A8 condition, with StepbaQ outperforming PTQD. StepbaQ's improvements, which reduce the FID from 44.76 23.92 and from 45.45 to 28. 14, effectively eliminate the visual artifacts associated with quantization errors, as shown in Fig. 7
Ablation Study. To assess the contribution of each component of StepbaQ; perform ablation study on Stable-Diffusion v1.5 under the W8A8 setting. Table detaills the results of the ablation study, examining individual components such as Temporal Information Alignment (TIA), Latent Adjustment (LA), Step Size Adaptation (SSA), and Error Accumulation (ACC). The results indicate that mere adjustments the temporal embedding yield only marginal improvements. In contrast, simultaneous corrections to both temporal embedding and latent variables lead to more substantial enhancements. The result also underscores the critical role of the step size adaptation. Correcting the step size results in significant FID improvements of 2.27 and 2.99 on the MS-COCO and SDprompts datasets, respectively. Finally, accounting for error accumulation enables the model to achieve further FIID reductions of 1.20 and 3.22 on the MS-COCO and SDprompts datasets, respectively. This highlights the value of considering error accumulation improving model performance. Fig. shows the magnitude of stepback for SD v1.5 on the SDprompts dataset under the W8A8 setting. Without considering error accumulation, steps with minor quantization errors are ignored, and therefore, corrections are not applied until the final six sampling steps. In contrast, StepbaQ, accounting for the error accumulation across multiple steps, corrects the sampling steps more frequently. Notably, both configurations exhibit greater magnitude of stepback during the final stages of the process. This observation implies that the latter sampling steps are particularly susceptible to quantization errors, confirmed by the steep SNR curve depicted in Fig. 2
Table 3: Quantization results of SD v1.4 on MS-
COCO and SDprompts.
Table 2: Ablation Study of SD v1.5 on MS- COCO and SDprompts under W8A8 setting.
MS-COCO
SDprompts
Method
bit(W/A)
FIDI,
CLIP
FID
CLIP
MS-COCO
SDprompts FIDJ CLIP
TIA LA SSA ACC
Q-Diffusion
8/8
8.74
26.63
9.54
29.20
FID CLIPt
PTQD
8/8
9.07
26.58
10.22
29.18
16.69 26.93 23.66
28.16
TFMQ-DM
8/8
8.63
26.60
9.41
29.21
16.25 27.01
23.21
28.24
StepbaQ
8/8
8,23
26.73
8.81
29.24
15.81 26.87
22.57
28.53
O-Diffusion
4/8
10.04
26.59
12.50
28.93
13.54
26.82
19.58
28.61
PTOD
4/8
10.23
26.51
11.82
28.92
12.34
27.01
16.36
28.69
TFMQ-DM
4/8
10.37
26.48
12.73
28.76
StepbaQ
4/8
9.61
26.68
11.19
29.02
5.2 Comprehensive Comparative Results
To demonstrate the superiority of StepbaQ over existing approaches, we conduct a comparative experiment against O-diffusion [33], TFMQ-DM [24), and PTQD [20]. We select Q-diffusion as the baseline and integrate StepbaQ and PTQD with it. The experiment is implemented based on the codebase of Q-diffusion O to note that we deactivate the timestep-specific quantization parameters utilized in TFMQ-DM. While the technique can potentially enhance the results, can be regarded as an orthogonal strategy that alters the quantization setting by introducing control of finer granularity.
adopting the exact quantization setting for fair comparison. is important
Stable-Diffusion +1.4 [8]. Table provides the results of
v1.4 using DDIM sampler with 20 sampling steps. Results show that StepbaQ surpasses previous works in terms of FID and CLIP-score across all tested scenarios. Observations indicate that PTQD generally produces results that are inferior to the Q-Diffusion baseline. Further examination revealed that the performance drop is attributable to their strategy of correlated noise elimination, which depends the correlation between the model's output and the quantization error. However, the observed I22 values are relatively low; indicating a weak correlation. Consequently, the estimated value utilized for correlated noise climination is unreliable and may inadvertently compromise the model's performance. On the other hand, TFMQ-DM outperforms Q-Diffusion only under the WSA8 setting through their temporal information aware reconstruction. This limitation stems from their oversight of the distribution shift in the latent space, which is a more critical issue for quantized diffusion models.
htttps://githubbo http:///iii.ooooooooooooeiiiggo
9

---

Conclusion
In this work, we introduce novel aspect that considers the temporal shift in the latent space caused by the quantization error as "stepback" in the denoising process. We present how this temporal shift would lead to deviation from the scheduled sampling trajectory and harm the performance of quantized diffusion models. To address this issue, we propose StepbaQ. which corrects the sampling steps to calibrate the sampling trajectory and alleviate quantization error accumulation. Our experimental results demonstrate that StepbaQ can serve as plug-and-play technique, enhancing the performance of a given quantized diffusion model without modifying its quantization settings. The comparative result also demonstrates the superiority of StepbaQ over existing methods. Notably, StepbaQ requires only statistics of quantization error derived from small calibration dataset and can be applied to both stochastic and deterministic samplers, demonstrating its applicability. However, due to the design of the stephack mechanism, StepbaQ does not apply single-step scenarios, such as 1 1-ste 3DXL-TTubo How to prevent performance drop of quantized diffusion model under a single-step scenario is an area that requires further exploration in future research.
Broader Impact.
StepbaQ significantly improves the performance of quantized diffusion models, thereby enabling users to generate high-quality results even on computationally limited edge devices. While this advancement broadens the accessibility of diffusion models to broader audience, it also increases the risk of these models being leveraged for malicious purposes.
Acknowledgement
This work was supported by the CA12 Department at MediaTek, which provided technical support and resources.
References
[1] ButerAhnestrlDiierrttSShhhheee htte
http::////ggiieeeeeoo/////////aa api/ http://cccuurrrreeeeeeeeaeettttt
[2] Intel Open Berelloo/coolss//pparrrroo......
[3] MediaTek NeuroPilot, httpss//eaarrolood..mdaaaa....a
[4]
NVIDIA TensorRT. httpps//ree..gee...vvaaaaaooooor
[5] SDXL-Turbo. http:////woooooooo.......//////a
[6] Snapdragon Neural Processing Engine SDK. https: |ttppp/sss///eeeepppp........oo
wtt.ddocsssppeeeeendeeeeee html
[7] Stable-diffusion
prompts dataset.
http:///ugginfaee.co//atttttt
[8] Stable-Diffusion v1.4.
https://hogginffeeee.o////////s
fiable-diffusionn
[9] Stable-Diffusion v1.5.
[10] Y. Cai, Z. Yao, Z. Dong, A. Gholami, M. W. Mahoney, and K. Keutzer. Zeroq: A novel zero shot quantization framework. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13169-13178 17788, 2020.
[11] Dhariwal and A. Nichol. Diffusion models beat gans on image synthesis. Advances neural information processing systems, 24:8780-8794, 2021.
10

---

[12] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, al. An image is worth 16x16 words: Transformers for image recognition at scale. International Conference on Learning Representations, 2021.
[13] S. K. Esser, J. L. McKinstry, D. Bablani, R. Appuswamy, and D. S. Modha. Learned step size quantization. International Conference on Learning Representations, 2020.
[14] G. Fang, X. Ma, and X. Wang. Structural pruning for diffusion models. Advances in neural information processing systems, 36, 2024.
[15] E. Frantar, S. Ashkboos, T. Hoefter, and D. Alistarh. Gptq: Accurate post-training quantization for generative pre- pre-trained transformers. arXiv preprint vrXiv:2210.173233 2022.
[16] A. Gholami, S, Kim, Z. Dong, Z. Yao, M. W. Mahoney, and K. Keutzer. A survey of quantization methods for efficient neural network inference. In Low-Power Computer Vision, pages 291-326. Chapman and Hal//CRC, 2022.
[17] R. Gong, X. Liu, S. Jiang, T. Li, P. Hu, J. Lin, F. Yu, and J. Yan. Differentiable soft quantiza- tion: Bridging full-precision and low-bit neural networks. In Proceedings of the IEEE/CVF international conference computer vision, pages 4852-4861, 2019.
[18] I. Goodfellow, J. Pouget-Abadie, M. Mirza, Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio. Generative adversarial nets. Advances in neural information processing systems, 27, 2014.
[19] Y. He, J. Liu, W. Wu, H. Zhou, and B. Zhuang. Efficientdm: Efficient quantization-awe aware fine-tuning of low-bit diffusion models. International Conference on Learning Representations, 2024.
[20] Y. He. L. Liu, J. Liu, W. Wu, H. Zhou, and B. Zhuang. Ptqd: Accurate post-training quantization for diffusion models. Advances in Neural Information Processing Systems, 36, 2024.
[21] J. Hessel, A. Holtzman, M. Forbes, R. L. Bras, and Y. Choi. Clipscore: A reference-free evaluation metric for image captioning. Conference on Empirical Methods in Natural Language Processing, pages 7514-7528, 2021.
[22] M. Heusel, H. Ramsauer, T. Unterthiner, B. Nessler, and S. Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. Advances in neural information processing systems, 30, 2017.
[23] Ho, A. Jain, and Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-688 6851, 2020.
[24] Y. Huang, R. Gong, J. Liu, T. Chen, and X. Liu. Tfmq- dm: Temporal feature maintenance quantization for diffusion models. Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, 2024.
[25] Hubara, Y. Nahshan, Y. Hanani, R. Banner, and Soudry. Accurate post training quantization with small calibration sets. In International Conference on Machine Learning, pages 4466-4475 PMLR, 2021.
[26] B. Jacob, S. Kligys, B. Chen, M. Zhu, M. Tang, A. Howard, H. Adam, and D. Kalenichenko. Quantization and training of neural networks for efficient tnnger--rithmetic-nn/ inference. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2704-2713, 2018
[27] Jain, A. Gural, M. Wu, and C. Dick. Trained quantization thresholds for accurate and efficient fixed-point inference of deep neural networks. Proceedings of Machine Learning and Systems, 2:1 1:112- 2020.
[28] Z. Jiang, C. Mao, Y. Pan, Z. Han, and J. Zhang. Scedit: Efficient and controllable image diffusion generation via skip connection editing. Proceedings of the IEFF/CVF conference on computer vision and pattern Recognition, 2024.

---

[29] T. Karras, M. Aittala, T. Aila, and S. Laine. Elucidating the design space of diffusion based generative models. Advances in Neural Information Processing Systems, 35:26565-26577, 2022.
[30] D. P. Kingma and M. Welling, Auto-encoding variational bayes. International Conference Learning Representations, 2014.
[31] R. Krishnamoorthi. Quantizing deep convolutional networks for efficient inference: A whitepa- per. arXiv preprint arXiv: 08342, 2018.
[32] B. Li. K. Xue, B. Liu, and Y.-K. Lai. Bbdm: Image-to-image translation with brownian bridge diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern Recognition, pages 1952- 1961, 2023.
[33] X. Li, Y. Liu, Lian, Yang. Z. Dong, D. Kang, S. Zhang, and Keutzer. Q-diffusion: Quantizing diffusion models. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 17535-17545, 2023.
[34] Y. Li, R. Gong, X. Tan, Y. Yang. P. Hu, Q. Zhang, F. Yu, W. Wang, and Gu. Breeq: Pushing the limit of post-training quantization by block reconstruction. International Conference on Learning Representations, 2021.
[35] Y. Li. S. Xu, X. Cao, X. Sun, and B. Zhang. Q-dm: An efficient low-bit quantized diffusion model. Advances in Neural Information Processing Systems, 36, 2024.
[36] T.-Y. Lin, M. Maire, Belongie, J. Hays, P. Perona, D. Ramanan, Dollár, and C. Zitnick. Microsoft coco: Common objects in context. In Computer ECCo 2014: 13th European Conference, Zurich, Swifzerland, September 6-12, 2014, Proceedings, Part V 13, pages 740-755. Springer, 2014.
[37] X. Lin, J. He, Z. Chen, Z. Lyu, B. Fei, B. Dai, W. Ouyang, Y. Qiao, and C. Dong. Diffbir: To- wards blind image restoration with generative diffusion prior. arXiv preprint arXiv:2308. 15070, 2023.
[38] L. Liu, Y. Ren, Z. Lin, and Z. Zhao. Pseudo numerical methods for diffusion models on manifolds. International Conference on Learning Representations, 2022.
[39] C. Lu, Y. Zhou, F. Bao, J. Chen, C. Li, and J. Zhu. Dpm-solver: A fast ode solver for diffusion probabilistic model sampling in around 10 steps. Advances in Neural Information Processing Systems, 35:5775-5787, 2022.
[40] C. Luo. Understanding diffusion models: A unified perspective. arXiv:2208.1 11970, 2022.
arXiv preprint
[41] S. Luo, Y. Tan, L. Huang, Li, and H. Zhao. Latent consistency models: Synthesizing high-resolution images with few-step inference. arXiv preprint arXiv:2310.04378, 2023.
[42] M. Nagel, R. A. Amjad, M. Van Baalen, Louizos, and T. Blankevoort. Up or down? adaptive rounding for post-training quantization. In International Conference on Machine Learning. pages 7197-7206. PMLR, 2020.
[43] Y. Nahshan, B. Chmiel, C. Baskin, Zheltonozhskii, R. Banner, A. M. Bronstein, and A. Mendelson. Loss aware post-training quantization. Machine Learning, 110(11):3445-322. 2021.
[44] A. Q. Nichol and Dhariwal. Improved denoising diffusion probabilistic models. In Interna- tional conference machine learning, pages 8162-8171 8171. PMLR, 2021.
[45] R. Rombach, A. Blattmann, D. Lorenz. P. Esser, and B. Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 11684-10695, 2022.
[46] A. Sauer, Lorenz, A. Blattmann, and R. Rombach. Adversarial diffusion distillation. arXiv preprint arXiv:2311.17042. 2023.
12

---

[47] Y. Shang. Z. Yuan, B. Xie, B. Wu, and Y. Yan. Post-training quantization on diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pages 1972-1981, 2023.
[48] J. So, J. Lee, D. Ahn, H. Kim, and E. Park. Temporal dynamic quantization for diffusion models. Advances in Neural Information Processing Systems, 36, 2024.
[49] J. Song, C. Meng, and S. Ermon, Denoising diffusion implicit models. International Conference on Learning Representations, 2021.
[50] S. Tang, X. Wang, H. Chen, C. Guan, Z. Wu, Y. Tang, and W. Zhu. Post-training quantization with progressive calibration and activation relaxing for text-to-image diffusion models. arXiv preprint arXiv:2311.06322 2023.
[51] W. Tukey al. Exploratory data analysis, volume Springer, 1977.
[52] C. Wang, Z. Wang, X. Xu, Y. Tang, J. Zhou, and J. Lu. Towards accurate post-training quantization for diffusion models. Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, 2024.
[53] J. Wang, Z. Yue, S. Zhou, K. C. Chan, and C. C. Loy. Exploiting diffusion prior for real-world image super-resolution. arXiv preprini arXiv:2305.07000 2023.
[54] B. Xia, Y. Zhang, S. Wang, Y. Wang, X. Wu, Tian, W. Yang, and Van Gool. Diffir: Efficient diffusion model for image restoration. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 13095-13105 2023.
[55] G. Xiao, J. Lin, M. Seznec, H. Wu, J. Demouth, and Han. Smoothquant: Accurate and efficient post-training quantization for large language models. In International Conference on Machine Learning, pages 38087-38099. PMLR, 2023.
[56] S. Xie, Z. Zhang, Z. Lin, T. Hinz. and K. Zhang. Smartbrush: Text and shape guided object inpainting with diffusion model. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 22428-22437, 2023.
[57] Y. Yang, X. Dai, J. Wang, P. Zhang, and H. Zhang. Efficient quantization strategies for latent diffusion models. arXiv preprint arXivv2312.05431, 2023.
[58] Z. Yue, Wang, and C. C. Loy. Resshift: Efficient diffusion model for image super-resolution by residual shifting. Advances in Neural Information Processing Systems, 36, 2024.
13

---

SDv1.4 SDv1.5
SDXL-Turbo
1300
3200
2008
300
Figure 4: The distribution of quantization errors collected under W8A8 setting.
Analysis of Quantization Error
Our assumption that quantization error follows a Gaussian distribution is inspired by the empirical findings presented in PTQD [20], where the authors verify the Gaussian nature quantization error through statistical tests detailed in Appendix B of their paper. In our own empirical investigations, we observed that the distribution of quantization error typically exhibits a symmetric, bell-shaped curve, as depicted in FigH] This observation aligns with the result shown in Fig.3 of the PTQD paper.
To delve deeper into the characteristies of quantization emrrr visualize the error distribution in and present an analysis of both the kurtosis (Fisher) and skewness of the error distribution in Table 4 averaged across all steps for each model under the W8A8 setting. Our findings indicate that the skewness values are notably small, confirming our observation that the quantization error follows symmetric, bell-shaped distribution. Regarding kurtosis, the values observed are greater than zero for each model, suggesting that the distribution of quantization error exhibits relatively fat tails. These fat tails likely originate from clipping errors, which occur less frequently but are more significant than rounding errors.
Although the quantization error exhibits higher kurtosis than Gaussian distribution, proposed StepbaQ method still significantly enhances the performance of diffusion models, as evidenced by the results in Tables [ad 3 of our main paper. This underscores the robustness and applicability of our approach, demonstrating that StepbaQ can effectively improve model performance under realistic conditions.
Table 4: The kurtosis and skewness of the quantization errors collected under W8A8 setting.
SDv1.4
SDv1.5 SDXL-TTurbo
Kurtosis 2.627
3.429
0.502
Skewness 0.036
-0.020
-0,007
B Discussion of Quantization Error Stepback
Here. we provide another derivation of Eq. g from the signal-to-noise ratio (SNR) perspective. Combining Eq. 2 and Eq. 5 the quantized latent variable it; can be expressed as:
i,, Dt, A Tt + v1 a, A;
Given that the sum of two independent Gaussian random variables remains Gaussian with mean being the sum of the individual means and variance being the sum the individual variances, the quantized latent & can be represented as:
ir
Vor, + 1- oz; + oft',
d'~N(0,1)
(10)
From Eq. 5 it is evident that the quantization error amplifies the variance of the quantized latent variable Zec- This increase in variance, in turn, diminishes the SNR of the latent variable, which
defined as SNR 5 44). resulting in:
SN
Cte
SN
ar
(1I)
Ctt.
Co, + +07
14

---

12300
10000
3000
2500
Figure 5: The distribution of quantization error collected from W4A8 SDXI--Turbo on SDprompts dataset, outliers are clipped by Tukey's 's fence with set as 1.7.
Since diffusion models exhibit an increasing SNR in the denoising process, this reduction in SNR can be interpreted as a "stepback," as depicted in Fig. 2 Therefore, we can find a step Tę that better aligns with the SNR of fig. such that:
Ti = arg min|| minl|)NN(())) - SN.() [ M]
(12)
This allows us approximate C., if sampled from the distribution at the T,-th step. By reformulat- ing Eq. 12 as follows:
Ti arg min| jjj Oj
a1.
Oie
+ o
(13)
arg minllaj o
11. [L...., M]
we arrive at the same objective described in Eq.
C Extending StepbaQ to SDXL-Turbo
We have demonstrated the application StepbaQ diffusion models using DDIM sampler in Sec. Here, we introduce several modifications to adapt StepbaQ for SDXL-Turbo, which employs the EuterAAnesttaiDiisreeeechhhh
4
(Euler-a) as its sampler.
Statistical Analysis of Quantization Error. In our analysis of SDXL-Turbo's 's quantization error, find outliers with large values, making the distribution of quantization error very long-tailed. To mitigate the influence of these outliers on the variance measurement, we employ Tukey's fence [51) empirically setting the value of l to 1.7, which effectively clips out these extreme values, shown in Fig. 5
Finding the Corrected Step. The Euler-a scheduler determines the noise level at each step using the
equation (t+) T-a;, To identify the corrected sampling step т; that more accurately represents
on
the noise level of the quantized latent variable dz.. we solve the following equation:
Ti arg minllo(()² (a + Il j [ M]
(14)
where of represents the variance of the quantization error.
Temporal Information Alignment. As discussed in Sec. 4 the time embedding input provides crucial information about the noise level to the noise prediction network. Aligning this temporal information with the latent variable is generally beneficial for improving noise prediction. However, for SDXL-Turbo, aligning temporal information does not enhance model prediction. We hypothesize that this because the training procedure of SDXL-Turbo (46) involves only four selected steps. In this case, correcting the temporal input would introduce inputs unseen during training. Therefore, we deactivate the temporal information alignment component when integrating StepbaQ with SDXL. Turbo.
15

---

Latent Adjustment With Euler-a as its sampler, SuXL-Turbo rescales the latent variable z before feeding it into the noise prediction network by factor s(t;) (Eq.170 (2 To account for the increased noise level in the quantized latent variable Et.. we use the scaling factor of the corrected step (T;), which more accurately reflects the noise level.
Step Size Correction The original step size defined by Euler-a for the 4,-th step moving toward the target step with noise level down is defined as att; down. To accommodate the increased noise level of the quantized latent variable Tt,, we employ larger step size, O(T.) down- This adjustment ensures that the resulting noise level aligns with the intended target.
D Qualitative Results
In this section, we present the qualitative results of our experiments. Figure displays the outcomes for SD v1.5, which quantized using NeuroPilot on the SDprompts dataset under the W8A8 setting. Compared to the results from Naive PTQ and PTQD, our proposed StepbaQ method effectively addresses the inconsistent visual artifacts introduced by quantization errors while preserving the overall structural integrity of the images.
Figure illustrates the results for SDXL-Turbo, also quantized by NeuroPilot on SDprompts but under the more challenging W4A8 setting. In this scenario, images processed with Naive PTQ display noticeable crack-like artifacts. Both PTQD and StepbaQ manage smooth these visual disturbances. However, the images produced by StepbaQ demonstrate higher degree of visual smoothness and greater similarity to those generated by the floating-point model, underscoring the effectiveness of StepbaQ.
Figure shows the comparative results for SD v1.4 SDprompts under the W4A8 setting. This experiment was conducted using the Q-Diffusion codebase, which represents a less challenging quantization scenario as described in Sec. [5.1 Despite the less pronounced differences in image quality, StepbaQ consistently delivers results that more closely resemble those of the floating-poin model, avoiding the artifacts that are sometimes evident in outputs from other methods.
16

---

FP
PTQ
PTQD
StepbaQ
Figure 6: Qualitative results of SD v1.5 on SDprompts under W8A8 setting.
17

---

FP
PTQ
PTQD
StepbaQ
Figure 7: Qualitative results of SDXL-Turbo on SDprompts under W4A8 setting.
18

---

FP
Q-Diffusion
PTQD
TFMQ
StepbaQ
Figure 8: Qualitative results of SD v1.4 on SDprompts under W4A8 setting.
19

---

NeurIPS Paper Checklist
1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer:
[Yes]
Justification:
We include all the claims and contributions in the abstract and introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We include the limitations of the proposed approach in the conclusion section.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken low lighting. Or speech-to-ter -reech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
20

---

Justification: We make assumption about the distribution of quantization error following prior research, and provide complete proof of why the distribution shift caused by quantiza- tion error can be considered as stepback.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We provide information on the tool, codebase, data, and evaluation metrics used in our experiments and also provide pseudo code with detailed information.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer this question will be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided not.
If the contribution a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access a hosted model (e.g., the case of a large language model), releasing of model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions provide some reasonable avenue for reproducibility. which may depend the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-soures models, it may be that access to the model is limited in some way (e.g.. to registered users), but it should be possible for other researchers to have some path to reproducing verifying the results.
5. Open access to data and code
21

---

Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [No]
Currently, we cannot release the source code due to confidentiality and intellectual property concerns.
Justification:
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips... cc/
public/qui des CodeSubm asi onPolicy for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https Sww.edeeee/odSSummmmm. sa ssinPolicy) for more details.
//nips. cc
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We provide all the quantization settings and hyperparameters required in the main paper and appendices.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer:
[No]
Justification:
We do not report error bars.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the emoo bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
22

---

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors). of errors is not verified.
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
It is OK to report 1 Isiggma error bars, but one should state it. The authors should preferably report 2-sigma enror bar than state that they have 96% CI, if the hypothesis of Normality
For asymmetric distributions, the authors should be careful not show in tables figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars are reported tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yss]
Justification: The information on computation resources is provided in the implementation details.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't t make into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification:
We follow each point mentioned in the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethies.
The authors should make sure to preserve anonymity (c.g., if there is a special considera- tion due to laws regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive socictal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We include the Broader Impacts section in our paper.
Guidelines:
The answer NA means that there is societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
23

---

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g.. deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to partieular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out, For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mecha- nisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [NA]
Justification: Our paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification:
We include references of every code, data, or model used in our paper.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, if possible, include URL..
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperrsii thcode. com/da httr/datasetss. has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
24

---

For existing datasets that are packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: Our paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets. Researchers should communicate the details of the dataset/code/model as part of their sub- missions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: Our paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: Our paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
25


Reviews:
[{"review_id": "uDVGvT41zI", "reviewer": "Reviewer_1NNd", "summary": "This paper is well organized and logically understandable. The author reconsidered the quantization error as a \"stepback\" problem and provided concrete theoretical illustration. The author looked into latent variables behind general quantization errors in sampling trajectories, which impressed significantly when first reading. Qualitative experimental results proved the effectiveness of the proposed method.  This is literally a creative, persuasive and well-completed work.", "questions": "1.Moving to table1, the fid calculated on sd v1.5 with W4A8 drops obviously while sdxl-turbo slightly. Why there will be significant differences  on these 2 datasets?\n2. The ablation study shows less improvement on clip score, maybe another measurement like is score should be adopted to further prove the effectiveness. \n3. what exactly the classifier-free guidance scale is used with sd v1.5 in this work? have you tried different scales?", "limitations": "Despite the author gives an interesting insight behind the quantization error arisen in sampling stage of diffusion models, experiments seems to be slightly coarse as some details are not well presented.  More comparison between existing methods and more model setting details are strongly recommended.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Contents are well organized leading to an easy-reading form. The author shows mature writing style that makes this work logically understandable. \nThe core insight that quantization error can be reviewed as \"stepback\" is interesting and creative. Authors have presented detailed proof for their theories and conduct extensive experiments.", "weaknesses": "Despite varies experiments on 2 mainstream diffusion models are conducted, the comparison with other existing methods seems to be slightly thick.  Experimental results on sd v1.5 and sdxl-turbo only takes native PTQ and PTQD into consideration, which makes the result less competitive.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the positive review and constructive comments. We provide our responses as follows.\n\n> Experimental results on sd v1.5 and sdxl-turbo only takes native PTQ and PTQD into consideration, which makes the result less competitive.\n\nThe experiments presented in Section 5.1 are designed to evaluate the enhancements StepbaQ brings to off-the-shelf quantization toolkits. While these toolkits provide a streamlined process for model quantization, not all of them support reconstruction-based quantization algorithms like AdaRound. In scenarios where it is necessary to augment models quantized by these toolkits without altering their established quantization workflows, only statistical methods such as PTQD and StepbaQ can be effortlessly integrated.\nConsequently, within Section 5.1, we limit our comparisons to StepbaQ, Naive PTQ, and PTQD. It is important to highlight that we have conducted a comparative analysis involving other quantization methods on the SDv1.4 model, as shown in Table 3. These comparisons adhere to the experimental settings used in prior studies to ensure a fair comparison.\n> The fid calculated on sd v1.5 with W4A8 drops obviously while sdxl-turbo slightly. Why there will be significant differences on these 2 datasets?\n\nWe observe that SDXL-Turbo, compared to SD v1.5, demonstrates higher resistance to noise. This can be attributed to the differences in the training process of these two models. Specifically, SDXL-Turbo undergoes an additional fine-tuning phase through knowledge distillation to facilitate a few-step sampling. This supplementary fine-tuning process seemingly contributes to the model's increased tolerance to quantization errors.\n\n> The ablation study shows less improvement on clip score, maybe another measurement like is score should be adopted to further prove the effectiveness.\n\nWe do not use the IS score as it may provide misleading results when applied to datasets other than ImageNet, as indicated in [A]. The improvement in CLIP-score is by nature less significant even when FID is greatly improved. This is because CLIP-score focuses more on the alignment between images and textual descriptions, which may not be as responsive to certain types of image quality enhancements. Furthermore, when comparing our results to those reported in Table 5 of the TFMQ-DM paper, the improvements we have achieved are already notably substantial.\n\n> What exactly the classifier-free guidance scale is used with sd v1.5 in this work? have you tried different scales?\n\nWe focus on evaluating the model under the default setting, and therefore we only use the default value of 7.5 as the classifier-free guidance scale for the SDv1.5 model.\n\n> Experiments seems to be slightly coarse as some details are not well presented.\n\nWe apologize for any oversight in detailing specific aspects of our method. Unless otherwise mentioned, we adhere to the default hyperparameter settings of the diffusion models for our evaluations. Regarding the quantization settings, for the experiments presented in Section 5.1, we kindly direct readers to the \"Quantization Settings\" subsection within that section for comprehensive information. For the experiments conducted in Section 5.2, additional details can be found within the Q-Diffusion codebase.\nThe details of the proposed StepbaQ method are thoroughly elaborated in Algorithm 1. For information on how StepbaQ is adapted for use with SDXL-Turbo, please refer to Appendix B of our paper.\n\n[A] A Note on the Inception Score. ICML 2018 Workshop", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "myIg08TXjO", "reviewer": "Reviewer_hn4w", "summary": "The paper introduces a novel method to address the issue of accumulated quantization errors in quantized diffusion models. It attributes the quantization error to a \"stepback\" in the denoising process, and it introduces a sampling step correction mechanism to mitigate the adverse effects of accumulated quantization error. Experimental results demonstrate significant performance improvements in terms of FID on the SDprompts dataset, particularly under challenging quantization settings.", "questions": "See above.", "limitations": "Based on the standard metrics, the algorithm performs well. But further justification on Equation (5) is needed. Furthermore, it would be interesting to see how this method works in the setting that only a few sampling steps are taken, such as in the context of consistency models.", "rating": 7, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper presents a novel perspective by interpreting quantization errors as a temporal shift or \"stepback\" in the denoising process. This insight provides a clear understanding of how quantization errors impact model performance.\n\n2.  By calibrating the sampling trajectory, the StepbaQ effectively addressed the accumulated quantization error.", "weaknesses": "1. The author should provide more justification to the argument in Equation (5) that the quantization error can be modeled as a Gaussian random variable.\n\n2. Can the calibrated statistics obtained from one model be directly adapted to different models?\n\n3. Does the method work in consistency models, which perform sampling with only a few steps?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the positive review and constructive comments. We provide our responses as follows.\n> The author should provide more justification for the argument in Equation (5) that the quantization error can be modeled as a Gaussian random variable.\n\nPlease see \"Author Rebuttal by Authors\" above.\n> Can the calibrated statistics obtained from one model be directly adapted to different models?\n\nAs the estimated quantization error varies between different models, the corrected steps optimized for one model are unlikely to yield optimal results when directly applied to another. We highly recommend implementing an independent correction process for each model, given that the correction procedure of StepbaQ does not impose a significant time burden.\n\n> Does the method work in consistency models, which perform sampling with only a few steps?\n\nThanks for the suggestion. Although we have already shown the effectiveness of the proposed StepbaQ under a few-step sampling setting through experiments on SDXL-Turbo with a 4-step sampling process, as detailed in Section 5.1, the prospect of applying StepbaQ to consistency models is indeed intriguing. Consequently, during the rebuttal phase, we conducted additional experiments to assess the performance of StepbaQ on the LCM-SDXL model (https://huggingface.co/latent-consistency/lcm-sdxl). These experiments were conducted using a 4-step sampling process with the guidance scale set to 8, following the provided sample code. All other hyperparameters were maintained at their default settings, and the quantization parameters were consistent with those detailed in Section 5.1 with input prompt embedding also quantized to 16-bit as SDXL-Turbo.\n\nThe results indicate that StepbaQ consistently enhances the performance of quantized diffusion models even under a few-step sampling setting. This finding demonstrates the robustness and applicability of the proposed StepbaQ.\n\n| Method  | Precision  | COCO-FID  | COCO-CLIP  | SDprompts-FID  | SDprompts-CLIP  |\n|---------|:---------:|:---------:|:---------:|:---------:|:---------:|\n| Naive PTQ | W8A8 | 18.47 | 25.46 | 18.61 | 29.08 |\n| StepbaQ | W8A8 | **17.87** | **25.50** | **17.62** | **29.12** |\n| Naive PTQ | W4A8 | 97.28 | 22.80 | 73.36 | 24.58 |\n| StepbaQ | W4A8 | **80.41** | **23.33** | **61.33** | **26.18** |", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qnUSSoS0GR", "reviewer": "Reviewer_twKC", "summary": "This paper proposes the StepbaQ, a general strategy designed to enhance the performance of quantized diffusion models which employs a sampling step correction technique to realign the sampling trajectory and eliminate the accumulation of quantization error. Experiments show that the proposed StepbaQ improves performances of diffusion models quantized by off-the-shelf tools.", "questions": "See comments in Weaknesses.", "limitations": "Not applicable.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "+ This paper is well-written and the topic seems quite interesting and is of high practical value.\n+ The numerical results are convincing (but limited).\n+ Ablation studies are provided to show importance of different components.", "weaknesses": "- It is not clear that how the definition of stepback will impact performances if the stepback consists of multiple steps.\n- It would be good if the authors can provide computational cost and corresponding comparisons with baselines.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the positive review and constructive comments. We provide our responses as follows.\n\n> It is not clear that how the definition of stepback will impact performances if the stepback consists of multiple steps.\n\nHere we provide an example to illustrate cases where stepback consists of multiple steps. In Figure 3, the magnitude of stepback at the first step is $10$, indicating that $\\tau_1 - t_1 = 10$. Given that $t_1=1$, this implies that the distribution of $ \\hat{x}_1 $, is more closely aligned with the distribution of $x{_1}{_1}$ rather than the distribution of $x_1$. Therefore, we can use $\\tau_1 = 11$ and $\\alpha{_1}{_1}$ for sampling (line 4 in Algorithm 1) to reflect the increased noise level of the quantized latent variable and obtain improved results.\n\n> It would be good if the authors can provide computational cost and corresponding comparisons with baselines.\n\nPlease see \"Author Rebuttal by Authors\" above.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qf3eXdeaPY", "reviewer": "Reviewer_oUSD", "summary": "This paper proposes a novel perspective of quantization error in diffusion models being equivalent to a \"step back\" in the denoising process. The authors are show that one can effectively quantify this stepback using a small calibration dataset, and continue the diffusion process as usual after correcting for the new effective timestep. This technique doesn't make many assumptions, and hence is very general and can be integrated with many off-the-shelf quantized diffusion solvers. The authors also show convincing experimental evidence on effectiveness of their method.", "questions": "1) How was the calibration for estimating the stepback performed? Was there any analysis done on how many samples was sufficient to estimate it with high confidence? I also wonder if different time steps show varying difficulty of estimating the associated variance.\n2) How does one explain the sudden jump in the stepback magnitude for t=1 (Figure 3)? Yes, it makes sense that the last few steps are important, but 10 step correction seems too drastic compared to 1-2 steps for previous time steps?\n3) On the same note, does this high stepback number for t=1 mean that the final drawn sample from the model is heavily miscalibrated? Are there ways to remedy this?\n4) Could you please elaborate on why it was okay to deactivate the timestep-specific quantization parameters in TFMQ-DM for comparison?", "limitations": "Yes.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The main ideas introduced in the paper are novel and elegant. The perspective of thinking of quantization error as stepback enables the author to devise simple, clean, general techniques to help correct for step size and error accumulation. The empirical evaluation is convincing.", "weaknesses": "While the experimental results (like the ablation study) are helpful and convincing, the paper could benefit from a more thorough analysis/interpretation of the method. For instance, how hard is to estimate the quantization error variance for different models/denoising methods? What does this say about the Gaussian assumption in the first place? What does it mean (in terms of making progress) to take one denoising step if the calculated stepback is >=1 step?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the positive review and constructive comments. We provide our responses as follows.\n> How hard is it to estimate the quantization error variance for different models/denoising methods?\n\nEstimating the variance of quantization error is a straightforward process. We begin by comparing the sampled latent variable from the floating-point model with those from the quantized model to calculate the quantization error, and then we assess its variance. This procedure is model-agnostic, meaning it does not introduce any additional complexity when estimating the variance of quantization error across different models or denoising methods.\n\n> What does this say about the Gaussian assumption in the first place?\n\nPlease see \"Author Rebuttal by Authors\" above.\n> What does it mean (in terms of making progress) to take one denoising step if the calculated stepback is >=1 step?\n\nHere we provide an example to illustrate cases where stepback is >= 1. In Figure 3, the magnitude of stepback at the first step is $10$, indicating that $\\tau_1 - t_1 = 10$. Given that $t_1=1$, this implies that the distribution of $ \\hat{x}_1 $, is more closely aligned with the distribution of $x{_1}{_1}$ rather than the distribution of $x_1$. Therefore, we can use $\\tau_1 = 11$ and $\\alpha{_1}{_1}$ during the sampling process (line 4 in Algorithm 1) to reflect the increased noise level of the quantized latent variable and obtain improved results.\n> Was there any analysis done on how many samples were sufficient to estimate it with high confidence?\n\nAs detailed in the Implementation Details section of our paper, our findings indicate that utilizing 128 samples for StepbaQ correction is sufficient, and increasing the sample size to 1024 does not result in a noticeable improvement in performance. Here, we provide additional results that utilize 16, 128, and 1024 samples for StepbaQ correction on the SDv1.5 model under the W8A8 setting, which demonstrate that these settings yield comparable results. In fact, they derive identical corrected sampling steps. The minor variances in performance can be attributed to the slight differences in the estimated error mean.\n\nThese observations suggest that a relatively small sample size is adequate to implement StepbaQ correction effectively. This efficiency is likely due to the large dimensionality of the latent variable, which is $64 \\times 64 \\times 4 = 16384$, providing a substantial number of data points for accurate quantization error estimation. Nevertheless, for our experiments, we have chosen to use 128 samples rather than 16 to mitigate any potential concerns that might arise from using an excessively small sample size.\n| Sample Num  | COCO-FID  | COCO-CLIP  | SDprompts-FID  | SDprompts-CLIP  |\n|:---------:|:---------:|:---------:|:---------:|:---------:|\n| 16 | 12.21 | 27.07 | 16.69 | 28.72 |\n| 128 | 12.34 | 27.01 | 16.36 | 28.69 |\n| 1024 | 12.26 | 26.95 | 16.32 | 28.78 |\n> I also wonder if different time steps show varying difficulty of estimating the associated variance\n\nThere is no varying difficulty in estimating the associated variance since the estimation process is the same across all steps. Please refer to line 6-7 of Algorithm 1 in our paper.\n> How does one explain the sudden jump in the stepback magnitude for t=1 (Figure 3)? \n\nThe drastic change in the stepback magnitude can be attributed to the sensitive nature of SNR when t is close to 0, which can be observed from the steep SNR curve in Figure 2. By definition, $\\alpha$ approaches 1 as t close to 0. In that case, the denominator of the SNR formula, $1-\\alpha_{t_i}$ (Equation 11 in Appendix A), becomes extremely small. Therefore this small denominator can be significantly influenced by the quantization error term $\\sigma_i$, leading to substantial changes in the magnitude of stepback.\n> Does this high stepback number for t=1 mean that the final drawn sample from the model is heavily miscalibrated? Are there ways to remedy this?\n\nAs discussed in the previous response, the large stepback value is a consequence of the sensitive nature of the SNR. Therefore, this observation is not considered to be a result of miscalibration. To further alleviate the impacts of quantization error, potential approaches involve exploring alternative strategies for sampling calibration data, implementing finer granularity in the quantization process, or integrating more advanced PTQ algorithms, which we plan to explore in our future work.\n\n> Could you please elaborate on why it was okay to deactivate the timestep-specific quantization parameters in TFMQ-DM for comparison?\n\nIt is generally understood that employing finer granularity in quantization can enhance the performance of a quantized model, as it allows for a more precise representation of information within the constraints of a limited number of bits. For instance, some studies on LLM quantization employ techniques such as group-wise quantization or token-wise quantization [B], while certain approaches to diffusion model quantization utilize timestep-specific quantization parameters, as discussed in lines 120-123 in our paper.\nHowever, finer granularity typically introduces additional computational overhead and may provide an unfair advantage compared to methods that do not employ such granularity. Therefore, to ensure a fair comparison that excludes the trivial benefits derived from using finer granularity, we have chosen to deactivate the timestep-specific quantization parameters in TFMQ-DM. This allows us to focus on measuring the effectiveness of the temporal information-aware reconstruction of TFMQ-DM, which is the primary contribution of their approach.\n\n[B] ZeroQuant: Efficient and affordable post-training quantization for large-scale transformers. NeurIPS 2022", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "SAunO7a17z", "reviewer": "Reviewer_5uyz", "summary": "This paper introduces a new perspective on quantization in diffusion models. It views the quantization error as causing a \"stepback\" in time within the latent space during the denoising process. The paper proposes StepbaQ which adjusts the sampling steps to correct the sampling path and reduce the buildup of quantization errors.", "questions": "Please see weaknesses.", "limitations": "The method assumes quantization errors follow a Gaussian distribution, it actually requires the scaled latent variable to share the same mean as the quantized latent variable. Thus, it necessitates both a tailored diffusion process design and an accurate error model, which may restrict its applicability or effectiveness.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* The paper is well-organized, making it easy for readers to understand and follow the main ideas.", "weaknesses": "* This paper does not provide solid theoretical guarantees or an in-depth analysis of how the temporal shift impacts the scheduled sampling trajectory. \n\n* The study lacks a comparison of computational efficiency with other quantization approaches.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the positive review and constructive comments. We provide our responses as follows.\n\n> This paper does not provide solid theoretical guarantees or an in-depth analysis of how the temporal shift impacts the scheduled sampling trajectory.\n\nWe would like to clarify this concern. Our manuscript indeed presents a comprehensive analysis of the impact that temporal shifts have on the scheduled sampling trajectory of diffusion models. Under the assumption that quantization error is Gaussian, Section 4.1 of our paper lays the theoretical foundation by explaining how quantization error can lead to temporal shifts in the latent space.\n\nSubsequently, in Section 4.2, we delve into the consequences of these temporal shifts on the scheduled sampling trajectory. We demonstrate that such shifts are primarily responsible for the accumulation of errors, as they introduce an unexpected \"stepback\" in the denoising process, as depicted in Figure 1(b). This divergence from the scheduled sampling trajectory can significantly compromise the quality of the generated results.\n\nTo address this issue, Section 4.3 discusses calibration techniques involving the correction of sampling steps. Our empirical findings strongly support our analytical discourse on sampling trajectory deviations; the proposed sampling step correction method substantially enhances the performance of diffusion models. This evidence underscores the validity of our analysis and the effectiveness of our proposed correction strategy.\n> The study lacks a comparison of computational efficiency with other quantization approaches.\n\nPlease see \"Author Rebuttal by Authors\" above.\n\n> The method assumes quantization errors follow a Gaussian distribution, it actually requires the scaled latent variable to share the same mean as the quantized latent variable. Thus, it necessitates both a tailored diffusion process design and an accurate error model, which may restrict its applicability or effectiveness.\n\nRegarding the concern about the assumption that quantization error follows a Gaussian distribution, please refer to the \"Author Rebuttal by Authors\" section above for a detailed explanation.\nAs for the requirement that the scaled latent variable shares the same mean as the quantized latent variable, it can be readily achieved by removing the mean of quantization error. Consequently, the proposed StepbaQ can be effectively implemented across various diffusion process designs, demonstrating its broad applicability.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We would like to thank all the reviewers for their constructive comments. Common questions asked by multiple reviewers would be replied here in a unified manner.
>  Justification of modeling quantization error as a Gaussian random variable

Our assumption that quantization error follows a Gaussian distribution is inspired by the empirical findings presented in PTQD, where the authors verify the Gaussian nature of quantization error through statistical tests detailed in Appendix B of their paper. In our own empirical investigations, we observed that the distribution of quantization error typically exhibits a symmetric, bell-shaped curve, as depicted in Figure 4 in Appendix B of our paper and the attached pdf file. This observation aligns with the result shown in Figure 3 of the PTQD paper.

To delve deeper into the characteristics of quantization error, we present an analysis of both the kurtosis (Fisher) and skewness of the error distribution, averaged across all steps for each model under the W8A8 setting. Our findings indicate that the skewness values are notably small, confirming our observation that the quantization error follows a symmetric, bell-shaped distribution. Regarding kurtosis, the values observed are greater than zero for each model, suggesting that the distribution of quantization error exhibits relatively fat tails. These fat tails likely originate from clipping errors, which occur less frequently but are more significant than rounding errors.

Although the quantization error exhibits higher kurtosis than a Gaussian distribution, our proposed StepbaQ method still significantly enhances the performance of diffusion models, as evidenced by the results in Tables 1 and 3 of our main paper. This underscores the robustness and applicability of our approach, demonstrating that StepbaQ can effectively improve model performance under realistic conditions.

|  | SDv1.4 | SDv1.5 | SDXL-Turbo |
|----------|:----------:|:----------:|:----------:|
| Kurtosis| 2.627 | 3.429 | 0.502 |
| Skewness | 0.036 | -0.020 | -0.007 |

> Computational efficiency and comparisons with other quantization approaches

In our paper, we provide the runtime of StepbaQ for the 20-step SD v1.5 model at line 265. Here we further provide the approximate runtime of each quantization method applied to the SDv1.4 model, using a single A6000 GPU. For reconstruction-based methods such as Q-Diffusion and TFMQ-DM, calibration data is uniformly sampled every 20 steps, totaling 3,200 samples (128 samples per step). This sample size is reduced from that reported in the Q-Diffusion paper to expedite the quantization process. In contrast, for statistical methods like PTQD and StepbaQ, the calibration set size is set to 128.

According to the table, it is evident that reconstruction-based approaches, including Q-Diffusion and TFMQ-DM, require significantly more computation time.  This discrepancy can be attributed to the utilization of AdaRound or BRECQ, which are notably time-consuming, especially when applied to large-scale models. This has resulted in their diminished popularity in the current landscape, where model sizes are increasing, as discussed in the GPTQ[15] paper. On the other hand, statistical approaches such as PTQD and StepbaQ exhibit superior computational efficiency, as they only involve collecting the statistics of quantization error.

However, it may not be appropriate to directly compare the computation time between reconstruction-based and statistical approaches, given their different objectives. Reconstruction-based methods aim to optimize quantization parameters to minimize error at each step, whereas statistical methods adjust the sampling process to reduce the overall error. As such, these two types of approaches can be combined to enhance performance, which is demonstrated in Table 3 of our paper.

|  | Q-Diffusion | TFMQ-DM | PTQD | StepbaQ |
|----------|:----------:|:----------:|:----------:|:----------:|
| Time | 54hr | 54hr | 20min | 20min |

Meta-review: This is a cool paper on fixing quantization errors in diffusions. The paper gives a novel interpretation of quantization errors as "stepbacks" in the denoising process. The reviewers liked the paper since it has good insights, is well-written and has good experimental results. Some minor questions were well addressed by the authors during the rebuttal phase.
Decision: Accept (poster)