Leveraging Early-Stage Robustness in Diffusion Models for Efficient and High-Quality Image Synthesis
Authors: Yulhwa Kim
Dongwon Jo
Hyesung Jeon
Taesu Kim
Daehyun Ahn
Hyungjun Kim
jae-joon kim
Venue: NeurIPS 2023 poster (2023)
Keywords: diffusion models
post-training quantization

Abstract: While diffusion models have demonstrated exceptional image generation capabilities, the iterative noise estimation process required for these models is compute-intensive and their practical implementation is limited by slow sampling speeds. In this paper, we propose a novel approach to speed up the noise estimation network by leveraging the robustness of early-stage diffusion models. Our findings indicate that inaccurate computation during the early-stage of the reverse diffusion process has minimal impact on the quality of generated images, as this stage primarily outlines the image while later stages handle the finer details that require more sensitive information. To improve computational efficiency, we combine our findings with post-training quantization (PTQ) to introduce a method that utilizes low-bit activation for the early reverse diffusion process while maintaining high-bit activation for the later stages. Experimental results show that the proposed method can accelerate the early-stage computation without sacrificing the quality of the generated images.

Leveraging Early-Stage Robustness in Diffusion Models for Efficient and High-Quality Image Synthesis
Yulhwa Kim', Dongwon Jo Hyesung Jeon', Taesu Kim², Daehyun Ahn',
Hyungjun Kim², Jac-Joon Kim!
Seoul National University, "SqueezeBits Inc.
(yulhwakim, dongwonjo, hjeon2k, kimjaejoon)/annu.. ac kr, (taesu. kim, daehyun.. ahn, byungjun.kim/ kim) com
Abstract
While diffusion models have demonstrated exceptional image generation capabili- ties, the iterative noise estimation process required for these models is compute- intensive and their practical implementation is limited by slow sampling speeds. In this paper, we propose novel approach to speed up the noise estimation network by leveraging the robustness of early-stage diffusion models. Our findings indicate that inaccurate computation during the early-stage the reverse diffusion process has minimal impact on the quality of generated images, as this stage primarily outlines the image while later stages handle the finer details that require more sen- sitive information. To improve computational efficiency, we combine our findings with post-training quantization (PTQ) and introduce a method that utilizes low-bit bit activations for the early reverse diffusion process while maintaining high-bit acti- vations for the later stages. Experimental results show that the proposed method accelerate the ctrly-stage computation without sacrificing the quality of the generated images.
Introduction
Diffusion models years due to their exceptional proficiency in generating high quality and diverse images. A key feature of the diffusion models is that they gradually dissipate noise from latent variables across a sequence of diffusion steps [1] However, despite their remarkable image synthesis capabilities, diffusson models suffer from slow image synthesis process. This is primarily caused by the compute-intensive nature of the denoising network (3] 19, which is integral part of the diffusion model. The iterative process of applying the denoising network across multiple diffusion steps consumes substantial computational resources, resulting in very long synthesis time [6]. As result, this limitation poses a challenge for real-time applications or scenarios that require quick image synthesis.
for image synthesis have gained significant attention recent
Quantization IO 12 is widely used technique for improving the computational efficiency of neural networks. Quantization reduces memory usage by converting high-precisioe floating- ppint weight/activationn values into low-precision integer values [13] It also enables more efficient pro- cessing by leveraging efficient integer operations instead computationally expensive floating-point operations [14. As a result, there is growing research trend focusing on applying quantization techniques to denoising networks. Two notable studies, PTQ4DM [I5) and Q-diffusion [16), explored the applications of LAPQ [17] and BRECQ [18). which are commonly used Post- Training Quanti- zation (PTQ) frameworks, to the diffusion models. Denoising networks differ from conventional neural networks they are iteratively used during the reverse diffusion process, and their output distributions vary with diffusion step. Considering this characteristic, the aforementioned studies primarily explored methods for constructing calibration dataset effectively apply PTQ to diffusion
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Forward diffusion proces
U-Net (Denoising Network)
Xp
No.1
Xi
Xy
Xy:
Xy
Reverse diffusion proces
Xy
Xy+x
X2
No
Figure 1: (a) Overview of the diffusion process (b) U-Net architecture of the denoising network
models. While they have successfully achieved a reduction in weight precision to low 4 bits, reducing activation bits below bits still remains a challenge.
In this paper, we investigate the computational characteristies of diffusion models during their iterative diffusion steps and present a novel strategy for quantizing diffusion models based on the analysis. Our analysis of the diffusion model reveals the high tolerance of carly-stage reverse diffusion in the presence of computational errors. In the reverse diffusion process, the early stage focuses on capturing the outlines of the images, while the later stage refines the details to improve the overall quality of the generated images. As a result, modifications made the early stages primarily affect the structural aspects of the generated images, while the quality remains mostly unaffected. Based on observation, we propose leveraging the early-stage robustness observed the diffusion process to optimize activation bit precision, thereby reducing computational overhead while maintaining the fidelity and diversity of the generated images. By strategically quantizing the activation values reduce the effective bit precision of the activations of the denoising
in step-wise manner, networks without sacrificing the overall quality of the generated images.
Background
2.1 Diffusion Models
The generation of an image diffusion models is represented as series of diffusion steps (Fig. aa)) Н3. In the forward diffusion process, cach step adds noise to the data, eventually making the data follow a Gaussian distribution. To generate an image zo from a Gaussian noise Iy (00, 1). diffusion models reverse the forward process and sample latent variables from a learned conditional distribution po po which approximates the true reverse conditional distribution follows:
pa(a,-11 (x.) = N(x, 1: He (Z,, o?
More specifically, by combining Eq. and DDIM sampling method 66, the reverse diffusion process yields the following equation to generate sample:
(12)) + V1- co(2,, oze Vai
24-1
04
(2)
where e, N(0, 7) is Gaussian noise independent of xq. Here, is coefficient determined by the noise scheduling policy of the diffusion process, and o, is a hyperparameter that controls the degree of stochasticity of the diffusion process. Note that 2,, the output of the diffusion step t, becomes the input of the next diffusion step 1, and this process continues until the image zo is reconstructed from the noise ZT.
(at,, t) in Eq. represents a trainable denoising network diffusion model. Diffusion models often use the U-Net architecture (Fig. (b)) for their denoising network (9]. The U-Net consists of an encoder-decoder structure which the encoder progressively decreases the spatial dimensionality of the input image, extracting high-level features. Conversely, the decoder performs upsampling operations on the feature maps and incorporates skip connections from the encoder to reconstruct the original image resolution while generating low-level features. In the context of diffusion models,

---

the U-Net denoising model is trained to reduce noise in the latent variables at each diffusion step. By iteratively applying the denoising model across multiple diffusion steps. the generated images gradually exhibit improved clarity. However, it is worthwhile to note that the compute-intensive nature of the denoising network makes the generation process of diffusion models very time-consuming.
2.2 Post-Traine Post-Training Quantization
A typical uniform quantization function for converting floating-point value æ into integer value Zint as follows:
Fint = 8. clamppl I 27, milguaant maZguantt
21,
where 8 a scaling factor and z is the zero point. Post-training quantization (PTQ) TO 12
17
18
refers to technique for compressing neural networks that is typically applied after the network has been trained unlike quantization-aware training (QAT) [T] 19 20, 21, 22), which considers quantization while training the network. One common PTQ strategy is to collect calibration data from the training data and calibrate the scaling factor or the rounding scheme (c.g. AdaRound (TO)..
Recent studies, PTQ4DM 15 and Q-diffusion [16]. have proposed PTQ methodologies for denoising network of diffusion models. PTQ4DM and Q-Diffusion provided insights into the methodology for constructing the PTQ calibration dataset for diffusion models. Specifically, since the denoising network is applied across multiple diffusion steps, these papers discussed how to consider the diffusion steps when constructing the calibration data. Both works intensively analyzed the distribution of activations in different diffusion steps, but they differ in the approaches to constructing the calibration dataset. To improve the quality of the calibration dataset, PTQ4DM adopts method of extracting the diffusion steps from a distribution (J,, 0.5T)) where μt S 0.57' drawing more samples from the diffusion steps closer to 0.07' On the other hand, Q-diffusion divides the diffusion steps evenly into constant intervals to construct the calibration set, leveraging the similarity of activation distributions of adjacent diffusion steps. As result, PTQ4DM achieved INT8 weights and INT8 activations for high-quality quantized diffusion models, while Q-diffusion models demonstrated INT4 weights and INT8 activations.
Previous works on quantizing diffusion models demonstrate that optimizing the construction of the calibration set can successfully reduce the bit precision of weights to as low as 4 bits [16). However, they still require 8-bit activations, which limits network efficiency. To further reduce the activation precision, we first investigate the robustness of each diffusion step in the reverse diffusion process.
3 Etrly-Stage Robustness of Reverse Diffusion Process
3.1 Properties of Reverse Diffusion Process
The reverse diffusion process generates images progressively starting from Gaussian noise (8) As shown in Fig. 2[a), a). the carly stage of the reverse diffusion process captures the outlines of the images. In this stage, sampled images are rough and blurry sketches, so the shapes of the objects are hardly recognizable. In the later stage, because the overall structure of the images are properly shaped, the reverse diffusion process draws details of the images improve the quality of the generation.
To quantitatively analyze the characteristics of the generated latent variables 2, at each diffusion step of the reverse diffusion process, we use entropy output at each step. Note that a higher entropy indicates higher randomness in the pixel values of the images, meaning they are blurrier. The entropy values of the latent variables Iy at each step are summarized in Fig. b). The carly steps ranging from 1.0l' to 0.5l' maintain high entropy values, indicating that sampled images are noisy with high randomness. On the other hand, the entropy rapidly decreases after 0.57' step. These latter steps generate details of the image to enhance the quality of image generation, so the sampled images become sharper and their randomness decreases.
measure of the randomness exhibited by the
Since the characteristies of the generated images vary each step of the reverse diffusion process, we can expect that the computational requirements for each step may differ as well. Specifically, the entropy of the sampled images remains constant with relatively high value the early stage, suggesting that the computations performed during this stage may not need to be as precise as that for the later stages. However, in later stages where the computations have a greater impact on the entropy results, more accurate computations are necessary.
3

---

1.OT
0.OT
Entropy Transition
4.65
4.60
Entropy
4.55
4.50
LSUN-Dedrooms
Chur-churches
CelebÁ
FFHQ
4.45
1.0T
0.8T 0.6T 0.4T 0.2T 0.0T
Diffusion Step (t)
(a)
(b)
Figure 2: Studies the reverse diffusion process with respect to diffusion steps. (a) Examples of 256x256 LSUN-Bedromms, LSUN- SUU-Churches, and CelebA-HQ generated each step the reverse diffusion process. (b) Entropy transition across the diffusion steps for various image generation tasks.
32x32 Cifar-10
256x256 CUUNChhrchess
150
Noise Injection
Noise Injection
0.T
0.7T
0.2T
0.7T
0.3T
0.8T
0.3T
0.8T
100
0.4T
0.9T
FID
50
5
0.05
0.20
0.35
0.50
0.05
0.10
0.15
0,20
T of Injected Noise
T Injected Noise
Figure 3: Image generation quality (FID) after noise injection. 5k images of (left) 32x32 CIFAR-10 and (right) 256 L.SUN-Churchee are generated for the evaluation. The magnitude of the noise controlled by r using Eq. The diffusion step exhibits lower FID after noise injection indicates higher level of resilience to imaccurate computations.
3.2 Early-Stage Robustness of Reverse Diffusion Process
In this section, we evaluate the robustness of the reverse diffusion process at each diffusion step. To measure the robustness, we inject random noise into the output of the targer diffusion step / (Eq.2). and assess the quality of the generated image To after the entire reverse diffusion process using the Fréchet Inception Distance (FID) [23]. The FID helps to understand how much the computational inaccuracy at step affects the quality of overall diffusion process. The injected noise follows Gaussian distribution N (0, o.. 'T), where o,, is the standard deviation of z and controls the noise standard deviation. The noise injection modifies the latent variable o; as follows:
Fo.noisy F. +
N(O, (az, :7)
(4)
The analysis results are summarized in Fig. The cases of 32x CIFAR- and 256x256 LSUN- Churches generation exhibit different responses to the noise injection. This discrepancy is attributed the higher resolution of LSUN-Churches images, which introduces finer details into the generation process. Nevertheless, in both cases, there is consistent pattern: the standard deviation of injected noise (7) increases, the FID also increases, while the correlation between noise and FID varies signficantly depending the diffusion step. In the early diffusion stages, close to 1.07', observe

---

w/o
w/ noise injected at
w/o noise injection (baseline)
noise
1.0T
0.OT
w/ early stage noise injection
(a)
(b)
Figure 4: Examples of image generation after noise injection. (a) Comparison of the CIFAR-10 images generated without noise and with noise injected in different diffusion steps. (b) Comparison of the generated images with and without early-stage noise injection. The early-stage noise injection changes the overall structure of the images rather than their quality.
that the quality of the image is relatively tolerant noise. The early stages exhibit resilience high magnitude of noise, as can be found by the maintained FID even with high r values. However, the later stages more susceptible the injected noise, leading degradation in the FID score. This analysis confirms that the early stages are more robust to the noise than the latter stages.
To gain more intuitive understanding of this phenomenon, we examine how the final output images change when noise is injected at each diffusion step (Fig. 4). When noise is added to the back end of the diffusion steps to capture image details, the resulting image maintains a similar overall structure but exhibits lower quality due to the injected noise. contrast, when noise is added in the early stages, high-resolution image details remain intact while the shape of the image changes due to the noise which affects the process of capturing the overall outline. For example, as shown in Fig. for CIFAR-10 (24 image generation, injecting noise in the initial stage causes the image at the bottom row (originally a green airplane) to become a green automobile or a slightly different shaped airplane. The diffusion model is trained to generate images similar those belonging to the CIFAR-10 dataset, regardless of the specific class. Since both airplane and automobiles fall within the CIFAR-10 class categories, the diffusion model generates images as intended. Similarly, in the case of bedrromms, LSUN-churches [25] CelebA-HQ (26), and FFHQ (27] generation (b). injecting noise in the early stage leads to changes the overall image shape, rather than impacting image quality significantly. The effect of shape alteration while maintaining image quality similar to that of modifying the random seed 3r of the diffusion. The diffusion model initiates image generation from FT and gradually acquires overall shapes and colors of the images. Therefore, modification in the early stage would introduce changes to the overall structure of the generated images, not the quality.
This finding highlights the importance of understanding the varying robustness of diffusion models across different sampling steps, which can be a critical factor in improving the efficiency of denoising networks. When trying speed up a neural network computations, approximated operations are often employed. For instance, quantization techniques approximate FP operations as integer operations to enhance processing efficiently. However, the fact that robustness varies across each step suggests that the degree to which operations can be approximated also differs across each step. Therefore, to improve efficiency without compromising the generation quality of the diffusion model, there is great need to analyze and optimize each step individually.
4 Robustness-Aware Quantization of Diffusion Models
4.1 Impact of Weight and Activation Quantization on Diffusion Models
As the first step of quantizing the diffusion model, we evaluate the impact of quantization on each component of the denoising network. We begin by examining the impact of weight and activation
5

---

Weight Quantization
Activation Quantization
12
11.41
332.9
341.3
193.8
10
10a
FID
8.34
FID
22.5
6.05
6.43
6.07
6.22
10a
6.05
6.4
5
Bit Precision
Bit Precision
Figure 5: Image generation quality as measured by FID after (left) weight quantization (right) activation quantization of the denoising network. The diffusion model LDM-8 [8] is used generate 5k images of 256x LSUN-Churches for the evaluation. The dashed gray lines indicate FID with the full-precision network. Note that the y-axis of the left graph is on linear scale while the y-axis of the right graph is on logarithmic scale.
1.0T,0.8T 044:0.27 027.0.0T 1.01.000
Quantized Steps
10a
1.000.00
0.20:000T
0.4T:02TT
0.6T0.4T
0.8T:0.6T
1.07:0.8T
101
Bit Precision
Figure 6: (left) Examples and (right) FID of 256 L.SN-Chhrches generation with activation quantization across different diffusion steps. Example images are generated by applying 4-bit activation quantization in the target diffusion steps, and FID is measured after 5k image generation. The dashed gray lines indicate FID for the full-precision network
quantization. We BRECQ 18 a commonly utilized PTQ framework, for weight quantization and the calibration dataset is constructed based on the method proposed in Q-diffusion [16). For activation quantization, we employ the uniform quantization function described in Eq. 3 Similar to Q-diffusion, we apply quantization for all layers of the denoising network that perform matrix multiplication, including convolution, linear, and attention layers, while keeping the activation values of attention score matrix to 8 bits. Fig. shows that FID degradation is negligible values so that low-bit weight quantization is relatively straightforward. contrast, FIDD deteriorates more rapidly when the activation quantization bit decreases. Hence, we focus on reducing the number of activation bits.
to 4 bit weight
4.2 Dynamic Nature of Activation Quantization across Diffusion Steps
Activation quantization introduces error when converting floating-point activations to integer values. In the case of uniform quantization, where a finite range of data is evenly divided using specified number of bits, the quantization error of the target data is inversely proportional to the number of quantization bits. Furthermore, as discussed Section 3 the robustness of diffusion models varies across different diffusion steps. Hence, we can expect that the bit precision of activations at each diffusion step will have distinct effects on FID. To investigate the impact of activation quantization on different diffusion steps, we divide the diffusion steps into five intervals [1.07:0.3 8T, 0.87:00677
0.27 :0.0 00.07", and selectively apply activation quantization to target step interval while keeping the floating-point activation values in the other steps. For instance, if the target interval is set as 1.07:0.87. activation quantization is applied within this interval only, while activations in the other diffusion steps remain unquantized. Fig, indicate that the carly-stage diffusion steps closer to 1.07

---

exhibit higher resilience to quantization errors. In the case of applying 4-bit activation quantization to the 0.2T:O.0T diffusion steps in 5k Chuhhhhe generation, the resulting FID is 34.30. However, when the same 4-bit activation quantization is applied to the 1.07:0.87 steps, the resulting FID is 6.10, which is close to the FID of 6.05 achieved through image generation using a full-precision network.
worthwhile to emphasize that the dynamic nature of activation quantization during network processing eliminates the requirement of using identical activation bits for each diffusion step. This characteristic alleviates burden on adjustment of network parameters or other factors to reduce the activation bits. Consequently, by leveraging the robustness observed in the early stages, it becomes possible further reduce activation bits, leading to more efficient processing of the denoising network.
4.3 Proposed Robustness-Aware Quantization
The previous section has shown that using low activation bits (e.g. 4 bits) for the denoising network in the early diffusion steps does not have a significant impact on the quality of image generation. However, the reverse diffuuion process progresses, it becomes increasingly challenging to lower the activation bits while maintaining the generated image quality. Therefore, we propose a quantization technique that leverages robustness observed in the early stages to minimize the activation bits as much as possible.
Algorithm 1 Robustness-awar Quantization
Require: Weight-quantized denoising network 60,
Require: Lower bound of the activation bit a
Require: Number of sampled images for evaluation and threshold of FID Arid b> Dacti Dictionary for matching steps and activation bits D Optimizing activation bits for each step
Bilact =o, Dact = ()
for L to do
while (1 not in Dart.keys() or (Doetll) f Bilnet) do Duce[t]= Bitact
Sample n images with Eq, applying Duct for acitvations if FID of sampled images > T/ then
Bitast Bitact +
end if end while end for
The proposed robustness-aware quantization (RAQ) is presented in Algorithm It begins with pre-quantizing the weights of the denoising network, utilizing 4 bits for weight quantization, as weights are relatively easier to quantize. The optimization of activation bits starts from diffusion step T and proceeds sequentially for each step. At each step, the dictionary Duet is updated to establish the correspondence between the given activation bit Bitact and the diffusion step t. This dictionary is used by the denoising network adjust activation bits. FP32 activations are employed for diffusion steps that have not yet been updated in the dictionary. Subsequently, using the updated bit information, the diffusion model generates small subset images, and their FID is evaluated. If the FID exceeds the desired threshold, the activation bit Biluet is increased by 1, and the image quality re-evaluated. Note that RAQ employs the FID of diffusion models without activation quantization as the FID threshold to preserve the quality of image sampling after the quantization. Nonetheless, there exists the flexibility to modify the FID threshold according to the user-defined level of FID tolerance, enabling greater reduction in activation bitwidths. The iterative process continues until the desired FID is achieved. Once the FID meets the desired criterion, the optimization proceeds to the next diffusion step. Notably, the optimization starts from the previously adopted activation bit and does not explore lower bit values. This strategy acknowledges the increasing difficulty of lowering activation bits as diffusion steps progress. As result, the time complexity for optimizing activation bits at each step reduced from O(m²) ) to O(m + T), where m represents the number of bit candidates for activation.
7

---

Table Unconditional image generation results of the baselines and the proposed RAQ. (W: weight bidwidth, A: effective activation bitwidth, TBOPs: Tera Bit Operations)
Baseline
LSUN-Churches (256x256)
LSUN Bedrooos (256x256)
Model
W/A
FID4
TBOPs
Model
W/A
FID
TBOPs
LDM-8 8
32/32
4.09
4285.2
LDM-4 4
8
32/32
2.96
20725.8
4/32
4.46
769.2
4/32
4.13
3120.7
Q-diffusion 16
Q-diffusion 16
4/8
4.45
148.6
4/8
4.17
681.2
Proposed RAQ (w/ 4-bit weights)
LSUN-Churches (256x256)
LSUN- BUN-Bedrooms (256x256)
A
Dant
FID
TBOPs
A
Dant
FID4
TBOPs
6.00
entire range- 6b
5.31
108.8
6.00
entire range 6b
5.58
504.8
[1.00T:0.80%] 4b
11.000:0.90T7
4b
[0.801:0.600 60T" 5b
[0.90T 5b
6.00
:0.60] 6b
4.64
108.8
6.00
[0.602 6b
3.99
504.8
[0.50T:0.107]
[0.40T:0.107]]
7b
[0.107:0.007]]
[0.107:0.007]
[1.00T:0.757]]
[1.00T:0.907] 4b
00.757:0.6007 5bb
5.60
[0.60T:0.22) 257"
5.12
101.3
5.50
[0.90T:0.601] 5b
4.94
461.6
[0.60T:0.000T) 6b
[0.25T:0.00T] 7b
[1.00T:0.75]] -
[1.007:0.857] 4b
5.35
(0.757:0.607)) - 5b
5.51
96.4
5.45
[0.857:0.607) :0067) 5b
5.32
457.3
[0.60T:0.00T]] 6b
[0.600:0.007"] 6b
Experiment
5.1 Unconditional Image Generation with Proposed Robustness-Awar Quantization
We first evaluate the effectiveness of the proposed method for unconditional image generation. We use Latent Diffusion Model (LDM) 8 with DDIM sampling [6] for 256x256 image generations, with total of 200 diffusion steps. The lower bound of the activation bit is set 4, as we observed that using 3 bits significantly degrades the quality of the generated images even with the proposed RAQ. As we find that consecutive timesteps within 0.05T range exhibit similar sensitivity to noise injection, we set the granularity of Bit update as 0.057'. We sample 5,000 images for the optimization of the quantization bits, but we use a larger sample size of 50,000 images for the final evaluation. We evaluate the performance of our approach using two datasets: 256x256 LSUN Bedrooms and LSUN Churches. Note that the generation of LSUN Bedrooms use diffusion model LDM-4 and LSUN Churches use a diffusion model LDM-8 S The quality of the generated images evaluated using the FID metric. To estimate the efficiency improvement of RAQ, we introduce the concept of Bit Operations (BOPs), where BOPs are calculated as the product of OPs (Operations) with weight bitwidth and activation bitwidth.
The evaluation results of the baseline method, which does not employ the step-wise bitwidth opti- mization, are presented in the top of Table Despite optimizing the activation bitwidth with 0.05T granularity, certain consecutive intervals exhibit similar sensitivity to quantization, resulting in the the RAQ results being composed of only or fewer intervals. As demonstrated in Q-diffusion [16), utilizing 4-bit weights and 8-bit activations preserves FID at a similar level to that of floating- point models. However,
observed that FID substantially increases when 6-bit activation values are used in the entire range of the steps. In contrast, when applying our proposed RAQ method, observe FID scores comparable those obtained with 8-bit activations even when the effective bit count of the activation is (Table I. Thus, for
the proposed RAQ

---

Full Precision
W4A6
W4A8 (Q-Diffusion)
W4A6/8 (Proposed)
Figure Text-guided 512x512 image generation results with Stable Diffusion. The denoising model with full- cellppecciioo, 8-bit (Q-diffusion), and 6-bit activations, and scalable-bit acitvations with the proposed RAQ method are evaluated for the comparison. Here, W4 denotes adopting 4-bit weights, An denotes n-bit activations, and An/m indicates adopting n-bit activations in the early stage of the diffusion stages, while the later stage adopts m-bit activations
achieves 1 times smaller BOPs compared to the full full-precision baseline, while Q-diffusion only achieves 28.8/ 18.8/30.4 .4 times smaller BOPs compared to the full-precision baseline. Furthermore, by slightly compromising FID, becomes possible to further reduce the effective activation bits. These experimental results clearly demonstrate that leveraging the early-stage robustness of the diffusion model contributes to maintaining the quality of image generation even with low-bit activations.
5.2 Conditional Image Generation with Proposed Robustness-Awar Quantization
In this section, we assess the effectiveness of the proposed method for conditional image generation using Stable Diffusion [8. We employ the PLMS sampler (28 with 50 diffusion steps to generate 512x512 images. Since there is no established metric to quantitatively measure the image generation quality in this case, rely qualitative comparison for each image. As presented in Figure even when the proposed RAQ method not applied, 8-bit activations produce high-quality images that are comparable with full- full-precision models. However, there is a significant degradation in image quality

---

when the activation bits are reduced to 6 bits. On the other hand, by applying 6-bit activations only to the [1.0T:0.8T] diffusion steps and retaining 8-bit activations for the remaining steps. the diffusion models are able to generate high-quality images. This result underscores the importance of leveraging the early-stage robustness of the diffusion process.
Meanwhile, unlike the unconditional image generation cases in Section 5.1 we had difficulties in applying 4-bit activations to the early-stage diffusion process of Stable Diffusion. This challenge arises from the large dynamic range of the activation values in Stable Diffusion. Hence, we plan to explore more sophisticated techniques to address this issue in the future work.
6 Limitation
An important limitation of the proposed RAQ is that current results do not enhance running time efficiency on GPUs. This is primarily due to the substantial presence irregular activation bitwidths, such as 6 bits [T and accelerating diffusion models with irregular bitwidths GPU does not lead to performance improvement due to the lack of corresponding arithmetic units in GPU.
However. we would like to emphasize that our main contribution is to show a direction that we can reduce bit resolution for a part of parameters, and our approach is not fundamentally limited to a certain bit resolution. Hence, if other works that can further reduce the overall bit resolution of the network are developed independent of our scheme, then it can combined with our proposed scheme, and there is a chance that portion of the 4-bit parameters in our scheme can be increased to see realistic performance benefits on GPU. For example, our current approach involves a basic min/max-basee quantization mechanism for activation quantization, leading to an 8-bit quantization for the baseline fixed bitwidth case. Then, the allocation of activation bits using the RAQ method is distributed as (4b, 5b, 6b, 7b, 8b) (20%. 20%, 10%, 40%, 10%) for LSUN-churches (Table 1). However, in the scenario where an advanced quantization mechanism enables 6-bit activation quantization even for the bascline fixed bitwidth case, there exists the possibility of increasing the proportion of 4-bit activations with the proposed RAQ. such as (4b, 5b, 6b) (50%, 30%, 20%)., which can then be effectively accelerated on GPUs.
Moreover, the utilization of specialized hardware, like bit-scalable accelerators, could offer promis- ing solution for processing these models [29]. These accelerators are purpose-buil to harness the benefits of quantization on a bit-by-bit basis, resulting in a nearly linear improvement in computing efficiency, encompassing both latency and energy consumption, with decreasing bit width. To show potential benefit from such bit-scalable hardware, we analyze the BOPs Table [ This metric allows to estimate the performance gain achievable through the RAQ method. In the case of LSUN-Churche Chu-hhhre the full-precision baseline and Q-diffusion necessitates 4285.2 and 148.6 TBOPs for a single image generation respectively, while the proposed RAQ method only requires 108.8 TBOPs. This indicates that with the utilization of specialized accelerators, the implementation of the RAQ approach could potentially lead to more than 39.4 times speedup and energy savings compared to the full-precision baseline, while Q-diffusion can achieve 28.8 times improvement compared to the full- full-precision baseline.
7 Conclusion
In this paper, we present a comprehensive analysis of the reverse diffusion process and introduce strategy optimize diffusion models using quantization techniques. Leveraging the observed robustness of the early stages in the diffusion process, we successfully optimize the activation bitwidth for each diffusion step, leading to reduced effective activation bit precision without compromising the fidelity and diversity of the generated images. Experimental results show that our proposed method (RAQ) effectively reduces the effective bitwidth of activations to 6 bits while maintaining image quality comparable to conventional approaches that employ 8- activations. Additionally, we also evaluate the effectiveness of the proposed method the context of conditional image generation using Stable Diffusion, state-of-the-ar diffusion model.
10

---

Acknowledgements
This work was supported in part by Institute of Information & communications Technology Planning & Evaluation (ITP)) grant funded by the Korea government (MSIT) (No. 2021-0-0010S: Development of model compression framework for scalable on-device Al computing on Edge applications IITP- 2023-RS-2023-4 RRS-2023--0225081 artificial intelligence semiconductor support program to nurture the best talents, No. 2021-0-01343: Artificial Intelligence Graduate School Program (Seoul National Artificial Intelligence Innovation Hub), Basic Science Research Program through the National Research Foundation of Korea (NRF) funded by the Ministry of Education (2022R A6A3 BK21 FOUR program and Inter-university Semiconductor Research Center at Seoul National University. (Corresponding Author: Jae-Joon Kim)
University), fo.20011--02000
References
[1] Jascha Sohl-Dickstein, Eric A. Weiss, Niru Maheswaranathan, Surya Ganguli, "Deep Unsupervised Learning using Nonequilibrium Thermodynamics" International Conference on Machine Learning (ICML), 2015.
[2] Yang Song and Stefano Ermon. "Generative modeling by estimating gradients of the data distribution". Advances in Neural Information Processing Systems, (NeurIPS), 2019.
[3] Jonathan Ho, Ajay Jain, Pieter Abbeel, "Denoising Diffusion Probabilistic Models", Advances in Neural Information Processing Systems, (NeurIPS), 2020.
[4] Alex Nichol, Prafulla Dhariwal, "Improved Denoising Diffusion Probabilistic Models", International Conference on Machine Learning (ICML), 2021.
[5] Yang Song. Jascha "Score-Base Basee Generative Modeling through Stochastic Differential Equations", International Conference on Learning Representations (ICLR), 2021.
Diederik P. Kingma, Abhishek Kumar, Stefano Ermon, Ben Poole,
[6] Jiaming Song, Chenlin Meng, and Stefano Ermon, "Denoising diffusion implicit models", International Conference Learning Representations (ICLR), 2021.
[7] Prafulla Dhariwal, Alex Nichol, "Diffusion Models Beat GANs on Image Synthesis", Advances in Neural Information Processing Systems, (NeurIPS), 2021.
[8] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, Bjöm Ommer, FHigh-Resolution Image Synthesis With Latent Diffusion Models", IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
[9] Olaf Ronneberger, Philipp Fischer, Thomas Brox, Tijmen Blankevoort, "U-Net: Convolutional Networks for Biomedical Image Segmentation". Medical Image Computing and Computer-Assisted Intervention (MICCAI), 2015.
[10] Markus Nagel, Rana Ali Amjad, Mart van Baalen, Christos Louizos, Tijmen Blankevoort, "Up or Down? Adaptive Rounding for Post-Training Quantization", International Conference on Machine Learning (ICML), 2020.
[11] Steven K. Esser, Jeffrey L. McKinstry, Deepika Bablani, Rathinakumar Appuswany, Dharmendra S. Modha, "Leamedd Step Size Quantization", International Conference on Learning Representations (ICLR), 2020.
[12] Xiuying Wei, Ruihao Gong, Yuhang Li, Xianglong Liu, Fengwei Yu, "QDrop: Randomly Dropping Quantization for Extremcly Low-bit Post-Training Quantization", International Conference on Learning Representations (ICLR), 2022.
[13] Sein Park, Junhyuk So, Juncheol Shin, Eunhyeok Park. "Profit: novel training method for sub-4-bit mobilenet models", European Conference on Computer Vision (ECCV), 2020.
[14] Kuan Wang. Zhijian Liu, Yujun Lin, Ji Lin, Song Han, "HAQ: Hardware-Awarc Automated Quantization with Mixed Precision". IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
[15] Yuzhang Shang and Zhihang Yuan and Bin Xie and Bingzhe Wu and Yan Yan, "Post-training Quantization Diffusion Models". IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
[16] Xiuyu Li, Long Lian, Yijiang Liu, Huanrui Yang, Zhen Dong, Daniel Kang, Shanghang Zhang, Kurt Keutzer, "Q-Diffusion: Quantizing Diffusion Models", arXiv preprint arX 2023.
11

---

[17] Yury Nahshan, Brian Chmiel, Chaim Baskin, Exgenii Zheltonozhskii, Ron Banner. Alex M. Bronstein, Avi Mendelson, "Loss Aware Post-training Quantization", arXiv preprini 1rXiv:191..7790, 2019.
[18] Yuhang Li, Ruihao Gong, Xu Tan, Yang Yang. Peng Hu, Qi Zhang, Fengwei Yu, Wei Wang, Shi Gu, "BRECQ: Pushing the Limit of Post- Post-Trainin Training Quantization by Block Reconstruction", International Confer- ence on Learning Representations (ICLR), 2021.
[19] Yash Bhalgat, Jinwon Lee. Markus Nagel, Tijmen Blankevoort, Nojun Kwak, "LSQ+; Improving low-bit quantization through learnable offsets and better initialization", IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), 2020.
[20] Jungwook Choi, Zhuo Wang. Swagath Venkataramani, Pience I-Jen Chuang, Vijayalakshmi Srinivasan, Kailash Gopalakrishnan, "PACT: Parameterized Clipping Activation for Quantized Neural Networks", arXiv preprint arXiv: 180.006055 2018
[21] Sangil Jung. Changyong Son, Seohyung Lee, Jinwoo Son, Youngjun Kwak, Jae-Joon Han, Sung Ju Hwang. Changkyu Choi, "Learning to Quantize Deep Networks by Optimizing Quantization Intervals with Task Loss", JEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
[22] Ruihao Gong, Xianglong Liu, Shenghu Jiang, Tianxiang Li, Peng Hu, Jiazhen Lin, Fengwei Yu, Junjic Yan, "Differentiable Soft Quantization: Bridging Full-Precision and Low-Bi Bit Neural Networks", IEEE International Conference on Computer Vision (ICCV), 2019.
[23] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bemhard Nessler, Sepp Hochreiter, "GANs Trained by Two Time-Seale Update Rule Converge to Local Nash Equilibrium" Advances Neural Information Processing Systems, (NeurIPS), 2017.
[24] Krizhevsky, Alex, and Geoffrey Hinton, "Learning Multiple Layers Features from Tiny Images", 2009.
[25] Fisher Yu, Ari Seff, Yinda Zhang, Shuran Song, Thomas Funkhouser, Jianxiong Xiao, "LSUN: Con- struction of Large-scale Image Dataset using Deep Learning with Humans the Loop". arXiv preprint arXiv:1506.0320 2015.
[26] Tero Karras, Timo Aila, Samuli Laine, Jaakko Lehtinen, "Progressive Growing GANs Improved Quality, Stability, and Variation", International Conference on Learning Representations (ICLR), 2018.
[27] Tero Karras, Samuli Laine, Timo Aila, Style-Bash Based Generator Architecture Generative Adversarial Networks". JEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
[28] Luping Liu, Yi Ren, Zhijie Lin, Zhou Zhao, "Pseudo Numerical Methods for Difffusion Models on Manifolds", International Conference on Learning Representations (ICLR), 2022.
[29] Yonggan Fu, Yang Zhao, Qixuan Yu, Chaojian Yingyan Lin, "2-in-l accelerator: Enabling random pre- cision switch for winning both adversarial robustness and efficiency"", 54th Annual IEEE/ACM International Symposium on Microarchitecture (MICRO), 2021.
12

---

A Experimental Details
A.1 Details of Diffusion Models
In order conduct comprehensive analysis of the early-stage robustness in diffusion models, we employ five different diffusion models to generate various datasets, including CIFAR [24). FFHQ (2) CelebA-HQ (26). SSUN-bedrooms, and LSUN-churches [25]. To meausre the Fréchet Inception Distance (FID), we utilize the torch-fidelity library following the methodology established previous works 66] 8 Subsequently, we apply the proposed RAQ method to perform high-
resolution image generation tasks, encompassing both unconditional image generation for LSUN- Bodrooms/LSUN-chunee of the diffusion models utilized in these experiments are thoroughly presented in Table Z Note that
and conditional image generation using Stable Diffusion [8] The details
the calculation of o in Eq. 2 performed using v) in Table as specified in the following equation 6
ot = V(1 a a--1)/(111 (L)aaaa))(aaaa)/ as/a-11
(5)
Table 2: Implementation specifications of the diffusion models
FFHQ/ CelebA-HQ
LSUN- Bedrooms
LSUN- Churches
Conditional Generation
CIFAR-10
Image Size
32x32
256 256
256 x256
256x256
512x512
Architecture
DDIM
LDM-4
LDM-4
LDM-8
Stable Diffusion v1.4
Sampler
DDIM
6
DDIM
DDIM
DDIM
PLMS 28
Step Count
100
200
200
200
50
7
0/1
0
0
A.2 Details of Entropy Analysis
In Section 3.1 calculate the entropy of the latent variables H, for each diffuusion step. To facilitate this calculation, we transform the values of a, into histogram bins. Specifically, we map 3e to histogram bin using the following equation:
(xx))) = clamp() Et
1, -3, +3)
66)
256
This equation ensures that the values of the histogram bins are constrained within the range of -3 to +3, allowing us to effectively create a histogram with 256 bins, Once we have the histogram representation of Tr, calculate the entropy using the equation
H(X) Σ (og))
A.3 Intermediate Image Prediction during Reverse Diffiusion Process
In Fig. a[a) and Fig- 2 we showcase the intermediate image prediction results during the reverse diffusion process, aiming to illustrate the characteristics of diffusion models. To visualize the prediction results, we utilize the following zo prediction of each diffusion step as stated in [6]:
Lt
VI (Ico
(8)
'https://github. 'ntts://gitub..oo/ossssst com/hoshas/orrcc-ffdiiit
"https://githubb com/errmongroup/ddimm /twpo//////ttttttooo//o/oooooooo
3 Phutp::/githubb http:///gghuuuooooooooooooo cwm/Comp Vis/latent-diffssion
4 "htrps://github httpps//giiuu..om/c... com/Comp Vis/stalleddiffssion
"Cluuee Elwood Shannon. "A Mathematical Theory of Communication" Bell system technical journal, 1948
13

---

A.4 Details of Activation Quantization
During the activation quantization process, we observed that the skip connections of ResBlocks, the first convolutional layer responsible for transforming the latent variable into the input of the denoising network, and the last convolutional layer responsible for transforming the output of the denoising network (Fig. [))) had significant impact on the quality of the final image. However, these components constitute a negligible fraction the overall computation. Therefore, in order to balance computational efficiency and image quality, the proposed RAQ adjust the activation bits of the diffusion models to the desired bit precision while keeping the activation bits of these three components fixed at 8 bits.
B
Additional Results
B.1
Activation Quantization across Diffusion Steps
Fig. 9 complements the image generation results presented in Fig. 6 Fig. showcases image genera- tion with 4-bit activation quantization different diffusion steps, alongside the image generation with floating poating-ppintt activations. The results of the activation quantization demonstrate consistent trend with the noise injection test 4 When 4-bit activation quantization is applied to the early stages, the resulting images closely resemble those generated using floating-poin activations, showcasing high quality with minor shape variations. However, applying 4-bit activation quantization to the entire diffusion process leads to significant compromise in the generated image quality. This is primarily due to the degradation image quality caused by the quantization applied the later diffusion steps.
FP Act
LUM!
Quanticed
1.0T0.8T
Quantized
Quantized
0.6T0.4T
Quantized
0.4T:02TT
Quantized
0.2T0.0T
Quantized
1.0T0.0T
Figure 8: Examples of 256 LSUUNCCurches generation with FP32 activation or activation quantization across different diffusion steps. Example images with activation quantization are generated by applying 4-bit activation quantization in the target diffusion steps.
B.2 Explanation on FID Improvement with Proposed RAQ on LSUN-Bedrooms.
Table of Section 5.1
we observe that the
images generated with the proposed
RAQ exhibit slightly better FID scores compared to Q-diffusion with W4A32 and W4A8. To investigate the reason behind this FID improvement, we conduct detailed comparison of the images generated using full-precision activations and 4-bit activations in the early stage of the diffusion
14

---

(a) FP32 Activation
(b) 4-bit Activation for [1.07:0.87]
Figure 9: Examples of 256x256 ASUU--eeroome generation with different activation precision.
process. We find that the models with full-precision activations sometimes generate images with complex structures that are not easily recognizable as bedrooms. However, when 4-bit activation quantization is applied to the early stage, it simplifies the complex structures and results in images that more closely resemble bedrooms. This observation suggests that the step-wise activation quantization strategy employed the proposed RAQ method helps refine the generated images, leading to dataset.
improved quality and better alignment with the target
B.3 Activation Quantization of Stable Diffusion
For the conditional image generation with Stable Diffusion, we utilize the prompt examples that publicly available online 6 The prompts used in Section 5.2 are as follows:
1. a puppy wearing a hat
2. cluttered house in the woods in anime oil painting style*
3. Old photo of Clint Eastwood dressed htwboy.18000 centered, by professional photogra- pher, wide-angle lens, background saloon*
4. interior design, open plan, kitchen and living room, modular furniture with cotton textiles, wooden floor, high ceiling, large steel windows viewing city Artstation and Antonio Jacobsen and Edward Moran, (long shot), clear blue sky, intricated details, 4k*
5. tree on the hill, bright scene, highly detailed, realistic photo
6. highly detailed, majestic royal tall ship calm sea,realistic painting, by Charles Gregory
7. medium shot side profile portrait photo of the Takeshi Kaneshiro warrior chief, tribal panther make up, blue on red, looking away, serious eyes, 50mm portrait, photography, hard rim lighting photography -ar 2:3 beta -uphtaa
8. picture of dimly lit living room, minimalist furniture, vaulted ceiling, huge room, floor to ceiling window with an ocean view, nighttim*
9. an armchair in the shape of an avocado, an armchair imitating an avocado*
In this section, we additionally present non-cherry-picked samples generated using Stable Diffusion with and without activation quantization. We use the prompts that are highlighted with asterisk (*). For each prompt, generate four images to demonstrate the variety and quality of the generated results (Fig. in the next page).
"hunp:/staabeddiftuiiooofooooo
'nttss//mpoostioobestttt sttl-sttae-diiffssiiiiii prompts-the the-mmott-beuufffuu ai-text-to sest-to-immage to-image- co-imagge-rommtte
15

---

(a) Full Precision
(b) W4A8 (Q-Diffusion)
(c) W4A6
(d) W4A6/8 (Proposed)
Figure 10: Text-guided 512x512 image generation results with Stable Diffusion.
16


Reviews:
[{"review_id": "LuyHpwFVgo", "reviewer": "Reviewer_eumA", "summary": "In this submission, the authors propose a novel approach to speed up the noise estimation network by leveraging the robustness of early-stage diffusion models. Specifically, they present an algorithm to modify the quantization bit width according to the diffusion step. The proposed method shows positive results in reducing activation bits below 8 bits.", "questions": "•\tHow does RTQ perform in accelerating diffusion models in real-world scenarios?\n\n•\tHow do you set a reliable FID threshold for the RTQ algorithm?\n", "limitations": "The choice of FID threshold in the RTQ algorithm is unclear.", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "•\tThe writing of this manuscript is easy to follow, and the illustrations are clear.\n\n•\tThis work is well-motivated. Based on the analysis, the authors provide insights on the different roles that different diffusion steps play and show the room to improve the PTQ process by treating early and later steps differently.\n\n•\tThe experiments show positive results of the proposed method.\n", "weaknesses": "•\tThe real-world benefits of reducing activation bits. With advanced samplers, the sampling steps of diffusion models are significantly reduced, e.g., to 50 steps or lower. Thus, the gain achieved through low bit width calculation in the early steps may be marginal in real-world evaluation. On the other hand, bit width is usually a power of two. To my knowledge, some execution cores are designed to process 8-bit-only or 4-bit-only data. Irregular bit widths like 6 bits are treated as standard bit widths by padding zeros. Thus, the benefits of reducing to irregular bit widths (e.g., 6 bits) instead of standard bit widths (e.g., 4 bits) are questionable from the perspective of hardware. The authors are encouraged to provide real-world evidence of the benefits of RTQ or a discussion of the above concerns.\n\n•\tThe choice of FID threshold. In the RTQ algorithm, the choice of FID threshold is critical since it determines the final bit width dictionary and thus the quantization gain. How do you set this hyperparameter for a new dataset?\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Response eumA-1: The influence of RAQ on accelerating diffusion models**\n\nAs the reviewer rightly pointed out, advanced samplers have been recently presented to reduce the sampling steps, and we agree that accelerating diffusion models quantized with irregular bit widths on a GPU poses a challenge in terms of performance improvement in real world scenarios. Nonetheless, we believe that the proposed RAQ provides a meaningful direction for advancing the acceleration of diffusion models. Let us discuss more in detail below.\n\nFirst, the approaches of reducing sampling steps and the proposed RAQ are orthogonal methodologies for accelerating diffusion models unless an extreme reduction in sampling steps is undertaken, such as employing a single-step sampling strategy. Recent works for reducing sampling steps typically fine-tune diffusion models so that multiple sequential sampling actions can be encompassed by fewer sampling steps. Even in such scenarios, the fundamental property of diffusion models, where Gaussian noise progressively transforms into desired images across sampling steps, remains applicable. This intrinsic characteristic enables each step to display unique responses to quantization, thereby making our concept applicable even in situations with diminished sampling steps.\n\nSecond, while we agree that accelerating diffusion models quantized with irregular bit widths (e.g., 6 bits) on a GPU is indeed challenging due to the lack of corresponding arithmetic units, we would like to emphasize that our main contribution is to show a direction that we can reduce bit resolution for a part of parameters, and our approach is not fundamentally limited to a certain bit resolution. Hence, if other works that can further reduce the overall bit resolution of the network are developed independent of our scheme, then it can be combined with our proposed scheme, and there is a chance that portion of the 4-bit parameters in our scheme can be increased to see realistic performance benefits. For example, our current approach involves a basic min/max-based quantization mechanism for activation quantization, leading to an 8-bit quantization for the baseline fixed bitwidth case. Then, the allocation of activation bits using the RAQ method is distributed as (4b, 5b, 6b, 7b, 8b) = (20%, 20%, 10%, 40%, 10%) for LSUN-churches (Table 1). However, in the scenario where an advanced quantization mechanism enables 6-bit activation quantization even for the baseline fixed bitwidth case, there exists the possibility of increasing the proportion of 4-bit activations with the proposed RAQ, such as (4b, 5b, 6b) = (50%, 30%, 20%). \n\nThird, while accelerating diffusion models quantized with irregular bit widths (e.g., 6 bits) on a GPU is indeed challenging, the utilization of specialized hardware, like bit-scalable accelerators, could offer a promising solution for processing these models [1]. These accelerators are purpose-built to harness the benefits of quantization on a bit-by-bit basis, resulting in a nearly linear improvement in computing efficiency, encompassing both latency and energy consumption, with decreasing bit width. We introduce the concept of Bit Operations (BOPs), where BOPs are calculated as the product of OPs with weight bitwidth and activation bitwidth. This metric allows us to estimate the performance gain achievable through the RAQ method. In the case of LSUN-Churches, the full-precision baseline and Q-diffusion necessitates 4285.2 and 148.6 TBOPs (TBOPs: Tera BOPs) for a single image generation respectively, while the proposed RAQ method only requires 108.8 TBOPs. This indicates that with the utilization of specialized accelerators, the implementation of the RAQ approach could potentially lead to more than 39.4 times speedup and energy savings compared to the full-precision baseline, while Q-diffusion can achieve 28.8 times improvement compared to the full-precision baseline.\n\nTable A-1. LSUN-Churches (256x256) generation results\n| Model |W/A | FID | TBOPs|\n|---|---|---|---|\n| LDM-8 |32/32 | 4.09 | 4285.2 |\n| Q-diffusion | 4/8 | 4.45 | 148.6 |\n| RAQ | 4/6 | 4.64 | 108.8 |\n\nTable A-2.LSUN-Bedrooms (256x256) generation results\n| Model |W/A | FID | TBOPs|\n|---|---|---|---|\n| LDM-4 |32/32 | 2.96 | 20725.8 |\n| Q-diffusion |4/8 | 4.17 | 681.2 |\n| RAQ |4/6 | 3.99 | 504.8 |\n\n[1] Fu, Yonggan, et al. \"2-in-1 accelerator: Enabling random precision switch for winning both adversarial robustness and efficiency.\" MICRO 2021.\n\n**Response eumA-2: FID threshold setting for the RAQ algorithm** \n\nThe RAQ algorithm employs the FID of diffusion models without activation quantization as the FID threshold. This aligns with the central objective of the RAQ algorithm, which is to adaptively vary the activation quantization bitwidth across the different sampling steps while preserving the quality of image sampling.\n\n For example, when applying the RAQ algorithm to the LSUN-Churches dataset, we initially compute the FID of the LSUN-Churches dataset without activation quantization while generating 5,000 samples. This approach ensures that the optimization of activation quantization maintains FID at an acceptable level. Consequently, as illustrated in Table 1, the proposed RAQ technique achieves an effective activation bitwidth of 6 without compromising FID.\n\nOn the other hand, we can achieve lower activation bitwidth by slightly compromising FID. In this case, it becomes necessary to make adjustments to the FID threshold. The amount of the adjustment is dependent on the predetermined level of FID tolerance. For example, if a 10% increase in FID is acceptable, then the FID threshold is increased by 10%. For example, in the context of LSUN-church cases, we achieved an effective activation bit count of 5.60, resulting in an FID of 5.12. This FID value represents an approximate 10% increase compared to that of the full precision activation bit configuration, where the FID is 4.45 (Table 1).\n\nThanks for the comments."}, {"review_id": "fRIUXVoFK7", "reviewer": "Reviewer_rGNp", "summary": "The author initially notes that errors in the early stages of the reverse diffusion process result in minimal disturbance to the final generated image. As a solution, they suggest employing low-bit activations for the initial reverse diffusion process while preserving high-bit activations for the subsequent stages, in conjunction with PTQ.", "questions": "- Why is the images in Figure (a) is mirrored?\n\n- What does the term W4A6/8 mean in Figure 7", "limitations": "See weaknesses.", "rating": 5, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "- The idea is clear and easy to understand\n\n- The proposed RAQ method outperforms the other methods such as Q-diffusion\n", "weaknesses": "- Could you the authors explain how is the entropy calculated and why higher randomness in the pixel values will cause the images blurrier?\n\n\n- Cpmparison to other methods. The authors mentioned two PTQ methods PTQ4DM and Q-diffusion, but only provide quantitative and qualitative comparison to baseline and Q-diffusion.\n\n\n- In section 3.2, it seems obvious that add the same amound of noise to a noisier image will have less influence than to a less noiser image?  \n\n- And the authors did not explain why in figure 3, the performance on two different dataset are so different.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for the constructive comment.\n\nWe will now discuss 1) relationship between entropy and image clarity, 2) exclusion of comparison with PTQ4DM, 3) the effects of noise addition on images with varied clarity levels, 4) the performance difference between two datasets in Figure 3, 5) correction of mirrored images in Figure 1(a), and 6) clarification  of notation “W4A6/8” in Figure 7\n\n**Response rGNp-1: Relationship between entropy and image clarity**\n\nEntropy quantifies the level of randomness or uncertainty within a dataset, and when applied to image generation, it reflects the diversity or randomness in the pixel values of the generated images.The entropy of the random variable $X$ is defined as following equation [1]:\n\n$H(X)=\\sum_{x}-p(X)\\log p(X)$\n\nHigher entropy indicates a wider range of pixel values, which can introduce more noise and randomness to the images. This noise can lead to fluctuations in pixel values that do not align with the actual image structure, ultimately causing a decrease in image sharpness and clarity. In contrast, lower entropy suggests more structured and consistent pixel values, contributing to clearer and sharper images. \n\nFor detailed information regarding the calculation of entropy within the context of our study, please refer to Supplementary Material A.2.\n\n[1] Shannon, Claude Elwood. \"A mathematical theory of communication.\" The Bell system technical journal 27.3 (1948): 379-423.\n\n**Response rGNp-2: Exclusion of comparison with PTQ4DM**\n\nThere are two key reasons behind our decision not to include a comparison with PTQ4DM in this study.\n\nFirstly, PTQ4DM primarily concentrates on quantizing both weights and activations to 8 bits. In contrast, the Q-diffusion approach achieves a more advanced level of quantization by reducing the bitwidth of weights to 4 bits while retaining activations at 8 bits. This advancement positions Q-diffusion as a state-of-the-art quantization technique for diffusion models.\n\nSecondly, PTQ4DM focuses on low-resolution image generation using DDIM-based models. The reported FID results in their work correspond to image resolutions such as 32x32 for CIFAR-10 and 64x64 for ImageNet. On the other hand, our paper places its focus on generating high-resolution images, specifically at a resolution of 256x256 for LSUN images and 512x512 for Stable Diffusion. The significant discrepancy in image resolution creates challenges in directly comparing our method and PTQ4DM due to the absence of compatible data points.\n\n**Response rGNp-3:  The effects of noise addition on images with varied clarity levels**\n\nAs the reviewer correctly indicated, the analysis presented in Section 3 leads us to an additional conclusion: the influence of adding noise to already noisy images is relatively less significant compared to adding noise to images that possess greater clarity. These conclusions stem from  two main insights from our analysis. Firstly, images generated during the initial phases of the diffusion process show heightened noise levels compared to those generated later. Secondly, the early-stage diffusion process exhibits a higher resilience against the introduction of noise.\n\nHence, based on these insights, we can reasonably infer that a more aggressive quantization approach could be applied during the early stages of the process, as more aggressive quantization causes higher quantization noise. This strategic choice aligns with the inherent strengths of the early diffusion steps and their ability to accommodate higher quantization noise resulting from more aggressive quantization.\n\n**Response rGNp-4:  The performance difference between two datasets in Figure 3**\n\nWe believe that the variation in performance between the two datasets is attributed to the significant difference in their image resolutions. Specifically, the CIFAR-10 dataset comprises images with a resolution of 32x32, while the LSUN-churches dataset contains images at a higher resolution of 256x256. The increased resolution of LSUN-churches images introduces finer details into the generation process, making the image generation process more susceptible to noise. However, it's noteworthy that both datasets exhibit a similar early-stage resilience to noise injection, as demonstrated in our analyses.\n\n**Response rGNp-5:  Correction of mirrored images in Figure 1(a)**\n\nWe appreciate your observation regarding the mirrored images in Figure 1(a). It appears that the mirroring occurred unexpectedly during the figure preparation process. \nWe will correct the mistake and update the images in the final version of the paper. Thank you for bringing this to our attention.\n\n**Response rGNp-6:  Clarification  of notation “W4A6/8” in Figure 7**\n\nWe acknowledge the concern you raised regarding the notation \"W4A6/8\" in Figure 7. We aimed to convey that the activation bits of the stable diffusion model were combined with both 6 bits and 8 bits precision. However, we recognize that this notation might lead to confusion among readers.\nTo address this issue, we will update the explanation of the \"W4A6/8\" notation in the caption of the Figure 7 as follows: “Here, W4 denotes adopting 4-bit weights, A$n$ denotes $n$-bit activations, and A$n/m$ indicates adopting $n$-bit activations in the early stage of the diffusion stages, while the later stage adopts $m$-bit activations”. We appreciate your valuable feedback, and thank you for bringing this to our attention.\n\nThanks again for the comments."}, {"review_id": "ILqZu9wuqg", "reviewer": "Reviewer_ZFyZ", "summary": "This paper presents robustness-aware quantization (RAQ), a novel strategy to use mixed precisions for activations when quantizing diffusion models. The authors found that inaccurate computation during the early stages of the reverse diffusion process has minimal impact on the quality of generated images, and propose to use low-bit activations for the early reverse diffusion process while maintaining high-bit activations for the later stages. Experiments have been conducted for both unconditional and conditional generation using latent diffusion and stable diffusion on various datasets.", "questions": "How is $Bit_{act}$ updated in Algorithm 1 with respect to t? It seems like the intervals are always multiples of 0.05T. Was $Bit_{act}$ only updated every 0.05T?", "limitations": "The authors adequately addressed the limitations.", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- The paper is well-structured and presents a clear motivation for leveraging the robustness of early-stage diffusion models to use lower-bit activations at those time steps to further improve the computation efficiency.\n- Experimental results show that the proposed method can use lower precisions for early-stage computation without sacrificing the quality of the generated images. \n- The experiments with stable diffusion indicate the effectiveness of the proposed methods on text-to-image applications.", "weaknesses": "My biggest concern with the proposed RAQ approach is its practicality. The method suggests using low-bit activations for the early denoising process and high-bit activations for the later stages. However, the paper does not provide sufficient arguments on how this varying precision can be efficiently implemented and how much additional benefits it can bring compared to the simple W4A8 cases. In real-world applications, changing activation precisions could introduce complexities in designing and implementing corresponding kernels for different stages of the process, as the weight precisions need to be always upcasted to the activation precisions when performing the compute on conventional GPUs (e.g. the compute will always be WyAy for WxAy precisions, where x=4 and y>=4 for the settings discussed in the paper). Consequently, this could limit the practical utility and impact of the RAQ approach. An analysis of the theoretical speed up or memory saving should be done to show that changing activation precisions for early stages can indeed bring substantial improvements in compute efficiency (so the extra efforts for kernels implementation can be justified), and providing some additional simple experimental results will be preferred.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for the constructive comment.\n\nWe will now discuss 1) the practicality of the proposed RAQ, and 2) the granularity of Bit_act update.\n\n**Response ZFyZ-1: The practicality of the proposed RAQ.**\n\nAs the reviewer rightly pointed out, we agree that accelerating diffusion models quantized with irregular bit widths on a GPU poses a challenge in terms of performance improvement in real world scenarios. Nonetheless, we believe that the proposed RAQ provides a meaningful direction for advancing the acceleration of diffusion models. Let us discuss more in detail below.\n\nFirst, while we agree that accelerating diffusion models quantized with irregular bit widths (e.g., 6 bits) on a GPU is indeed challenging due to the lack of corresponding arithmetic units, we would like to emphasize that our main contribution is to show a direction that we can reduce bit resolution for a part of parameters, and our approach is not fundamentally limited to a certain bit resolution. Hence, if other works that can further reduce the overall bit resolution of the network are developed independent of our scheme, then it can be combined with our proposed scheme, and there is a chance that portion of the 4-bit parameters in our scheme can be increased to see realistic performance benefits. For example, our current approach involves a basic min/max-based quantization mechanism for activation quantization, leading to an 8-bit quantization for the baseline fixed bitwidth case. Then, the allocation of activation bits using the RAQ method is distributed as (4b, 5b, 6b, 7b, 8b) = (20%, 20%, 10%, 40%, 10%) for LSUN-churches (Table 1). However, in the scenario where an advanced quantization mechanism enables 6-bit activation quantization even for the baseline fixed bitwidth case, there exists the possibility of increasing the proportion of 4-bit activations with the proposed RAQ, such as (4b, 5b, 6b) = (50%, 30%, 20%). \n\nSecond, while accelerating diffusion models quantized with irregular bit widths (e.g., 6 bits) on a GPU is indeed challenging, the utilization of specialized hardware, like bit-scalable accelerators, could offer a promising solution for processing these models [1]. These accelerators are purpose-built to harness the benefits of quantization on a bit-by-bit basis, resulting in a nearly linear improvement in computing efficiency, encompassing both latency and energy consumption, with decreasing bit width. We introduce the concept of Bit Operations (BOPs), where BOPs are calculated as the product of OPs with weight bitwidth and activation bitwidth. This metric allows us to estimate the performance gain achievable through the RAQ method. In the case of LSUN-Churches, the full-precision baseline and Q-diffusion necessitates 4285.2 and 148.6 TBOPs (TBOPs: Tera BOPs) for a single image generation respectively, while the proposed RAQ method only requires 108.8 TBOPs. This indicates that with the utilization of specialized accelerators, the implementation of the RAQ approach could potentially lead to more than 39.4 times speedup and energy savings compared to the full-precision baseline, while Q-diffusion can achieve 28.8 times improvement compared to the full-precision baseline.\n\nTable A-1. LSUN-Churches (256x256) generation results\n| Model |W/A | FID | TBOPs|\n|---|---|---|---|\n| LDM-8 |32/32 | 4.09 | 4285.2 |\n| Q-diffusion | 4/8 | 4.45 | 148.6 |\n| RAQ | 4/6 | 4.64 | 108.8 |\n\nTable A-2.LSUN-Bedrooms (256x256) generation results\n| Model |W/A | FID | TBOPs|\n|---|---|---|---|\n| LDM-4 |32/32 | 2.96 | 20725.8 |\n| Q-diffusion |4/8 | 4.17 | 681.2 |\n| RAQ |4/6 | 3.99 | 504.8 |\n\n[1] Fu, Yonggan, et al. \"2-in-1 accelerator: Enabling random precision switch for winning both adversarial robustness and efficiency.\" MICRO-54: 54th Annual IEEE/ACM International Symposium on Microarchitecture. 2021.\n\n**Response ZFyZ-2: The granularity of Bit_act update.**\n\nIn the context of the RAQ method outlined in Algorithm 1, choosing a finer granularity for Bit_act update necessitates a larger number of sampled images for the optimization process. Meanwhile, our investigation revealed that consecutive timesteps within a 0.05T range exhibit similar sensitivity to noise injection. Therefore, our experimental configuration concentrated on optimizing the activation bitwidth with intervals of 0.05T for the Bit_act update. This approach ensures effective optimization while managing computational demands.\n\nThanks again for the comments."}, {"review_id": "UhljVueHyG", "reviewer": "Reviewer_4DBQ", "summary": "This paper proposes to quantize diffusion models to a different extent along the iterative process for image generation. The main motivation of the proposed approach is that diffusion model is robust to input distortion at early stages (i.e. noisy stages) of the iterative process. Therefore, the proposed approach starts with a 4-bit quantization, and gradually increase activation bits along the iterations. Experiments show that the proposed approach achieves improved performance with the same effective bitwidth.  ", "questions": "Overall, I think the dynamic quantization in this paper is a legitimate approach for improving the efficiency of diffusion model sampling. Please refer to the weakness sections for the questions. ", "limitations": "Please consider discussing the limitations and potential societal impact of the proposed approach in the paper. ", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This paper has a good motivation. The empirical experiments show that it is legitimate to apply different rates of quantization at different stages of different diffusion process.\n\n2. The proposed method effectively improves the performance with a reduced bitwidth, as shown in Table 1.\n\n3. The idea is simple and is easy to follow.  ", "weaknesses": "1. From Table 1, the bitwidth for each timestep is model-specific. That means optimization has to be done for each model. It would be good to have analysis on the robustness of the bitwidth selection.\n\n2.  As one of the main objectives is to improve the sampling efficiency, the comparison of runtime should be included. This is important for readers to understand the improvement brought by the proposed method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for the constructive comment.\n\nWe will now discuss 1) the bitwidth selection for different models, and 2) improvement of the sampling efficiency with RAQ.\n\n**Response 4DBQ-1: The bitwidth selection for different models**\n\nAs you correctly highlighted, the bitwidth optimization with the proposed RAQ is a model-specific optimization. The experimental results presented in Table 1 have been attained through the activation bitwidth optimization process as illustrated in Algorithm 1. The optimization of bitwidth is executed iteratively for each model, involving FID measurement. However, it is worth noting, as detailed in Section 4.3, that the proposed RAQ method holds the potential to reduce the optimization time required for model-specific bitwidth optimization from $O(m^T)$ to $O(m+T)$\n\n**Response 4DBQ-2: Improvement of the sampling efficiency with RAQ**\n\nWhile the accelerating diffusion models quantized with irregular bit widths (e.g., 6bits) on a GPU presents challenges, the utilization of specialized hardware, like bit-scalable accelerators, could offer a promising solution for processing these models [1]. These accelerators are purpose-built to harness the benefits of quantization on a bit-by-bit basis, resulting in a nearly linear improvement in computing efficiency, encompassing both latency and energy consumption, with decreasing bit width. We introduce the concept of Bit Operations (BOPs), where BOPs are calculated as the product of OPs with weight bitwidth and activation bitwidth. This metric allows us to estimate the performance gain achievable through the RAQ method. In the case of LSUN-Churches, the full-precision baseline and Q-diffusion necessitates 4285.2 and 148.6 TBOPs (TBOPs: Tera BOPs) for a single image generation respectively, while the proposed RAQ method only requires 108.8 TBOPs. This indicates that with the utilization of specialized accelerators, the implementation of the RAQ approach could potentially lead to more than 39.4 times speedup and energy savings compared to the full-precision baseline, while Q-diffusion can achieve 28.8 times improvement compared to the full-precision baseline.\n\nTable A-1. LSUN-Churches (256x256) generation results\n| Model |W/A | FID | TBOPs|\n|---|---|---|---|\n| LDM-8 |32/32 | 4.09 | 4285.2 |\n| Q-diffusion | 4/8 | 4.45 | 148.6 |\n| RAQ | 4/6 | 4.64 | 108.8 |\n\nTable A-2.LSUN-Bedrooms (256x256) generation results\n| Model |W/A | FID | TBOPs|\n|---|---|---|---|\n| LDM-4 |32/32 | 2.96 | 20725.8 |\n| Q-diffusion |4/8 | 4.17 | 681.2 |\n| RAQ |4/6 | 3.99 | 504.8 |\n\n[1] Fu, Yonggan, et al. \"2-in-1 accelerator: Enabling random precision switch for winning both adversarial robustness and efficiency.\" MICRO-54: 54th Annual IEEE/ACM International Symposium on Microarchitecture. 2021.\n"}, {"review_id": "b9E4FqGWs3", "reviewer": "Reviewer_ctiM", "summary": "The authors design robustness-aware quantization (RAQ) to speed up the noise estimation network by leveraging the robustness of early-stage diffusion models. Specifically, the authors found that the quality of generated images is less affected by the early-stage. Therefore, they reduce the bitwidth of activations for the early-stage, and maintain high-bit activations for the later-stage. Experiments show that the proposed method can speed up early computations while maintaining generation quality.", "questions": "1. Can performance and computation be further optimized with finer-grained (beyond five intervals, even one interval per step) settings?\n2. In Tab. 1, the FID of RAQ (3.99) with smaller activation bitwidth is better than Q-diffusion (4.17) about LSUN-Bedrooms (256x256). Please give some analyses about these results.\n3. In Fig. 7, the generation quality of the W4A6/8 has a large difference compared with W4A8 and full precision (especially the two images in the lower right corner and upper left corner). This result is different from the quantitative comparison in Tab. 1, where RAQ has comparable (even better) FIDs but smaller activation bitwidth. Please give some explanations.", "limitations": "The authors do not discuss its limitations or potential negative societal in separate section. It is better if the authors add some discussion.", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "1. The idea of the paper is simple yet effective, promoting the application of Post-Training Quantization (PTQ) in the diffusion model.\n2. The analyses in the paper are extensive. The authors demonstrate the early-stage robustness through entropy transition across steps (Fig. 2) and noise injection. \n3. The paper is well-organized and easy to read. \n4. The authors also provide the code for results reproduction, showing the solidness of the work.", "weaknesses": "1. In Tab. 1, RAQ only sets different bitwidths in five intervals, which is inconsistent with Algorithm 1, which sets different bitwidths in each step. Some explanation is needed.\n2. The paper only reduces the activation bitwidth on the base of Q-diffusion. However, compared with LDM-4, the FID of Q-diffusion increased by 1.21 about LSUN-Bedrooms (256x256) in Tab. 1. Therefore, the activation bitwidth should not be reduced only. It is better to further apply RAQ to weight bidwidth to obtain a better trade-off between performance and efficiency.\n3. The authors propose RAQ to accelerate the early-stage computation. However, the running time, FLOPs, and model size are not provided to demonstrate the effectiveness of the proposed method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Response ctiM-1: The granularity of bitwidth optimization**\n\nIn the context of the RAQ method outlined in Algorithm 1, choosing a finer granularity for Bit_act update necessitates a larger number of sampled images for the optimization process. Meanwhile, our investigation revealed that consecutive timesteps within a 0.05T range exhibit similar sensitivity to noise injection. Therefore, our experimental configuration concentrated on optimizing the activation bitwidth with intervals of 0.05T for the Bit_act update. This approach ensures effective optimization while managing computational demands.\nOn the other hand, despite optimizing the activation bitwidth with a granularity of 0.05T, some consecutive intervals present the same level of sensitivity to the quantization and thus share the optimal bitwidth. Therefore, the experimental results summarized in Table 1 are composed of only 5 intervals or even fewer intervals.\n\n**Response ctiM-2: Applying RAQ to weight bitwidth optimization**\n\nWe agree with your opinion that, by implementing the RAQ method on weight quantization, it's possible to prevent FID increase in scenarios involving Q-diffusion with 4-bit weight quantization. However, it is important to note that applying the RAQ method on the weight quantization is not within the scope of this paper. The principal aim of this research is to enhance the diffusion inference process without introducing supplementary costs into the inference stage. This is especially achievable in the context of implementing the RAQ method on activation quantization, it does not entail any extra costs such as additional memory requirements.\n\nConversely, applying RAQ to weight quantization increases the memory requirements due to the potential increase in the number of parameters, as we have to save copies of parameters with different resolutions. \n\n**Response ctiM-3: The effectiveness of the proposed RAQ**\n\nAs the reviewer pointed out, the operation counts will be updated in the final version.\n\nWe introduce the concept of Bit Operations (BOPs), where BOPs are calculated as the product of OPs with weight and activation bitwidths. This metric allows us to estimate the performance gain achievable through the RAQ method. In the case of LSUN-Churches, the full-precision baseline and Q-diffusion necessitates 4285.2 and 148.6 TBOPs (TBOPs: Tera BOPs) for a single image generation respectively, while the proposed RAQ method only requires 108.8 TBOPs. This indicates that the RAQ approach could potentially lead to more than 39.4 times speedup and energy savings compared to the full-precision baseline, while Q-diffusion can achieve 28.8 times improvement compared to the full-precision baseline.\n\nTable A-1. LSUN-Churches (256x256) generation results\n| Model |W/A | FID | TBOPs|\n|---|---|---|---|\n| LDM-8 |32/32 | 4.09 | 4285.2 |\n| Q-diffusion | 4/8 | 4.45 | 148.6 |\n| RAQ | 4/6 | 4.64 | 108.8 |\n\nTable A-2.LSUN-Bedrooms (256x256) generation results\n| Model |W/A | FID | TBOPs|\n|---|---|---|---|\n| LDM-4 |32/32 | 2.96 | 20725.8 |\n| Q-diffusion |4/8 | 4.17 | 681.2 |\n| RAQ |4/6 | 3.99 | 504.8 |\n\n**Response ctiM-4: The impact of bitwidth optimization with finger-grained settings**\n\nThank you for the insightful question. However, further optimization is hardly achievable with finer-grained settings. As mentioned in **Response ctiM-1** regarding interval granularity, the optimization process was indeed carried out with a finer-grained approach. However, the optimization process resulted in only 5 intervals or even fewer intervals, because consecutive time steps exhibit similar sensitivity to quantization. Consequently, the potential for achieving further optimization through finer-grained settings becomes limited.\n\n**Response ctiM-5: Explanation on FID improvement with proposed RAQ on LSUN-bedrooms**\n\nAs you pointed out, it is interesting that LSUN-Bedrooms images generated with the proposed RAQ exhibit slightly better FID scores compared to Q-diffusion with W4A32 and W4A8. To investigate the reason behind this FID improvement, we conduct a detailed comparison of the images generated using full-precision activations and 4-bit activations in the early stage of the diffusion process. process. We find that the models with full-precision activations sometimes generate images with complex structures that are not easily recognizable as bedrooms. However, when 4-bit activation quantization is applied to the early stage, it simplifies the complex structures and results in images that more closely resemble bedrooms. This observation suggests that the step-wise activation quantization strategy employed in the proposed RAQ method helps refine the generated images, leading to improved quality and better alignment with the target LSUN-Bedrooms dataset.\nThe detailed discussion with sampled images are updated in Supplementary Material B.2.\n\n\n**Response ctiM-6: The generation quality of the W4A6/8 Stable Diffusion model**\n\nAs you highlighted, there is a quality issue when generating high-resolution (512x512) images using the Stable Diffusion model with the W4A6/8 configuration.This is due to the intricate nature of capturing fine details in high-resolution images, which demands exceptionally accurate computations. Consequently, this case becomes more sensitive to quantization effects compared to the 256x256 image generation instances reported in Table 1.\n\nHowever, it is important to note that our experimental results consistently emphasize the effectiveness of deploying a mixed precision strategy through the RAQ method. Specifically, the mixed-precision quantization achieved by the RAQ approach (W4A6/8) significantly enhances the quality of the generated images when contrasted with scenarios where activation bits are fixed at 6 bits (W4A6).\n\nThanks for the constructive comments."}]

Author rebuttal:

Meta-review: This paper presents a robustness-aware quantization (RAQ) method to speed up diffusion models by finding that the early-stage denoising steps can be done with fewer bit in activation. The paper received five reviews, most of which are neutral positive (except that one review is 4).   

In the paper, the finding that the inaccurate activation (or activation with fewer bits) in early stages of diffusion steps does not affect the accuracy of diffusion model is novel. Based on this finding, the paper proposed a robustness-aware quantization algorithms which dynamically adjusts the bits for activations during the diffusion steps. The rebuttal clarified most of the questions raised by the reviewers. The main remaining concern from the reviewers is the practicality of the proposed RAQ methods on GPUs (since varying bit widths is not easy to implement on current GPU architectures). While this is a valid point especially for practical usage, as the authors pointed out, the paper showed a direction to reduce bit resolution for future new GPU  architectures. Moreover, the authors did provide some explanations and potential implementation with some specialized hardware (e.g., bit-scalable acelrators [1]).

 Considering all the reviews, the rebuttal, and the discussions, the AC decided to accept the paper.
Decision: Accept (poster)