SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Yulhwa  $\mathbf{K}\mathbf{m}^{1}$ , Dongwon  $\mathbf{J}\mathbf{o}^{1}$ , Hyesung Jeon $^{1}$ , Taesu  $\mathbf{K}\mathbf{m}^{2}$ , Daehyun Ahn $^{2}$ , Hyungjun  $\mathbf{K}\mathbf{m}^{2}$ , Jae-Joon  $\mathbf{K}\mathbf{m}^{1}$
$^{1}$ Seoul National University,  $^{2}$ SqueezeBits Inc.
{yulhwakim, dongwonjo, hjeon2k, kimjaejoon}@snu.ac.kr,
{taesu.kim, daehyun.ahn, hyungjun.kim}@squeezebits.com
While diffusion models have demonstrated exceptional image generation capabilities, the iterative noise estimation process required for these models is computationally intensive and their practical implementation is limited by slow sampling speeds. In this paper, we propose a novel approach to speed up the noise estimation network by leveraging the robustness of early-stage diffusion models. Our findings indicate that inaccurate computation during the early-stage of the reverse diffusion process has minimal impact on the quality of generated images, as this stage primarily outlines the image while later stages handle the finer details that require more sensitive information. To improve computational efficiency, we combine our findings with post-training quantization (PTQ) and introduce a method that utilizes low-bit activations for the early reverse diffusion process while maintaining high-bit activations for the later stages. Experimental results show that the proposed method can accelerate the early-stage computation without sacrificing the quality of the generated images.
Diffusion models [1, 2, 3, 4, 5, 6, 7, 8] for image synthesis have gained significant attention in recent years due to their exceptional proficiency in generating high quality and diverse images. A key feature of the diffusion models is that they gradually dissipate noise from latent variables across a sequence of diffusion steps [1]. However, despite their remarkable image synthesis capabilities, diffusion models suffer from slow image synthesis process. This is primarily caused by the compute-intensive nature of the denoising network [3, 9], which is an integral part of the diffusion model. The iterative process of applying the denoising network across multiple diffusion steps consumes substantial computational resources, resulting in very long synthesis time [6]. As a result, this limitation poses a challenge for real-time applications or scenarios that require quick image synthesis.
Quantization [10, 11, 12] is a widely used technique for improving the computational efficiency of neural networks. Quantization reduces memory usage by converting high-precision floating-point weight/activation values into low-precision integer values [13]. It also enables more efficient processing by leveraging efficient integer operations instead of computationally expensive floating-point operations [14]. As a result, there is a growing research trend focusing on applying quantization techniques to denoising networks. Two notable studies, PTQ4DM [15] and Q-diffusion [16], explored the applications of LAPQ [17] and BRECQ [18], which are commonly used Post-Training Quantization (PTQ) frameworks, to the diffusion models. Denoising networks differ from conventional neural networks as they are iteratively used during the reverse diffusion process, and their output distributions vary with diffusion step. Considering this characteristic, the aforementioned studies primarily explored methods for constructing a calibration dataset to effectively apply PTQ to diffusion
(b)
Figure 1: (a) Overview of the diffusion process (b) U-Net architecture of the denoising network
models. While they have successfully achieved a reduction in weight precision to as low as 4 bits, reducing activation bits below 8 bits still remains a challenge.
In this paper, we investigate the computational characteristics of diffusion models during their iterative diffusion steps and present a novel strategy for quantizing diffusion models based on the analysis. Our analysis of the diffusion model reveals the high tolerance of early-stage reverse diffusion in the presence of computational errors. In the reverse diffusion process, the early stage focuses on capturing the outlines of the images, while the later stage refines the details to improve the overall quality of the generated images. As a result, modifications made in the early stages primarily affect the structural aspects of the generated images, while the quality remains mostly unaffected. Based on the observation, we propose leveraging the early-stage robustness observed in the diffusion process to optimize activation bit precision, thereby reducing computational overhead while maintaining the fidelity and diversity of the generated images. By strategically quantizing the activation values in a step-wise manner, we can reduce the effective bit precision of the activations of the denoising networks without sacrificing the overall quality of the generated images.
The generation of an image in diffusion models is represented as a series of diffusion steps (Fig. 1(a)) [1, 3]. In the forward diffusion process, each step adds noise to the data, eventually making the data follow a Gaussian distribution. To generate an image  $x_0$  from a Gaussian noise  $x_{T} \sim N(0,I)$ , diffusion models reverse the forward process and sample latent variables  $x_{t}$  from a learned conditional distribution  $p_{\theta}(x_{t - 1}|x_t)$ , which approximates the true reverse conditional distribution as follows:
$$
p _ {\theta} \left(x _ {t - 1} \mid x _ {t}\right) = N \left(x _ {t - 1}; \mu_ {\theta} \left(x _ {t}, t\right), \sigma_ {t} ^ {2} I\right) \tag {1}
$$
More specifically, by combining Eq. 1 and DDIM sampling method [6], the reverse diffusion process yields the following equation to generate a sample:
$$
x _ {t - 1} = \sqrt {\alpha_ {t - 1}} \left(\frac {x _ {t} - \sqrt {1 - \alpha_ {t}} \epsilon_ {\theta} (x _ {t} , t)}{\sqrt {\alpha_ {t}}}\right) + \sqrt {1 - \alpha_ {t - 1} - \sigma_ {t} ^ {2}} \cdot \epsilon_ {\theta} (x _ {t}, t) + \sigma_ {t} \epsilon_ {t} \tag {2}
$$
where  $\epsilon_t \sim N(0, I)$  is Gaussian noise independent of  $x_t$ . Here,  $\alpha_t$  is a coefficient determined by the noise scheduling policy of the diffusion process, and  $\sigma_t$  is a hyperparameter that controls the degree of stochasticity of the diffusion process. Note that  $x_t$ , the output of the diffusion step  $t$ , becomes the input of the next diffusion step  $t - 1$ , and this process continues until the image  $x_0$  is reconstructed from the noise  $x_T$ .
$\epsilon_{\theta}(x_t,t)$  in Eq. 2 represents a trainable denoising network of a diffusion model. Diffusion models often use the U-Net architecture (Fig. 1(b)) for their denoising network [9]. The U-Net consists of an encoder-decoder structure in which the encoder progressively decreases the spatial dimensionality of the input image, extracting high-level features. Conversely, the decoder performs upsampling operations on the feature maps and incorporates skip connections from the encoder to reconstruct the original image resolution while generating low-level features. In the context of diffusion models,
the U-Net denoising model is trained to reduce noise in the latent variables at each diffusion step. By iteratively applying the denoising model across multiple diffusion steps, the generated images gradually exhibit improved clarity. However, it is worthwhile to note that the compute-intensive nature of the denoising network makes the generation process of diffusion models very time-consuming.
2.2 Post-Training Quantization
A typical uniform quantization function for converting floating-point value  $x$  into integer value  $x_{int}$  is as follows:
$$
x _ {i n t} = s \cdot \operatorname {c l a m p} \left(\lfloor \frac {x - z}{s} \rceil , \operatorname {m i n} _ {\text {q u a n t}}, \operatorname {m a x} _ {\text {q u a n t}}\right) \tag {3}
$$
where  $s$  is a scaling factor and  $z$  is the zero point. Post-training quantization (PTQ) [10, 12, 17, 18] refers to a technique for compressing neural networks that is typically applied after the network has been trained unlike quantization-aware training (QAT) [11, 19, 20, 21, 22], which considers quantization while training the network. One common PTQ strategy is to collect calibration data from the training data and calibrate the scaling factor or the rounding scheme (e.g. AdaRound [10]).
Recent studies, PTQ4DM [15] and Q-diffusion [16], have proposed PTQ methodologies for denoising network of diffusion models. PTQ4DM and Q-Diffusion provided insights into the methodology for constructing the PTQ calibration dataset for diffusion models. Specifically, since the denoising network is applied across multiple diffusion steps, these papers discussed how to consider the diffusion steps when constructing the calibration data. Both works intensively analyzed the distribution of activations in different diffusion steps, but they differ in the approaches to constructing the calibration dataset. To improve the quality of the calibration dataset, PTQ4DM adopts a method of extracting the diffusion steps from a distribution  $N(\mu, 0.5T)$  where  $\mu \leq 0.5T$ , drawing more samples from the diffusion steps closer to  $0.0T$ . On the other hand, Q-diffusion divides the diffusion steps evenly into constant intervals to construct the calibration set, leveraging the similarity of activation distributions of adjacent diffusion steps. As a result, PTQ4DM achieved INT8 weights and INT8 activations for high-quality quantized diffusion models, while Q-diffusion models demonstrated INT4 weights and INT8 activations.
Previous works on quantizing diffusion models demonstrate that optimizing the construction of the calibration set can successfully reduce the bit precision of weights to as low as 4 bits [16]. However, they still require 8-bit activations, which limits network efficiency. To further reduce the activation precision, we first investigate the robustness of each diffusion step in the reverse diffusion process.
3 Early-Stage Robustness of Reverse Diffusion Process
3.1 Properties of Reverse Diffusion Process
The reverse diffusion process generates images progressively starting from Gaussian noise [8]. As shown in Fig. 2(a), the early stage of the reverse diffusion process captures the outlines of the images. In this stage, sampled images are rough and blurry sketches, so the shapes of the objects are hardly recognizable. In the later stage, because the overall structure of the images are properly shaped, the reverse diffusion process draws details of the images to improve the quality of the generation.
To quantitatively analyze the characteristics of the generated latent variables  $x_{t}$  at each diffusion step of the reverse diffusion process, we use entropy as a measure of the randomness exhibited by the output at each step. Note that a higher entropy indicates higher randomness in the pixel values of the images, meaning they are blurrier. The entropy values of the latent variables  $x_{t}$  at each step are summarized in Fig. 2(b). The early steps ranging from  $1.0T$  to  $0.5T$  maintain high entropy values, indicating that sampled images are noisy with high randomness. On the other hand, the entropy rapidly decreases after  $0.5T$  step. These latter steps generate details of the image to enhance the quality of image generation, so the sampled images become sharper and their randomness decreases.
Since the characteristics of the generated images vary at each step of the reverse diffusion process, we can expect that the computational requirements for each step may differ as well. Specifically, the entropy of the sampled images remains constant with relatively high value in the early stage, suggesting that the computations performed during this stage may not need to be as precise as that for the later stages. However, in later stages where the computations have a greater impact on the entropy results, more accurate computations are necessary.
Figure 2: Studies on the reverse diffusion process with respect to diffusion steps. (a) Examples of  $256 \times 256$  LSUN-Bedrooms, LSUN-Churches, and CelebA-HQ generated in each step of the reverse diffusion process. (b) Entropy transition across the diffusion steps for various image generation tasks.
Figure 3: Image generation quality (FID) after noise injection. 5k images of (left)  $32 \times 32$  CIFAR-10 and (right)  $256 \times 256$  LSUN-Churches are generated for the evaluation. The magnitude of the noise is controlled by  $\tau$  using Eq. 4. The diffusion step exhibits lower FID after noise injection indicates a higher level of resilience to inaccurate computations.
3.2 Early-Stage Robustness of Reverse Diffusion Process
In this section, we evaluate the robustness of the reverse diffusion process at each diffusion step. To measure the robustness, we inject random noise into  $x_{t}$ , the output of the target diffusion step  $t$  (Eq. 2), and assess the quality of the generated image  $x_{0}$  after the entire reverse diffusion process using the Fréchet Inception Distance (FID) [23]. The FID helps us to understand how much the computational inaccuracy at step  $t$  affects the quality of overall diffusion process. The injected noise follows a Gaussian distribution  $N(0,\sigma_{x_t}\cdot \tau)$ , where  $\sigma_{x_t}$  is the standard deviation of  $x_{t}$ , and  $\tau$  controls the noise standard deviation. The noise injection modifies the latent variable  $x_{t}$  as follows:
$$
x _ {t, n o i s y} = x _ {t} + \epsilon , \quad \epsilon \sim N \left(0, \left(\sigma_ {x _ {t}} \cdot \tau\right) ^ {2}\right) \tag {4}
$$
The analysis results are summarized in Fig. 3. The cases of  $32 \times 32$  CIFAR-10 and  $256 \times 256$  LSUN-Churches generation exhibit different responses to the noise injection. This discrepancy is attributed to the higher resolution of LSUN-Churches images, which introduces finer details into the generation process. Nevertheless, in both cases, there is a consistent pattern: as the standard deviation of injected noise  $(\tau)$  increases, the FID also increases, while the correlation between noise and FID varies significantly depending on the diffusion step. In the early diffusion stages, close to  $1.0T$ , we observe
Figure 4: Examples of image generation after noise injection. (a) Comparison of the CIFAR-10 images generated without noise and with noise injected in different diffusion steps. (b) Comparison of the generated images with and without early-stage noise injection. The early-stage noise injection changes the overall structure of the images rather than their quality.
that the quality of the image is relatively tolerant to noise. The early stages exhibit resilience to high magnitude of noise, as can be found by the maintained FID even with high  $\tau$  values. However, the later stages are more susceptible to the injected noise, leading to a degradation in the FID score. This analysis confirms that the early stages are more robust to the noise than the latter stages.
To gain a more intuitive understanding of this phenomenon, we examine how the final output images change when noise is injected at each diffusion step (Fig. 4). When noise is added to the back end of the diffusion steps to capture image details, the resulting image maintains a similar overall structure but exhibits lower quality due to the injected noise. In contrast, when noise is added in the early stages, high-resolution image details remain intact while the shape of the image changes due to the noise which affects the process of capturing the overall outline. For example, as shown in Fig. 4(a) for CIFAR-10 [24] image generation, injecting noise in the initial stage causes the image at the bottom row (originally a green airplane) to become a green automobile or a slightly different shaped airplane. The diffusion model is trained to generate images similar to those belonging to the CIFAR-10 dataset, regardless of the specific class. Since both airplane and automobiles fall within the CIFAR-10 class categories, the diffusion model generates images as intended. Similarly, in the case of LSUN-bedroom, LSUN-churches [25], CelebA-HQ [26], and FFHQ [27] generation (Fig. 4(b)), injecting noise in the early stage leads to changes in the overall image shape, rather than impacting image quality significantly. The effect of shape alteration while maintaining image quality is similar to that of modifying the random seed  $x_{T}$  of the diffusion. The diffusion model initiates image generation from  $x_{T}$  and gradually acquires overall shapes and colors of the images. Therefore, modification in the early stage would introduce changes to the overall structure of the generated images, not the quality.
This finding highlights the importance of understanding the varying robustness of diffusion models across different sampling steps, which can be a critical factor in improving the efficiency of denoising networks. When trying to speed up a neural network computations, approximated operations are often employed. For instance, quantization techniques approximate FP operations as integer operations to enhance processing efficiently. However, the fact that robustness varies across each step suggests that the degree to which operations can be approximated also differs across each step. Therefore, to improve efficiency without compromising the generation quality of the diffusion model, there is a great need to analyze and optimize each step individually.
4 Robustness-Aware Quantization of Diffusion Models
4.1 Impact of Weight and Activation Quantization on Diffusion Models
As the first step of quantizing the diffusion model, we evaluate the impact of quantization on each component of the denoising network. We begin by examining the impact of weight and activation
Figure 5: Image generation quality as measured by FID after (left) weight quantization (right) activation quantization of the denoising network. The diffusion model LDM-8 [8] is used to generate  $5\mathrm{k}$  images of  $256\times 256$  LSUN-Churches for the evaluation. The dashed gray lines indicate FID with the full-precision network. Note that the y-axis of the left graph is on a linear scale while the y-axis of the right graph is on a logarithmic scale.
Figure 6: (left) Examples and (right) FID of  $256 \times 256$  LSUN-Churches generation with activation quantization across different diffusion steps. Example images are generated by applying 4-bit activation quantization in the target diffusion steps, and FID is measured after 5k image generation. The dashed gray lines indicate FID for the full-precision network
quantization. We use BRECQ [18], a commonly utilized PTQ framework, for weight quantization and the calibration dataset is constructed based on the method proposed in Q-diffusion [16]. For activation quantization, we employ the uniform quantization function described in Eq. 3. Similar to Q-diffusion, we apply quantization for all layers of the denoising network that perform matrix multiplication, including convolution, linear, and attention layers, while keeping the activation values of attention score matrix to 8 bits. Fig. 5 shows that FID degradation is negligible up to 4 bit weight values so that low-bit weight quantization is relatively straightforward. In contrast, FID deteriorates more rapidly when the activation quantization bit decreases. Hence, we focus on reducing the number of activation bits.
4.2 Dynamic Nature of Activation Quantization across Diffusion Steps
Activation quantization introduces error when converting floating-point activations to integer values. In the case of uniform quantization, where a finite range of data is evenly divided using a specified number of bits, the quantization error of the target data is inversely proportional to the number of quantization bits. Furthermore, as discussed in Section 3.2, the robustness of diffusion models varies across different diffusion steps. Hence, we can expect that the bit precision of activations at each diffusion step will have distinct effects on FID. To investigate the impact of activation quantization on different diffusion steps, we divide the diffusion steps into five intervals [1.0T:0.8T, 0.8T:0.6T, ..., 0.2T:0.0T], and selectively apply activation quantization to the target step interval while keeping the floating-point activation values in the other steps. For instance, if the target interval is set as  $1.0T:0.8T$ , activation quantization is applied within this interval only, while activations in the other diffusion steps remain unquantized. Fig. 6 indicate that the early-stage diffusion steps closer to  $1.0T$
exhibit higher resilience to quantization errors. In the case of applying 4-bit activation quantization to the  $0.2T:0.0T$  diffusion steps in 5k LSUN-Churches generation, the resulting FID is 34.30. However, when the same 4-bit activation quantization is applied to the  $1.0T:0.8T$  steps, the resulting FID is 6.10, which is close to the FID of 6.05 achieved through image generation using a full-precision network.
It is worthwhile to emphasize that the dynamic nature of activation quantization during network processing eliminates the requirement of using identical activation bits for each diffusion step. This characteristic alleviates burden on adjustment of network parameters or other factors to reduce the activation bits. Consequently, by leveraging the robustness observed in the early stages, it becomes possible to further reduce activation bits, leading to more efficient processing of the denoising network.
4.3 Proposed Robustness-Aware Quantization
The previous section has shown that using low activation bits (e.g. 4 bits) for the denoising network in the early diffusion steps does not have a significant impact on the quality of image generation. However, as the reverse diffusion process progresses, it becomes increasingly challenging to lower the activation bits while maintaining the generated image quality. Therefore, we propose a quantization technique that leverages robustness observed in the early stages to minimize the activation bits as much as possible.
Algorithm 1 Robustness-aware Quantization
Require: Weight-quantized denoising network  $\epsilon_{\theta_q}$    
Require: Lower bound of the activation bit  $a$    
Require: Number of sampled images  $n$  for evaluation and threshold of FID  $\eta_{fid}$    
Bitact  $= a$ $D_{act} = \{\}$  Dact: Dictionary for matching steps and activation bits for  $t = T$  to 1 do Optimizing activation bits for each step while (t not in  $D_{act}.keys()$  or  $(D_{act}[t]\neq Bit_{act})$  do  $D_{act}[t] = Bit_{act}$  Sample n images with  $\epsilon_{\theta_q}$  applying  $D_{act}$  for acitvations if FID of sampled images  $>\eta_{fid}$  then  $Bit_{act} = Bit_{act} + 1$  end if end while end for