SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Gongfan Fang Xinyin Ma Xinchao Wang*
gongfan@u.nus.edu, maxinyin@u.nus.edu, xinchao@nus.edu.sg
Generative modeling has recently undergone remarkable advancements, primarily propelled by the transformative implications of Diffusion Probabilistic Models (DPMs). The impressive capability of these models, however, often entails significant computational overhead during both training and inference. To tackle this challenge, we present Diff-Pruning, an efficient compression method tailored for learning lightweight diffusion models from pre-existing ones, without the need for extensive re-training. The essence of Diff-Pruning is encapsulated in a Taylor expansion over pruned timesteps, a process that disregards non-contributory diffusion steps and ensembles informative gradients to identify important weights. Our empirical assessment, undertaken across several datasets highlights two primary benefits of our proposed method: 1) Efficiency: it enables approximately a  $50\%$  reduction in FLOPs at a mere  $10\%$  to  $20\%$  of the original training expenditure; 2) Consistency: the pruned diffusion models inherently preserve generative behavior congruent with their pre-trained models. Code is available at https://github.com/VainF/Diff-Pruning.
Generative modeling has undergone significant advancements in the past few years, largely propelled by the advent of Diffusion Probabilistic Models (DPMs) [18, 41, 37]. These models have derived numerous applications ranging from text-to-image generation [40], image editing [58], image translation[45], and even discriminative tasks [2, 1]. The incredible power of DPMs, however, often comes at the expense of considerable computational overhead during both training [49] and inference [43]. This trade-off between performance and efficiency presents a critical challenge in the broader application of these models, particularly in resource-constrained environments.
In the literature, huge efforts have been made to improve diffusion models, which primarily revolved around three broad themes: improving model architectures [41, 39, 52], optimizing training methods [49, 11] and accelerating sampling [46, 43, 12]. As a result, a multitude of well-trained diffusion models has been created in these valuable works, showcasing their potential for various applications [48]. However, the notable challenge still remains: the absence of a general compression method that enables the efficient reuse and customization of these pre-existing models without heavy re-training. Overcoming this gap is of paramount importance to fully harness the power of pre-trained diffusion models and facilitate their widespread application across different domains and tasks.
In this work, we demonstrate the remarkable effectiveness of structural pruning [23, 8, 26, 4] as a method for compressing diffusion models, which offers a flexible trade-off between efficiency and quality. Structural pruning is a classic technique that effectively reduces model sizes by eliminating redundant parameters and sub-structures from networks. While it has been extensively studied in discriminative tasks such as classification [16], detection [54], and segmentation [13], applying structural pruning techniques to Diffusion Probabilistic Models poses unique challenges that necessitate
a rethinking of traditional pruning strategies. For example, the iterative nature of the generative process in DPMs, the models' sensitivity to small perturbations in different timesteps, and the intricate interplay in the diffusion process collectively create a landscape where conventional pruning strategies often fall short.
To this end, we introduce a novel approach called Diff-Pruning, explicitly tailored for the compression of diffusion models. Our method is motivated by the observation in previous works [41, 52] that different stages in the diffusion process contribute variably to the generated samples. At the heart of our method lies a Taylor expansion over pruned timesteps, which deftly balances the image content, details, and the negative impact of noisy diffusion steps during pruning. Initially, we show that the objective of diffusion models at late timesteps  $(t\to T)$  prioritize the high-level content of the generated images during pruning, while the early ones  $(t\to 0)$  refine the images with finer details. However, it is also observed that, when using Taylor expansion for pruning, the noisy stages with large  $t$  can not provide informative gradients for importance estimation and can even harm the compressed performance. Therefore, we propose to model the trade-off between contents, details, and noises as a pruning problem of the diffusion timesteps, which leads to an efficient and flexible pruning algorithm for diffusion models.
Through extensive empirical evaluations across diverse datasets, we demonstrate that our method achieves substantial compression rates while preserving and in some cases even improving the generative quality of the models. Our experiments also highlight two significant features of Diff-Pruning: efficiency and consistency. For example, when applying our method to an off-the-shelf diffusion model pre-trained on LSUN Church [57], we achieve an impressive compression rate of  $50\%$  FLOPs, with only  $10\%$  of the training cost required by the original models, equating to 0.5 million steps compared to the 4.4 million steps of the pre-existing models. Furthermore, we have thoroughly assessed the generative behavior of the compressed models both qualitatively and quantitatively. Our evaluations demonstrate that the compressed model can effectively preserve a similar generation behavior as the pre-trained model, meaning that when provided with the same inputs, both models yield consistent outputs. Such consistency further reveals the practicality and reliability of Diff-Pruning as a compression method for diffusion models.
In summary, this paper introduces Diff-Pruning as an efficient method for compressing Diffusion Probabilistic Models, which is able to achieve compression with only  $10\%$  to  $20\%$  of the training costs compared to pre-training. This work may serve as an initial baseline and provide a foundation for future research aiming to enhance the quality and consistency of compressed diffusion models.
2 Ralted Works
Efficient Diffusion Models The existing methodologies principally address the efficiency issues associated with diffusion models via three primary strategies: the refinement of network architectures [41, 52, 37], the enhancement of training procedures [11, 49], and the acceleration of sampling [18, 27, 12]. Diffusion models typically employ U-Net models as denoisers, of which the efficiency can be improved via the introduction of hierarchical designs [40] or by executing the training within a novel latent space [41, 19, 25]. Recent studies also suggest integrating more efficient layers or structures into the denoiser to bolster the performance of the U-Net model [52, 39], thereby facilitating superior image quality learning during the training phase. Moreover, a considerable number of studies concentrate on amplifying the training efficiency of diffusion models, with some demonstrating that the diffusion training can be expedited by modulating the weights allocated to distinct timesteps [43, 11]. The training efficiency can also be advanced by learning diffusion models at the patch level [49]. In addition, some approaches underscore the efficiency of sampling, which typically does not necessitate the retraining of diffusion models [27]. In this area, numerous studies aim to diminish the required steps through methods such as early stopping [34] or distillation [43].
Network Pruning In recent years, the field of network acceleration [59, 3, 20, 53, 51, 29, 30] has seen notable progress through the deployment of network pruning techniques [31, 16, 33, 23, 14, 5, 15]. The taxonomy of pruning methodologies typically bifurcates into two main categories: structural pruning [23, 6, 56, 26, 56] and unstructured pruning [38, 7, 44, 22]. The distinguishing trait of structural pruning is its ability to physically eliminate parameters and substructures from networks, while unstructured pruning essentially masks parameters by zeroing them out [8, 4]. However, the preponderance of network pruning research is primarily focused on discriminative
tasks, particularly classification tasks [16]. A limited number of studies have ventured into examining the effectiveness of pruning in generative tasks, such as GAN compression [24, 47]. Moreover, the application of structural pruning techniques to Diffusion Probabilistic Models introduces unique challenges that demand a reevaluation of conventional pruning strategies. In this work, we introduce the first dedicated method explicitly designed for pruning diffusion models, which may serve as a useful baseline for future works.
3 Diffusion Model Objectives
Given a data distribution  $q(\pmb{x})$ , diffusion models aim to model a generative distribution  $p_{\theta}(\pmb{x})$  to approximate  $q(\pmb{x})$ , taking the form
$$
p _ {\theta} (\boldsymbol {x}) = \int p _ {\theta} \left(\boldsymbol {x} _ {0: T}\right) d \boldsymbol {x} _ {1: T}, \quad \text {w h e r e} \quad p _ {\theta} \left(\boldsymbol {x} _ {0: T}\right) := p \left(\boldsymbol {x} _ {T}\right) \prod_ {t = 1} ^ {T} p _ {\theta} \left(\boldsymbol {x} _ {t - 1} \mid \boldsymbol {x} _ {t}\right) \tag {1}
$$
And  $\pmb{x}_1, \dots, \pmb{x}_T$  refer to the latent variables, which contribute to the joint distribution  $p_{\theta}(\pmb{x}_{0:T})$  with learned Gaussian transitions  $p_{\theta}(\pmb{x}_{t-1}|\pmb{x}_t) = \mathcal{N}(\pmb{x}_{t-1}; \mu_{\theta}(\pmb{x}_t,t), \Sigma_{\theta}(\pmb{x}_t,t))$ . Diffusion Models involve two opposite processes: a forward (diffusion) process  $q(\pmb{x}_t|\pmb{x}_{t-1}) = \mathcal{N}(\pmb{x}_t; \sqrt{1 - \beta_t}\pmb{x}_{t-1}, \beta_tI)$  that adds noises to the  $\pmb{x}_{t-1}$ , based on a pre-defined variance schedule  $\beta_{1:T}$ ; and a reverse process  $q(\pmb{x}_{t-1}|\pmb{x}_t)$  which "denoises" the observation  $\pmb{x}_t$  to get  $\pmb{x}_{t-1}$ . Using the notation  $\alpha_t = 1 - \beta_t$  and  $\bar{\alpha}_t = \prod_{s=1}^t \alpha_s$ , DDPMs [18] trains a noise predictor with the objective:
$$
\mathcal {L} (\boldsymbol {\theta}) := \mathbb {E} _ {t, \boldsymbol {x} _ {0} \sim q (\boldsymbol {x}), \epsilon \sim \mathcal {N} (0, 1)} \left[ \| \boldsymbol {\epsilon} - \boldsymbol {\epsilon} _ {\boldsymbol {\theta}} \left(\sqrt {\bar {\alpha} _ {t}} \boldsymbol {x} _ {0} + \sqrt {1 - \bar {\alpha} _ {t}} \boldsymbol {\epsilon}, t\right) \| ^ {2} \right] \tag {2}
$$
where  $\epsilon$  is a random noise drawn from a fixed Gaussian distribution and  $\epsilon_{\theta}$  refers to a learned noise predictor, which is usually an U-Net autoencoder [42] in practice. After training, synthetic images  $x_0$  can be sampled through an iterative process from a noise  $x_{T} \sim \mathcal{N}(\mathbf{0}, \mathbf{1})$  with the formular:
$$
\boldsymbol {x} _ {t - 1} = \frac {1}{\sqrt {\alpha_ {t}}} \left(\boldsymbol {x} _ {t} - \frac {\beta_ {t}}{\sqrt {1 - \bar {\alpha} _ {t}}} \boldsymbol {\epsilon} _ {\boldsymbol {\theta}} (\boldsymbol {x} _ {t}, t)\right) + \sigma_ {t} \boldsymbol {z} \tag {3}
$$
where  $z \sim \mathcal{N}(\mathbf{0}, \mathbf{I})$  for steps  $t > 1$  and  $z = \mathbf{0}$  for  $t = 1$ . In this work, we aim to craft a lightweight  $\epsilon_{\theta'}$  by removing redundant parameters of  $\epsilon_{\theta}$ , which are expected to produce similar  $x_0$  while the same  $x_T$  are presented.
4 Structural Pruning for Diffusion Models
Given the parameter  $\theta$  of a pre-trained diffusion model, our goal is to craft a lightweight  $\theta^{\prime}$  by removing sub-structures from the network following existing paradigms [35, 8]. Without loss of generality, we assume that the parameter  $\theta$  is a simple 2-D matrix, where each sub-structure  $\theta_{i} = [\theta_{i0},\theta_{i1},\dots,\theta_{iK}]$  is a row vector that contains  $K$  scalar parameters. Structural pruning aims to find a sparse parameter matrix  $\theta^{\prime}$  that maximally preserves the original performance. Thus, a natural choice is to optimize the loss disruption caused by pruning:
$$
\min  _ {\boldsymbol {\theta} ^ {\prime}} \left| \mathcal {L} \left(\boldsymbol {\theta} ^ {\prime}\right) - \mathcal {L} (\boldsymbol {\theta}) \right|, \quad \text {s . t .} \| \boldsymbol {\theta} ^ {\prime} \| _ {0} \leq s \tag {4}
$$
The term  $|\theta'|_0$  denotes the L-0 norm of the parameters, which counts the number of non-zero row vectors, and  $s$  represents the sparsity of the pruned model. Nevertheless, due to the iterative nature intrinsic to diffusion models, the training objective, denoted by  $\mathcal{L}$ , can be perceived as a composition of  $T$  interconnected tasks:  $\{\mathcal{L}_1, \mathcal{L}_2, \dots, \mathcal{L}_T\}$ . Each task affects and depends on the others, thereby posing a new challenge distinct from traditional pruning problems, which primarily concentrate on optimizing a single objective. In light of the pruning objective as defined in Equation 4, we initially delve into the individual contributions of each loss component,  $\mathcal{L}_t$  in pruning, and subsequently propose a tailored method, Diff-Pruning, designed for diffusion models pruning.
Taylor Expansion at  $\mathcal{L}_t$  Initially, we need to model the contribution of  $\mathcal{L}_t$  for structural pruning. This work leverages Taylor expansion [35] on  $\mathcal{L}_t$  to linearly approximate the loss disruption:
$$
\begin{array}{l} \mathcal {L} _ {t} \left(\boldsymbol {\theta} ^ {\prime}\right) = \mathcal {L} _ {t} (\boldsymbol {\theta}) + \nabla \mathcal {L} _ {t} (\boldsymbol {\theta}) \left(\boldsymbol {\theta} ^ {\prime} - \boldsymbol {\theta}\right) + O \left(\left\| \boldsymbol {\theta} ^ {\prime} - \boldsymbol {\theta} \right\| ^ {2}\right) \tag {5} \\ \Rightarrow \mathcal {L} _ {t} \left(\boldsymbol {\theta} ^ {\prime}\right) - \mathcal {L} _ {t} (\boldsymbol {\theta}) = \nabla \mathcal {L} _ {t} (\boldsymbol {\theta}) \left(\boldsymbol {\theta} ^ {\prime} - \boldsymbol {\theta}\right) + O \left(\left\| \boldsymbol {\theta} ^ {\prime} - \boldsymbol {\theta} \right\| ^ {2}\right) \\ \end{array}
$$
Figure 1: Diff-Pruning leverages Taylor expansion at pruned timesteps to estimate the importance of weights, where early steps focus on local details like edges and color and later ones pay more attention to contents such as object and shape. We propose a simple thresholding method to trade off these factors with a binary weight  $\alpha_{t} \in \{0,1\}$ , leading to a practical algorithm for diffusion models. The generated images produced by  $5\%$  -pruned DDPMs (without post-training) are illustrated.
Taylor expansion offers a robust framework for network pruning, as it can estimate the loss disruption using first-order gradients. To evaluate the importance of an individual weight  $\theta_{ik}$ , we can simply set  $\theta^{\prime}_{ik} = 0$  in Equation 5, which results in the following importance criterion:
$$
\begin{array}{l} \mathcal {I} _ {t} \left(\boldsymbol {\theta} _ {i k}, \boldsymbol {x}\right) = | \mathcal {L} _ {t} (\boldsymbol {\theta} | _ {\boldsymbol {\theta} _ {i k} = 0}) - \mathcal {L} _ {t} (\boldsymbol {\theta}) | \\ = \left| \left(\boldsymbol {\theta} _ {i 0} - \boldsymbol {\theta} _ {i 0}\right) \cdot \nabla_ {\boldsymbol {\theta} _ {i 0}} + \dots + \left(0 - \boldsymbol {\theta} _ {i k}\right) \cdot \nabla_ {\boldsymbol {\theta} _ {i k}} + \dots + \left(\boldsymbol {\theta} _ {i K} - \boldsymbol {\theta} _ {i K}\right) \cdot \nabla_ {\boldsymbol {\theta} _ {i K}} \right| \tag {6} \\ = \left| \boldsymbol {\theta} _ {i k} \cdot \nabla_ {\boldsymbol {\theta} _ {i k}} \mathcal {L} _ {t} (\boldsymbol {\theta}, \boldsymbol {x}) \right| \\ \end{array}
$$
where  $\nabla_{\theta_{ik}}$  refers to  $\nabla_{\theta_{ik}}\mathcal{L}_t(\pmb {\theta},\pmb {x})$ . In structural pruning, we aim to remove the entire vector  $\pmb{\theta^{\prime}}_i$  concurrently. The standard Taylor expansion for multiple variables, as described in the literature [9], advocates using  $|\sum_{k}\pmb{\theta}_{ik}\cdot \nabla_{\theta_{ik}}\mathcal{L}_t(\pmb {\theta},\pmb {x})|$  for importance estimation. This method exclusively takes into account the loss difference between the initial state  $\pmb{\theta}$  and the final states  $\pmb{\theta}^{\prime}$ . However, considering the iterative nature of diffusion models, even minor fluctuations in loss can influence the final generation results. To this end, we propose to aggregate the influence of removing each parameter as the final importance. This modification models cumulative loss disturbance induced by each  $\pmb{\theta}_{ik}$ 's removal and leads to a slightly different score function for structural pruning:
$$
\mathcal {I} _ {t} \left(\boldsymbol {\theta} _ {i}, \boldsymbol {x}\right) = \sum_ {k} \left| \mathcal {L} _ {t} \left(\boldsymbol {\theta} \mid_ {\boldsymbol {\theta} _ {i k} = \boldsymbol {0}}\right) - \mathcal {L} _ {t} (\boldsymbol {\theta}) \right| = \sum_ {k} \left| \boldsymbol {\theta} _ {i k} \cdot \nabla_ {\boldsymbol {\theta} _ {i k}} \mathcal {L} _ {t} (\boldsymbol {\theta}, \boldsymbol {x}) \right| \tag {7}
$$
In the following sections, we utilize Equation 7 as the importance function to identify non-critical parameters in diffusion models.
The Contribution of  $\mathcal{L}_t$ . With the Taylor expansion framework, we further explore the contribution of different loss terms  $\{\mathcal{L}_1,\dots,\mathcal{L}_T\}$  during pruning. We consider the functional error  $\delta_t = \epsilon_{\theta'}(\boldsymbol{x},t) - \epsilon_\theta (\boldsymbol{x},t)$  which represents the prediction error for the same inputs at time step  $t$ . The reverse process allows us to exam the effects  $\delta_{t\to 0}$  on the generated images  $x_0$  by iteratively applying the Equation 3 starting from  $\epsilon_{\theta^{\prime}}(\pmb {x},t) = \epsilon_{\theta}(\pmb {x},t) + \delta_{t}$ . At the  $t - 1$  step, it leads to the error  $\delta_{t - 1}$  derived as:
$$
\begin{array}{l} \delta_ {t - 1} = \left[ \frac {1}{\sqrt {\alpha_ {t}}} \left(\boldsymbol {x} _ {t} - \frac {\beta_ {t}}{\sqrt {1 - \bar {\alpha} _ {t}}} \boldsymbol {\epsilon} _ {\boldsymbol {\theta}} (\boldsymbol {x} _ {t}, t)\right) + \sigma_ {t} \boldsymbol {z} \right] - \left[ \frac {1}{\sqrt {\alpha_ {t}}} \left(\boldsymbol {x} _ {t} - \frac {\beta_ {t}}{\sqrt {1 - \bar {\alpha} _ {t}}} (\boldsymbol {\epsilon} _ {\boldsymbol {\theta}} (\boldsymbol {x} _ {t}, t) + \delta_ {t})\right) + \sigma_ {t} \boldsymbol {z} \right] \\ = \frac {1}{\sqrt {\alpha_ {t}}} \frac {\beta_ {t}}{\sqrt {1 \bar {\alpha} _ {t}}} \delta_ {t} \tag {8} \\ \end{array}
$$
This error has a direct impact on the subsequent input, given by  $x_{t-1}' = x_{t-1} + \delta_{t-1}$ . By checking Equation 3, we can observe that these perturbed inputs can further trigger a chained effect through both  $\frac{1}{\sqrt{\alpha_{t-1}}} x_{t-1}'$  and  $-\frac{1}{\sqrt{\alpha_{t-1}}} \frac{\beta_{t-1}}{\sqrt{1 - \bar{\alpha}_{t-1}}} \epsilon_{\theta'}(x_{t-1}', t-1)$ . In the first term, the distortion progressively amplifies by a factor  $\frac{1}{\sqrt{\alpha_{t-1}}} > 1$ , which means that this error will be enhanced throughout the generation process. Regarding the second term, pruning affects both the functionality parameterized by  $\theta'$  and the inputs  $x_{t-1}'$ , which contributes to the final results in a nonlinear and more complicated manner, resulting in a more substantial disturbance on the generated images.
Algorithm 1 Diff-Pruning
Input: A pretrained diffusion model  $\theta$ , a dataset  $X$ , a threshold  $\mathcal{T}$  and a pruning ratio  $p\%$ . Output: The pruned diffusion model  $\theta'$
1:  $\mathcal{L}_{max} = \mathbf{0}$
2:  $x = \mathrm{mini}$  -batch(X);
3:  $\epsilon \sim \mathcal{N}(0,1)$
4:  $\triangleright$  Accumulating gradients over partial steps with the threshold  $\mathcal{T}$
5: for  $t$  in  $[0,1,2,\dots,T]$  do:
6:  $\mathcal{L}_t = \| \pmb {\epsilon} - \pmb {\epsilon}_\theta (\sqrt{\bar{\alpha}_t}\pmb {x} + \sqrt{1 - \bar{\alpha}_t}\pmb {\epsilon},t)\|^2;$
7:  $\mathcal{L}_{max} = \max (\mathcal{L}_{max},\mathcal{L}_t)$
8: if  $\mathcal{L}_t / \mathcal{L}_{max} \leq \mathcal{T}$  then
9: break;  $\triangleright$  The threshold in Equation 10
11:  $\nabla_{\pmb{\theta}_{ik}}\mathcal{L}_t(\pmb {\theta},x) =$  back-propagation  $(\mathcal{L}_t(\pmb {\theta},x))$
13:  $\triangleright$  Estimating the importance of sub-structure  $\theta_{i}$  with the accumulated  $t$ -step gradients
14:  $\mathcal{I}(\pmb{\theta}_i, x) = \sum_k |\pmb{\theta}_{ik} \cdot \sum_{s=0}^t \nabla_{\pmb{\theta}_{ik}} \mathcal{L}_s(\pmb{\theta}, x)|$
15:  $\triangleright$  Pruning and finetuning
16: Remove  $p\%$  channels in each layer to obtain  $\theta'$ .
17: Finetune the pruned model  $\theta^{\prime}$  on  $X$
18: return  $\theta^{\prime}$
As a result, prediction errors occurring at larger  $t$  tend to have a larger impact on the images due to the chain effect, which might change the global content of generated images. Conversely, smaller  $t$  values focus on refining the images with relatively small modifications. These findings align with our empirical examination using Taylor expansion as illustrated in Figure 1, as well as the observation in previous works [18, 52], which shows that diffusion models tend to generate object-level information at larger  $t$  values and fine-tune the features at smaller ones. To this end, we model the pruning problem as a weighted trade-off between contents and details by introducing  $\alpha_{t}$ , which acts as a weighting variable for different timesteps  $t$ . Nevertheless, unconstrained reweighting can be highly inefficient, as it entails exploring a large parameter space for  $\alpha_{t}$  and requires at least  $T$  forward-backward passes for Taylor expansion. This results in a vast sampling space and can lead to inaccuracies in the linear approximation. To address this issue, we simplify the re-weighting strategy by treating it as a "pruning problem", where  $\alpha_{t}$  takes the value of either 0 or 1 for all steps, allowing us to only leverage partial steps for pruning. The general importance metric is modeled as the following.
$$
\mathcal {I} \left(\boldsymbol {\theta} _ {i}, \mathbf {x}\right) = \sum_ {k} \left| \boldsymbol {\theta} _ {i k} \cdot \sum_ {t} \alpha_ {t} \nabla_ {\boldsymbol {\theta} _ {i k}} \mathcal {L} _ {t} (\boldsymbol {\theta}, \mathbf {x}) \right|, \quad \text {s . t .} \alpha_ {t} \in \{0, 1 \} \tag {9}
$$
Taylor Score over Pruned Timesteps. In Equation 9, we try to remove some "unimportant" timesteps in the diffusion process so as to enable an efficient and stable approximation for partial steps. Our empirical results, as will be discussed in the experiments, indicate two key findings. Firstly, we note that the timesteps responsible for generating content are not exclusively found towards the end of the diffusion process  $(t\rightarrow T)$ . Instead, there are numerous noisy and redundant timesteps that contribute minorly to the overall generation, which is similar to the observations in the related work [34]. Secondly, we discovered that employing the full-step objective can sometimes yield suboptimal results compared to using a partial objective. We attribute this negative impact to the presence of converged gradients in the noisy steps  $(t\to T)$ . Taylor approximation in Equation 5 comprises both first-order gradients and higher-order terms. When the loss  $\mathcal{L}_t$  converges, the loss curve is predominantly influenced by the higher-order terms rather than the first-order gradients we utilize. Our experiments on several datasets and diffusion models show that the loss term  $\mathcal{L}_t$  rapidly approaches 0 as  $t\to T$ . For example in Figure 5, the relative loss  $\frac{\mathcal{L}_t}{\mathcal{L}_{max}}$  of a pre-trained diffusion model for CIFAR-10 decreases to 0.05 when  $t = 250$ . Consequently, a full Taylor expansion can accumulate a considerable amount of noisy gradients from these converged or unimportant steps, resulting in an inaccurate estimation of weight importance.