Structural Pruning for Diffusion Models
Authors: Gongfan Fang
Xinyin Ma
Xinchao Wang
Venue: NeurIPS 2023 poster (2023)
Keywords: Diffusion Model
Network Pruning
Model Compression
Efficient Deep Learning

Abstract: Generative modeling has recently undergone remarkable advancements, primarily propelled by the transformative implications of Diffusion Probabilistic Models (DPMs). The impressive capability of these models, however, often entails significant computational overhead during both training and inference. To tackle this challenge, we present Diff-Pruning, an efficient compression method tailored for learning lightweight diffusion models from pre-existing ones, without the need for extensive re-training. The essence of Diff-Pruning is encapsulated in a Taylor expansion over pruned timesteps, a process that disregards non-contributory diffusion steps and ensembles informative gradients to identify important weights. Our empirical assessment, undertaken across several datasets highlights two primary benefits of our proposed method: 1) Efficiency: it enables approximately a 50\% reduction in FLOPs at a mere 10% to 20% of the original training expenditure; 2) Consistency: the pruned diffusion models inherently preserve generative behavior congruent with their pre-trained models.

Structural Pruning for Diffusion Models
Gongfan Fang
Xinyin Ma Xinchao Wang"
National University Singapore
gongfandi www..uu..eei... maxinyindu. mus edu, xinchao edu.
Abstract
Generative modeling has recently undergone remarkable advancements, primarily propelled by the transformative implications of Diffusion Probabilistic Models (DPMs). The impressive capability of these models, however, often entails signif- icant computational overhead during both training and inference. To tackle this challenge, we present siff-PPruning. efficient compression method tailored for learning lightweight diffusion models from pre-existing ones, without the need for extensive re-training. The essence of Diff-Pruning is encapsulated in a Taylor expansion over pruned timesteps, a process that disregards non non-contributory dif- fusion steps and ensembles informative gradients to identify important weights. Our empirical assessment, undertaken across several datasets highlights two pri- mary benefits of our proposed method: 1) Efficiency: it enables approximately a 50% reduction FLOPs at a mere 10% to 20% of the original training ex- penditure; 2) Consistency: the pruned diffusion models inherently preserve gen- erative behavior congruent with their pre-trained models. Code is available at https: https://gittub. wtw.ttt..cooooooeecooommo www.//aia//Dii.....ac.mmmmmo Pruniin
Introduction
Generative modeling has undergone significant advancements in the past few years, largely pro-
pelled by the advent of Diffusion Probabilistic Models (DPMs) [18 derived numerous applications ranging from text-to-image generation
41 37 These models have
image editing [58] im-
age translation 45,, and even discriminative tasks 1 'he incredible power of DPMs, however, often comes
expense of considerable computational overhead during both training [49 and
inference [43] This trade-off between performance and efficiency presents critical challenge in the broader application these models, particularly in resource-constraimg environments.
In the literature, huge efforts have been made to improve diffusion models, which primarily revolved around three broad themes: improving model architectures 41 39 52],
optimizing training meth- ods 49 I and accelerating sampling [46] 43 [2] As result, a multitude of well-trained diffusion models has been created these valuable works, showcasing their potential for various applications 48 However, the notable challenge still remains: the absence of a general compression method that enables the efficient reuse and customization of these pre-existing models without heavy re-training. Overcoming this gap is of paramount importance to fully harness the power of pre-trained diffusion models and facilitate their widespread application across different domains and tasks.
In this work, we demonstrate the remarkable effectiveness of structural pruning 23 method for compressing diffusion models, which offers flexible trade-off between efficiency and quality. Structural pruning is a classic technique that effectively reduces model sizes by eliminating redundant parameters and atbsstrucstres from networks. While has been extensively studied in discriminative tasks such as classification [16]. detection [54], and segmentation [13, applying struc- tural pruning techniques to Diffusion Probabilistic Models poses unique challenges that necessitate
26
as
CCoreesondingg author
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

rethinking of traditional pruning strategies. For example, the iterative nature of the generative process in DPMs, the models' sensitivity to small perturbations in different timesteps, and the intricate interplay in the diffusion process collectively create landscape where conventional pruning strategies often fall short.
To this end, we introduce a novel approach called Diff-Pruning, explicitly tailored for the compression of diffusion models. Our method is motivated by the observation in previous works 41 [2 that different stages in the diffusion process contribute variably to the generated samples. At the heart of our method lies a Taylor expansion over pruned timesteps, which deftly balances the image content, details, and the negative impact of noisy diffusion steps during pruning. Initially, we show that the objective of diffusion models at late timesteps (t + T) prioritize the high-level content of the generated images during pruning, while the early ones (t - 0) refine the images with finer details. However, it also observed that, when using Taylor expansion for pruning, the noisy stages with large t can not provide informative gradients for importance estimation and can even harm the compressed performance. Therefore, we propose to model the trade-off between contents, details, and noises as a pruning problem of the diffusion timesteps, which leads to an efficient and flexible pruning algorithm for diffusion models.
Through extensive empirical evaluations across diverse datasets, we demonstrate that our method achieves substantial compression rates while preserving and in some cases even improving the generative quality of the models. Our experiments also highlight two significant features of Diff- Pruning: efficiency and consistency. For example, when applying our method to an off-the-shelf diffusion model pre-trained on LSUN Church 157, we achieve an impressive compression rate of 50% FLOPS, with only 10% of the training cost required by the original models, equating to 0.5 million steps compared to the 4.4 million steps of the pre-existing models. Furthermore, we have thoroughly assessed the generative behavior of the compressed models both qualitatively and quantitatively. Our evaluations demonstrate that the compressed model can effectively preserve similar generation behavior as the pre-trained model, meaning that when provided with the same inputs, both models yield consistent outputs. Such consistency further reveals the practicality and reliability of Diff- fiff-Pruning as a compression method for diffusion models.
In summary, this paper introduces Diff-Pruning as an efficient method for compressing Diffusion Probabilistic Models, which is able to achieve compression with only 10% to 20% of the training costs compared to pre-training. This work may serve as an initial baseline and provide a foundation for future research aiming to enhance the quality and consistency of compressed diffusion models.
2 Ralted Works
Efficient Diffusion Models The existing methodologies principally address the efficiency issues associated with diffusion models via three primary strategies: the refinement of network archi-
tectures [1
52
37 the enhancement of training procedures [1 49, and the acceleration of
sampling 18 27 I2). Diffusion models typically employ U-Net models as denoisers, of which
the efficiency can improved via the introduction of hierarchical designs (40) or by executing the
training within novel latent space 41 19 25). Recent studies also suggest integrating more efficient
layers or structures into the denoiser to bolster the performance of the U-Net model (52 39), thereby facilitating superior image quality learning during the training phase. Moreover, a considerable number of studies concentrate on amplifying the training efficiency of diffusion models, with some demonstrating that the diffusion training can be expedited by modulating the weights allocated to distinct timesteps [43 [1]. The training efficiency can also be advanced by learning diffusion models at the patch level 49 In addition, some approaches underscore the efficiency of sampling, which
typically does not necessitate the retraining of diffusion models (27). In this area, numerous studies aim to diminish the required steps through methods such early stopping (34 or distillation (3).
Network Pruning In recent years, the field of network acceleration 159
3
20
53
51
29
30
has seen notable progress through the deployment of network pruming techniques
B1
33 23
14 5 15]. The taxonomy of pruning methodologies typically bifurcates into two main categories:
7 44 22 The distinguishing trait of structural pruning is its ability to physically eliminate parameters and substructures from networks, while unstructured pruning essentially masks parameters by zeroing them out R 4. However, the preponderance of network pruning research is primarily focused on discriminative
structural pruning 23 56 26 56) and unstructured pruning [38
2

---

Given a data distribution g(z). diffusion models aim to model a generative distribution po(x:) to tasks, particularly classification tasks [[]. A limited number of studies have ventured into examining the effectiveness of pruning generative tasks, such as GAN compression 474 Moreover, the application of structural pruning techniques to Diffusion Probabilistic Models introduces unique challenges that demand a reevaluation of conventional pruning strategies. In this work, we introduce
the first dedicated method explicitly designed for pruning diffusion models, which may serve as useful baseline for future works.
Diffusion Model Objectives
approximate g(x). taking the form
po(x)) f po (20. (ooorrddrr.
where (x p(TT) II po po t=1
(I)
And Ei,..., Zy refer to the latent variables, which contribute to the joint distribution po (zo:)) with learned Gaussian transitions p xx;) = -V (x t). Ex,,, Diffusion Models in- volve two opposite processes: forward (diffusion) process g( = Naax; v1 3, that adds noises to the 2;-1, based on pre-defined variance schedule B1.:: and a reverse process (a2; (2:;) which "denoises" the observation 2; to get T2-1- Using the notation o₁ = 1 8g and
āe IT= Og. DDPMs [18] trains noise predictor with the objective:
L(0) == F,, F0, [[le --oooo + 11-,,6,1113
where a random noise drawn from fixed Gaussian distribution and Eş refers to learned noise predictor, which is usually an U-Net autoencoder [2] in practice. After training, synthetic images Do can be sampled through iterative process from noise TT (O, 1) with the formular:
3.
at
60 c0( az
+ 0:z
VI
where z ~ (0, 1) for steps t > 1 and x = 0 for f 1. this work, we aim to craft a lightweight Eor by removing redundant parameters of cø, which expected to produce similar Ty while the same 2r are presented.
Structrual Pruning for Diffusion Models
Given the parameter 0 of pre-trained diffusion model, our goal is to craft lightweight 0' by removing sub-structures from the network following existing paradigms [35] 8. Without loss of generality, we assume that the parameter 0 is simple 2-1) matrix, where each sub-structure 0, = [Oso, f....., O:k] is row vector that contains K scalar parameters. Structural pruning aims to find sparse parameter matrix O' that maximally preserves the original performance. Thus, natural choice is to optimize the loss disruption caused by pruning:
min [C(0') L())), s.t. 110' S 6'
The term 10']o denotes the L-0 norm of the parameters, which counts the number of non-zero row vectors, and represents the sparsity of the pruned model. Nevertheless, due to the iterative nature intrinsic to diffusion models, the training objective, denoted by L, can be perceived as a composition of T' interconnected tasks: {C1, L2, }r Each task affects and depends on the others. thereby posing new challenge distinct from traditional pruning problems, which primarily concentrate on optimizing a single objective. In light of the pruning objective as defined in Equation [4] we initially delve into the individual contributions of each loss component, L. in pruning, and subsequently propose a tailored method, Diff- fiff-Pruning, designed for diffusion models pruning.
Taylor Expansion at C, Initially, we need model the contribution of L, for structural pruning. This work leverages Taylor expansion (35) on L, to linearly approximate the loss disruption:
(,'))) (0) + VL,(0) 0) O(l 0
(5)
C,(0') - C.()) = (L.( - -0) + OO10'' - 01l2)
3

---

Preserving Contents
Contents Details
Preserving Details
lo.ca 10 9 Leal
Importance ln 18 Ee Re Rydeel
10 = 1 Va Lool
no (x,-11
XT
Xt2
Xt1
X11-1
Xeo
Xo
q(x,1x,-1)
Pruned Noisy Steps
Figure 1: Diff-- Pruning leverages Taylor expansion at pruned timesteps to estimate the importance of weights, where early steps focus local details like edges and color and later ones pay more attention to contents such as object and shape. We propose a simple thresholding method to trade off these factors with binary weight ag (0, 1), leading practical algorithm for diffusion models. The generated images produced by 5%-pruned DDPMs (without post-training) are illustrated.
Taylor expansion offers robust framework for network pruning, it can estimate the loss disruption using first-order gradients. To evaluate the importance of an individual weight Oik. we can simply set O' = 0 in Equation 5 which results in the following importance criterion:
I, (0 x) = [ ( o., o.:=o - (0)
(0 0.0) Vo..
0;k) Von
(0ik Oik) Vou
(6)
= 0ik (0)
where Vo., refer to Vo,, (0, In structural pruning, we aim to remove the entire vector 0'; concurrently. The standard Taylor expansion for multiple variables, as described in the literature 19] advocates using Σx 0.x C, [0)l for importance estimation. This method exclusively takes into account the loss difference between the initial state O and the final states O'. However, considering the iterative nature diffusion models, even minor fluctuations in loss can influence the final generation results. To this end, we propose to aggregate the influence of removing each parameter as the final importance. This modification models cumulative loss disturbance induced by each 0 's removal and leads to slightly different score function for structural pruning:
I,,0,, Σ (0,(0 -=(0)==>) Σ
E10 Ozx Vo Ci,, x)
(7)
In the following sections, utilize Equation the importance function identify non-critical parameters in diffusion models.
The Contribution of L.. With the Taylor expansion framework, further explore the contribution of different loss terms {L1, Lr during pruning. We consider the functional error d = Coy (x: t) co (x, which represents the prediction error for the same inputs at time step t. The reverse process allows us to exam the effects ō, on the generated images zo by iteratively applying the Equation starting from eg (ar, Eo (xx + òr. At the L step, it leads to the error d derived as:
8
D4
V1
a. co(
+ 012
- Vor
xt
v1 -1- a
(co( + 6.) 0.2
B.
5z
V
vlā.
(8)
This eиτоr has direct impact on the subsequent input, given by 2/-1 Zt-1 + By checking Equation] 3 we can observe that these perturbed inputs can further trigger a chained effect through both
and
8--1 Var-i -1-1 6or (#/_1> 1- 1). In the first term, the distortion progressively
which means that this error will be enhanced throughout the generation process. Regarding the second term, pruning affects both the functionality parameterized by 0' and the inputs a/-1- which contributes to the final results in a nonlinear and more complicated manner, resulting in more substantial disturbance the generated images.
amplifies by a factor f

---

Algorithm Diff-Pruning
Input: A pretrained diffusion model 0, dataset X, a threshold T and a pruning ratio p% Output: The pruned diffusion model 6'
1: Lmaz =
2:
(Xni-batch(X);;
3: E <~ ~~N(0,1) 1)
4: Accumulating gradients over partial steps with the threshold T do:
5: for 1. in [0, 1, 2,,, ,
6: Li = EØly 0 V-- 0,C, 1)l1:
volvara Lmaa max Lmaz: L()
Equation
7:
8:
if Li/Cmax S T then break;
9:
>> The threshold in Equation 10
end if
11: Vo.. L((0.x) back- (ropppaattiio((C(((..)))
end for
13: >> Estimating the importance of sub-structure 0 with the accumulated t-step gradients
14: I(0;, = Ex 00x. Ei= Vo, L. (0,
& Equation 10
Pruning and finetuning
16: Remove p% channels in each layer to obtain 0'.
17: Finetune the pruned model 0' on
18: return 0'
As result, prediction errors occurring at larger tend to have a larger impact on the images due to the chain effect, which might change the global content of generated images. Conversely, smaller t values focus on refining the images with relatively small modifications. These findings align with our empirical examination using Taylor expansion illustrated in Figure [ as well the observation in previous works 18] (2), which shows that diffusion models tend to generate object-level information larger / values and fine-tune the features smaller ones. To this end, model the pruning problem as a weighted trade-off between contents and details by introducing at, which acts as a weighting variable for different timesteps f. Nevertheless, unconstrained reweighting can be highly inefficient, as it entails exploring a large parameter space for a₁ and requires at least T forward-backward passes for Taylor expansion. This results in a vast sampling space and can lead to inaccuracies in the linear approximation. To address this issue, we simplify the re- re-weighting strategy by treating it as "pruning problem". where a takes the value of either 1 for all steps, allowing us only leverage partial steps for pruning. The general importance metric is modeled as the following.
I(O,. Σ Osk [oveex.
s.t. 0g E (0.1)
Taylor Score over Pruned Timesteps. In Equation we try remove some "unimportant" timesteps in the diffusion process so to enable an efficient and stable approximation for partial steps. Our empirical results, as will be discussed in the experiments, indicate two key findings. Firstly, we note that the timesteps responsible for generating content are not exclusively found towards the end the diffusion process (t + T)). Instead, there are numerous noisy and redundant timesteps that contribute minorly to the overall generation, which is similar to the observations in the related work [4]. Secondly, we discovered that employing the full-step objective can sometimes yield suboptimal results compared to using partial objective. We attribute this negative impact to the presence of converged gradients in the noisy steps (t - T). Taylor approximation in Equation 5 comprises both first-order gradients and higher-order terms. When the loss C, converges, the loss curve is predominantly influenced by the higher-order terms rather than the first-order gradients we utilize. Our experiments on several datasets and diffusion models show that the loss term L, rapidly approaches 0 as f - T. For example in Figure 5 the relative loss سيع of pre-trained diffusion model for CIFAR-10 10 decreases to 0.05 when / 250. Consequently, full Taylor expansion can accumulate a considerable amount of noisy gradients from these converged or unimportant steps, resulting in an inaccurate estimation of weight importance.
5

---

Considering the significant impact of larger timesteps, it is necessary to incorporate them for im- portance estimation. To address this problem. Equation 9 nnaturrlly provides simple and practical thresholding strategy for pruning. To achieve this, we introduce a threshold parameter 'I based on the relative loss Emaz' Those timesteps with a relative loss below this threshold, i.e., _يع < T, Emax are considered uninformative and are disregarded by setting at = 0, which yields the finalized importance score:
I(0;, = Σ O:k-
Σ (1) Ziaz >T}
Von. Vo (0, x)
(10)
In practice, we need to select an appropriately large value for T to strike a well well-balanced preservation of details and content, while also avoiding uninformative gradients from noisy loss terms. The full algorithm is summarized in Alg.
5
Experiments
5.1 Settings
Datasets and Models The efficacy of Diff-Pruning is empirically validated across six diverse datasets, including CIFAR-10 (32 (11) CelebA-HQ (44))4) LSUN Church (556x566), LSUN Bedroom (256x256) (57] and ImageNet- IK (256x256). We focus on two popular DPMs in our experiments, i.c., Denoising Diffusion Probability Models (DDPMs) [18] and Latent Diffusion
Models (LDMs) 4]. For the sake of reproducibility, we utilize off-the-shelf DPMs from [8 and 41 as pre-trained models and prune these models in a one-shot fashion 23
Evaluation Metries In this paper, we concentrate primarily on three types of metries: 1) Efficiency metrics, which include the number of parameters (#Params) and Multiply-Add Accumulation (MACs); 2) Quality metric, namely the Frechet Inception Distance (FID) 17]; and 3) Consistency metric, represented by Structural Similarity (SSIM) [50. Unlike previous generative tasks that lacked reference images, we employ the SSIM index to evaluate the similarity between images generated by pre-trained models and pruned models, given identical noise inputs. We deplpy a 250-step DDIM sampler (46) for ImageNet and a 100-step DDIM sampler for other experiments.
5.2 An Simple Benchmark for Diffusion Pruning
Scratch Training v.s. Pruning. Table shows our results CIFAR-10 and CelebA-HQ. The first baseline method that piques our interest is scratch training. Numerous studies on network pruning 100 suggest that training compact network from scratch can be formidable competitor. To ensure fair comparison, we create randomly initialized networks with the same architecture as the pruned ones for scratch training. Our results reveal that scratch training demands relatively more steps to reach convergence. This suggests that training lightweight models from scratch may not be an efficient and economical approach, given its training cost is comparable to that of pre- treitrained models. Conversely, we observe that all pruning methods are able to converge within approximately 100K steps and outperform scratch training in terms of FID and SSIM scores. Thus, pruning emerges as potent technique for compressing pre-trained Diffusion Models.
Pruning Criteria. A significant aspect of network pruning is the formulation of pruning criteria, which serve to identify superfluous parameters within networks. Due to the absence of dedicated work on Diffusion model pruning, we adapted three basic pruning methods from discriminative tasks: random pruning, magnitude- based pruning [16), and Taylor-bascl pruning [36], which we refer to as Random, Magnitude, and Taylor respectively in subsequent sections. For given parameter O. Random assigns importance scores derived from uniform distribution to each 0; randomly, denoted as I(@)= (0, 1). This results in straightforward baseline devoid of any prior or bias, and has been shown to be competitive baseline for pruning (28). Magnitude subscribes to the "smaller-norm- less-informative" hypothesis 22 (55, modelling the weight importance as I(0) 0 contrast, Taylor is a data-driven criterion that measures importance as I(O, x)) = 10- VoL(æ, 0)). which aims to minmmize loss change as discussed in our method. As shown in T an intriguing phenomenon that these three baseline methods do not maintain consistent ranking these two datasets. For

---

CIFAR-10 32 32 (100 DDIM steps)
Method
#Paarms
MACs
FID 4
SSIM f
Train Steps
Pretrained
35.7M
6.1G
4.19
1.000
800K
Scratch Training
9.88
0.887
100K
Scratch Training
5.68
0.905
500K
Scratch Training
5.39
0,905
800K
19.8M
3.4G
Random Pruning
5.62
0,926
100K
Magnitude Pruning
5.48
0,929
100K
Taylor Pruning
5.56
0.928
100K
Ours (T 0..00)
5.49
0.932
100K
Ours (T 0.02)
19.8M
3.4G
5.44
0,931
100K
Ours (T = 0.05)
5.29
0.932
100K
CelebA-HQ 64
64 MACs
(100
DDIM steps)
Method
#Params
FID
SSIM
Train Steps
Pretrained
78.7M
23.9G
6.48
1.000
500K
Seratch Training
7.08
0.833
100K
Scratch Training
6.73
0.867
300K
Scratch Training
6.71
0.869
500K
43.7M
13.3G
Random Pruning
6.70
0,874
100K
Magnitude Pruning
7.08
0.870
100K
Taylor Pruning
6.64
0,880
100K
Ours (T 0..00)
6.24
0.885
100K
Ours (T = 0.02)
43.7M
13.3G
6.45
0.878
100K
Ours (T = 0.05)
6.52
0.878
100K
Diffusion pruning on CIFAR-I0 and CelebA. We leverage Frechet Inception Distance (FID) and Structural Similarity (SSIM) estimate the quality and similarity of generated samples under the same random seed. A larger SSIM score means more consistent generation.
Table 1:
instance, while Magnitude achieves the best FID performance among the three on CIFAR-10, performs poorly CelebA datasets. In contrast, our method delivers stable improvements baseline methods, demonstrating superior performance on both datasets. Remarkably, our method even surpasses the pre-trained model on CelebA-HQ, with only 100K optimizations. Nonetheless, performance degradation is observed on CIFAR-10, which can be attributed to its more complex scene and larger number of categories.
5.3 Pruning at Higher Resolutions
DDPMs on LSUN To further validate the efficiency and effectiveness of our proposed Diff- Pruning, we perform pruning experiments on two 256x256 scene datasets, LSUN Church, and LSUN Bedroom [57]. The pre-trained models from [88 require around 2. 2.4M and 4.4M training steps, which can quite time-consuming in practice. We demonstrate that Diff-Pruning can compress these pre-existing models using only 10% of the standard training resources. We report the number of parameters, MACs, and FID scores Table 2 and compare the pruned methods to the pre-trained well as those trained from scratch. Results show that the pruned model converges with passable FID score in 10% of the standard steps, while a model trained from scratch still severely under-fitted. Nevertheless, such as LSUN Bedroom, which contains 300K images, proves to be quite challenging with a very limited number of training steps. We show that, in the supplementary materials, the FID scores be further improved with more training steps. Moreover, we also visualize the generated images in Figure and report the single-image SSIM score to measure the similarity of generated images. By nature, the pruned models can preserve similar generation capabilities since they inherit most parameters from the pre-trained models.
also discover that compressing model trained on large-scale datasets,
Conditional LDMs on ImageNet Table and Figure illustrate the pruning results of L.DM pre-trained on ImageNet-IK. IK. An LDM consists of an encoder, a decoder, and a U-Net model. Around
7

---

0.814
SSIM=
SSIM 0.817
SSIM = 0.792
SSIM TAM=00 0.821
SSIM = 0.952
=
=
SSIM 0.907
SSIM=
SSIM 0.907
SSIM = 0.881
Figure 2: Generated images of the pre-trained models (18 (left) and the pruned models (right) on
LSUN Church and LSUN Bedroom. SSIM measures the similarity between generated images.
LSUN-Church 256 x 256 (DDIM 100
Steps)
LSUN-Bedroom 256 x 256 (DDIM 100 Steps)
Method
#Params
MACs
FID
Steps
Method
#Params
MACs
FID
Steps
Pretrained
113.7M
248.7G
10.6
4.4M
Pretrained
113.7M
248.7G
6.9
2.4M
Scratch Training
63.2M
138.8G
40.2
0.5M
Scratch Training
63.2M
138.8G
50.3
0.2M
Ours (T 0.01)
63.2M
138.8G
13,9
0.5M
Ours (T 0.01)
63.2M
138.8G
18.6
0.2M
Table 2: Pruning diffusion models on LSUN Church and LSUN Bedroom.
400M parameters come from the U-Net architecture and only 55M from the autoencoder. Therefore, we mainly focus on the pruning the U-Net model. We used the threshold T = 0.1 to ignore those converged layers and make the pruming process more efficient. With T 0.1, only 534 steps participate the pruning process. After importance estimation, we apply pre-defined channel sparsity of 30% to all layers, leading to lightweight U-Net with 189.4 499.43M parameters. Finally, finetune the pruned model for only epochs with the official training script, with a scaled learning rate of 0.1 braeee
5.4 Ablation Study
Pruned Timesteps. First, we conduct an empirical study evaluating the partial Taylor expansion over pruned timesteps. This approach prioritizes steps with larger gradients and strives to preserve as much content and detail possible, thereby enabling more accurate and efficient pruning. The impacts of timestep pruning are demonstrated in Figure We seek to prune pre-trained diffusion model over a range of steps, spanning from 50 1000, after which we utilize the SSIM metric to gauge the distortion induced by pruning. In diffusion models, earlier steps (1 - 0) usually present larger gradients compared the later ones (t ~ T) 41. This inherently leads gradients that have reached convergence when & is large. In the CIFAR- 10 dataset, we find that the optimal SSIM score

---

Figure Images sampled from the pruned conditional LDM on ImageNet- K-256 1-26
Method
#Params +
MACs +
FTD +
IS f
Train Steps +
Pretrained LDM
400.92M
99.80G
3.60
247.67
2000K
Scratch Training
51.45
25.69
100K
Taylor Pruning
189.43M
52.71G
11.18
138.97
100K
Ours (T 0.1)
9.16
201.81
100K
Table 3: Compressing conditional Latent Diffusion Models on ImageNet- IK (256 x 256)
Pruning Ratios
Thresholding
Ratio
#Params
MACs
FID
SSIM 1
Threshold
Steps
FID 1
SSIM t
0%
35.7M
6.1G
4.19
1.000
T 0.00
1000
5.49
0.932
16%
27.5M
5.1G
4.62
0.942
0.01
707
5.41
0.932
44%
19.8M
3.4G
5.29
0,932
T = 0.02
433
5.44
0.931
56%
14.3M
2.7G
6.36
0.922
T = 0.05
244
5.29
0.932
70%
8.6M
1.5G
9.33
0.909
T = 0.10
127
5.31
0.931
Table 4: Pruning with different ratios
Table 5: Pruning with different threshold T
can attained at around 250 steps, and adding more steps can slightly deteriorate the quality of the synthetic images. This primarily stems from the inaccuracy of the first-order Taylor expansion at converged points, where the gradient no longer provides useful information and can even distort informative gradients through accumulation. However, observe that the situation differs slightly with the CelebA dataset, where more steps can be beneficial for importance estimation.
Pruning Ratios. Table presents the #Params, MACs, FID, and SSIM scores of models subjected to various pruning ratios based on MACs. Notably, our findings reveal that, unlike CNNs employed in discriminative models, diffusion models exhibit significant sensitivity to changes in model size. Even modest pruning ratio of 16% leads noticeable degradation in FID score (4.19 4.62). In classification tasks, a perturbation loss does not necessarily impact the final accuracy; it may only undermine prediction confidence while leaving classification accuracy unaffected. However, in generative models, the FID score is very sensitive, making more susceptible to domain shift.
Thresholding. In addition, we conducted experiments to investigate the impact of the thresholding parameter T. Setting T = O corresponds to full Taylor expansion at all steps, while T > 0 denotes pruning of certain timesteps during importance estimation. The quantitative findings presented in Table 5) align with the SSIM results depicted in Figure [5] Notably. Diff--runins attains optimal performance when the quality of generated images reaches its peak. For datasets such as CIFAR-I0. 10, we observed that a 200-step Taylor expansion is sufficient to achieve satisfactory results. Besides, using a full Taylor expansion, this case, can detrimental, as accumulates noisy gradients over approximately 700 steps, which obscures the correct gradient information from earlier steps.

---

w/o Pruming SSIM 1,000
Random
Magnitude SSIM 0.391
Taylor SSIM = 0.758
Ours (T)))) SSIM = 0,857
SSIM 0.905
Figure 4: Generated images of 5%-pruned models using different important criteria. We report the SSIM of batched images without post-training.
CIFAR-10
ColoA HQ
0.84
1.0
Relative Loss
1.0
Relatwe Loss
SSIM
0.82
SSIM
0.7
SSIM
Relalive
200
400
600
800
1000
200
400
600
800
1000
Steps
Steps
Figure 5: The SSIM of models pruned with different numbers of timesteps. For CIFAR-10, most of the late timesteps can be pruned safely. For CelebA-HQ, using more steps is consistently beneficial.
Visualization of Different Importance Criteria. Figure f visualizes the images generated by pruned models using different pruning criteria, including the proposed method with timestep pruning) and T > 0. The SSIM scores of the generated samples are reported for quantitative comparison. The Diff- fuff-Pruning method with > 0 achieves superior visual quality, with an SSIM score of 0.905 after pruning. It is observed that employing more timesteps in our method could have a negative impact, leading to greater distortion in both textures and contents.
(w/o
Conclusion
This work introduces biff-Pruning, a dedicated method for compressing diffusion models. utilizes Taylor expansion pruned timesteps identify and remove non-critical parameters. The proposed approach is capable crafting lightweight yet consistent models from pre-trained ones, incurring only about 10% to 20% of the cost compared to pre-training. This work may set an initial baseline for future research that aims at improving both the generation quality and the consistency of pruned diffusion models.
Acknowledgment
This research is supported by the Ministry of Education, Singapore, under its Academic Research Fund Tier 2 (Award Number: 22002222222122000000
10

---

References
[1] Tomer Amit, Tal Shaharbany, Eliya Nachmani, and Lior Wolf. Segdiff: Image segmentation with diffusion prohabilistic models. arXiv preprint 1rXiv:21/2.003000 2021.
[2] Shoufa Chen, Peize Sun, Yibing Song, and Ping Luo. Diffusiondet: Diffusion model for object detection. arXiv preprint arXiv:2211.07788 2022.
[3] Tianyi Chen, Luming Liang, Tianyu Ding, and Ilya Zharkov. Towards automatic neural architecture search within general super-networks. arXiv preprint arXiv:2305. 18030, 2023.
[4] Tianyi Chen, Luming Liang. Tianyu Ding, Zhihui Zhu, and Ilya Zharkov. Otov2: Automatic, generic, user-friendly. arXiv preprint arXiv:2303.066. 0666622..662.... 2023.
[5] Ting- Wu Chin, Ruizhou Ding, Cha Zhang, and Diana Marculescu. Towards efficient model compression via learned global ranking. In Proceedings of the IEEE/CVF conference on computer vision and pattem recognition, pages 1518-1522 2020.
[6] Xiaohan Ding. Guiguang Ding, Yuchen Guo, and Jungong Han. Centripetal sgd for pruning very deep convolutional networks with complicated structure. In Proceedings of the IEEE/CVF conference on computer vision and pattem recognition, pages 1943-4955 2019.
[7] Xin Dong, Shangyu Chen, and Sinno Pan. Learning to prune deep neural networks via layer-wise optimal brain surgoon. Advances in Neural Information Processing Systems, 30, 2017.
[8] Gongfan Fang, Xinyin Ma, Mingli Song, Michael Bi Mi, and Xinehao Wang. Dopgraph: Towards any structural pruning. The IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023.
[9] Gerald B Folland. Higher-order derivatives and taylor's formula in several variables. Preprint, pages 1-4, 2005.
[10] Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. arXiv preprini arXiv:1803.000 880.00335, 2018
[11] Tiankai Hang, Shuyang Gu, Chen Li, Jianmin Bao, Dong Chen, Han Hu, Xin Geng, and Baining Guo. Efficient diffusion training via min-snr weighting strategy. arXiv preprint arXiv:2303.09556 2023.
[12] En hao Liu, Xuefei Ning, Zi-Han Lin, Huazhong Yang, and Yu Wang. Oms-dpm: Optimizing the model schedule for diffuusion probabilistic models. International Conference on Machine Learning. 2023.
[13] Wei He, Meiqing Wu, Mingfu Liang. and Siew-Kei Lam. Cap: Context-aware pruning for semantic segmentation. In Proceedings of the IEEE/CVF winter conference on applications of computer vision, pages 960- 969, 2021.
[14] Yang He. Ping Liu, Ziwei Wang. Zhilan Hu, and Yi Yang. Filter pruning via geometric median for deep convolutional neural networks acceleration. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 4340-4349, 2019.
[15] Yihui Hc, Ji Lin, Zhijian Liu, Hanrui Wang, Li-Jia Li, and Song Han. Amc: Automl for model compression and acceleration on mobile devices. In Proceedings of the European conference computer vision (ECCV), pages 784-80 844-800,22 2018.
[16] Yihui He, Xiangyu Zhang, and Jian Sun. Channel pruning for accelerating very deep neural networks. In Proceedings of the IEFE international conference computer vision, pages 1389-1397, 2017
[17] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by two time-scale update rule converge to a local nash cquilibrium. Advances in neural information processing systems, 30, 2017.
[18] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 33:684006851, 2020.
2020.
[19] Bowen Jing, Gabriele Corso, Renato Berlinghieri, and Tommi Jaakkola. Subspace diffusion generative models. Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXIII, pages 274-289. Springer, 2022.
[20] Yongcheng Jing. Efficient Representation Learning With Graph Neural Networks. PhD thesis, 2023.
[21] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
[22] Namhoon Lee, Thalaiyasingam Ajanthan, Stephen Gould, and Philip HS Torr A signal propagation perspective for pruning neural networks at initialization. arXiv preprint urXiv:1906.066 063077 2019.
[23] Hao Li, Asim Kadav, Igor Durdanovic, Hanan Samet, and Hans Peter Graf. Pruning filters for efficient convnets. arXiv preprint arXiv: 08710, 2016.
[24] Muyang Li, Ji Lin, Yaoyao Ding. Zhijian Liu, Jun-Yan Yan Zhu, and Song Han. Gan compression: Efficient architectures for interactive conditional gans. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages
2020.

---

[25] Daochang Liu, Qiyue Li, Anh-Dung Dinh, Tingting Jiang, Mubarak Shah, and Chang Xu. Diffusion action segmentation. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 10139-10149, 2023.
[26] Liyang Liu, Shilong Zhang, Zhanghui Kuang, Aojun Zhou, Jing-Hao Xue, Xinjiang Wang, Yimin Chen, Wenming Yang. Qingmin Liao, and Wayne Zhang. Group fisher pruning for practical network compression. In International Conference on Machine Learning, pages 7021-7032. PMLR, 2021.
[27] Luping Liu, Yi Ren, Zhijie Lin, and Zhou Zhao. Pseudo numerical methods for diffusion models on manifolds. arXiv preprint (XXi:2222.009722 2022.
[28] Shiwei Liu, Tianlong Chen, Xiaohan Chen, Li Shen, Decebal C Mocanu, Zhangyang Wang, and Mykola Pechenizkiy. The unrcasonable effectiveness of random pruning: Return of the most naive baseline for sparse training. In International Conference on Learning Representations, ICLR 2022, 2022.
[29] Songhua Liu, Kai Wang. Xingyi Yang. Jingwen Ye, and Xinchao Wang. Dataset distillation via factorization.
In Advances in Neural Information Processing Systems, 2022.
[30] Songhua Liu, Jingwen Ye. Runpeng Yu, and Xinchao Wang. Slimmable dataset condensation. In JEEE/CVE Conference on Computer Vision and Pattern Recognition, 2023.
[31] Zhuang Liu, Jianguo Li, Zhiqiang Shen, Gao Huang, Shoumeng Yan, and Changshui Zhang. Learning efficient convolutional networks through network slimming. In Proceedings of the IEEE international conference on computer vision, pages 2736-2744, 2017.
[32] Ziwei Liu,, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Large-scale celebfaces attributes (celeba) dataset. Retrieved August, 15(2018):11, 2018.
[33] Ja Luo, Jianxin Wu, and Weiyao Lin. Thinet: A filter level pruning method for deep neural network compression. In Proceedings of the IEEE international conference on computer vision, pages 5058-500 5066, 2017.
[34] Zhaoyang Lyu, Xudong Xu, Ceyuan Yang, Dahua Lin, and Bo Dai. Accelerating diffusion models via early stop of the diffusion process. arXiv preprint arXiv:2205. 12522 2022.
[35] Pavlo Molchanov, Arun Mallya, Stephen Tyree luri Frosio, and Jan Kautz. Importance estimation for neural network pruning. In Proceedings of the IEEE/CVF conference on computer vision and patter recognition, pages 1 11264-112777 11272, 2019.
[36] Pavlo Molchanov, Stephen Tyree, Tero Karras, Timo Aila, and Jan Kautz. Pruning convolutional neural networks for resource efficient inference. arXiv preprint arXiv:1611.060 1.04440, 2016.
[37] Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffussion probabilistic models. In International Conference on Machine Learning, pages 8162-8171. PMLR, 2021.
[38] Sejun Park. Jaeho Lee, Sangwoo Mo, and Jinwoo Shin. Lookahead: a far-sighted alternative of magnitude- based pruning. arXiv preprint arXiv:2002. 04809, 2020.
[39] Hao Phung, Quan Dao, and Anh Tran. Wavelet diffusion models are fast and scalable image generators. arXiv preprint arXiv:2211.1615: 2022.
[40] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.0661 06125, 2022.
[41] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. Highhressoution image synthesis with latent diffuusonn models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10684-10695, 2022.
[42] Olaf Ronneberger. Philipp Fischer, and Thomas Brox U-net: Convolutional networks for biomedical image segmentation. In Medical Image Computing and Computer-Assiee Assisteed Citrrentiio-MMICCC 2015: 18th International Conference, Munich, Germany, October 5-9, 2015, Proceedings. Part III 18, pages 234-241 Springer, 2015.
[43] Tim Salimans and Jonathan Ho. Progressive distillation for fast sampling of diffusion models. arXiv preprint arXiv:2202. 00512,022222 2022.
[44] Victor Sanh, Thomas Wolf, and Alexander Rush. Movement pruning: Adaptive sparsity by fine-tuning. Advances in Neural Information Processing Systems, 33:20378 20389, 2020.
[45] Hiroshi Sasaki, Chris G Willcocks, and Toby P Breckon. Unit-ddpm: Unpaired image translation with denoising diffusion probabilistic models. arXiv preprint arXiv:2104.05358, 2021.
[46] Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models, arXiv preprint arXiv:2010.0 02502, 2020.
[47] Duc Minh Vo, Akihiro Sugimoto, and Hideki Nakayama. Pped-gan: Progressive pruning and class- aware distillation for large-scale conditional gans compression. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 2436 2444, 2022.
12

---

[48] Patrick von Platen. Suraj Patil, Anton Lozhkov. Pedro Cuenca, Nathan Lambert, Kashif Rasul, Mishig Davaadorj. and Thomas Wolf. Diffusers: State-of-the-art diffusion models. lnttps://github. com/
huggingfacee/diffuerss 2022.
[49]
Zhendong Wang. Yifan Jiang, Huangjie Zheng, Peihao Wang. Pengcheng He, Zhangyang Wang. Weizhu Chen, and Mingyuan Zhou. Patch diffusion: Faster and more data-efficient training of diffusion models. arXiv preprint arXiv:2304. 12526, 2023.
[50] Zhou Wang, A.C. Bovik, H.R. Sheikh, and E.P. Simoncelli. Image quality assessment: from error visibility to structural similarity. IEEE Transactions Image Processing, 13(4):600-6) 612, 2004.
[51] Xingyi Yang. Jingwen Yc, and Xinchao Wang. Factorizing knowledge in neural networks. In European Conference on Computer Vision, 2022.
[52] Xingyi Yang, Daquan Zhou, Jiashi Feng, and Xinchao Wang. Diffusion probahilistic model made slim. In IEEF/CVF Conference on Computer Vision and Pattern Recognition, 2023.
[53] Xingyi Yang, Daquan Zhou, Songhua Liu, Jingwen Yc, and Xinchao Wang. Deep model reassembly. In Advances in Neural Information Processing Systems, 2022.
[54] Lewei Yao, Renjie Pi, Hang Xu, Wei Zhang. Zhenguo Li, and Tong Zhang. Joint-detnas: upgrade your detector with nas, pruning and dynamic distillation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10175-10181 2021.
[55] Jianbo Ye, Xin Lu, Zhe Lin, and James Z Wang. Rethinking the smaller mmaller-norm-iaae norm- wwwm--ess-infommmattee assumption in channel pruning of convolution layers, arXiv preprint urXiv:1802.00124 2018.
[56] Zhonghui You, Kun Yan, Jinmian Ye, Meng Ma, and Ping Wang. Gate decorator: Global filter pruning method for accelerating deep convolutional neural networks. Advances in neural information processing systems, 32, 2019.
[57] Fisher Yu, Ari Seff, Yinda Zhang, Shuran Song. Thomas Funkhouser, and Jianxiong Xiao. Lsun: Con- struction a large-scale image dataset using deep learning with humans in the loop. arXiv preprint arXiv: 1506.033666,6 2015.
[58] Lvmin Zhang and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. arXiv preprint arXiv:2302.08 0.554.55.,,,.... 2023.
[59] Xunyu 7hu, Jian Li, Yong Liu, Can Ma, and Weiping Wang. A survey model compression for large language models. arXiv preprint arXiv:2308.0744 076333, 2023.
13


Reviews:
[{"review_id": "00L41fkFGu", "reviewer": "Reviewer_S8k8", "summary": "The paper proposed a method to prune diffusion models to achieve 50% FLOPS reduction at 10% to 20% of original training budget. The authors show that prune and fine-tune strategy based on the random selection, magnitude based selection, Taylor expansion based selection all lead to non-ideal / suboptimal performance. They develops a method based on the modification of Taylor expansion based selection and show it achieves good performance-quality tradeoff. They quantitatively evaluated on standard datasets used for diffusion models at small and medium resolution (from 32^2 to 256^2). ", "questions": "I don't have other questions", "limitations": "I have a feeling that authors tried very hard to prove their technique working to some extent. It would be helpful as a research publication if the authors can present more results telling when and where their method fails or explore more on the boundary of quality and performance trade-offs. ", "rating": 4, "confidence": 2, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "The paper works on a popular task in recent literature of generative models and presented a practice to prune diffusion models while preserves its generative capability. They demonstrated empirical results on 256^2 resolution unconditional image generation, which seems promising.  ", "weaknesses": "The lack of objective quantitative metrics in image generation literature makes the evaluation of pruned model very difficult. In addition to FID score, the authors also use SSIM score to compare the image generated from the full model and the pruned model under the same random seed is used. However, this is still hard for me to evaluate the quality of empirical results given limited examples presented in the paper. It would be otherwise more informative, if the authors can demonstrate their method being valid for conditional diffusion models, or even latent diffusion models. \n\n\nThe technical contribution is also limited, as the method authors used already exist in the literature but has not been adopted for diffusion models. \n\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> **Q1:** The lack of objective quantitative metrics, and the effectiveness on conditional diffusion models like LDMs.\n\n**A1:** Thanks for your valuable comments. In the table below, we provide the results of pruning Conditional Latent Diffusion Models on ImageNet-1K (256$\\times$ 256). We follow the same training protocol as provided in \\cite{rombach2022high} to prune and fine-tune an off-the-shelf LDM. Due to the time limits of this rebuttal period, we only fine-tuned the pruned models for 4 epochs. The generated images can be found in the attached pdf file. \n\n|  Method         |  #Params $\\downarrow$ | MACs $\\downarrow$   |  FID $\\downarrow$ |  IS $\\uparrow$ |  Train Steps $\\downarrow$ |\n|--------------------------|---------------------------|-------------------------|----------------------|-------------------|------------------------------|\n| Pretrained LDM           | 400.92M                   | 99.80G                  | 3.60                 | 247.67            | 2000K                        |\n| Scratch Training         | 189.43M  | 52.71G | 51.45                | 25.69             | 100K                         |\n| Taylor Pruning           |       189.43M                      |       52.71G                   | 11.18                | 138.97            | 100K                         |\n| Ours ($\\mathcal{T}=0.1$) |         189.43M                    |         52.71G                 | 9.16                 | 201.81            | 100K                         |\n\n**Details of LDM Pruning**\nHere we also provide the details of LDM pruning: LDM consists of an encoder, a decoder, and a UNet diffusion model. Around 400M parameters come from the UNet architecture and only 55M from the autoencoder. So we only prune the UNet model for acceleration. Moreover, we are dealing with a **conditional** model. During importance estimation, we randomly sample classes and images to accumulate gradients for Taylor expansion. We used the threshold $\\mathcal{T}=0.1$ to ignore those vanished gradients from large timesteps, which also makes the pruning process more efficient. With $T=0.1$, only 534 steps participate in the pruning process. After importance estimation, we apply a pre-defined channel sparsity of 30\\% to all layers, leading to a lightweight UNet with 189.43M parameters. Subsequent fine-tuning follows official training scripts, using a scaled learning rate of $0.1\\times lr_{base}$.  After fine-tuning, we sample 50 images per class and report the FID and IS score of the pruned models. Besides, We only report the #Params and MACs of the UNet. Similar to the hyper-parameter tables in the appendix, we outline our training configurations as follows:\n\n|  Dataset | Img Size |  Pruning Ratio (MACs)    |  Lr  |  Batch |  $\\text{Step}_f/\\text{Step}_p$ |  weight decay |\n|-------------|----------|--------------------------------------|---------|-----------|-----------------------------------|--------|\n|    ImageNet-1K    | 256      | 47.2\\% (99.80G $\\rightarrow$ 52.71G) | 1.28e-5 | 64        | 5\\% (100K/2000K)                  | 0      |\n\n---\n\n> **Q2:** The technical contribution is also limited, as the method authors used already exists in the literature but has not been adopted for diffusion models.\n\nA2: Thanks for the comments. Pruning diffusion models is quite a new topic and there is a large design space to explore, such as importance estimation, fine-tuning, and pruning strategies. This work indeed adopts some popular methods like Magnitude Pruning and Taylor Pruning to diffusion models.  However, as illustrated in Table 1, they do not show significant improvements even compared to random pruning. As an initial exploration, we aim to make the method as simple and practical as possible. This also brings the benefit that many existing techniques like second-order approximation can be further leveraged to boost the performance of pruning. We will follow your suggestion to make it a more \"diffusion-style\" work.\n\n---\n\n> **Q3:** It would be helpful as a research publication if the authors can present more results telling when and where their method fails or explore more on the boundary of quality and performance trade-offs.\n\n**A3:** Thanks for the comment. We provided some failure cases in Fig. 4 of the appendix and also discussed some limitations in the controllability, such as the missing watermark in Line 252. In other words, diff-pruning only aims to minimize image distortion but does not know what kind of information is important in the images. We will provide a limitation section in the revised version. And for the boundary of quality and performance trade-offs, Table 3 provides some insights into the trade-off between performance and efficiency, where we observed prominent performance degradation with high pruning ratios. However, the pruning of diffusion models as well as generative models is till a new topic. I agree that more explorations on the boundary of quality and performance will be valuable.\n"}, {"review_id": "GwKes1BkCK", "reviewer": "Reviewer_PXRJ", "summary": "This paper presents Diff-Pruning, a novel structural compression technique for learning efficient diffusion models from pre-trained ones. The fundamental idea behind Diff-Pruning is the utilization of Taylor expansion on pruned timesteps, which effectively combines informative and clean gradients to estimate the importance of weights. The authors demonstrate that Diff-Pruning not only achieves compression of pre-trained models within a few training iterations but also maintains the original generation capabilities intact.", "questions": "Please refer to the weaknesses above.", "limitations": "This work has no negative societal impact.", "rating": 5, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "- The experimental results demonstrate that structural pruning can serve as a powerful and efficient compressor for diffusion models. An important advantage is that the pruned model inherently retains the generative behavior of the pre-trained model. Considering the time-consuming nature of training new diffusion models, this work proves valuable for various downstream applications.\n- The proposed method, which employs the Taylor expansion over pruned timesteps, is both well-motivated and practical. The idea of balancing the contribution of different stages with binary weighting is interesting and easy to practice.\n- The authors provide extensive experimental evidence that substantiates their claims regarding the efficiency and consistency of Diff-Pruning.", "weaknesses": "- Extra comparison of the memory requirement for different pruning methods might provide a more complete assessment of the proposed methods.\n- The experiments reveal that structural pruning can potentially have a negative impact on the performance of pre-trained models. This observation deviates slightly from the results seen in discriminative tasks such as classification, where lossless compression can be achieved for certain networks like ResNets. It would be beneficial for the authors to provide further clarification on this phenomenon", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "\n\n> **Q1:** Extra comparison of the memory requirement.   \n\n**A1:** Thanks for the suggestion. Methods like Random Pruning and Magnitude Pruning do not require additional memory during pruning. Taylor pruning and the proposed diff-pruning require $O(N)$ space to store the gradient, where $N$ is the number of parameters. In addition, the memory consumption of pruned models, as well as their training and inference FPS (Frames Per Seconds), can be found in the table below. We tested a pruned DDPM and a conditional LDM (Please refer to the Q1 of Reviewer S8k8) on 256$\\times$256 images, i.e., LSUN-Church and ImageNet-1K, with a single NVIDIA RTX A5000. All experiments were repeated for 30 times and the average results are reported.\n\n| Method             |  \\#Params |  MACs | Inference Mem.  | Training Mem. | Inference FPS  | Training FPS  |\n|------------------------|---------------------------|-----------------------|--------------------------------|-----------------------------|------------------------------|--------------------------|\n| Pretrained LDM         | 400.92M                   | 99.80G                | 11.03GB                        | 14.64GB                     | 12.87                        | 4.26                     |\n| Pruned LDM             | 189.43M                   | 52.71G                | 9.58GB                         | 11.95GB                     | 19.83                        | 6.37                     |\n| Pretrained DDPM        | 113.7M                    | 248.7G                | 3.35GB                         | 5.59GB                      | 28.66                        | 11.02                    |\n| Pruned DDPM            | 46.5M                     | 100.7G                | 2.43GB                         | 4.13GB                      | 32.92                        | 9.07                     |\n\n--- \n\n> **Q2:** The negative impact on the performance of pre-trained models. Why Lossless compression can not be achieved.\n\n**A2:** Thank you for your insightful comments. We propose two factors that may cause this phenomenon:\n\n* Model Capacity: The model capacity required for generative models is typically larger compared to discriminative models [1,2,3]. In discriminative tasks, images are often downsampled and filtered hierarchically, resulting in less information encoded within the network. Conversely, generative tasks face challenges in training small networks while achieving high generation quality, which causes performance lost during pruning.\n\n* Sensitivity of Metrics: Classification metrics like accuracy are generally not very sensitive to slight distortion in model predictions, as long as the samples maintain their correct positioning in relation to the decision boundary. In other words, Accuracy is discrete. However, for diffusion models, we use FID, a continuously changed value, to evaluate the performance of models. Pruning will introduce distortions in the generated images, which are immediately reflected in the FID score.\n\n[1] Kang, Minguk, et al. \"Scaling up gans for text-to-image synthesis.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.   \n[2] Brock, Andrew, Jeff Donahue, and Karen Simonyan. \"Large scale GAN training for high fidelity natural image synthesis.\" arXiv preprint arXiv:1809.11096 (2018).    \n[3] Rombach, Robin, et al. \"High-resolution image synthesis with latent diffusion models.\" Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2022.\n"}, {"review_id": "kGm7UjZGd1", "reviewer": "Reviewer_i6fA", "summary": "The paper introduces a pruning-based method aimed at addressing the efficiency challenges of diffusion models. Unlike existing approaches that focus on accelerating sampling or enhancing architectures, it specifically focus on the time cost of compression and the consistency in generated images. The authors propose a novel variant of Taylor approximation as the importance function for pruning, which effectively preserves both high-level content and low-level details of generated images, while minimizing the impact of noisy steps. Experimental results demonstrate that the compressed diffusion model successfully generates similar images to the pre-trained model.", "questions": "It would be valuable for the author to include discussions regarding the controllability of the observed distortions or changes, as mentioned in the weaknesses.", "limitations": "N/A", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "I agree that efficiency and consistency are indeed crucial considerations in compressing diffusion models, which have received limited attention in prior works. This work contributes to this field by building several initial experiments on datasets such as Cifar, CelebA, and LSUN, effectively showcasing the advantages of the method. This work establishes a solid baseline for future explorations into efficient diffusion models. Moreover, the impressive ability of the pruned model to generate consistent results is particularly noteworthy, which is user-friendly in deployment.", "weaknesses": "It is observed that in certain cases, the generated images may still exhibit some distortions or changes, such as the watermarks shown in Figure 2. It is worth noting that some of these elements may contain important information that should ideally be preserved. To address this, it would be beneficial to explore methods that allow for controllable preservation of such information while pruning the models. This ability would provide the flexibility to focus on specific parts of the generated images based on different scenarios. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> **Q1:** the generated images may still exhibit some distortions or changes, such as the watermarks shown in Figure 2. A discussion about controllable preservation of such information. it would be beneficial to explore methods that allow for controllable preservation of such information while pruning the models. This ability would provide the flexibility to focus on specific parts of the generated images based on different scenarios.\n\n**A1:** Thanks for your valuable comments. This work focuses on minimizing distortion at the pixel level and does not inherently offer direct controllability at a high level. Nevertheless, the idea of incorporating such functionality is highly valuable. To achieve this, one solution is to extend our method by utilizing a mask that prioritizes important regions, employing the following equation adapted from Equation 2 in the main paper:\n\n$$\n\\mathcal{L}( {\\theta} ):= \\mathbb{E}_{t, x_0\\sim q(x), \\epsilon\\sim \\mathcal{N}(0,1)} [ || m \\odot ( \\epsilon  -  \\epsilon^{\\prime}(\\sqrt{\\bar{\\alpha}_t}x_0 + \\sqrt{1-\\bar{\\alpha}_t}\\epsilon, t) ) ||^2 ]\n$$\n\nwhere $m$ is a binary mask in the image space and $\\odot$ is an element-wise multiplication. Here we use $\\epsilon^{\\prime}$ instead of \"\\epsilon_{\\theta}\" because there is a display bug in OpenReview. This method allows the pruner to perverse those highlighted regions like the watermarks. However, it is important to note that this improvement relies on the availability of additional annotations. To address this limitation, we can explore the possibility of decoupling objects in images and achieving a balanced quality for each object. A naive way is to segment the images with general models like SAM [1]. We intend to include a discussion section in the appendix that delves into the controllability of image quality and its implications.\n\n[1] Kirillov, Alexander, et al. \"Segment anything.\" arXiv preprint arXiv:2304.02643 (2023)."}, {"review_id": "1RGqa5nceJ", "reviewer": "Reviewer_5jdL", "summary": "This work introduces a structural pruning method for diffusion models, called Diff-pruning. The authors leverage Taylor expansion on each term of the ELBO loss as the creteria to decide the pruned weights. By calculating the error of the final image induced by the pruning at timestep t and discussing about the effect of converged loss on the higher order terms of the Taylor expansion, this work manually drop the pruning creteria of timesteps near noise by setting a threshold $\\mathcal{T}$ related to the relative loss. Experiments compare the performance of the proposed pruning method with other common pruning methods, as well as training from scratch.", "questions": "1. How is eq.9 derived and why the choice of $\\alpha_t$ according to this equation is opposite to the proposed method (as described in Weaknesses)?", "limitations": "The authors do not include the limitations and potential negative societal impact in the paper.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. This paper proposes a novel pruning method that utilizes the multi-step property of the diffusion model.\n2. The experiment results demonstrate the effectiveness of the proposed method compared with baseline methods. ", "weaknesses": "1. My major concern is the motivation of the method is not clearly explained: \n  - The authors conclude an important conclusion from eq.9. However, the derivation process of eq.9 has mistakes and typos. There is a neural inference term with $x_t$ as input in eq.3, the final error $\\delta_0$ can not be derived in such a simple form. Moreover, the meaning of the symbol \"$\\delta$\" is ambiguous. It seems to represent noise prediction error in line 146, but $\\delta_{t-1}$ and $\\delta_0$ seems to represent the error of $x_t$ and $x_0$ caused by pruning. And the subscript $s$ of $\\alpha_s$ is wrongly written in eq.8. \n  - How could Equ. 9 derive the conclusion \"prediction errors occurring at larger t primarily impact the high-level content of generated images, while smaller t values concentrate on refining the images with relatively small modifications\"? Relevant paper presents that the main difference between high-level contents and details is that they come from different frequency components of the whole image [1] , while Equ.9 focus on the \"amplitude of the error $\\delta_0$\". It requires further clarification. \n  - The authors claim that the final distortion is progressively magnified according to eq.9, which indicates that the error at timesteps near the noise has a greater impact on the final image. However, the proposed method truncates the pruning criteria at large t, which does not match such observation.\n\n2. Lack of detailed description of the baseline method ToMe.\n\n[1] Diffusion probabilistic model made slim. arXiv preprint arXiv:2211.17106, 2022.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> **Q1:** The motivation of the method is not clearly explained.  \n\n**A1:** In essence, we harness the informative gradients stemming from early timesteps ($t\\rightarrow 0$) to identify unimportant parameters for the pruning process. As demonstrated in our experiments, we disregard larger timesteps ($t\\rightarrow T$) due to the presence of vanished gradients, rendering them incapable of furnishing valuable insights for importance assessment.\n\n--- \n\n> **Q2:** The derivation process of eq.9 has mistakes and typos. (a) There is a neural inference term with $x_t$ as input in eq.3, the final error $\\delta_0$ can not be derived in such a simple form. (b) Moreover, the meaning of the symbol \"$\\delta$\" is ambiguous. It seems to represent noise prediction error in line 146, but $\\delta_{t-1}$ and $\\delta_0$ seems to represent the error of $x_t$ and $x_0$ caused by pruning. (3) And the subscript $s$ of $\\alpha_s$ is wrongly written in eq.8.\n\n**A2:** I apologize for any confusion that may have arisen. In Equation 9, $\\delta_t=\\epsilon_{\\theta^\\prime }(x, t) - \\epsilon_{\\theta}({x}, t)$ (as defined in Line 146) denotes the error that emerges under the same input. On the other hand, $\\delta_0$ refers to the distortion resulting from the pruning error $\\delta_t$, which **does not include the error associated with input shift** due to the difficulty in analyzing the non-linearity of network forwarding. In order to enhance clarity, we would like to improve the notation by substituting $\\delta_0$ with $\\delta_{0\\leftarrow t}$. The subsequent discussion employs this refined notation for clarification.\n\n* (a) Indeed, there's a network inference $\\epsilon_\\theta (x_{t-1} + \\delta_{t-1}, t-1)$ in different timessteps. At Line 147, we assume \"no additional prediction error emerges in other steps,\" indicating consistent inference across timesteps without introducing new pruning errors. Besides, we don't concentrate on changes caused by shifted inputs as they can't reflect the **functional shifts** due to pruning. Instead, we are interested in the pruning error as defined above, that is $\\delta_{t-1}=\\epsilon_{\\theta'}(x_{t-1} + \\delta_{t-1}, t-1) - \\epsilon_{\\theta}(x_{t-1} + \\delta_{t-1}, t-1)$ . In this case, a simple functional error can be linearly derived as $\\delta_{0\\leftarrow t}$ in Eq. 9. However, there are still error terms caused by the input shift during inference, i.e., $\\epsilon_{\\theta}(x_{t-1} + \\delta_{t-1}, t-1) - \\epsilon_{\\theta}(x_{t-1}, t-1)$. Analyzing this is challenging due to non-linearity. We'll revise Eq. 9 per your suggestion to address these concerns.\n\n* (b) Thanks for the comments. As defined in Line 146, $\\delta_t=\\epsilon_{\\theta^\\prime}(x, t) - \\epsilon_\\theta (x, t)$. It refers to the functional error with the same inputs. \n\n* (c) Thanks. The $\\alpha_s$ should be $\\alpha_t$. We will make the necessary correction to fix it.\n \n---\n\n> **Q3:** How could Equ. 9 derive the conclusion \"prediction errors occurring at larger t primarily impact the high-level content of generated images, while smaller t values concentrate on refining the images with relatively small modifications\"?.\n\n**A3:** Thanks for the comment. Due to the scaling factor mentioned in Line 149, \"the final distortion induced by $\\delta_t$ is progressively magnified by a factor of $\\frac{1}{\\sqrt{\\alpha_s}}>1$ along the sampling path\". For a larger $t$, more scaling factors will be applied. Therefore the distortion at a large $t$ has a larger impact ($\\delta_{0\\leftarrow t}$) on the image space than that at a small $t$.\n\n---\n\n> **Q4:** The authors claim that the final distortion is progressively magnified according to eq.9, which indicates that the error at timesteps near the noise has a greater impact on the final image. However, the proposed method truncates the pruning criteria at large t, which does not match such observation.\n\n**A4:** Thanks for the comment. The impact have two main components: the scaling factor $\\frac{\\beta_t}{\\sqrt{\\bar{\\alpha}_t \\cdot (1-\\bar{\\alpha}_t) } }$ and the initial error $\\epsilon_t$ at step $t$. While it might seem reasonable to prioritize focusing on larger steps $t$ as a potential solution, there are certain challenges associated with this approach. As detailed in Line 172-177 and depicted in Figure 4, the gradients tend to vanish rapidly as $t$ approaches $T$, leading to inaccuracies in the applied Taylor expansion for $\\delta_t$. Consequently, it becomes essential to truncate certain steps to prevent the accumulation of unreliable gradients.\n\n---\n\n> **Q5:** Lack of detailed description of the baseline method To Me.\n\n**A5:** Thank you for your valuable feedback. The specific details of the baseline methods can be found in Lines 218-228 of the main paper. We will follow your suggestion to make it an independent paragraph in the revised version."}, {"review_id": "W3dr7LviGy", "reviewer": "Reviewer_oE14", "summary": "This paper presents Diff-Pruning, an efficient compression method for learning lightweight diffusion models from pre-existing ones. DPMs have shown impressive capabilities in generative modeling, but they often come with significant computational overhead during training  and inference. Diff-Pruning addresses this challenge by introducing a  Taylor expansion over pruned timesteps, which eliminates  non-contributory diffusion steps and combines informative gradients to  identify important weights. The authors empirically evaluate the proposed method on four diverse datasets, highlighting two primary  benefits: 1) Efficiency, with a 50% reduction in FLOPs at a fraction of  the original training expenditure, and 2) Consistency, as the pruned  diffusion models retain generative behavior congruent with their pre-trained counterparts.\n", "questions": "see Weaknesses.", "limitations": "The paper briefly mentions limitations but does not provide detailed explanations. Expanding on the limitations section will help readers understand the potential constraints or assumptions of the proposed method and its applicability in different scenarios. Additionally, it is necessary to discuss specific technical shortcomings.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper introduces a novel method, Diff-Pruning,  **specifically designed** for compressing diffusion models. \n- The authors conduct empirical assessments on  four diverse datasets, providing a comprehensive analysis of the  proposed method's performance. The evaluation demonstrates the  effectiveness of Diff-Pruning in terms of efficiency and consistency.\n- The paper appears to be well-written overall.\n", "weaknesses": "- Provide additional details about the pruning process: Although this paper describes the essence of diffo-pruning as Taylor expansions at pruning time steps, providing more specific details about the pruning process itself would enhance the clarity of the proposed method. Providing step-by-step explanations or pseudocode algorithms would help readers understand and replicate this approach.\n\n- In Eq. (4), (5), (6), (10), (11), the symbols $|\\cdot|$, $||\\cdot||$, and $||\\cdot||_0$ appear. Are they representing the same thing? The author needs to carefully check if the symbols in the formulas are correct. Additionally, the notation $\\nabla L_t(\\mathbf{\\theta})(\\mathbf{\\theta}^\\prime - \\theta)$ should be reviewed for its validity since $\\theta$ is a vector. Furthermore, what does the \"$\\cdot$\" symbol represent in Equation (6)? Is it the dot product? If so, why is \"$|\\cdot|$\" included?\n\n- I am unsure how Eq. (6) is derived from Eq. (5). Why can it serve as an importance criterion? Does it have a direct relationship with the model's performance? If so, the author should demonstrate the relationship between this criterion and the performance of the diffusion model, such as through correlation analysis, and so on.\n\n- Why does the pruned model exhibit better performance than the pretrained model? Can the author explain this phenomenon?\n\n- I would like to know if the FID and SSIM metrics are sensitive to the pruned model.\n\n- Can the MACs metric demonstrate the superiority of the proposed method in terms of performance acceleration? I am interested in understanding the effectiveness of the proposed method on training or inference speed under different GPU, e.g. Frame Per Second (FPS).\n\nOverall, this article is well-written and easy to follow, but there are some questions that need to be addressed. If the author can address these questions, I would consider increasing the score.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> **Q1:** Provide additional details about the pruning process like step-by-step explanations or pseudocode algorithms.\n\n**A1:** Thanks for the advice. Section 4 of the appendix provides a brief discussion about the pruning pipeline and training configurations. We will make it more detailed following your suggestions and provide pseudocode in the appendix or the main paper. \n\n---\n\n> **Q2:** (a) The meaning of symbols $|\\cdot|$, $||\\cdot||$ and $||\\cdot||_0$.   \n> (b) Additionally, the notation  $\\nabla L_t(\\theta) (\\theta^\\prime - \\theta)$ should be reviewed for its validity since  $\\theta$ is a vector.   \n> (c) Furthermore, what does the \"$\\cdot$\" symbol represent in Equation (6)? Is it the dot product? If so, why is \"$|\\cdot|$\" included?\n\n**A2:** Apologies for any confusion. We address the concerns as follows:\n\n* (a) In our formulation, $|\\cdot|$ indicates the absolute value. The symbol $||\\cdot||$, unless specified, corresponds to the L-2 Norm. Also, $||\\cdot||$ denotes the L-0 norm, counting nonzero elements. To prevent confusion, we'll clarify this at the method's start. A typo in Line 118, $|\\theta^\\prime|_0$, will be corrected to $||\\theta^\\prime||_0$.\n\n* (b) In Eq. 5, $\\nabla L_t(\\theta) (\\theta^\\prime - \\theta)$ is the dot product of vectors, yielding a scalar. For structural pruning, rows or columns of weight matrices are removed, warranting vector input for Taylor expansion.\n\n* (c) \"$\\cdot$\" implies scalar multiplication. Eq. 6 gauges a weight's importance in the matrix. To estimate importance, we use the absolute value, as loss change can be negative (<0) or positive (>0), both impacting the model negatively.\n\n---\n\n> **Q3:** (a) How Eq. (6) is derived from Eq. (5). Why can it serve as an importance criterion?   \n> (b) Does it have a direct relationship with the model's performance? If so, the author should demonstrate the relationship.\n\n**A3:** The Taylor expansion in Eq. 2 measures the loss damage caused by pruning. By setting a single parameter scalar $ \\theta_{ik} $ to 0 and keeping others unchanged. We will obtained a pruned weight vector $ (\\theta^\\prime_i - \\theta_i) = [0, 0, ..., -\\theta_{ik}, 0, 0, ..., 0] $, where all elements except $k$-th one are zero. If we apply the Taylor expansion in equation 2 and take the absolute value, we can obtain the importance for a parameter scalar:\n\n$\\mathcal{I}(\\theta_{ik}, x)= | \\mathcal{L}_t(\\theta^\\prime) - \\mathcal{L}_t(\\theta) |$\n\n$ = | (\\theta_{i0}-\\theta_{i0}) \\cdot \\nabla_{\\theta_{i0}}  + \\dots + (0 - \\theta_{ik}) \\cdot \\nabla_{\\theta_{ik}} + \\dots + (\\theta_{iK}-\\theta_{iK}) \\cdot \\nabla_{\\theta_{iK}} | $\n\n$= |\\theta_{ik} \\cdot \\nabla_{\\theta_{ik}} |$\n\nFor simplicity, we drop high-order term and use $\\nabla_{\\theta_{ik}}$ for $\\nabla_{\\theta_{ik}} L(\\theta, x)$. If we set the whole vector  $\\theta_i = [0, 0, ...]$, we get the \"vector\" criterion in Line 133: $| \\sum_k \\theta_{ik} \\cdot \\nabla_{\\theta_{ik}} \\mathcal{L}_t(\\theta, x) |$. Thus, this criterion directly estimates performance damage: $| \\mathcal{L}_t(\\theta^\\prime) - \\mathcal{L}_t(\\theta) |$. We show this relation in the PDF figure depicting generation quality from various pruning algorithms without tuning. Our algorithms outperform other methods in generation quality.\n\n---\n\n> **Q4:** Why does the pruned model exhibit better performance than the pretrained model? Can the author explain this phenomenon?\n\n**A4**: This is akin to double descent [1], where pre-trained networks are oversized for certain datasets like CelebA, and pruning mitigates overfitting. CelebA images share similar features, allowing fewer parameters in the network. But this phenomenon was not observed in datasets like CIFAR-10, Church, and Bedrooms due to their varied content and complexity.\n\n[1] Sparse double descent: Where network pruning aggravates overfitting. ICML, 2022.\n\n---\n\n> **Q5:** I would like to know if the FID and SSIM metrics are sensitive to the pruned model.\n\n**A5**: We provide an empirical study in Table 3 of the main paper. FID and SSIM (Structural Similarity) are both sensitive to the pruning ratio. With pruning increasing from 0% to 70%, FID climbs from 4.19 to 9.33, while SSIM drops from 1.00 to 0.909. So, FID and SSIM can indeed reveal the effectiveness of pruning.\n\n---\n\n> **Q6**: Can the MACs metric demonstrate the superiority of the proposed method in terms of performance acceleration? I am interested in understanding the effectiveness of the proposed method on training or inference speed.\n\n**A6:** Thanks for the comments. Following the reviewer's advice, we provide more results in the Table below. Models are trained on 4090 but tested on A6000. MACs indeed reveals the actual speed-up.\n\n| Method    |  \\#Params |  MACs | Inference Mem.  | Training Mem. | Inference FPS  | Training FPS  |\n|--|-----|----|----|--|--|--|\n| Pretrained LDM  | 400.92M  | 99.80G | 11.03GB | 14.64GB | 12.87 | 4.26 |\n| Pruned LDM  | 189.43M  | 52.71G | 9.58GB | 11.95GB| 19.83| 6.37 |\n| Pretrained DDPM | 113.7M| 248.7G| 3.35GB| 5.59GB| 28.66| 11.02|\n| Pruned DDPM| 46.5M| 100.7G| 2.43GB | 4.13GB | 32.92 | 9.07 |\n\n---\n\n> **Q7:** The paper briefly mentions limitations but does not provide detailed explanations. Expanding on the limitations section will help readers understand the potential constraints or assumptions.\n\n**A7:** Thanks for the suggestion. We discussed some failure cases in Fig.4 of the appendix and in the experiments part of the main paper, such as Line 246 (about performance) and Line 253 (about distortion). We will provide a limitation section in the revised version for the following limitations and technical shortcomings:\n\n1. Performance: it is still very difficult to preserve the original performance after pruning, especially on large-scale datasets. \n2. Distortion: the algorithm only tries to minimize the global distortion, but is unaware of important semantic information in the model, such as watermarks.\n"}]

Author rebuttal:
We thank all reviewers for their constructive comments. Following the advice of Reviewer S8k8, we report our results on conditional LDMs on ImageNet-1K 256. Generated images are available in the attached PDF file. 

|  Method         |  #Params $\downarrow$ | MACs $\downarrow$   |  FID $\downarrow$ |  IS $\uparrow$ |  Train Steps $\downarrow$ |
|--------------------------|---------------------------|-------------------------|----------------------|-------------------|------------------------------|
| Pretrained LDM           | 400.92M                   | 99.80G                  | 3.60                 | 247.67            | 2000K                        |
| Scratch Training         | 189.43M  | 52.71G | 51.45                | 25.69             | 100K                         |
| Taylor Pruning           |       189.43M                      |       52.71G                   | 11.18                | 138.97            | 100K                         |
| Ours ($\mathcal{T}=0.1$) |         189.43M                    |         52.71G                 | 9.16                 | 201.81            | 100K                         |

|  Dataset | Img Size |  Pruning Ratio (MACs)    |  Lr  |  Batch |  $\text{Step}_f/\text{Step}_p$ |  weight decay |
|-------------|----------|--------------------------------------|---------|-----------|-----------------------------------|--------|
|    ImageNet-1K    | 256      | 47.2\% (99.80G $\rightarrow$ 52.71G) | 1.28e-5 | 64        | 5\% (100K/2000K)                  | 0      |

Meta-review: The paper introduces Diff-Pruning, a novel structural compression technique for learning efficient diffusion models from pre-trained ones. The method utilizes Taylor expansion on pruned timesteps to estimate the importance of weights, achieving a good performance-quality tradeoff. Diff-Pruning successfully compresses pre-trained models within a few training iterations while maintaining the original generation capabilities intact. 

The reviewers raised several concerns regarding the evaluation metrics, the technical contribution, the impact of structural pruning on performance, the preservation of important information in generated images, the derivation and clarification of equations, the lack of detailed baseline description, and the relationship between the proposed criterion and model performance. 

After rebuttal, the authors have addressed most of the concerns. As a result, the paper received a score of 4-5-5-6-7, with the majority of reviewers agreeing to accept it. Considering the reviewer who gave a score of 4 expressed lower confidence (2), the AC has agreed to accept the paper and advises the authors to consider the reviewers' feedback and incorporate the rebuttal content into the final version.
Decision: Accept (poster)