Denoising Diffusion Step-aware Models
Authors: Shuai Yang
Yukang Chen
Luozhou Wang
Shu Liu
Ying-Cong Chen
Venue: ICLR 2024 poster (2024)
Keywords: diffusion models
efficiency
network pruning
Primary area: generative models

Abstract: Denoising Diffusion Probabilistic Models (DDPMs) have garnered popularity for data generation across various domains. However, a significant bottleneck is the necessity for whole-network computation during every step of the generative process, leading to high computational overheads. This paper presents a novel framework, Denoising Diffusion Step-aware Models (DDSM), to address this challenge. Unlike conventional approaches, DDSM employs a spectrum of neural networks whose sizes are adapted according to the importance of each generative step, as determined through evolutionary search. This step-wise network variation effectively circumvents redundant computational efforts, particularly in less critical steps, thereby enhancing the efficiency of the diffusion model. Furthermore, the step-aware design can be seamlessly integrated with other efficiency-geared diffusion models such as DDIMs and latent diffusion, thus broadening the scope of computational savings. Empirical evaluations demonstrate that DDSM achieves computational savings of 49% for CIFAR-10, 61% for CelebA-HQ, 59% for LSUN-bedroom, 71% for AFHQ, and 76% for ImageNet, all without compromising the generation quality. Our code and models are available at https://github.com/EnVision-Research/DDSM.

Published as a conference paper at ICLR 2024
DENOISING DIFFUSION STEP-AWARE MODELS
Shuai Yang 1,3
Yukang Chen 2
Luozhou Wang 1,3
Shu Liu
Yingeong Chen
HHUUS(GZZ) 'SmartMore
CCUHK
HKUUSTGZZ) SmartMore Joint Lab
*HKUST
ABSTRACT
Denoising Diffusion Probabilistic Models (DDPMs) have garnered popularity for data generation across various domains. However, significant bottleneck is the necessity for whole-network computation during every step of the genera- tive process, leading to high computational overheads. This paper presents novel framework, Denoising Diffusion Step-aware Models (DDSM), to address this challenge. Unlike conventional approaches. DDSM employs spectrum of neural networks whose sizes are adapted according to the importance of each generative step, as determined through evolutionary search. This step-wise net- work variation effectively circumvents redundant computational efforts, partic- ularly in less critical steps, thereby enhancing the efficiency of the diffusion model. Furthermore, the step-aware design can be seamlessly integrated with other efficiency-geartee geared diffusion models such as DDIMs and latent diffusion, thus broadening the scope of computational savings. Empirical evaluations demon- strate that DDSM achieves computational savings of 49% for CIFAR-10, 61% for CelehA-HQ. 59% for LSUUN-beroom, 71% % for AFHQ, and 76% for ImageNet, all without compromising the generation quality. Our code and models are avail- able bttptts: htttps://githb. Cooo/EnViiiion Research/DDSM
INTRODUCTION
Denoising diffusion probabilistic models (DDPMs) have emerged powerful tool in the genera- tion of high-quality samples across a range of media. The core idea hinges on a predefined diffusion process that gradually соmupрs the original information by injecting Gaussian noise into the data. Subsequently, a neural network learns denoising procedure to systematically recover the original data step by step.
However, a notable drawback of DDPMs lies their ef- ficiency. The generative phase requires hundreds thou- sands of iterations to reverse the noise and retrieve sam- ples, significantly lagging behind the efficiency of other generative models like Generative Adversarial Networks (GANs) that only necessitate a single pass through the network. Moreover, the iterative nature of diffusion mod- els requires full pass through the neural network at every denoising step to yield single sample, raising questions regarding its efficiency and practicality.
DDPM, 48.8FPO
Proper, 40.54GFLOPs
t-250
Improper, 40.54GFLOPs
Given these challenges, natural question arises: do all diffusion steps require equal computational resources? We hypothesize that different diffusion steps have vary- ing levels importance. In certain less critical steps, a smaller model might suffice. It is, therefore, imprudent to allocate equal computational resources across all steps. To substantiate this hypothesis, crafted a fusion of
t-750
Small model Lorge model
Combinations of different models along steps. The proper combi- nation leads to high-quality samples and computational saving.
Figure 1:
"Corresoonding Author

---

Published as a conference paper at ICLR 2024
large and small model into four distinct model combinations for various steps. Our findings reveal a notable variance in performance across different combinations, even under identical computa- tional budgets. As illustrated in Figure [ good model combination produces favorable outcomes, whereas an inappropriate one leads to a less pleasing generation. This insight underscores the fact that the significance varies across different steps, pinpointing redundancy in contemporary DDPMs. Thus, the quest for an astute step-aware model combination can help pare down redundancies and expedite the diffusion model's operations.
Based on these observations, we propose using different networks for different steps. We delve into the formulation of diffusion models and extend of this approach and present the Denoising Diffusion Step-aware Models (DDSM). In DDSM, the neural network variable and slimmable at different steps to avoid redundant computation at unim- portant steps. We determine the capacity of the neural network at each step via evolutionary search and prune the network to various scales accordingly. The pruned network requires no re-training and shares weights with the intact one.
more general one. We show the feasibility
In our comprehensive empirical evaluation, we showcase the benefits of DDSM by conducting ex- tensive experiments on five distinct datasets: CIFAR-IO, CelebA-HQ, LSUN-bedroom, AFHQ, and ImageNet. We successfully achieved FLOPs acceleration rates 49% for CIFAR-10, 61% for CelebA-HQ. 59% for LSUN-bedroom. 71% for AFHQ, and 76% for ImageNet. The search result varies according to the attributes of different datasets, further indicating the significance of step- aware search.
In addition, our findings indicate that DDSM, a unique method focusing on pruning network steps, is orthogonal and easily compatible with established diffusion acceleration techniques like DDIM (Song et al. 2020a) and latent diffusion (Rombach et al. 2022). Our experiments show that integrating DDSM can further boost their efficiency."
2 RELATED WORK
Diffusion Models
Diffusion models DickseDickstein et al., 2015 Song & Ermon 2019 Song
et al. 20206] Ho et al. 2020, Nichol & Dhariwal [202]] Vahdat et al. 2021 are generative models that transform simple Gaussian distribution into more complex data distribution. This is done progressively, enabling the generation of high-quality samples from the target distribution. However, a major concern with the diffusion model the high computational cost resulting from the repeated use of a large U-net model, which typically consists of more than 35.75M parameters and requires 12. 14T FLOPS to produce a sample for CIFAR-10.
Diffusion Acceleration Several existing works have sought to mitigate the computational costs associated with the extended step. For instance, DDIM (Song al.] 2020a). PNDM (Liu et al., 2022), and DPM solver (Lu et al. 2022a b) have introduced first-order and high-order solvers. These solvers drastically cut down the number of steps required during inference. Additionally, knowledge distillation techniques have been employed by researchers (Salimans & 2022 Luhman & Luh- man 202I) to train student model from a teacher model, effectively reducing the step. Another innovative approach involves compressing the data prior to training the denoising model (Rombach et al. 2022). By working with a reduced dimension of input data, this method diminishes the com- putational demands of inference. AutoDiffusion (Li et al. 2023). concurrent study, suggests non-uniform skipping of steps and blocks within the generation sequence, offering further accel- eration DDPMs. In our study, we tackle the challenge from unique angle. We aim to curtail the overall computation of the diffusion generation process by adaptively pruning the U-net at each step. It's important to highlight that our strategy orthogonal the aforementioned methods and can seamlessly integrate with them. Several notable works share a close relationship with ours. OMS-DPM (Liu et al. 2023) firstly proposes an effective medictor-based algorithm to optimize the model-schedule. Compared to OMS-DPM, our work advances this field by implementing a super- net, efficiently scaling up the model zoo to considerably larger size. Spectral Diffusion (Yang et al.) 2022), endeavors to expedite diffusion models by harnessing denoising loss to train dynamic gate for acceleration. Our method, however, adopts a more straightforward metric: the FID. While Spec- tral Diffusion incorporates channel filtering, leading to sparse feature maps, we utilize slimmable networks and channel slicing. This results in continuously pruned channels, optimizing GPU ef- ficiency. While eDiff-1 Balaji al. (2023) adopts multiple expert models at each step improve
2

---

Published as a conference paper at ICLR 2024
Combination A
Combination B
1000
250
Combination C
Combination D
1000
750
500
1000
750
Tiny model
Large model
Figure 2: The diagrams of different model combinations. Combinations A, B, C, and D have the same computation cost. The only difference between them is the steps of using tiny models.
image quality, it does not enhance efficiency, as all these expert models are uniformly sized. Our method distinguishes itself by utilizing models of varying sizes for acceleration purposes.
Network Pruning Model pruning has rich history in the literature of neural networks. In the early period, model pruning (Han et al., 2016 is limited to sparsify connections in the network architecture, while the actual speedup from these methods might be constrained by hardware. Latter methods (Ye et al. 2018 Luo et al. 2017 Liu et al., (017) extend the pruning scope to channels, filters, layers. Slimmable network (Yu & Huang 2019b) further includes knowledge distillation into pruning. It trains a supernet network that is executable at various widths, which enables adaptive accuracy-efficieng, trade-offs during inference. Network pruning focuses the architecture in single inference time while ours makes to be step-aware in various denoising steps.
DISCUSSION OF STER-AWARE NETWORKS
3.1 PIL.TT STUDY
Conventional diffusion models use heavy network for all denoising steps, ignoring the differences between the steps. However, we hypothesize that some steps the generation process may be easier to process, and that even lightweight network can handle these steps. Treating all steps equally leads to redundancies.
To verify our hypothesis, we conducted experiments on CIFAR-10. with two pre-trained networks of different model sizes. We first tried replacing all steps with smaller models. As shown in line 1 and line 2 of Table [ the large model obviously outperformed the tiny one. Generally, the large model demonstrates its stronger ability to denoising. Uniform pruning to the tiny model leads to drops in image quality.
However, when we only prune some specific steps of the process, the results seem to vary. We manually composed four model combinations with both the large and tiny models and evaluated their performance. To ensure the same computation cost, all combinations used the large model for 750 steps and the tiny model for 250 steps, but the range of steps using tiny models varied across the combinations. Combinations A, B, C, and D employed tiny models for the step ranges [0, 250), [250, 500), [500, 750), and [750, 1000), respectively. The diagrams of these combinations are shown in Figure 2
Table 1: Evaluation of the model combinations on CIFAR-10.
The surprising evaluation results showed that different combinations led to different out- comes, as shown in Table Proper model com- binations achieved comparable, or even slightly better, image quality than the baseline large model, while taking less computation cost. Im- proper model combinations led to significant drops in performance, even though they had the same FLOPs as the proper ones.
Method
FID
GFLOPs
DDPM-tiny
18.64
0.38
DDPM-large
3.71
12.1
Combination A
10.61
9.20
Combination
4.38
9.20
Combination C
3.30
9.20
Combination 3.62
9.20
These experiments validated our hypothesis and gave three conclusions:
It is not necessary large models for every step.
3

---

Published as a conference paper at ICLR 2024
The importance different diffusion steps varies.
If we fully utilize the tiny model for its suitable steps, we can achieve excellent results with less computational cost.
These exciting conclusions have led to a way to improve the diffusion model efficiency. We eagerly seeking an excellent step-awand strategy to remove redundancies and accelerate the model.
3.2 FORMULATION ANALYSIS
In this section, we show that it is possible to combine different networks in a diffusion process. We first review the mathematical model of the diffusion process, and then expand it to generalized version that supports step-aware diffusion.
Diffusion models are generative models that use latent variables. They define Markov chain of diffusion steps that gradually add noise to data, and then train a neural network to approximate the reverse diffusion distribution construct desired data samples from the noise (Sohl-Dickstein al., 2015
Ho et al. 2020).
Suppose the data distribution is ((20) where To denotes the original data. Given training data sample To ~ q(xp), the forward diffusion process aims to produce a series of noisy latents T2, TT by the following Markovian process,
2T
(( xt-1) = (ax;; v 11,
(1)
where t is the step number of the diffusion process. T denotes the set of the steps, 3r (0, represents the variance in the diffusion process. I is the identity matrix with the same dimensions as the input To. and Nx; μt, 0)) means the normal distribution with mean and covariance a.
To generate new data sample, diffusion models sample ør from standard normal distribution and then gradually remove noise using the intractable reverse distribution g( 2z). Tradition- ally, diffusion models learn general neural network pa for all steps to approximate the reverse distribution, as shown in Eq.
Po pa x.)) = -i t), Eg(r,, f)):
where y and E are the trainable mean and covariance functions, respectively.
natural expand this to use a different network for different steps. The reverse distribution does not necessarily require the same network for the entire process, as long as the network can ap- proximate the reverse distribution p. To release this constraint, we only need to change the network 0 in Eq. to functional F(t), shown in Eq. The Ft) returns different networks 0 for different steps t. In this formulation, the traditional method can be viewed as a special case of ours, where F(t) is constant functional that returns the same function 0 for all steps.
(r() xx) = Ni t), (2), t)),
Therefore, with the expanded version of the diffusion model in Eq. 3 it is theoretically possible to combine different networks in a diffusion generation process.
4 STEP-AWARE NETWORK FOR DIFFUSION MODELS
4.1 STEP-AWARE NETWORK TRAINING
In Section 3.I.
we show that a human-designed model combination of two different sub-networks with various model sizes can improve efficiency. However, this design still has two constraints. First, the design's 's flexibility is limited by the number of sub-networks. Fewer sub-networks may not provide enough flexibility adjust network size for different steps, while more sub-networks would require training each of them, significantly increasing training costs. Second, a human -eesgned ensemble strategy may result in sub-optimal results.
4

---

Published as a conference paper at ICLR 2024
Subentwonks
Strategics
insining 1 forr
Strokgy
OF
6G
maiining
1 fors
Strutegy
62
Evolation Scanch
Strategy F(R.F)
mashring
Strategy
Forward Updane
Slimmable Training
Stepoaware Strategy
The overall procedure of DDSM, including training and searching. In this figure, the color green represents the activated weights within the network, whereas blue represents the inactive weights. The varying proportions of blue and green in different networks indicate networks of various sizes. The supernet, which has all of its weights activated, depicted entirely in green. In the training stage, update the slimmable network via its sub-networks optimization. When the slimmable network converges, it can be executed at any arbitrary size. In the searching stage, we aim find step-aware strategy that assigns different models their suitable steps. We employ the evolution algorithm to search for the optimal strategy.
Figure 3:
Algorithm DDSM Training
1: g(xo) is the data distribution, S is the the step set, ā is the noise schedule parameter.
2: θ is the slimmable network, and 0g is the largest and the smallest sub-network O.
3: repeat
4: To ~ ((()), S,c,,(O,
5: 0, ~ @ Sample random sub-network
6: for 0,, 0,, 0, do
7: Take gradient descent step
V ||e - es( Vara + VI āze, t)]
8:
9: end for
10:
until converged
To address these constraints, introduce the slimmable network and the step-aware evolutionary search. slimmable network (Yu et al. 2018 Yu & Huang (019b]/a is a neural network that can be executed at arbitrary model sizes. When training the slimmable network, we simultaneously push up the lower and upper bounds of its sub-networks, which can implicitly boost the performance of all sub-networks.
Specifically, for each training iteration, we sample random sub-network 0, from the slimmable network e. Then, we take the DDPM training step on the largest sub-network 6;, the smallest sub-network 0,, and the sampled sub-network f,. By optimizing these sub-networks, we can get slimmable network that contains large number of networks with fine- one-grained model size options, without the need for multiple training runs. The training procedure for DDSM slimmable network shown Algorithm
Our method applies the slimmable network to diffusion denoising tasks instead of image recognition. Through empirical testing, we have found that the in-place distillation and switchable batch normal- ization, which were proposed by the slimmable network, do not improve network performance the diffusion task. Therefore, we have removed these techniques for simplicity.
4.2 STEP-AWARE EVOLUTIONARY SEARCH
After training the slimmable network, we perform a multi-objective evolutionary search within the slimmable network to minimize the model size and maximize performance under different compu- tational constraints. NSGA-II (Deb et al. 2002) is the most commonly used method for this type
of search Jozefowicz et al. 2015 Pietron al. 2021 Elsken al. 2019 Guo et al. 2020) due

---

Published as a conference paper at ICLR 2024
Algorithm 2 DDSM Evolutionary Searching
Input: the slimmable network θ, number of generation G, population size P, mutation probability m, FLOPs objective weight wM, source dataset X
2: Generate the initial population p of strategies
3: for in range(GG do
4:
for each individual in do
5: score(() = FID(i, θ, X) wM x FLOPs(i)
6: assign a rank to the individual
7: end for
8: selection(p. P) # select new population
9: crossover(p) # crossover among individuals
10:
mutationnp. m) # mutate individuals according to m end for
12: find the best individual F with the highest score
Output: the best step-aware strategy
to its non-dominated sorting method, which helps balance conflicting objectives. This objective- oriented method eliminates the need for extensive human efforts in step- aep-awarre strategy designing and demonstrates strong ability in finding solutions.
To conduct the search, we use NSGA-II as the search agent. We initialize the NSGA-II with some uniform non-step-awed sstre strategies and some random strategies in the first generation. Our search algorithm adopts single-point crossover on a random position and random choice mutation to en- hance generation diversity. When mutation occurs, selected step model size changes another available choice randomly. The searching algorithm can be seen in Algorithm 2
Once obtain optimal step-aware strategy F, For each sampling, we employ our step-aware strategy F to select the most suitable sub-network 0 from the slimmable network O and use 0 denoise single diffusion step. The overall procedure of DDSM is presented in Figure 3 including training and searching.
it to speed up the sampling process.
4.3 COMPATIBILITY WITH OTHER ACCELERATION
Our method accelerates diffusion models by using diffcrent-sized models to process different steps. It is compatible with existing methods, making it a lun-andd-play module that is easy use. When combined with latent diffusion models (Rombach et al. 2022), we only need to train the slimmable network the latent space. When combined with step-skippin methods (Song al.. 2020a Liu
et al. 2022 Lu et al. 2022a b), we only need to reduce the evolutionary search space to the skipped one.
Our experiments in Section 5.4 show that our method works well with DDIM and I Latent Diffusion, producing much more efficient diffusion model compared the baselines. This makes our ap- proach effective and practical. It provides new angle to accelerate diffusion while keeping quality high and computational costs low.
EXPERIMENT
We conduct experiments on five image datasets on different domains. ranging from small scale to large scale. They are CIFARIO (Krizhevsky et al. 2009), CelebA-HQ (64x64, 128)128) (Liu et al. LSUN-bedroom (Yu et al. (016)), AFHQ (Choi al. 2020). and ImageNet (Deng et al. 2009 Due to the length limitation, we present the LSUN bedrooe, AFHQ, CelebA-HQ-123 and conditional CIFAR-10 results the appendix.
2015|.
5.1
IMPLEMENTATION DETAIL
Data settings For the CIFAR-10 dataset. utilized the standard split of 50,000 images desig- nated for training. These images were normalized and resized to a resolution of 32x32 pixels. For the CelebA-HQ-64, AFHQ, and SUNNhedroom datasets, adhered to the typical training data

---

Published as a conference paper at ICLR 2024
splits, normalizing and resizing the images to 64x64 pixels. As for ImageNet, aligned our pro- cessing with the unconditional ImageNet-O 64 settings as described in Nichol Dhariwal (202T). During training for all these datasets, incorporated random horizontal flipping data augmen- tation technique.
Training settings Our network architecture followed the setting in ADM (Dhariwal Nichol 2021]. without the classifier guidance. We used U-Net (Ronneberger et al. 2015) with an attention mechanism to down-sane wwn--sample imput images and then up-sample them back. For more training details, we present them in the Section D of the appendix.
Search and evaluation For our search algorithm, we utilized NSGA-II (Deb et al. 2002). and implemented it leveraging the open-source tools pymoo (Blank & Deb 2020) and pytorch (Paszke et al.] 2017). For assessing image quality, we adopted the FID metric (Heusel et al 2017). To evalu- speed, calculated the average Floating Point Operations (FL.Ps) across all steps throughout the image generation process. Additionally, we assessed real GPU latency to demonstrate the gen- uine acceleration capabilities of our method. The GPU latency is the time cost of generating one image with single NVIDIA RTX 3090. For more details about search and evaluation, we present them in the Section of the appendix.
Table comparative analysis of ADM and DDSM models on CIFAR-I0, CelebA-HQ, and Im- ageNet datasets. GLOPs are measured as the average over 1,000 generation steps, and latency represents the GPU processing time (in seconds) for generating a single image on an NVIDIA RTX-
3090.
CIFAR-10
CelebA-HQ GFLOPs
ImageNet
method
FID
GFLOPs
latency
FID
latency
FID
GFLOPs
latency
ADM
3.713
12.14
0.93
6.755
49.88
2.25
27.000
82.06
4.75
DDSM
3.552
6.20
0.59
6.039
19.58
1.13
24.714
19.40
3.22
5.2
RESULT
Table 2 shows the quantitative results of DDSM compared to ADM, on CIFAR-10, CelebA-HQ, and ImageNet. Our DDSM shows its superiority on speed while maintaining high generative quality. also noteworthy that our method not only reduce the FLOPs of diffusion inference, but also achieve actual acceleration. When pruning the networks with the step-aware strategy, we slice the channel contiguously. This is GPU friendly operation and could provide actual acceleration.
On CIFAR-10 and ImageNet, DDSM has only 51 % and 24% FI.OPs of the ADM but still achieves better image quality. CelebA-HQ, DDSM only takes about 39 % FLOPS of the original ADM, yet it produces images with comparable quality.
5.3 ABLATION STUDY ON THE STEP-AWARE STRATEGIES
In this section, we compared DDSM to dif- ferent strategies. The ADM-large denotes the most common baseline ADM, which takes the whole network to sample an image. The ADM-mid, ADM-small and ADM-tiny uni- formly prune the network for each of the steps with different pruning densities. They still take the same network denoise and treat steps equally. The random strategy means com- pletely choose a random model size for each step. DDSM denotes the strategy searched by our proposed method.
Table 3: Comparison of different step strategies.
The experiments are conducted on CIFAR-I0.
The result of the random strategy is calculated by averaging three different random strategies.
Strategy 3.713
FID)
avg. GFLOPs
ADM-large
12.14
ADM-mid
4.697
8.84
ADM-small
5.936
3.04
ADM-tiny
18.638
0.38
Random
7.727
5.64
DDSM
3.552
6.20
The result is shown in Table 3 The ADM-large takes heavy average FLOPs of 12.14 to pro- duce image. The ADM-mid, ADM-small
7

---

Published as a conference paper at ICLR 2024
and ADM-tiny manage reduce their computational cost, but they scarify image quality. The ran- dom strategy even achieves worse results than the ADM-mid strategies, indicating that an aimless combination of networks without step awareness could even decrease the performance. Compared to all of these baselines, our DDSM strategy demonstrates its superiority in both image quality and speed.
5.4 COMBINING WITH OTHER ACCELERATION
In this section, we present our work's compatibility with the step-skipping acceleration methods and latent diffusion. We choose the most commonly used scheduler, DDIM and a recent sampler EDM to conduct experiments. More details can be found in our appendix.
DDIM models the denoising diffusion process in a non-Markovian process and proposes to respace the sampling procedure. skips the steps of the diffusion generation process while generating high- quality images. Our method is theoretically compatible with DDIM since we only focus on model width rather than steps. integrate DDIM, we searched the step-aware strategy on the respaced time steps. We employed DDSM in different step-skipping degrees of DDIM, as shown in Table It shows that our method achieves further acceleration compared to separately using DDIM, no matter what skipping degree DDIM adopts.
Table 4: Combining DDSM with DDIM. The upper part of the table presents the FID score. The lower part of the table presents the total FLOPs of the generation process. Our DDSM can further accelerate DDIM all step-skipping conditions without quality compromise. The result is reported on CelebA.
T=10
T=50
T= 100
T=1000
DDIM
28.72
9.385
7.912
6.755
FID
+DDSM
28.33
8.826
7.132
6.039
DDIM
0.498
2.494
4.988
49.88
Total TFLOPs
+DDSM
0.260
1.376
2.384
19.58
We also present our DDSM can be combining with the famous Latent Diffusion Model (L.M),, and further accomplish an excellent acceleration. For simplicity. we directly employ Stable Diffusion's s encoder obtain the latent vectors. The result is shown in Table(5) It shows that our DDSM combine well with the LDM on the CelebA dataset.
5.5 ANALYSIS OF THE SEARCHED RESULTS
In this section, we present the DDSM searched strategies using smoothed color bars and line graphs. The result is shown in Figure (a) and (b). In the color bars, yellow denotes the smallest model, while green denotes the largest model. Middle-sized models between these two are presented with an interpolation of yellow and green.
Table 5: Combining DDSM with Latent Diffu- sion Model(LDM). The experiment is conducted on the CelebA dataset.
Method
FID
avg. GFLOPs
LDM
10.91
5.52
LDM DDSM
10.76
2.22
For CIFAR-10 and ImageNet, DDSM generally tends to use larger models when the step number goes to zero. However, for CelebA, DDSM has a completely different tendency. It employs larger models at the beginning steps and leans towards smaller models as the step number decreases.
We attribute this phenomenon to the difference in the dataset attributes. In the beginning steps of generation, diffusion models are responsible for coarse structure generation (Choi et al.) 2022). CIFAR-10 and ImageNet are multi-classes datasets of natural images with large variety. Due to the image diversity, various structures are included in the dataset, so the restriction generating coarse structures of these images is very loose. In this case, the early-stage diffusion model can give any type of coarse structure, just as it is in Figure [ (b). No matter what structure it generates, it always find proper object suitable for this structure. All it needs to do is to produce some inspiration for the latter generation. Therefore, the coarse structure generation task is easy in CIFAR- 10 and ImageNet, resulting in small model usage in the DDSM's search result. On the other hand,
8

---

Published as a conference paper at ICLR 2024
Image Net CIFAR.I. 10
(+) Search resudt (sroesched codor bine!)
(1) Suarch reash (fine graphs)
(c) Generation resueetee
and
Celebx
small
large
Figure 4: (a) The search results are presented in color bars. The yellow color denotes small models. The green color denotes large models. (b) The search results are presented in line graphs. (c) The generation sequences of CIFAR-10 and CelebA
in CelebA, the coarse structure of images is relatively uniform. Every image of CelebA should have an oval-like contour in the center, and the basic position of organs like eyes, nose, ears, and mouth should be fixed some particular points, as shown in Figure (b). Once the coarse structure goes wrong, the latter steps of the diffusion model can rarely fix this mistake. This restriction forces the diffusion model to possess a stronger structure-caption ability in the early stage. resulting in large model usage in the DDSM's search result.
In the latter steps of generation, diffusion models take charge of detail generation. For CIFAR-10 10 and ImageNet, every detail matters when considering image quality. When generating cars, wheels must be the right shape and color. When generating birds, the beak and wings must have a correct pattern. However, in CelebA, these pixel-level details on faces do not largely affect the general image quality. Under the large resolution generation, the organs and structures have already been generated in the previous steps. Detail generation be viewed as simple interpolation of colors in CclebA generation.
In appendix, the search outcomes of LSSUN-berrodd results paralleled those of CIFAR-10 and ImageNet, due to their shared high variety, while AFHQ echoed the patterns in CelebA-HQ, as both centrally feature face contours.
and AFHQ further prove hypothesis. The
We also conduct a strategy swap on these two datasets. The result is in Table [ Applying CIFAR-10's strategy to CelebA leads to great drop in performance, and vice versa, indicat- ing the significance of searching for an optimal strategy each dataset.
Table 6: FID) scores of swapping the searched strategy in CIFARIO and CelebA. Different rows denote different strategies. Different columns de- note different datasets.
Dataset
Overall, the step's 's importance distribution is highly dependent on the dataset. We must con- sider the influence of the dataset when trying to design step-aware strategy. For concrete anal- ysis of the behaviors of different steps, more theoretical proof and empirical efforts are required. We leave them for future work. We believe method can be used good tool to find the importance of different steps. It may assist in analyzing the in-depth theory the responsibility of each step in the future.
CelebA
CIFAR-10
Strategy CelebA
6.039
8. 3.140
CIFAR-10
7,431
3.552
CONCLUSION
In this paper, we re-examine the data generation steps in DDPMs and observe that the whole-network computation in all steps might be unnecessary and inefficient. For acceleration, we propose the step- aware network, DDSMs, as the solution. DDSMs, we adopt the neural network in various sizes at different steps. according to generation difficulties. We conduct extensive experiments and show its effectiveness on CelebA, CIFAR- 10, LSUN-bedroom, AFHQ, and ImageNet. DDSMs can present high-quality samples, with most 76% computational cost saving.

---

Published as a conference paper at ICLR 2024
REFERENCES
Yogesh Balaji, Scungjun Nah, Xun Huang, Arash Vahdat, Jiaming Song, Qinsheng Zhang, Karsten Kreis, Miika Aittala, Timo Aila, Samuli Laine, Bryan Catanzaro, Tero Karras, and Ming- Yu Liu. ediff-i: Text-to-image diffusion models with an ensemble of expert denoisers, 2023.
J. Blank and K. Deb. pymoo: Multi-objective optimization in python. IEEE Access, 8:89497-89509, 2020.
Jooyoung Choi, Jungbeom Lee, Chachun Shin, Sungwon Kim, Hyunwoo Kim, and Sungroh Yoon. Perception prioritized training of diffusion models. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp- 11472-11481, 2022.
Yunjey Choi, Youngjung Uh, Jaejun Yoo, and Jung- Woo Ha. Stargan v2: Diverse image synthesis for multiple domains. In Proceedings the IEEE Conference Computer Vision and Pattern Recognition, 2020.
Kalyanmoy Deb, Amrit Pratap, Sameer Agarwal, and TAMT Meyarivan. A fast and elitist mul- tiobjective genetic algorithm: Nsga-ii. IEEE transactions on evolutionary computation, 6(2): 182-197, 2002.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hi- erarchical image database. In 2009 IEEE conference computer vision and pattern recognition, pp. 248-255 leee, 2009.
Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. Advances in neural information processing systems, 34:8780-8794, 2021.
Thomas Elsken, Jan Hendrik Metzen, and Frank Hutter. Neural architecture search: A survey, The Journal of Machine Learning Research, 1091):1977-2017, 2019.
Gongfan Fang, Xinyin Ma, Mingli Song, Michael Bi Mi, and Xinchao Wang. Depgraph: Towards any structural pruning. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 16091-16101 2023a.
Gongfan Fang, Xinyin Ma, and Xinchao Wang. Structural pruning for diffusion models. In Advances in Neural Information Processing Systems, 2023b.
Ronghao Guo, Chen Lin, Chuming Li, Keyu Tian, Ming Sun, Lu Sheng, and Junjie Yan. Powering one-shot topological nas with stabilized share- paramerrr proxy. In Computer Vision-ECCV 2020; 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XIV, pp. 625- 641. Springer, 2020.
Song Han, Huizi Mao, and William J. Dally. Deep compression: Compressing deep neural network with pruning, trained quantization and huffman coding. In International Conference Learning Representations, 2016.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. NeurlPS, 2017.
Jonathan Ho, Ajay Jain, and Picter Abbeel. Denoising diffusion probabilistic models. NeurIPS, 2020.
Rafal Jozefowicz, Wojeiech Zaremba, and llya Sutskever. An empirical exploration of recurrent network architectures. In International conference on machine learning, pp. 2342-2350. PMLR, 2015.
Tero Karras, Miik Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion- based generative models. Proc. NeurIPS, 2022.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
10

---

Published as a conference paper at ICLR 2024
Lijiang Li, Huixia Li, Xiawu Zheng, Jie Wu, Xuefeng Xiao, Rui Wang, Min Zheng, Xin Pan, Fei Chao, and Rongrong Ji. Autodiffusion: Training-fre free optimization of time steps and architectures for automated diffusion model acceleration, 2023.
Enshu Liu, Xuefei Ning, Zinan Lin, Huazhong Yang, and Yu Wang. OMS-DPM, Optimizing the model schedule for diffusion probabilistic models. In Proceedings of the 40th International Con- ference on Machine Learning, pp. 21915-21936. PMLR, 2023.
Luping Liu, Yi Ren, Zhijie Lin, and Zhou Zhao. Pseudo numerical methods for diffusion models manifolds. In International Conference on Learning Representations, 2022.
Zhuang Liu, Jianguo Li, Zhiqiang Shen, Gao Huang, Shoumeng Yan, and Changshui Zhang. Learn- ing efficient convolutional networks through network slimming, In International Conference on Computer Vision, pp. 2755-2763, 2017.
Ziwei Liu, Ping Luo. Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In Proceedings of International Conference on Computer Vision (ICCV), December 2015.
Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. Dpm-solver: A fast ode solver for diffusion probabilistic model sampling in around 10 steps. arXiv preprint arXiv:2206.000 00927, 2022a.
Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. Dpm-solver++: Fast solver for guided sampling of diffusion probabilistic models. arXiv preprint arXiv:2211.01095, 2022b.
Eric Luhman and Troy Luhman. Knowledge distillation in iterative generative models for improved sampling speed. arXiv preprint S0tiv:2201,0233000 2021.
Jian-Hao Luo, Jianxin Wu, and Weiyao Lin. Thinet: A filter level pruning method for deep neural network compression. In International Conference on Computer Vision, pp. 5068-5076, 2017.
Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. In International Conference on Machine Learning, 2021.
Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. In NIPS-W, 2017.
M Pietroń, D Žurek, and Kamil Faber. Fast and scalable neuroevolution deep learning architecture search for multivariate anomaly detection. arXiv preprint arXiv:2112.05644 2021.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, 2022.
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomed- ical image segmentation. In Medical Image Computing and Computer-Asiisted Intervention- MICCAI 2015: 18th International Conference, Munich, Germany, October 5-9, 2015, Proceed- ings, Part III 18, pp. 234-241. Springer, 2015.
Tim Salimans and Jonathan Ho. Progressive distillation fast sampling of diffusion models. In International Conference Learning Representations, 2022.
Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. 2015.
Jiaming Song. Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint 1xXix:0000002000 2020a.
Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. Advances in Neural Information Processing Systems, 32, 2019.
11

---

Published as a conference paper at ICLR 2024
Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:2011.13456. 2020b.
Arash Vahdat, Karsten Kreis, and Jan Kautz. Score-based generative modeling in latent space. Advances in Neural Information Processing Systems, 34:1 1tt:1287-11302 2021.
Xingyi Yang, Daquan Zhou, Jiashi Feng, and Xinchao Wang. Diffusion probabilistic model made slim, 2022.
Jianbo Ye, Xin Lu, Zhe Lin, and James Z. Wang. Rethinking the assumption in channel pruning of convolution layers. In International Conference on Learning Representations, 2018.
Fisher Yu, Ari Seff, Yinda Zhang, Shuran Song, Thomas Funkhouser, and Jianxiong Xiao. Lsun: Construction large-scale image dataset using deep learning with humans in the loop, 2016.
Jiahui and Thomas Huang. Autoslim: Towards one-shot architecture search for channel numbers.
arXiv preprint arXiv: 1728, 2019a.
Jiahui Yu and Thomas S Huang. Universally slimmable networks and improved training techniques. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 1803-1811, 2019b.
Jiahui Yu, Linjie Yang. Ning Xu, Jianchao Yang, and Thomas Huang. Slimmable neural networks. 2018.
arXiv preprint
Wenliang Zhao, Lujia Bai, Yongming Rao, Jie Zhou, and Jiwen Lu. Unipe: A unified predictor- corrector framework for fast sampling of diffusson models. NeurIPS, 2023.
12

---

Published as a conference paper at ICLR 2024
APPENDIX
A EXPERIMENT ON LSUN-BEDROOM AND AFHQ
We've validated DDSM's effectiveness and scalability on two extra datasets, LSUN-bedroom (64x64) and AFHQ (64x64). LSUN-bedroom is entirely non-object-centric. While AFHQ is object- centric but from domain different from CelebA. SaDiffusion achieves 59% and 71% acceleration on the respective datasets without compromising image quality, as detailed in Table
The search results Figure 5 of dataset's s attributes. The LSUN- bebrroo results parallels those of CIFAR-10, due to their shared high variety, while AFHQ echoed the patterns seen CelebA-HQ, both centrally feature face contours.
and AFHQ further validate our analysis of the
Table 7: Evaluation on LSUN-bedroom and AFHQ
LSUN-bedroom
AFHQ
Method
FID
GFLOPs
FID
GFLOPs
ADM
5.027
49.88
7.244
49.88
DDSM
5.289
20.48
6.249
14.72
LSUN-bedroom
AFHQ
T
small
large
Figure 5: search results LSUN-hedroom and AFHQ
B COMPARISON WITH CONCURRENT WORK DIFF-PRUNING
We elucidate the differences and provide empirical comparison with the recent Diff- Diff-pruning Fang et al. (2023a b). Diff-pruning employs Taylor expansion over pruned timesteps to effectively reduce computational overhead. This approach primarily focuses on pruning the entire network. Our method, in contrast, adopts strategy of adaptively pruning the network at various timesteps. This approach allows for more nuanced and efficient solution. For clearer understanding of these differences,
conducted an empirical comparison with Diff-pruning, Table 8 showcases our experiment on CelebA 64x64 using 100 steps. Our DDSM not only achieves a FID comparable to Diff- Diff-pruning but also demonstrates superior efficiency.
Table 8: Experimental comparison with Diff-Pruning, CelebA 64x64, 100 steps. Metric
Baseline
Diff-pruning
DDSM
FID (Lower is better)
6.48
6.24
6.04
GFLOPS (Lower is better)
49,9
26.6
19.6
C COMBINING WITH THE EDM SAMPLER
Our DDSM is compatible with recent sampling schedulers, such DPM Lu et al. (2022a)
DPM++ Lu et al. (2022b), uniPC et al. (2023). , and EDM Karras et al. (202). These samplers primarily focused devising novel noise schedules to enhance the performance of diffusion models. Our method, which adaptively prunes at different steps, is theoretically compatible with these approaches. Among these samplers, we have chosen EDM for our experiments,
believe
13

---

Published as a conference paper at ICLR 2024
this selection could demonstrate method's suitability for similar methods. In the experiment, kept the pretrained weights of DDSM. We then replaced the original DDPM scheduler with EDM's Heun Discrete 2nd order method and initiated new search process. The results, illustrated in Table 9 show that DDSM effectively integrates with EDM on the unconditional CIFAR-10 dataset, achieving 45% and 56% total TFLOPs saving for 50 steps and 100 steps. Note that, the quality can be further improved when retraining the network with EDM's 's setting.
Table 9: Experiment result combining DDSM with EDM, on CIFAR- 10. EDMI00+DDSM
Metric
EDM50
EDM50+DDSM
EDMI00
FID
3.65
3.61
3,41
3.49
Total TFLOPs
0.61
0.34
1.22
0.54
D EXPERIMENT ON CELEBA-HQ 128x128
We conduct experiments on CelebA-HQ- 1Q-128x1288 to prove that our DDSM is applicable with higher resolution data. this experiment, we train a new slimmable supernet from scratch, and directly employ the search result on CelebA-HQ-64x664 to super network, Table [0 shows the result.
Table 10: Experiment result on CelebA- HQ 128x128.
Metric
ADM
DDSM
FID (Lower is better)
7.53
7.71
GFLOOSS (Lower is better)
194.00
76.18
E EXPERIMENT ON CONDITIONAL CIFAR-10
We also conduct experiments conditional 10FAR-10 to prove that our DDSM is applicable with guided diffusion. In this experiment, we train a guided diffusion slimmable supernet from scratch, and directly employ the search result on unconditional CIFAR-10 10 super network. Table I1 shows the result.
Table 11: Experiment result on conditional CIFAR-10.
Metric
ADM
DDSM
FID (Lower is better)
2.48
2.52
GFLOPS (Lower better)
12.14
6.20
F IMPLEMENTATION DETAILS
Architecture Our implementation of slimmable networks draws inspiration from the US-Net. These networks characterized by their ability operate at various widths, providing flexi- ble and universal solution for network scalability. During the training of slimmable networks, we focus on optimizing the smallest, largest, and a selection of randomly sampled middle-sized sub- networks. This approach implicitly enhances the performance of all potential networks within the supernet.
In practical terms, our slimmable UNet adheres to the structure of ADM, but with a significant modification: all convolution layers are replaced by slimmable convolutions. These specialized convolutions are capable of adaptively processing tensors with varying number of input channels.
To accommodate a broader range of sub-networks, we adjusted the group number in the group normalization layer from 32 to 16. Additionally, chose omit the Batch Normalization (BN) calibration stage, as proposed [11], since our diffusion UNet exclusively utilizes GroupNorm.
Regarding the sizes of the sub-networks, we offer seven different options, corresponding to $ 3 6 7 and & of the original ADM's width. To find more efficient strategies, we manually exclude
14

---

Published as a conference paper at ICLR 2024
some large width options. For example, in CIFAR-10, exclude the and options. In most cases, the total strategy space is of 7num,timescps.
Search and Evaluation In the searching phase, the FID was computed by comparing the entire training dataset with the generated images. Empirical findings indicate that the FID score offers more consistent performance measurement compared to the loss; therefore, we aimed to search for strategy yielding lower FID. For the search parameters, the process encompasses total of 10 iterations, with each iteration involving a population of 50, and maintaining a mutation rate of 0.001. The initial generation crucially includes mix of uniform non-step-aware strategies and some ran- dom strategies. This specific approach to initialization and mutation has been empirically found to facilitate easier convergence of the search algorithm. Furthermore, a weight parameter incorpo- rated, which multiplies the GFLOPs to strike a balance between image quality and computational efficiency. For CIFAR-10, we set the weight parameter to 0. to favor higher image quality, while for CelebA, the FLOPs weight parameter is adjusted to 0.25. These parameters were manually selected to ensure that there are no compromises in generation quality.
Compatibility Experiments For the DDIM experiment, we adopted the weights from the 1000- step ADM model and applied the DDIM sampling schedule. As indicated in Table 4, our method significantly boosts DDIM's speed by 48%, 45%, 53%, and 62% for 10, 50, 100, and 1000 steps, respectively. This suggests that while additional steps enhance generation quality, they also intro- duce excess computational load. Our approach proves increasingly beneficial as the number of steps increases. our latent diffusion experiment, we employed the AutoEncoderKL. of SD1.4 to trans- form images into latent vectors, upon which our DDSM trained. To adapt the reduced spatial size, we modified the U-Net downsampling from 2. Our DDSM achieved 60% acceleration in latent diffusion methods.
G DISCUSSION ON THE EXTRA COST
In this section, diseuss the extra cost introduced by DDSM in the training and searching stage. Although we introduce extra training and search cost, DDSM still shows its advantages in inference. In deep learning algorithms, common manner is train a model once and repeatedly use this model to infer results. Nowadays, well-known Diffusion models all conform to this manner, like StableDiffusion and Imagen. They require large amount of time train. But once they trained, they could repeatedly generate billions of images. Therefore, in terms of efficiency, we usually care more about the inference speed due its repetitiveness.
Our method aims to reduce the inference cost of diffusion models. We prune the diffusion inference process with its step-aware strategy, accomplishing most 76% acceleration. These cost savings could be counted repeatedly. The more samples we produce, the more benefits our framework demonstrates. In DDSM, we indeed introduce extra training and search cost, but these costs only take once. Concretely, the training cost of the slimmable network is about 2 times to 3 times of the normal diffusion model's training. The search cost depends on the search hyper-parameters In our setting, it is approximately the same time training normal diffusion model. In Table 12 present the time cost of our DDSM on NVIDIA RTX 3090's s GPU hours.
Table 12: Total GPU hours of our DDSM.
CIFAR-10
CelebA
DDPM-train
278
435
DDSM-train
502
1036
DDSM-search
320
524
H SEPARATED LINE GRAPHS OF SEARCH RESULTS
We plot the search result separately for better visualization, as shown in Figure
15

---

Published as a conference paper at ICLR 2024
73
Figure 6: Line graphs of the search results across all datasets.
I QUALITATIVE RESULT
We present the generation result of DDSM in Figure 7 It shows that the efficient DDSM could produce high-quality images.
(a) CIFAR-10
(b) CelebA
(c) ImageNet
(d) LSUN-bedroom
(e) AFHO
Figure 7: Generated images of DDSM on CIFAR-10, CelebA, ImageNet, LSUN-bedroom, and AFHQ.
16


Reviews:
[{"review_id": "6AD1Pn1GHj", "reviewer": "Reviewer_GL7V", "summary": "This paper proposes Denoising Diffusion Step-aware Models (DDSM), which utilize variable-sized neural networks for different steps of the diffusion generative process. The key insight is that diffusion steps have varying importance, so uniformly allocating computational resources is inefficient. The method trains a slimmable UNet that can be flexibly pruned to different capacities. An evolutionary search then determines the optimal per-step network size to balance efficiency and performance. Experiments demonstrate substantial computational savings on CIFAR-10, CelebA-HQ, LSUN-bedroom, AFHQ, and ImageNet versus conventional diffusion models, with minimal quality loss.", "questions": "You claim compatibility with methods like DDIM and latent diffusion, but details are limited. Could you provide in-depth quantitative experiments demonstrating accelerated performance when combining DDSM with these existing diffusion acceleration techniques?\n\nThe search cost and scalability of the evolutionary algorithm is unclear. Could you analyze the computational requirements of the search procedure and discuss how it scales with factors like dataset size?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- The core ideas are technically sound and offer a unique perspective on accelerating diffusion models.\n- Using evolutionary search to determine step-wise network requirements. \n- Compatibility with existing methods like DDIM.", "weaknesses": "- Using a different network at different timestep has been explored before, such as in e-diff-i. \n- The compatibility claims with DDIM and latent diffusion are fairly cursory. More detailed experiments showing accelerated performance combining these methods could better showcase modularity.\n- The evolutionary search itself requires non-trivial compute resources. Analysis of the search costs and scalability could be insightful.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "TXZNKiatUn", "reviewer": "Reviewer_339s", "summary": "This paper...\n- proposes to accelerate diffusion sampling by using diffusion models of different size at each time-step,\n- proposes to use evolutionary search to find a best step-aware strategy,\n- shows the effectiveness of the proposed approach on CIFAR-10, CelebA-HQ, and ImageNet sampling.", "questions": "- Figure 3 is unclear. What do networks with mixed colors (blue and green) mean?\n- In Table 3, why do DDSM outperform ADM-large? Shouldn't the performance of DDSM be bounded by the performance of the largest model?\n- Is DDSM compatible with guided diffusion? Can the authors provide some demonstrations?\n- Is DDSM compatible with recent fast solvers, such as EDM [2]?\n\n[2] Elucidating the Design Space of Diffusion-Based Generative Models, Karras et al., NeurIPS, 2022.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- Paper is easy to follow.\n- Using networks of different size (slimmable network) for each-time step is an unexplored approach in diffusion acceleration.\n- DDSM shows promising performance gains on a variety of generation tasks.", "weaknesses": "While the idea behind DDSM is interesting, I am inclined to give \"marginal reject\" due to weak experimental validation.\n\n- The paper lacks comparison with [1], which I think is a very relevant work.\n- The paper lacks experiments on higher resolution data. Can the authors provide results on $\\geq 128$ resolution images?\n\n[1] Structural Pruning for Diffusion Models, Fang et al., NeurIPS, 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "SOwpqNKRB3", "reviewer": "Reviewer_X3mN", "summary": "The paper proposes Denoising Diffusion Step-aware Models (DDSM) to improve efficiency of denoising diffusion probabilistic models (DDPMs) for image generation. Previously, DDPMs require compute-intensive iterative sampling, using the full model each step. DDSM hypothesizes different steps have varying importance, and uses a spectrum of networks with adapted sizes for each step, determined via evolutionary search. This avoids redundant computation in less critical steps.\nDDSM integrates with slimmable networks - trained simultaneously on sub-networks to enable execution at arbitrary sizes.", "questions": "Refer to weaknesses.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "DDSM accelerates diffusion models by avoiding uniform computation for all steps. The step-aware network design is shown to be efficient and effective across datasets.", "weaknesses": "The major concern is the pathway of denoising acceleration.  It's evident that the prevalent approach to enhancing diffusion speed, in the context of network compression, hinges on the application of post-training quantization techniques. These techniques enable the compression of neural networks in a manner that circumvents additional training [1,2]. However, I observe that your method necessitates training and, notably, falls short in performance when compared to methods that forego training. To illustrate, empirical assessments show that DDSM facilitates computational reductions of 49% for CIFAR-10 and 76% for ImageNet. In contrast, the techniques in [1,2] manage to achieve 4 or 8-bit quantization (surpassing DDSM in speed) without compromising the FID score. It's crucial to underscore that these methods achieve this efficiency entirely without the need for further training. Consequently, the approach employed by DDSM for accelerating denoising doesn't appear to be robust enough.\n\n\n[1] PTQ4DM, CVPR 2023\n\n[2] Q-dfiifusion, ICCV 2023", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "mEC6SDSO0M", "reviewer": "Reviewer_buDR", "summary": "This paper presents a novel approach to accelerating sampling from denoising diffusion models without sacrificing image quality. This approach is based on the idea that not every generation step is equally difficult nor equally important for the quality of the final image. The authors propose using *slimmable networks* to prune the network weights at less-important generation steps, and combine this with an evolutionary search method to identify which steps can be safely pruned without sacrificing image quality. The authors demonstrate that their method reduces the FLOPs required for multiple datasets while achieving comparable or better sample quality (FID) than the unpruned methods, and further show that it can be combined with the adaptive step-skipping of DDIM or with latent diffusion for further improvements. They also visualize the learned pruning schedules for different datasets, and show that the set of safely-prunable generation steps differs depending on dataset structure.", "questions": "Could you provide more details on the slimmable network architecture and on the configuration of the search algorithm, as I discuss in [W1] and [W2]? For the search algorithm, what was the value of $w_M$ used, and how critical is this choice? (Or, are the FID and FLOPs measurements automatically balanced by NGSA-II somehow?)\n\nFigure 4a and Figure A are quite interesting, but they also look somewhat blurry. I can sort of see some fuzzy \"bands\" at different points of the trajectory; are these actual changes in the slimmed model size or are these some sort of compression artifact in the figure image? Also, what is the resolution of the X axis here, e.g. where are the boundaries between different steps? It's hard to understand exactly what the plot is showing, and I think this plot would be more readable if it were presented as a line graph (instead of as a heatmap), since its only 1 dimensional.\n\nI found some very recent related work [\"Structural Pruning for Diffusion Models\" (Fang et al. 2023)](https://arxiv.org/abs/2305.10924) which was just accepted at NeurIPS. That work also considers pruning in order to speed up diffusion model inference, although I believe they only consider pruning the entire network rather than adaptively pruning at different timesteps. It might make sense to discuss this related work in your paper, and I'd also be curious how your approach compares to theirs in terms of FLOPs savings.\n\nHave you explored combining your DDSM approach with some of the more recent accelerated sampling approaches? And do you still observe runtime improvements for those methods?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "**[S1]** The idea of pruning the network adaptively based on the difficult of each generation step is insightful, and well-motivated by the introduction and the pilot study in section 3.1. This approach also seems quite novel. I'm not aware of any previous work that has considered adaptively using different network sizes at different timesteps.\n\n**[S2]** The authors show that their approach is actually complementary to existing acceleration methods like DDIM, and yields additional improvements when combined. I expect that the approach could also be combined with some of the more recent ODE-based acceleration methods like DPM-Solver, since the proposed DDSM technique is agnostic to the exact mathematical sampling process and instead adaptively shrinks the network itself.\n\n**[S3]** The empirical results are quite strong across multiple datasets. The authors show significant cost savings relative to sampling from the full model, and actually show slight *improvements* in sample quality for all but one of the datasets (LSUN-bedroom takes a small hit).\n\n**[S4]** The approach is easy to understand and seems like it would not be too difficult to implement. The authors also plan to release their code.\n\n**[S5]** I found the results of the pruning schedule search to be quite insightful. Different datasets show different amounts of pruning, and this pruning occurs at different times, which seems to match whether the dataset has more information in its high-frequency or low-frequency components.", "weaknesses": "**[W1]** The approach relies on \"slimmable networks\" and \"slimmable counterparts\" of standard convolution and normalization. However, these aren't discussed in much detail. I think the paper could benefit from some additional background on what slimmable networks are and how they work, and more details on the particular slimmable network architecture used for this work. (In particular, how many slimmable switches were used, and how do they fit into the U-Net architecture?)\n\n**[W2]** The objective for the evolutionary search algorithm is also presented at a fairly high level and could use more details. The authors mention using NSGA-II to \"balance conflicting objectives\", but it's not clear to me what objectives were used. Algorithm 2 suggests that the search objective was a linear combination of FID and FLOPS, but the details of the linear combination weights are not specified.\n\n**[W3]** Although the contributions of this work do seem orthogonal to some of the more recent work in accelerating diffusion model sampling, I think the experimental results would be more impressive if they could also be demonstrated for these more recent accelerated sampling approaches like DPM-Solver, [DPM-Solver++](https://arxiv.org/abs/2211.01095) or the recent [UniPC](https://arxiv.org/abs/2302.04867). My guess is that the methods could be combined, but it would be useful to see how much FLOPs can be saved when combined with these more recent samplers.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)