Faster Diffusion: Rethinking the Role of the Encoder for Diffusion Model Inference
Authors: Senmao Li
taihang Hu
Joost van de Weijer
Fahad Khan
Tao Liu
Linxuan Li
Shiqi Yang
Yaxing Wang
Ming-Ming Cheng
jian Yang
Venue: NeurIPS 2024 poster (2024)
Keywords: generative model
UNet
acceleration
inference phase
Primary area: diffusion_based_models
TL;DR: We introduce a simple yet effective encoder propagation (unet feature reuse) scheme to accelerate the diffusion (i.e., StableDiffusion, and DiT) sampling for a diverse set of tasks.

Abstract: One of the main drawback of diffusion models is the slow inference time for image generation. Among the most successful approaches to addressing this problem are distillation methods. However, these methods require considerable computational resources. In this paper, we take another approach to diffusion model acceleration. We conduct a comprehensive study of the UNet encoder and empirically analyze the encoder features. This provides insights regarding their changes during the inference process. In particular, we find that encoder features change minimally, whereas the decoder features exhibit substantial variations across different time-steps. This insight motivates us to omit encoder computation at certain adjacent time-steps and reuse encoder features of previous time-steps as input to the decoder in multiple time-steps. Importantly, this allows us to perform decoder computation in parallel, further accelerating the denoising process. Additionally,  we introduce a prior noise injection method to improve the texture details in the generated image. Besides the standard text-to-image task, we also validate our approach on other tasks: text-to-video, personalized generation and reference-guided generation. Without utilizing any knowledge distillation technique, our approach accelerates both the Stable Diffusion (SD) and DeepFloyd-IF model sampling by 41$\%$ and 24$\%$ respectively, and DiT model sampling by 34$\%$, while maintaining high-quality generation performance. Our code will be publicly released.

Faster Diffusion: Rethinking the Role of the Encoder for Diffusion Model Inference
Senmao Li Taihang Hu Joost van de Weijer", Fahad Shahbaz Khan 3.4, Tao Liu' Linxuan Li', Shiqi Yang?. Yaxing Wang'! Ming-Ming Cheng', Jian Yang' VCIP, CS, Nankai University, *Computer Vision Center, Universitat Autônoma de Barcelona Mohamed bin Zayed University of AI, *Linkoping University, 'Independent Researcher, Tokyo (senmaonk, hutaihang00, ltolcy0, 1inxuanli520, shiqi. yang147 hp77.pp@@ppp @pmagmlll..... joost&cve. ea, fahad, khan@1iu. se, (yaxing, cmm, cs cajyang) (enjyann)enanaai @nankai. edu. cn https://sen-maa wwt.eeattooooooooeeeeeee... gittub.iooFasterDifffsssii
Statle Extussn
Deep
20t 7-0x
DOPM
DPV-Stharee 16.5e CLeto Dituatonn
32.72 2291
142 41%6
Comm
Steeh ou
Dif uy Gam 3-33 seol
Contro chee my
Figure 1: Results of our method for diverse set of generation tasks. We significantly increase the image generation speed (second/image).
Abstract
One the main drawback diffusion models the slow inference time for image generation. Among the most successful approaches to addressing this problem are distillation methods. However, these methods require considerable computational resources. In this paper, we take another approach to diffusion model acceleration. We conduct a comprehensive study of the UNet encoder and empirically analyze the encoder features. This provides insights regarding their changes during the inference process. In particular, we find that encoder features change minimally, whereas the decoder features exhibit substantial variations across different time- steps. This insight motivates us to omit encoder computation at certain adjacent time-steps and reuse encoder features of previous time-steps input the decoder in multiple time-steps. Importantly, this allows us to perform decoder computation in parallel, further accelerating the denoising process. Additionally, we introduce a prior noise injection method to improve the texture details in the generated image. Besides the standard text-to-image task, also validate our approach
"Equal contribution. Author ordering determined by coin flip over Google Hangout. The corresponding author.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

other tasks: text-to-video, personalized generation and reference-guee guiiedd generation. Without utilizing any knowledge distillation technique, our approach accelerates both the Stable Diffusion (SD) and DeepFloyd-IF model sampling by 41% and 24% respectively, and DiT model sampling by 34%, while maintaining high-quality generation performance.
1 Introduction
One of the popular paradigms in image generation, Diffusion Models (DMs) [1, 2, 3] have recently achieved significant breakthroughs in various domains, including text-to-video generation [4, 5, 6], personalized image generation [7, 8, 9] and reference-guided image generation [10, 11. 12]. While diffusion models produce images of exceptional visual quality, their primary drawback lies in the prolonged inference time. The original diffusion model had an inference time several orders of magnitude slower than, for instance, GANs. One of the challenges hindering the acceleration of diffusion models is their inherent sequential denoising process, which limits the possibilities of effective parallelization.
To improve inference time speed of diffusion models, several methods have been developed, that can roughly be divided in two sets of approaches. Firstly, involving step reduction, the aim is to reduce the number of sampling steps within diffusion model inference, such as DDIM [13] and DPM-Solver [14], which have significantly reduced the number of sampling steps. Secondly, in contrast, knowledge distillation progressively distills a slow (many-step) teacher model into a faster (few-step) student model [15, 16]. Some recent works [17, 18, 19] excel at generating high-fidelity images in a few-step sampling scenario but face challenges in maintaining quality and diversity in one-step sampling. The main drawback of the distillation methods is that they require retraining to perform the distillation into faster diffusion models.
Orthogonal to these methods, we take a closer look at the sequential nature of the denoising process. We focus on the char-
acteristics of the encoder in pretrained diffusion models (c.g.. the SD and the DIT (211-) Interestingly, based on our anal- ysis presented in Sec 3.2, discover that encoder features change minimally (Fig. 3a) and have a high degree of similar- ity (Fig. 2 (top)), whereas the decoder fea- tures exhibit substantial variations across different time-steps (Fig. 3a and Fig. 2 (bot- tom)). This insight is relevant because it allows us to circumvent the computation of the encoder during multiple time-steps. As a consequence, the decoder computations which are based on the same encoder input can be performed in parallel. Instead, we reuse the computed encoder features computed at one time-step (since these change minimally) as input to adjacent decoders during the following time-steps. More recently, both DeepCache and CacheMe[23] leverage feature similarity to achieve acceleration. However, they rely on sequential denoising, as well as CacheMe requires fine-tuning. Unlike these methods, our approach supports parallel processing, which leads to faster inference (Tab. 2).
Figure 2: Visualising the hierarchical features We applied PCA to the hierarchical features following PnP [20] and used the top three leading components an RGB image for visualization. The encoder fea- tures change minimally and have similarities at many time-steps (top), while the decoder features exhibit sub- stantial variations across different time-steps (bottom).
We show that the proposed propagation scheme accelerates the SD sampling by 24% DeepFolyd-IF sampling by 18%, and DiT sampling by 27%. Furthermore, since the same encoder features (from previous time-steps) can be used as the input to the decoder of multiple later time-steps, this makes it possible to conduct multiple time-steps decoding concurrently. This parallel procession accelerates SD sampling by 41%, DeepFloyd-IF sampling by 24%, and DiT sampling by 34%. Furthermore, to alleviate the deterioration of the generated quality, we introduce a prior noise injection strategy to preserve the texture details in the generated images. With these contributions, our proposed method achieves improved sampling efficiency while maintaining high image generation quality.
'See the visualisation the hierarchical features for blocks in Appendix A "W define the first several transformer blocks of DiT as the encoder, and the remaining ones of the transform blocks as the decoder.
2

---

of
T 20
Aude
14
Aoda
Bots
6
6.09
6:09
6
pt:100
Timeslop
(e)
(b)
(c)
Figure 3: Analyzing the UNet in Diffusion Model. (a) Feature evolving across adjacent time-steps is measured by MSE. (b) We extract the hierarchical features output of different layers of the UNet at each time-step, average them along the channel dimension to obtain two-dimensional hicrarchical features, and then calculate their Frobenius norm. (c) The hierarchical features of the UNet encoder show a lower standard deviation, while those the decoder exhibit a higher standard deviation.
Importantly, our method can be combined with several approaches existing to speed up DMs. The main advantage of our method with respect distillation- -ased approaches, is that our method can be applied inference time, and does not require retraining new faster distillation model; process that is computationally very demanding and infeasible for actors with limited computational budget. Finally, we evaluate the effectiveness of our approach across wide range of conditional diffusion- based tasks, including text-to-video generation (e.g., Text2 [4] and VideoFusion [5]), personalized image generation (e.g., Dreambooth [7]) and reference-guiee image generation (e.g.. ControlNet [10]).
To summarize, we make the following contributions:
We conduct thorough empirical study of the features of the UNet in the diffusion model showing that encoder features vary minimally (whereas decoder feature vary significantly).
We propose parallel strategy for diffusion model sampling at adjacent time-steps that significantly accelerates the denoising process. Importantly, our method does not require any training or fine-tuning technique.
Furthermore, we also present a prior noise injection method to improve the image quality (mainly improving the quality of fighffrequency textures).
Our method can be combined with existing methods (like DDIM, and DPM-solver) to further accelerate diffusion model inference time.
2
Related Work
Denoising diffusion model. Recently, Text-to-image diffusion models [1, 24, 25, 26] have made significant advancements, Notably, Stable Diffusion and DecpFloyd-IF stand out as two of the most successful diffusion models available within the current open-source community. These models, building upon the UNet architecture, are versatile and can be applied to wide range of tasks, including image editing [27, 28]. super-resolution [29, 30), segmentation [31, 32]. and object detection [33, 34]. Given the strong scalability of transformer networks, DiT [21] investigates the transformer backbone for diffusion models.
Diffusion model acceleration. Diffusion models use iterative denoising with UNet for image gener-
ation, which is consuming. There are plenty of works trying to address this issue. One strategy involves employing efficient diffusion model solvers, such as DDIM [13] and DPM-Solver [14], which have demonstrated notable reductions in sampling steps. Additionally, ToMe [35] exploits token redundancy to minimize the computations necessary for attention operations [36]. Conversely, knowledge distillation methods, exemplified by techniques like progressive simplification by student models [15, 16], aim to streamline existing models. Some recent studies combine model compression with distillation to achieve faster sampling [37, 38]. Orthogonal to these approaches, we introduce a novel method for enhancing sampling efficiency in DMs inference. We show that our method can be combined with several existing speed-up methods for further acceleration.
DeepCache[22) and CacheMe|[2] are two recent works that leverage feature similarity to achieve acceleration. DeepCache [22] adopts strategy of rudely reusing features cached from the previous step, requiring iterative denoising. Furthermore, CacheMe [23] requires additional fine-tuning for
3

---

M sample
#-1
21
21%
BR
M
D
422
sample
(a) Standard SD sampling
(b) UNet architecture
propagation
M sample
D
71
~ samplle
41
11
(+++1
H1
Parallei encoder propagation.
(d) Decoder propagation
(e) Non-uniform encoder propagation
Figure 4: (a) Standard SD sampling. (b) UNet architecture.
(c) Encoder propagation. We omit
the encoder at certain adjacent time-steps and reuse in parallel the encoder features in the previous time-steps for the decoder. Applying encoder propagation for uniform strategy every two iterations.
Note, at time-step t-1, predicting noise does not require 2,.1 (i.e., Eq. 1: 2
V a--1 / = - 1, c)). (d) Decoder propagation. The generated images often fail cover some specific objects in the text prompt. For example, given one prompt case "A man with a beard wearing glasses and a beanie", this method fails to generate the glasses subject. See Appendix F for quantitative evaluation. (e) Applying encoder propagation for non- uniform strategy. By benefiting from our propagation scheme, we are able to perform the decoder in parallel at certain adjacent time-steps.
va--2 (v 1VV=L11
better performance. In contrast, our approach enables parallel processing, leading to considerably faster inference.
3 Method
We first briefly revisit the architecture of the Stable Diffusion (SD) (Sec. 3.1), and then conduct comprehensive analysis for the hierarchical features of the UNet (Sec. 3.2). Our analysis shows that possible to parallelize the diffusion model denoising process partially. Thus, we introduce a novel method to accelerate the diffusion sampling while still largely maintaining the generation quality and fidelity (Sec. 3.3).
3.1 Latent Diffusion Model
In the diffusion inference stage, the denoising network takes as input text embedding latent code 2t and time embedding, predicts noise, resulting in latent zt-1 using the DDIM scheduler [13]:
2t /or- -V
-eee--oolze.... t0(2),t,c).
(1)
where is predefined scalar function at time-step =t= T',.., .......... 1). The typical denoising network uses a UNet-based architecture. It consists of an encoder E, bottleneck B, and a decoder D, respectively (Fig. 4b). The hierarchical features extracted from the encoder & are injected into the decoder D by a skip connection (Fig. 4a). For convenience of description, we divide the UNet network into specific blocks: & = {E(-),}, B = {B(-)s), ), and D {D))),), where {8, 16, 32, 64) (see Fig. 4b). Both E()), and D(-), represent the block layers with input resolution in both encoder and decoder, respectively.
Diffusion Transformer (DiT) [21] is novel architecture for diffusion models. It replaces the UNet backbone with a transformer, which consists of 28 blocks. Based on our observations, we define the first 18 blocks as the encoder, and the remaining 10 blocks as the decoder (see Appendix A.3).
3.2 Analyzing the UNet in Diffusion Model
In this section, we take the UNet-based diffusion model as example to analyze the properties of the pretrained diffsuion model. We delve into the UNet which consists the encoder E, the bottleneck B,
Once replace the with specific inputs in E()).. s. we define that it represents the feature of E(-).

---

and the decoder D, for a deeper understanding of the different parts of the UNet. Note the following observed properties also exist in DiT (see Appendix A.3).
Feature evolution across time-steps. We experimentally observe that the encoder features exhibit a subtle variation at adjacent time-steps, whereas the decoder features exhibit substantial variations across different time-steps (see Fig. 3a and Fig. 2). Specifically, given a pretrained diffusion model, we iteratively produce a latent code 2t (see Eq. 1), and the corresponding hierarchical features: {E(ze, C, ). [B(2,, C, 1)8}, and (D(z,, (),} (s E {8, 16,32, 64)) as shown in Fig. 4b. Here, we analyze how the hierarchical features change at adjacent time-steps. To achieve this goal, we quantify the variation of the hierarchical features as follows:
(e कxेड (1E(,,, C, (). 1( 1):1
(2)
where d represents the number of channels in E(z,, c, t).. Similarly, we also compute ABC-js and AD(-).
As illustrated Fig. 3a, for both the encoder & and the decoder D. the curves exhibit a similar trend: in the wake of initial increase. the variation reaches a plateau and then decreases, followed by a continuing growth towards the end. However, the extent of change in AEC), and AD(), is quantitatively markedly different. For example, the maximum value and variance of the Ae(.), less than 0.4 and 0.05, respectively (Fig. 3a (zoom-in area)), while the corresponding values of the Ap..), are about 5 and 30, respectively (Fig. 3a). Furthermore, we find that Ap(-)ser the change of the last layer of the decoder, is close to zero. This is due to the output of the denoising network being similar adjacent time-steps [39]. In conclusion, the overall feature change Ae()) smaller than (D).). throughout the inference phase.
Feature evolution across layers. We experimentally observe that the feature characteristics are significantly different between the encoder and the decoder across all time-steps. For the encoder the intensity of the change is slight, whereas it is very drastic for the decoder D. Specifically we calculate the Frobenius norm for hierarchical features E(z,, C, f),, across all time-steps, dubbed as FE(). = (FE(er.o.T),,..... Fete,,e.)) }. Similarly, we compute FB())s and FD(),, respectivel.
Fig. 3b shows the feature cvolution across layers with a boxplot S. Specifically, for {Fe(),) and {FB(). }. the box is relatively compact, with a narrow range between their first-quartile and third- quartile values. For example, the maximum box height (F)(-) of these features is less than 5 (see Fig. 3b (zoom-in area)). This indicates that the features from both the encoder & and the bottleneck B slightly change. In contrast, the box heights corresponding to (D(-)x) are relatively large. For example, for the D(.)6s the box height over 150 between the first quartile and third quartile values (see Fig. 3b). Furthermore, we also provide standard deviation (Fig. 3c), which exhibits similar phenomena to Fig. 3b. These results show that the encoder features have relatively small discrepancy and a high degree of similarity across all layers. However, the decoder features evolve drastically.
Could we omit the Encoder at certain time-steps? As indicated by the previous experimental analysis, we observe that, during the denoising process, the decoder features change drastically, whereas the encoder & features change minimally, and have a high degree of similarities at certain adjacent time-steps. Therefore, as shown in Fig. 4c, We propose to omit the encoder at certain time-steps and use the same encoder features for several decoder steps. This allows us to compute these multiple decoder steps in parallel.
Specifically, we delete the encoder at time-step / (t - T), and the corresponding decoder (including the skip connections) takes input the hierarchical outputs of the encoder & from the previous time-step f. instead of the ones from the current time-step t like the standard SD sampling (for more detail, see Sec. 3.3).
When omitting the encoder at a certain time-step, we are able to generate similar images (Fig. 4c) like standard SD sampling (Fig. 4a, Tab. I (the first and second rows) and additional results in Appendix F). Alternatively, if we use a similar strategy for the decoder (i.e., decoder propagation). we find the
"The feature resolution is half of the previous one in the encoder and two times in the decoder. Note that the feature resolutions of E()s. B()ss and D(.)ss do not change in the SD model.
'Each boxplot contains the minimum (Uth percentile), the maximum (100th percentile), the median (50th percentile). the first quartile (25th percentile) and the third quartile (75th percentile) values of the feature Frobenius norm (e.g.. (Feley Fe Fe(e),)1))))
5

---

generated images often fail to cover some specific objects in the text prompt (Fig. 4d). For example, when provided with prompt "A man with a beard wearing glasses and beanie", the SD model fails to synthesize "glasses" when applying decoder propagation. This is due to the fact that the semantics are mainly contained in the features from the decoder rather than the encoder [40].
The encoder propagation, which uses encoder outputs from previous time-step as the input to the current decoder, could speed up the diffusion model sampling at inference time. In the following Sec. 3.3, we give further elaborate on encoder propagation.
3.3 Encoder propagation
Diffusion sampling, combining iterative denoising with transformers, is time-consuming. Therefore we propose a novel and practical diffusion sampling acceleration method. During the diffusion sampling process t= (T,,.. 1}, we refer to the time-steps where encoder propagation is deployed, t
1 as non-key time-steps denoted as gnonskey monkkey ..... -N.1 j. The remaining time-steps are
dubbed as 1 (kev= { key ( ..-.. -..-----NN J. In other words, we do not use the encoder at time-steps knon and leverage the hierarchical features of the encoder from the time-step gkey, Note we utilize the encoder at the initial time-step (1 = T). Thus, the diffusion inference time-steps could be reformulated as {they ,n gnon-kry), } where tkey U mony (T,.... 1] and they n gnon-key Ø. In the following, we introduce both uniform encoder propagation and non-uniform encoder propagation strategies.
As shown in Fig. 3a, The encoder feature change is larger in the initial inference phase compared to the later phases throughout the inference process. Therefore, we select more key time-steps in the initial inference phase, and less key time-steps in later phases. We experimentally define the key time-steps as (5e 49, 48, 47, 45, 40, 35, 25, 15) for SD model with DDIM, and { 100, 99, 98. 92, 91, 90, 85, 80, ..... 25, 20, 15, 14, 13..... 2, 1 {50, .9. 2, }} and {75, 73, 70, 66. 61, 55, 48, 40, 31, 21, 10) for three stage of DeepFloyd-IF (see detail key time-steps selection in Appendix F.2.. The remaining time-steps are categorized as non-key time-steps. We define this strategy as non-uniform encoder propagation (see Fig. 4e). As shown in Fig. 4c, we also explore the non-key time-step selection with fix stride (e.g, 2), dubbed as uniform encoder propagation.
Note that method does not reduce the number of sampling steps. During encoder propagation, the decoder computes for all time-steps, necessitating time embedding inputs for each time-step to maintain temporal coherence (see detail in Appendix D).
Tab. 5 reports the results of the ablation study, considering various combinations of key and non-key time-steps. These results indicate that the set of non-um non-uniform key simessteps performs better in generating images.
Parallel non-uniform encoder propagation. When applying the non-uniform encoder prop- agation strategy, at time-step / E ( mon-key the decoder inputs do not rely on the encoder out- puts at time-step (see Fig. 4c). Instead, it relies on the encoder output at the previous nearest key time-step. This allows us to perform par- allel non-uniform encoder propagation at these adjacent time-steps in koon-key, key, We perform de- coding in parallel from & to / - k + 1 time-steps. This technique further improves the inference efficiency since the decoder forward indicate this as parallel-bhe -bath non-key time- stees. As shown in Fig. 5 (right), this further reduces evaluation time by 41% for the SD model.
SID MOGIIN
Encodes pnopaattion
pRopegakion
Figure 5: Comparing with SD (left), encoder prop- agation reduces the sampling time by 24% (mid- dle). Furthermore, parallel encoder propagation achieves a 41% reduction in sampling time (right).
multiple time-steps could be conducted concurrently. We
Prior noise injection. Although the encoder propagation could improve the efficiency in the inference phase, we observe that it leads to slight loss of texture information in the generated results (see Fig. 6 (left, middle)). Inspired by related works [41, 42], we propose a prior noise
'The ellipsis in key denotes each time-step between the time-steps on either side of the ellipsis. For example. 80...25 means that every time-step between 80 and 25 is included.

---

injection strategy. It combines the initial latent code 2T into the generative process at subsequent time-step (i.e., 2,),, following 2, 2 + a 2T, if т, where a 0.003 is the scale parameter to control the impact of 27. And we start to use this injection mechanism from 7 = 25 step. This strategic incorporation successfully improves the texture information. Importantly, demands almost negligible extra computational resources. We observe that the loss of texture information occurs in all frequencies of the frequency domain (see Fig. 6 (right, red, and blue curves)). This approach ensures a close resemblance of generated results in the frequency domain to both SD and äт injection (see Fig. 6 (right, red and green curves)), with the generated images maintaining the desired fidelity (see Fig. 6 (left, bottom)).
4 Experiments
In our experiments, we assess the speed-up of our method compared to others for in- ference acceleration. We also explore com- bining our method with these approaches. We do not directly compare our method with distillation methods, which offer su- perior results but involve computationally expensive retraining.
Injecton
Ours ni by irieection
by Fredoo
by Eyectten
Datasets and evaluation metrics. We randomly select 10K prompts from the MS- MS-COCO2017 validation dataset [43] and feed them into the text-to-image diffusion model to obtain 10K generated images. For the transformer architecture diffusion model, we randomly generate 50K images from 1000 ImageNet [44] class labels. For other tasks, we use the same settings baselines (e.g.. VideoOidoo [4], VideoFusion [5], Dream- booth [7] and ControlNet [10]). We use the Fréchet Inception Distance (FID) [45] metric to assess the visual quality of the generated images, and the Clipscore [46] to measure the consistency between image content and text prompt. Furthermore, we report the average values for both the computational workload (GFLOPs/image) and sampling time (s/image) to represent the resource demands for single image. See more detailed implementation information on Appendix A.
Presuency
Figure 6: (left) We keep the image content through 2T injection, slightly compensating for the texture informa- tion loss caused by encoder propagation. (right) The amplitudes of the generated image through 27 injection closely resemble those from SD.
4.1 Text-to-image Generation
We first evaluate the proposed encoder propaga- tion method for the standard text-to-image gen- cration task on both the latent space (i.c., SD) and pixel space (i.e.. DeepFloyd-IF) diffusion models. As shown in Tab. 1, we significantly accelerate the diffusion sampling with negligi- ble performance degradation. Specifically, our proposed method decreases the computational burden (GFLOPs) by a large margin (27%) and greatly reduces sampling time to 41% when compared to standard DDIM sampling in SD. Similarly, in DeepFloyd-Le IF, the reduction in both computational burden and time reaches 15% and 24%, respectively. Furthermore, our method can be combined with the latest sampling techniques like DPM-Solver [14], DPM-Solver++ [47], and ToMe [35]. Our method enhances sampling ef- ficiency while preserving good model perfor-
Table I: Quantitative evaluation 2 for both SD and DeepFloyd-IF IF diffusion models.
s/image DM
Sampling Method
FID Clip-
GFLOPS image
score
DM
DDIM DDMM w/ Ours
50.755
37050
2.23
2.12
Bobble Diltusion
50
21.08 0.783
27350 279
1,21
46994
1.42 1.42219%
20
20.36. 0.780
14821
0.90
1.14
BPAL-Sovee
w/ Ours
20
21.221...000
0.779
11743 919
0.46
0.64
43%
RDDP- Solveree DPM--olvve++
20 20.51
0.782
0.782
14821
0.90
1.13
Ours
20.76
0.781
11743
0.64
DDIM + ToMe 50 20.2..2
0.782
35123
2.07
2.26
DDIM
Ours
so
20.73
0.781
26053
1.15 44%
1.33 21
DDPM DDPM
225
89 0.783
734825
33.91
34.15
w/ Ours
22523.73
0.782
626523,
25.61255%
26.27
DPM Solver 10000.79
0.784
330025
15.19
16.09
DPM--Sller Wol--oolerr w/ Ours
10000.85 0.785
(313381 [S% 12.02
12.97
20904
mance, with negligible variations of both FID and Clipscore values (Tab. I (the third eighth rows)).
Our method achieves good performance across different sampling steps (Fig. 7 and see Appendix D for quantitative results.). Importantly, these results show that our method is orthogonal and compatible
We use the official implementation of Clipscore [46] to obtain around 0.75 but around 0.3. See Appendix C.
7

---

Table 2: Comparison with DeepCache and
Table 3: Quantitative evaluation for DIT.
CacheMe. CacheMe is not open-source.
Sampling Method
Image Res.
FID 1sFID
Precision
RReaall
shimge
Sampling Method
T |Parallel| FID
Clipscore t s/image
DDIM w/ DeopCache 50
DDIM
21.53
21.75
0.770
0.773
1.00
2.42
DIT w/ Oun
DiT 250
250
256
256
233
2:27
4.60
4,55
276.05
278.24
0.83
0.82
0.57
0.57
3.622%%
5.13
DDIM w/ CacheMe
50
4.30
DIT w/ W/Omm Durs 250
250
512
3.25
3.04
500
5,05
240
245.13
0.84
0,83
0.54
0.51
17.353491
DDIM
50
21.62
0.775
0.50
Table 4: Quantitative evaluation on text-to- personalized generation and seerence-guided generation tasks. i and I indicate "edges" and "scribble" conditions, respectively.
Table 5: Quantitative evaluation in various propa- gation strategies on MS-COCO 2017 10K subset. FTC=FID TinFIDDXTie//Ciscccor..
Method
T FID scoret
Clip-
GFLOPS image.
simagel Unst SD
Propogaaton
strslegy
Clipsoore
GFLOPs
Ammge
Unet
shinage
PTC J
SD
SD
21.75
0,773
37080
2.23
2.42
68.1
Text2 zero
0.732
12.59/8
13,65/8
Tex22 Vidco-zcro
Ours
50
0.731
20690
2.568
10.54/ 25%
1
26, 24,22, 20, 18. 10, 12, 10, 6,4, 4.2)
[60, 48, 46, 44, 42, 4tl, 28, 36, 144, 31, (30, 28.
VideoFusion
224700
16.71/16
17.93/16
Caifeem
21,55
0.775
31011 Le9r... 1.62g7914
1.8) 28115
50.3
VideoFusion Ours
33%
12.2/16
+fr os [50, 44, 38, 32, 26, 20. 14, 2,
0.773
27335
1.26,p96
1.46,095
40.7
ControlNet Ours
ControlNet (+) 13.78 0.769
50 14.66 0.767
31400
49500
1963
3.09
1.52,1 %
3.20
111
24.81
0.766
26370
450, (8, 26, 14,2)
1. 1.36
43.7
A = (50, 40, 39, 38, 30, 25, 20, 15,6)
ControlNet (2) 50 16.17 0.775
56850
3.85
3.95
ControlNet (7)
50 16.42 0.775
35990
1.83
1.93,1 %
22.94
0.776
27335
1.26
1. 1.42419
41.9
Ours
Nonanifom
11
[50, 25, 20, 15, 5,,5,4,3
Dreambooth Drcambooth Ours
50
0.640
37050
2.23
2,42
35,25
0.742
277350
1525
1. 1.4244994
67.4
50
0.660
27350
1.21 (59
1.42, 96
+ = [50, A 37, (35, 22, 21, 18, 14,65
111
22.14
0.778
27350
1.222 1. 1.42am.
40.4
CustemDiifisioo.50
CustemDiffasion Ours
0.640
37050
2,21
2,42
IV (Durs)
s [50, 49, 47, 45,40, 35,25, 151
27764
11.6
21.08
0.783
27350arx__
1.21ap9r4 1. 1.42am
38.2
with these acceleration techniques. As shown in Fig. 1, we visualize the generated images with different sampling techniques. Our method still generates high-quality results (see Appendix F for additional results).
Our method allows to use multi-GPU to generate one image. With multi- GPU parallel, our proposed method further accelerates the SD sampling by 77%. whereas DeepCache [22] and CacheMe [23]
achieve speedups of 56% and 44%. respectively (See Tab. 2). These results indicate that we achieve superior acceleration compared to DeepCache [22] and CacheMe [23].
4.2 Diffusion Transformer
We also evaluate our approach on DiT. As reported in Tab. 3, we achieve accelerations of about 29% and 34% for DiT sampling with image resolution of 256 and 512, respectively, while preserving high-quality results (see Figs. I and 18).
4.3 Other tasks with text-guided diffusion model
Besides the standard text-to- text-to-image task, we also validate our proposed approach on other tasks: text-to-video generation, personalized generation, and reference- guided image generation.
Text-to-video. To evaluate our method, we combine with both Text2Video-zer [4] and VideoFu- sion [5]. As reported in Tab. 4 (the second and fourth rows), when combined with our method, the two methods have a reduction of approximately 22% 33% in both computational burden and generation time. These results indicate that we are able to enhance the efficiency of generative processes in the text-to-video task while preserving video fidelity at the same time (Fig. 1 (left, bottom)). As an example, when generating video using the prompt "Fireworks bloom in the night sky", the VideoFusion model takes 17.92 seconds with 16 frames for the task (1. (1.f2//fr,me) when combined with our method it only takes 12.27s (0.76s/frame) to generate high-quality video (Fig. I (left, bottom)).
Personalized image generation. Dreambooth [7] and Custom Diffusion [8] are two approaches for customizing tasks by fine-tuning text-to-image diffusion models. As reported in Tab. 4 (the ninth to twelfth rows), our method, combining with the two customization approaches, accelerates image
8

---

48%
44%
51%
52%
46%
39%
52%
56%
54%
61%
I0so (DOM loyu (00M)
Controlte: IiPrrond nod
so mum
Rot sasiddo
Persoolla and
3PM.
Figure 8: User study results.
Figure 7: Generated images at different time-steps.
generation and reduces computational demands. Visually, it maintains the ability to generate images with specific contextual relationships based on reference images. (Fig. (right))
Reference-guided image generation. ControlNet [10] incorporates a trainable encoder, successfully generates text-guided image, and preserves similar content with conditional information. Our approach can be applied concurrently to two encoders of ControNet. In this paper, we validate the proposed method with two conditional controls: edge and scribble. Tab. (the fifth eighth row) reports quantitative results. We observe that it leads to a significant decrease in both generation time and computational burden. Furthermore, Fig. (middle, bottom) qualitatively shows that our method successfully preserves the given structure information and achieves similar results as ControlNet.
User study. We conducted user study, as depicted in Fig. 8, and asked subjects select results. We apply pairwise comparisons (forced choice) with 18 users (35 pairs of images or videos/user). The results demonstrate that our method performs equally well the baseline methods.
4.4 Ablation study
We ablate the results with different selections of both uniform and non-uniform encoder prop- agation. Tab. 5 reports that the performance of the non-uniform setting outperforms the uni- form one in terms of both FID and Clipscore (see Tab. 5 (the third and eighth rows)). Further- more, we explore different configurations within the non- non-uniform strategy. The strategy, using the set of key time-steps we established, yields better results in the generation process (Tab. (the eighth row)). We further present qualita- tive results stemming from the above choices. As shown in Fig. 9, given the same number of key time-steps, the appearance of nine-step non- uniform strategy I, III and IIIIII settings do not align with the prompt "Fireflies dot the night sky". Although the generated image in the two- step setting exhibits a pleasing visual quality, its sampling efficiency is lower than our chosen setting (see Tab. 5 (the second and eighth rows)).
Uniform
(DDIM)
[Vtuurs
Figure 9: Generating image with uniform and non- uniform encoder propagation. The result of uni- form strategy II yields smooth and loses textual compared with SD. Both uniform strategy III and non-uniform strategy T, III and III generate images with unnatural saturation levels.
Effectiveness of prior noise injection. We evaluate the effectiveness of injecting initial 2T. As reported in Tab, 6, the differences in FID and Clipscores without 24 (the third column), when compared to DDIM and Ours (the second and fourth columns), are approximately 0.01%, which can be considered negligible. While this not the case for the visual expression of the generated image, it is observed that the output contains complete semantic information with smoothing texture (refer to Fig. 6 (left, the second row)). Injecting the 27 aids maintaining fidelity in the generated results during encoding propagation (see Fig. 6 (left, the third row) and Fig. 6 (right, red and green curves)).
Table 6: Quantitative evaluation for prior noise injection.
Sumpling Method
SD (DDIM) Ours SD (DDIM) Ours w/o By injection
(DDIM)
w/ 2g injection
FID 4
21.75
21.71
21.08
Clipscore +
0.773
0.779
0.783

---

Conclusion
In this work, We explore the characteristics of the encoder and decoder in UNet of the text-to- rext-to-image diffusion model and find that encoder feature variation minimal for many time-steps, while the decoder plays significant role across all time-steps. Building upon this finding, we propose encoder propagation for efficient diffusion sampling, reducing time on both the UNet-based and the transform- based diffusion models on a diverse set of generation tasks. We conduct extensive experiments and validate that our approach can achieve improved sampling efficiency while maintaining image quality, Limitations: Although our approach achieves efficient diffusion sampling, it faces challenges in generating quality when using a limited number of sampling steps (e.g.. 5). In addition, even though our proposed parallelization can also be applied to network distillation approaches [18, 17, 19], we have not explored this direction in this paper and leave to future research.
Acknowledgements
This work was supported by NSFC (NO. 62225604) and Youth Foundation (62202243). We acknowledge project PID2022- 1-2---13277BBBI00 I00, financed by the Spanish Government MCIN/AEE/10. 13039/50110 10001 1033 and FEDER. We acknowledge "Science and Technology Yongjiang 2035" key technology breakthrough plan project (2024Z120). Computation is supported by the Supercomputing Center of Nankai University (NKSC).
We would like to thank Kai Wang, postdoctoral researcher at the Computer Vision Center, Universitat Autònoma de Barcelona, for his helpful discussions and comments during the rebuttal period.
References
[1] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684-10695, 2022.
[2] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systemx, 33:6840-6855 6851, 2020.
[3] Yang Song, Jascha Sohl-Dickstein, Diederik Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. arXiv preprint 1.. 2020.
[4] Levon Khachatryan, Andranik Movsisyan, Vahram Tadevosyan, Roberto Henschel, Zhangyang Wang. Shant Navasardyan, and Humphrey Shi. Text2vidco zzro: Text-to-image diffusion models are zero-shot video generators. arXiv preprint arXiv:2303. 13439, 2023.
[5] Zhengxiong Luo, Dayou Chen, Yingya Zhang, Yan Huang, Liang Wang, Yujun Shen, Deli Zhao, Jingren Zhou, and Tieniu Tan. Videofusion: Decomposed diffusion models for high-quality video generation. In Proceedings the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10209- 10218, 2023.
[6] Michal Geyer. Omer Bar-Tal, Shai Bagon, and Tali Dekel. Tokenflow: Consistent diffusion features for consistent video editing. arXiv preprint arxiv:2307. 10373, 2023.
[7] Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yacl Pritch, Michael Rubinstein. and Ktir Aberman. Dream- booth: Fine tuning text-to-image diffusion models for subject-driven generation. Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 22500-22510, 2023.
[8] Nupur Kumari, Bingliang Zhang, Richard Zhang, Eli Shechtman, and Jun-Yan Zhu. Multi-concept customization text-to-image diffusion. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1931-1941, 2023.
[9] Rinon Gal, Yuval Alaluf, Yuval Atzmon, Or Patashnik, Amit Bermano, Gal Chechik, and Daniel Cohen-Or Or. An image is worth one word: Personalizing text-to-image generation using textual inversion. arXiv preprint arxiv:200 01618, 2022.
[10] Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 3836-3847, 2023.
10

---

[11] Chong Mou, Xintao Wang. Liangbin Xie, Jian Zhang, Zhongang Qi. Ying Shan, and Xiaohu Qie. T2i- adapter: Learning adapters to dig out more controllable ability for text-to-image diffusion models. arXiv preprint arXiv:2302.000 08453, 2023.
[12] Yuheng Li, Haotian Liu, Qingyang Wu, Fangzhou Mu, Jianwei Yang, Jianfeng Gao, Chunyuan Li, and Yong Jae Lee. Gligen: Open-set grounded text-to-image generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2 2023.
[13] Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint urXiv:2010.02502. 2020.
[14] Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. Dpm-solver: A fast ode solver for diffusion prohabilistic model sampling in around 10 steps. Advances in Neural Information Processing Systems, 35:5775-5787, 2022.
[15] Tim Salimans and Jonathan Ho. Progressive distillation for fast sampling of diffusion models. arXiv preprint arXi:
2022.
[16] Chenlin Meng, Robin Rombach, Ruiqi Gao, Diederik Kingma, Stefano Ermon, Jonathan Ho, and Tim Salimans. On distillation of guided diffusion models. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14297 14306, 2023.
[17] Simian Luo. Yiqin Tan, Longbo Huang. Jian Li, and Hang Zhao. Latent consistency models: Synthesizing high-resolution images with stw-step inference. arXiv preprint urXiv:2310.04378, 2023.
[18] Axel Sauer, Dominik Lorenz, Andreas Blattmann, and Robin Rombach. Adversarial diffusion distillation. arXiv preprint arXiv:2311. 2023.
[19] Thuan Hoang Nguyen and Anh Tran. Swiftbrush: One-step text-to-image diffusion model with variational score distillation. arXiv preprint ar..0:2312..55299, 2023.
[20] Narck Tumanyan, Michal Geyer, Shai Bagon, and Tali Dekel. Plug-and-play diffusion features for text- driven image-to-image translation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1921-1930, 2023.
[21] William Peebles and Saining Xie. Scalable diffusion models with transformers. arXiv preprint arXiv:2212.09748, 2022.
[22] Xinyin Ma. Gongfan Fang, and Xinchao Wang. Deepcache: Accelerating diffusion models for free, arXiv preprint arXi: 2023.
[23] Felix Wimhauer, Bichen Wu, Edgar Schoenfeld, Xiaoliang Dai, Ji Hou, Zijian He, Artsiom Sanakoyeu, Peizhao Zhang, Sam Tsai, Jonas Kohler, et al. Cache me if you can: Accelerating diffusion models through block caching. arXiv preprint arXiv:2312.03200 2023.
[24] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li. Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, al. Photorealistic text-to- image diffusion models with deep language understanding. Advances in Neural Information Processing Systems, 35:36479-36444 2022.
[25] Yogesh Balaji, Seungjun Nah, Xun Huang, Arash Vahdat, Jiaming Song, Karsten Kreis, Miika Aittala, Timo Aila. Samuli Laine, Bryan Catanzaro, al. ediffi: text-to-image diffusion models with ensemble of expert denoisers. arXiv preprint arXiv:2211.01324 2022.
[26] Xiaoliang Dai, Ji Hou, Chih-Yao Yao Ma, Sam Tsai, Jialiang Wang, Rui Wang, Peizhao Zhang, Simon Vandenhende, Xiaofang Wang, Abhimanyu Dubey, et al. Emu: Enhancing image generation models using photogenic needles in haystack. arXiv preprint arXiv:2309. 15807, 2023.
[27] Amir Hertz, Ron Mokady, Jay Tenenhaum, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or Or. Prompt-to- prompt image editing with cross attention control. arXiv preprint arXiv:2208. 026666 2022.
[28] Senmao Li, Joost van de Weijer. Taihang Hu, Fahad Shahbaz Khan, Qibin Hou, Yaxing Wang, and Jian Yang. Stylediffusion: Prompt-embedding inversion for text- based editing. arXiv preprint arXiv:2303 15649, 2023.
[29] Jianyi Wang. Zongsheng Yue, Shangchen Zhou, Kelvin CK Chan, and Chen Change Loy. Exploiting diffusion prior for real-world image super-resolution arXiv preprint 50X00 2023.
11

---

[30] Hyungjin Chung. Byeongsu Sim, and Jong Chul Ye. fonee-cloer-ddifus---sssee Accelerating conditional diffusion models for inverse problems through stochastic contraction. In Proceedings the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12413-12422, 2022.
[31] Dmitry Baranchuk, Ivan Rubachev, Andrey Voynov, Valentin Khrulkov, and Artem Babenko. Label- efficient semantic segmentation with diffusion models. arXiv preprint 1rXiv:21/2.0//1 2021.
[32] Julia Wollch, Robin Sandkühler, Florentin Bieder, Philippe Valmaggia, and Philippe C Cattin. Diffusion models for implicit image segmentation ensembles. In International Conference on Medical Imaging with Deep Learning, pages 1336-1348. PMLR, 2022.
[33] Walter HL Pinaya, Mark Graham, Robert Gray, Pedro F Da Costa, Petru-Daniel Daniel Tudosiu, Paul Wright, Yee H Mah, Andrew MacKinnon, James T Teo, Rolf Jager, et al. Fast unsupervised brain anomaly detection and segmentation with diffusion models. In International Conference on Medical Image Computing and Compater-Assisted Intervention, pages 705-714. Springer, 2022.
[34] Shoufa Chen, Peize Sun, Yibing Song, and Ping Luo. Diffusiondet: Diffusion model for object detection. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 19830-19843, 2023.
[35] Daniel Bolya and Judy Hoffman. Token merging for fast stable diffusion. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4598 4602, 2023.
[36] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
[37] Bo- Bo-Kyeong Kim, Hyoung-Kya Song, Thibault Castells, and Shinkook Choi. On architectural compression of text-to-image diffusion models. arXiv preprint arXiv:2305.15795 2023.
[38] Yanyu Li, Huan Wang, Qing Jin, Ju Hu, Pavlo Chemerys, Yun Fu, Yanzhi Wang, Sergey Tulyakov, and Jian Ren. Snapfusion: Text-to-image diffusion model on mobile devices within two seconds. arXiv preprint urXiv:2306. 00980. 2023.
[39] Daiki Miyake, Akihiro Iohara, Yu Saito, and Toshiyuki Tanaka. Negative-prompt inversion: Fast image inversion for editing with text-guided diffusion models. arXiv preprint arXiv:2305.16800 2023.
[40] Guanqi Zhan, Chuanxia Zheng, Weidi Xie, and Andrew Zisserman. What does stable diffusion know about the 3d scene? arXiv preprint arXiv:2310.068. 2023.
[41] Jooyoung Choi, Jungbeom Lec, Chachun Shin, Sungwon Kim, Hyunwoo Kim, and Sungroh Yoon. Perception prioritized training of diffusion models. In Proceedings of the JEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11472 11481, 2022.
[42] Mingi Kwon, Jaeseok Jeong, and Youngjung Uh. Diffusion models already have semantic latent space. arXiv preprint arXiv:2210.10960 2022.
[43] Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, pages 740 7000-555.60 Springer, 2014.
[44] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. Advances in neural information processing systems, 25, 2012.
[45] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by two time-scale update rule converge to a local nash equilibrium. Advances in neural information processing systems, 30, 2017.
[46] Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. Clipscore: A reference-free evaluation metric for image captioning. arXiv preprint 10Xiv:2104.087/4 18, 2021.
[47] Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. Dpm-solver++: Fast solver for guided sampling of diffusion probabilistic models. arXiv preprint arXiv:2211.0000 2022.
[48] Nupur Kumari, Bingliang Zhang, Sheng- Yu Wang, Eli Shechtman, Richard Zhang, and Jun-Yan Zhu. Ablating concepts in text-to-image diffusion models. In Proceedings of the IEEÉ/CVF International Conference on Computer Vision, pages 22691-22702. 2023.
12

---

[49] Tero Karras, Miika Aittala, Timo Aila. and Samuli Laine. Elucidating the design space of diffusion-based generative models. In Proc. NeurIPS, 2022.
[50] Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. Advances in neural information processing systems, 32. 2019.
[51] Jiazheng Xu, Xiao Liu, Yuchen Wu, Yuxuan Tong, Qinkai Li, Ming Ding, Jie Tang, and Yuxiao Dong. Imagereward: Learning and evaluating human preferences for text-to-image generation, 2023.
[52] Yuval Kirstain, Adam Polyak, Uriel Singer, Shahbuland Matiana, Joe Penna, and Omer Levy. Pick-a-pic: An open dataset of user preferences for text-to-image generation. 2023.
[53] Ziqi Huang, Tianxing Wu, Yuming Jiang, Kelvin C.K. Chan, and Ziwei Liu. ReVersion: Diffusion-based relation inversion from images. arXiv preprint arXiv:2303. 13495, 2023.
[54] Enshu Liu, Xuefei Ning, Zinan Lin, Huazhong Yang, and Yu Wang. Oms-dpm: Optimizing the model schedule for diffusion probabilistic models. In International Conference Machine Learning, pages 21915-21936. PMLR, 2023.
[55] Lijiang Li, Huixia Li, Xiawu Zheng, Jie Wu, Xuefeng Xiao, Rui Wang, Min Zheng, Xin Pan, Fei Chao, and Rongrong Ji. Autodiffusion: Training-free optimization of time steps and architectures for automated diffusion model acceleration. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 7105-7114, 7114,1112222 2023.
[56] Shuai Yang. Yukang Chen, Luozhou Wang, Shu Liu, and Yingcong Chen, Denoising diffusion step-aware models. arXiv preprint arXiv:2310.0333 2023.
13

---

Contents
Introduction
2 Related Work
3
Method
3.1 Latent Diffusion Model
3.2 Analyzing the UNet in Diffusion Model
3.3 Encoder propagation
Experiments
4.1
Text-to-image Generation
4.2
Diffusion Transformer
4.3
Other tasks with text-guided diffusion model
4.4
Ablation study
Conclusion
10
Appendix
15
Implementation Details
15
A.1 Configure
15
A.2 Details about the layers in the UNet
15
A.3 Details about the blocks the DIT
15
A.4 Time and memory consumption ratios
17
A.5 GFLOPs...
19
A.6 Baseline Implementations.
19
B Parameter Count and FLOPs of SD
20
The methodology for computing Clipscore
20
D Different from step-reduction methods
20
Difference between "prior noise injection" and "churn"
22
F Ablation Experiments and Additional Results
22
F11
Comparion with DeepCache
22
F2 The definition of key time-steps in various tasks
23
E3
The effectiveness of encoder propagation...
23
F.4
User study details
23
E5
Additional metrics
24
F.6
Additional results
25
14

---

F.7
Additional tasks
25
Automatic selection for key time steps
25
H Impact Statements
25
Appendix
In the appendix, provide detailed description of the experimental implementation (Appendix A). Subsequently, an analysis of the parameter quantities for the encoder and decoder is conducted (Appendix B).
Following that, we explain the methodology used to compute Clipscore (Appendix C), as well as highlight the distinction between our approach and methods that reduce sampling steps (Appendix D). Then, we present additional experiments and results, including more detailed ablation studies and comparative experiments (Appendix F). In the end, we provide statement of the potential broader impact our work (Appendix H).
A Implementation Details
As shown in Code. 1, in the standard SD sampling code, adding lines code with green comments can achieve encoder propagation.
Code 1: Encoder propagation for SD (DDIM)
from diffusers inport StabieDirfussooPPpp....
import torch
from inport reglater.parallet poglstar.ppralllllppppl.... register.
import package
model_id "runvaynlsstable- dit.olle-ddffuio---iidee
pipe StablebiffhucionPipplii httpaaiooo.pppaaaa..ooooerrrerre www.....rrrrriicccmeeeaeee (modeliid, torch_dtype-torch float16)
pipa ppip..to
pap.eer.ppplllllppppllpppl register poarsarrr unet) encoder propagation
2. enable parallal
pronpt photo astronaut riding
image pipe (prompt). images
image aave aastroonuut.. rides. horas. png
A.1 Configure
We use the Stable Diffuison v1.5 pre-trained model and DeepFloyd-IF 9, All of our inference experiments are conducted using an A40 GPU (48GB of VRAM).
We randomly select 100 captions from the MS-COCO 2017 validation dataset [43] as prompts for generating images. The analytical results presented Sec. 3.2 are based on the statistical outcomes derived from these 100 generated images.
A.2 Details about the layers in the UNet
The UNet in Stable Diffusion (SD) consists of an encoder E, a bottleneck B, and a decoder D, respectively. We divide the UNet into specifie blocks: & = (E(-))) where s E (8, 16, 32, 64). E()), and D(.), represent the block layers with input resolution s in the encoder and decoder, respectively. Tab. presents detailed information about the block architecture. Fig. 10 illustrates the hierarchical features for these blocks.
(B(-)s), and D (D(-).); }
A.3 Details about the blocks in the DiT
15

---

Table Detailed information about the layers of the encoder &, bottleneck B and decoder D in the UNet of SD.
Layer number
Input resolution of layer
OOtputt resolution of layer
UNet
Type layer
Layer name
resnets
down_blocks. www.bbllo...ssssssssssoooo O.rsseetss.
(320, 64, 64)
(320, 64, 64)
attention
down_blocks, S.akks.OO.amm O.tttentions.0
(320, 64, 64)
(320, 64, 64)
8()6a
resnet
down blocks. 1.rennts..
(320, 64, 64)
(320, 64, 64)
attention
down down_blocks. Dalccks...tttentions.
(320, 64, 64)
(320, 64, 64)
downsamplers down blocks. www.:oddddwwwsapplss.....
(320, 64, 64)
(320, 32, 32)
resnet
down_bllocke resmets.0
(320, 32, 32)
(640, 32, 32)
attention Nown_blocks. bloocks..attentionn..0
(640, 32, 32)
(640, 32, 32)
E()az
resnet
down_blocks.. Iresnets.
(640, 32, 32)
(640, 32, 32)
attention
down_block aattcttion..
(640, 32, 32)
(640, 32, 32)
downsamplers hown_blocks. .lo.ekssl...ww
(640, 32, 32)
(640, 16, 16)
10
resnet
down_ Nwwn_blocks 2rrsssstt
(640, 16, 16)
(1280, 16, 16)
atlention
down blocks.l..nnnt..... blockk.....tttenninno.... (1280, 16, 16)
(1280, 16, 16)
E()1
12
wwwn_blocks.22resnee
(1280, 16, 16)
(1280, 16, 16)
13
attention
down_ www.__locck..2aatcntiooo
(1280, 16, 16)
(1280, 16, 16)
14
downsamplers down_ www._bllcck....owwnsnii
(1280, 16, 16)
(1280, 8)
g())s
15
resnet
down blocks..
(1280, 8, 8)
(1280, 8, 8)
16
resnet
down reenbls.......esssss.ll
(1280, 8, 8)
(1280, 8, 8)
17
resnet
mid blocks.resnets..0
(1280, 8, 8)
(1280.
B
B(-)a
18
attention
mid www.bbllooccctiicoooooo
(1280, 8, 8)
(1280, 8, 8)
19
resnet
mnd_blocks.resnets..
(1280, 8, 8)
(1280, 8, 8)
20
resnet
up blocks. 0.rcks...resnets.O
(1280+1280, 8, 8)
(1280.
D(-)s
21
resnet
up_
(1280+1280, 8, 8)
(1280, 8, 8)
22
resnet
up_ Blockss..rresnpe
(1280+1280, 8, 8)
(1280, 8, 8)
23
upsamplers
up blwcks. blocks...upsamppeerss..
(1280, 8, 8)
(1280, 16, 16)
resnet
up_ hl_.block...nnm resnets.0 lp_blockss.Iresnets.0
(1280+1280, 16, 16)
(1280, 16, 16)
attention
up upp_bloks.. atteentinss 1
(1280, 16, 16)
(1280, 16, 16)
resnet
up ul__looks._iep Iresnest..
11280+1280, 16)
(1280, 16, 16)
D(-)1
27
atlention
up blocks. Ilockks...tteettinns...
(1280, 16, 16)
(1280, 16, 16)
28
resnet
up_ .resnets2
(1280+640, 16, 16)
(1280, 16, 16)
29
attention
up up__bloks.1l 1btecks...atcentions..2
(1280, 16, 16)
(1280, 16, 16)
30
upsamplers
up upp_block.11 b.l.cckk...ussapplerss.....
(1280, 16, 16)
(1280, 32, 32)
resnet
up blocks.2.
(1280+640, 32, 32)
(640, 32, 32)
D
attention
up 20o.cckk....
(640, 32, 32)
(640, 32, 32)
resnet
up ppp.blocks.2.rsnetts..
(640+640, 32,
(640, 32, 32)
D(-)aa
attention
mp__loccks.2aatieniionn
(640, 32, 32)
(640, 32, 32)
35
resnet
up_ ..essesks....sseet..22
(640+320,, 32, 32)
(640, 32, 32)
36
attention
up blaccks..2.ateettions.22
(640, 32, 32)
(640, 32, 32)
37
upsamplers
up_ hpp_blocks.2upsampphee
(640, 32, 32)
(640, 64, 64)
resnet
up up_ blocks.3.resnets..
(640+320, 64, 64)
(320, 64, 64)
39
attention
up pop.blccks...ttenniee
(320, 64, 64)
(320, 64, 64)
resnet
up blocks.3.resnets.1
(320+320, 64, 64)
(320. 64, 64)
D())4
41
attention
(320, 64, 64)
(320, 64, 64)
42
resnet
up_ b...bokk.....ssett.222
(320+320, 64, 64)
(320, 64, 64)
43
attention
up
(320, 64, 64)
(320, 64, 64)
16

---

Sib Ct
Ei city
Figure 10: Visualising the hierarchical features. We applied PCA to the hierarchical features following PnP [20] and used the top three leading components an RGB image for visualization. The encoder features changes minimally and has similarity at many time-steps (left), while the decoder features exhibit substantial variations across different time-steps (right).
Fig. 12 visualizes the hierarchical features for DiT [21] blocks, which includes 28 transformer blocks. Through visualization and statistical anal- ysis (see Fig. 12 and Fig. 11), we observe that the features in the first several transformer blocks change minimally (i.e., the first 18 blocks), similar to the Encoder SD, while the features in the remaining transformer blocks exhibit substantial variations (i.e., the remaining 10 blocks), akin the Decoder in SD. For ease of presentation, we refer to the first sev- eral transformer blocks of DiT as the Encoder and the remaining transformer blocks the Decoder. In Tab. 3, we demonstrate the accelerating performance of our method while applied to DiT-based generation models.
40
30
de 2
Figure 11: DiT feature statistics (F-norm)
A.4 Time and memory consumption ratios
We report the time and GPU memory consumption ratios for text-to-image task. As shown in Tab. 8, we significantly accelerate the diffusion sampling with encoder propagation while maintaining comparable memory demand the baselines (Tab. 8 (the last two columns)). Specifically, proposed method reduces the spending time (s/image) by 24% and requires a little additional memory compared standard DDIM sampling in SD (DDIM vs. Ours: 2.62GB vs. 2.64GB). The increased GPU memory requirement is for caching the features of the encoder from the previous time-step. Though applying parallel encoder propagation results in increase in memory requirements by 51%, it leads to a more remarkable acceleration of 41% (DDIM vs. Ours: 2.62GB vs. 3.95GB). In conclusion, applying encoder propagation reduces the sampling time, accompanied by a negligible increase in memory requirements. Parallel encoder propagation on text-to-image tasks yields sampling speed improvement of 20% to 43%, requiring an additional acceptable amount of memory.
Besides the standard text-to-image task, we also validate our proposed approach on other tasks: text-to-video generation (i.e., Text2Video-zero [4] and VideoFusion [5]), personalized generation (i.e., Dreambooth [7] and Custom Diffusion [8]) and reference-guite image generation (i.e., Con- trolNet [10]). We present the time and memory consumption ratios for these tasks in Tab. 9.
As reported in Tab, 9 (top), when combined with our method, there is a reduction in sampling time by 23% and 32% for Text2Video-zero [4] and VidcoFusion [5], respectively, while the memory requirements increased slightly by 3% (0.2GB) and 0.9% (0.1GGB).
The time spent by eeference-guided image generation (i.e., ControlNet [10]) is reduced by more than 20% with negligible increase in memory (1%). When integrated with our parallel encoder propagation, the sampling time in this task can be reduced by more than half (51%) (Tab. 9 (middle)). Dreambooth [7] and Custom Diffusion [8] are two approaches for customizing tasks by fine-tuning text-to-image diffusion models. As reported in Tab. 9 (bottom), our method, working in conjunction with the two customization approaches, accelerates the image generation with an acceptable increase in memory.
17

---

www.
Mee Ary
The Jook sem too 1O PORTIN
20N Sny STH ooo 37t doo 20r0AA SAHV SYRLING 2O0 BOO ScTW 32% SoO OPe PRN.O
1-50
1-40
4747
1-30
Figure 12: Visualising the hierarchical features. We apply PCA to the hierarchical features following PnP [20], and use the top three leading components as an RGB image for visualization. The output features of the first 18 blocks change slowly, and show similarity many time steps (top), while the ones in the remaining 10 blocks exhibit substantial variations across different time steps (bottom).
Note that our method, either utilizing encoder propagation or parallel encoder propagation, improves sampling speed without compromising image quality (Sec. 4.1 and Sec. 4.3). We
18

---

Table 8: Time and GPU memory consumption ratios in both SD model and DeepFloyd-1 IF diffu- sion model. t: Encoder propagation, I: Parallel encoder propagation.
Table 9: Time and GPU memory consumption ra- tios in text-to-video, personalized generation and reference-guided generated tasks. t: Encoder prop- agation, $ Parallel encoder propagation.
DM
Sampling Method
siimage
memory (GB)
Sampling Method
T
s/mmage
memory (GB)
DDIM [13]
2.42
2.62
Text2Vidoo-zoro [4]
13.65
6.59
DDIM [13]
1.82
2.64
Text2Video-zem [4]
50
20.54
6.79
w/ Ours
1.42,1%
3,95
w/ Ours
DPM-Solver [14]
1.14
2.62
VideoPusion
17.93
11.87
Stable Diffusion
DPM-Solver [14]
f
19921
2.64
VidcoFuusion [5] w/ Ours
50
12:27
11.98
w/ Ours
0.64
2.69
DPM-Solver++
1.13
2.61
CoutrolNet [10] (odges)
3.20
3.81
DPM-Solvcr++ (47]
1.91
2.68
ControlNer [10] (odges) w/
50
2.01
3.85
w/ Ours
0.64
2.68
Ours
1.52
5.09
DDIM
2.26
2.62
ControlNer [10] (scribble)
3.95
3,53
ToMe 135]
ControlNet [10] (scribble)
50
3.1820504
3.57
DDIM ToMe [35] w/ Ours
1.72
2.64
w/ Ours
1.93
4.45
.1964
3,95
Dreambooth [7]
2.42
2.61
DDPM [2]
34.55
40.5
Dieambooth [7] Ours
50)
1.81
2.65
DespFloyd-11
DDPM [2]
225
15
41.1
1.4241784
3.93
w/ Ours
26527
Custom Diffusion [8]
2.42
2.61
DPM Solvent [47]
16.09
40.5
Custom Diffusion [8] w/ Ours
50
1.82
2.64
DPM-Solver++ [47] w/ Ours
14.13
40.8
1.92
3.94
12.9720%4
conducted GPU memory consumption ratios using the official method provided by PyTorch, torch. cuda max_ aaaa__emorr_aallcaaeeooo which records the peak allocated memory since the start of the program.
A.5 GFLOPs
We use the !ycore!! library to calculate the GFLOPs required for single forward pass of the diffusion model. Multiplying this by the number of sampling steps gives us the total computational burden for sampling one image. Additionally, using fycore, we can determine the computational load required for each layer of the diffusion model. Based on the key time-steps set in our experiments, we subtract the computation we save from the original total computational load, which then represents the GFLOPs required by our method. Similarly, fvcore also supports parameter count statistics.
A.6 Baseline Implementations
For the comparisons of text-to-image generation in Sec, 4, we use the official implementation of
DPM-Solver [14], DPM-Solver++ [47] 12, and ToMe [35] For the other tasks with text- text-guided
diffusion model, we use the official implementation of -exoooideo [4] 14 VideoFusion [5] 15
ControlNet [10] Dreamboth [7] 17, and Custom Diffusion [8] 18, We maintain the original implementations of these baselines and directly integrate code into their existing implementations to
implement our method.
"https://pytorch hhatt:::/ppooor..orr//////////ts http:///cc//stttbbbb/eeaaaaaaadd torche wwoorch.uuaa...... max wwap:/emorryaaaaa awwocated.ddddooooooo html
"l "nttps://github.cco httppp///gttbbb..o////bobbbooob http://hhhhh..oooococoooeeeeeaaa
httpps//gggthabbb.ccc/////gggggp solver
19

---

Table 10: Model complexity comparison regarding the encoder &, the bottleneck B and the decoder D in terms of parameter count and FLOPs.
Parameter (billion)
FLOPs (million)
3 B
0.25 0.097
224.2+6.04
D
0.521.47x
504.42.2x
B Parameter Count and FLOPs of SD
We take into account model complexity in terms of parameter count and FLOPs (see Tab. 10). It's noteworthy that the decoder exhibits a significantly greater parameter count, totaling 0.51 billion. This figure is approximately 1.47 times the number of parameter combinations for the encoder & (250 million) and the bottleneck B (97 million). This substantial parameter discrepancy suggests that the decoder D carries more substantial load in terms of model complexity.
Furthermore, when we consider the computational load, during a single forward inference pass of the SD model, the decoder D incurs considerable 504.4 million FLOPs. This value is notably higher, approximately 2.2 times, than the cumulative computational load of the encoder ε, which amounts to 224.2 million FLOPs, and the bottleneck B, which requires 6.04 million FLOPs. This observation strongly implies that the decoder D plays a relatively more important role in processing and transforming the data within the UNet architecture, emphasizing its critical part in the overall functionality of the model.
C The methodology for computing Clipscore
Clipscore [46] is a metric for computing the consistency between text and image. The formula for computing Clip-score between image embedding v and text embedding as presented in the original paper, is as follows:
CLIP-S w max(cos(c, v)), o).
where re-scaling operation is performed using w, set to 2.5 in the official implementation of Clipscore [46]. The rationale behind the re-scaling operation, as provided by the official paper (excerpted from [46], Section 3, Footmote 6), is as follows:
While the cosine similarity, theory, can range from [-1, (1) we never observed a negative cosine similarity; and (2) we generally observe values ranging from roughly zero to roughly .4. The particular value of w we advocate for, w 2.5, attempts to stretch the range of the score distribution to [0, 1].
Therefore, adhering to the aforementioned setting, we employed the official implementation of Clipscore for evaluation, yielding Clipscore data distributed around 0.75. Some works [8. 48] employ re-scaling operations during evaluation, yielding Clipscore values around 0.75, while others [10, 4] do not, resulting in Clipscore values around 0.3. essence, both approaches are equivalent, differing only in whether re-scaling is applied at the end. We evaluate Clipscore using = 2.5 for re-scaling, hence yielding Clipscore data around 0.75, unless otherwise stated.
D Different from step-reduction methods
Several efficient diffusion model solvers, such as DDIM [13], DPM-Solver [14] and DPM- Slover++ [47], have significantly reduced sampling steps. Our method does not reduce the number of sampling steps. the encoder propagation, the decoder needs to compute for all time-steps, and we require time embedding inputs for all time-steps maintain temporal coherence.
Fig. 13 illustrates the qualitative comparison results of SD (DDIM) showcasing a reduction in steps (i.e., 9 and 25 time-steps). We present the results with 9 steps because the number of key time-steps for the SD model with DDIM in our approach The generation quality notably declines with step reduction, attributed to alterations in image structure and diminished attention detail generation,
20

---

Two girls holding flowers and wearing sunglasses
Children laughing and laughing happily
DDIM (9 steps)
DDIM (25 steps)
DDIM (50 steps)
DDIM w/ Ours (50 steps)
Figure 13: When lowering the time-steps in inference, the image quality noticeably deteriorates, while ours maintains a similar image quality to the original.
steps
steps w/ noise injection
25 steps
25 steps w/ noise injection
50 steps
steps w/ Ours
Figure 14: Fewer sampling steps with noise injection.
Table 12: Quantitative comparison for DPM- Solver/DPM- with fewer steps.
Table 11: Quantitative comparison for DDIM with
fewer steps.
Sampling method
Sampling method
T
FID
Clipscore
T FID J Clipscore
s/imagel
DPM-Solver
20 21.36
0.780
DDIM
50
21.75
0.773
2.42
DPM-Solver
10
22.41
0.768
DDIM
25
22.16
0.761
1.54
DDIM
DPM-Solver
25
21.89
0.761
1.54
20
21.25
0.779
w/ noise injection
w/ ours
DDIM
9
27.58
0.735
0.96
OPM-Solver++
20
0.782
DDIM
9
27.63
0.736
0.96
DPM-Solver++
10.21.25
0.771
w/ noise injection
DPM-Solvert+ 20 20.76
DDIM w/ ours
50)
21.08
0.783
1.42
0.781
w/ ours
as exemplified by the hands in Fig. 13 (the second cloumn). We increase the number of sampling steps (i.e., steps), and find that the sampling results do not perform as well as DDIM with 50 steps and its variation with FasterDiffusion (Fig. 13 and Tab. 11). As shown in Tab. 11, the FID and Clipscore SD (DDIM) with 25 time-steps on the MS-COCO 2017 subset are 22.16 16 and 0.761, respectively, significantly worse than the results obtained with 50 time-steps and ours (FIDJ: 21.75, 21 21.08; Clipscoret: 0.773, 0.783). This demonstrates that our method
simply reducing the number of sampling steps. With fewer steps (T=10), both DPM- and DPM-SO PMM-Solver++ also
21

---

exhibit worse FID and Clipscore (see Tab. 12). These results indicate that our method achieves better performance compared to simply reducing the sampling steps.
As shown on Tab. 11, we conduct DDIM scheduler inference time with various steps. Although FasterDiffusion (the last row) is slightly longer than DDIM scheduler with 9 steps, our sampling results are much closer the 50-step DDIM generation quality, whereas the 9-step DDIM results are much inferior (see Tab. 11, Tab. 12, and Fig. 13 for examples).
For directly applying noise injection to the generation phase, we show image generation examples in Fig. 14 in the rebuttal file. Fewer sampling steps equipped with prior noise injection result in almost no improvement in image quality. Tab. 11 further supports this conclusion. While in our case, FasterDiffusion working without the noise injection will lead to smoother textures, shown in Fig.6. The noise injection technique helps in preserving fidelity in the generated results.
E Difference between "prior noise injection" and "churn"
In EDM [49] (Karras et al.), they increase the noise level during ODE sampling to improve determin- istic sampling, which is referred as stochastic sampling (i.e., "churn"). Compared to deterministic sampling, it injects noise into the image at each step, which helps to enhance the quality of the generated images. Song et al. [50] first observed that perturbing data with random Gaussian noise makes the data distribution more amenable to score-based generative modeling. Increasing the noise level in EDM is based on Song's paper.
Their purpose for injecting noise is to perturb the data, whereas our purpose for injecting noise during sampling is to preserve
the diffusion model from removing details in the image during the denoising process, preventing information as noise (see Figure.6 6 in the main paper). In addition, our method FasterDiiffusion differs from them by only inserting noise in the later stage of the diffusion steps, while they insert noise to all time-steps.
F
Ablation Experiments and Additional Results
F1 Comparion with DeepCache
DeepCache is developed based the observation over the temporal consistency between high-level features. In this paper, we have a more thorough analytical study over the SD model features as shown in Fig.3, where we find out the encoder features change less, whereas the decoder features exhibit substantial variations across different time steps. This insight motivates us to omit encoder computation at certain adjacent time-steps and reuse encoder features of previous time-steps as input to the decoder in multiple time-steps. We further determine the key time steps in the T21 inference stage, which helps our method to skip the time steps in more scientific way.
is also important to note that DeepCache is not parallelizable on multiple GPUs since DeepCache needs to use all part of the encoder and decoder at every time step. FasterDiffusion, the other hand, only uses the encoder at the key time steps, which enables parallel processing at these time steps and faster inference time cost.
is also worth to notice that DasterDiffussio can be further combined with a wide range of diffusion model based tasks, including Text2 Video-zero, VideoFusion, Dreambooth, and ControlNet. In this case, DeepCache often shows much slower speed while applied to these tasks. As an example shown in Table. 13, when combined with ControlNet DeepCache is slower by 24% compared to the FasterDiffusion.
More specifically, since the ControlNet model requires an additional encoder, our method Faster- Diffusion is able to execute this extra encoder in a parallel manner and reuse it, and that makes the additional time negligible. On the other hand, DeepCache is reusing the decoder feature, which leads the ControlNet to wait for the additional encoder to complete computation at each time-step, resulting in almost no time saved from skipping the encoder in the standard SD models. The T21 generation results are shown in Figure. 15, FiffeiDi successfully preserves the given structure information and achieves similar results as the original ControlNet model.
22

---

Table 13: When combined with ControlNet (Edge) 50-step DDIM, our inference time shows significant advantage compared to DeepCache.
Clipscoret
FID
s/image
ContrINet
0.769
13.78
3.20
ContrINet w/ DeepCache ContrlNet w/ Ours
0.765
14.18
1.89 (1.69x)
0.767
14.65
1.52 (2.10x)
Imput image
Canny condition
Stable diffusion
DeepCache
Ours
Figure 15: Combine our method with ControlNet.
F22 The definition of key time-steps in various tasks
We utilize 50, 20 and 20 time-steps for DDIM scheduler [13], DPM-solver [14] and DPM- solver++ [47], respectively. We follow the official implementation of DeepFloyd-IR, setting the time-steps for the three sampling stages to 100, 50, and 75, respectively.
Text-to-image generation. We experimentally define the key time-steps kkey {50, 49, 48, 47, 45, 40, 35, 25, 15) for SD model with DDIM [13], and they = {100, 99, 98. ..... 92, 91, 90, 85, 80. .... 25, 20, 15, 14, 13, .... 2, 1). {50, 49, ........ 2, 1} and {75, 73, 70, 66, 61, 55, 48, 40, 31, 21, 10) for three stages of DeepFloyd-IF with DDPM [2]. For SD with both DPM-Solver [14] and DPM-Solver++ [47], we experimentally set the key time-steps to gkey = (20, 19, 18, 17, 15, 5).
Other tasks with text-guided diffusion model. In addition to standard text-to-image tasks, further validate our approach on other tasks with rext-guided diffusion model (Sec. 4.2). These tasks are all based the SD implementation of DDIM [13]. Through experiments,
set the key time-steps to they {50, 49, 48, 47, 45, 40. 35, 25, 15} for Text2 VicxtVVieo--e zero [4], VideoFusion [5], and ControlNet [10]. For personalized tasks (i.e., Dreambooth [7] and Custom Diffusion [8]), we set the key time steps to kky = {50, 49, 48, 47, 45, 40, 35, 25, 15, 10).
F.3 The effectiveness of encoder propagation
In Sec. 3.2, we have demonstrated that encoder propagation (Fig. 16c) can preserve semantic consistency with standard SD sampling (Fig. 16)). However, images generated through decoder propagation often fail match certain specific objects mentioned in the text prompt (Fig. 16d and Tab. 14 (the third row)).
We extended this strategy include encoder and decoder propagation (Fig. 16e) well as decoder and encoder dropping (Fig. 16f). Similarly, encoder and decoder propagation often fail to cover specific objects mentioned in the text prompt, leading degradation in the quality of the generated results (Fig. 16c and Tab. 14 (the fourth row)). On the other hand, decoder and encoder dropping are unable to completely denoise, resulting in the generation of images with noise (Fig. 16f and Tab. 14 (the fifth row)). Note that decoder and encoder dropping (Fig. 16f) is different from directly reducing the number of time-steps. Time embedding is required input for each time-step, even when using only the encoder or decoder at each time-step.
F.4
User study details
The study participants were volunteers from our college. The questionnaire consisted of 35 questions, each presenting two images: one from the baseline methods (including ControlNet, SD, DecpFloyd-
23

---

B
M sanple
& D
#11
E1-382
24330
80
PC-16
B
P
D
(a) Standard sampling
UNet architecture
M sample
t D
#+1
-
E
D
2t-1
2+-2
sample
f-1
(c) Encoder propagation
(d) Decoder propagation
~ Sample
E
c
D
#11
t22
- sample
E D
Ft
8-
f-1
(e) Encoder and decoder propagation
(f) Decoder and encoder dropping
Figure 16: (a) Standard SD sampling. (b) UNet architecture. (c) Encoder propagation. (d) Decoder
propagation. (e) Encoder and decoder propagation. (f) Decoder and encoder dropping.
Table 14: Quantitative evaluation for additional strategy on MS-COCO 2017 10K subset. Other propagation strategies can lead to the loss of some semantics under prompts and degradation of image quality (the third to fifth rows).
Sampling method
FID
Clipscore
DDIM
50
21.75
0.773
DDIM w/ Encoder propagation (Ours)
50
21.08
0.783
DDIM Decoder propagation DDIM w/ Encoder and decoder propagation
50
22.97
0.758
50
23.69
0.742
DDIM w/ Decoder and encoder dropping
50
199.48
0.679
IF, DPM-Solver++, Dreambooth, Custom Diffusion, Text-to-Video, etc.) and the other from our method FasterDiffusion (one example shown in Fig. 17).
Users required to select the image where the target
accurately portrayed, choose
"both equally good". A total of 18 users participated the questionnaire, resulting in totally 630 samples (35 questions 1 option 18 users). As the final results shown in Fig.8, the chosen percentages for SD, DeepFloyd-IF, DPM-Solver++, Text-to- Video, ControlNet, and Personalize were 48%, 44%, 51%, 52%, 46%, and 39%, respectively. These results show that our method performs on par with the baseline methods and demonstrate that FiffurDiffusion retains the T2I generation quality while reducing the inference time.
F.5 Additional metries
We showcase our experimental results with the ImageReward [51] and PickScore [52] evaluation metrics over the MS-COC02017 10K dataset, as shown in the Tab. 15. Our method FifterDiffusion enhances sampling efficiency while preserving the original model performance.
24

---

12 Please select a high-quality image that matches the specified prompt.
30 Please select high-quality video that matches the specified prompt.
cndy catlin nestled
stowy mountain landscope
The tide coming the sandy beach
Option
Option Option have
Option have sighificnt difforences.
signficant ciffornnces.
Figure 17: User study examples.
ImageRewardf
PickScoref
s/image
SD (DDIM)
0.149
52.
2.42
SD (DDIM) w/ Ours
0.162
47.90%
1.42
Table 15: Metries in ImageReward and PickScore.
F.6
Additional results
Fig. 18 shows additional results generated by DIT. In Figs. 19, 20 and 21, show additional text-to-image generation results. Further results regarding other tasks with text-guided diffusion model are illustrated in Figs. 22, 23 and 24.
F.7 Additional tasks
Versioo [53] is relation inversion method that relies on the Stable Diffusion (SD). Our method retains the capacity to generate images with specific relations based on exemplar images, as illustrated Fig. 25 (top).
P2P [27] is an image editing method guided solely by text. When combined with our approach, it enhances sampling efficiency while preserving the editing effect (Fig. 25 (bottom)).
G Automatic selection for key time steps
To identify key time steps, conducted empirical analysis of feature changes at adjacent time steps in multistep Stable Diffusion models (50-step model as an example). This analysis is based on statistics from the distribution of features across 100 random prompts, which does not impose high time cost. We found that the encoder features change minimally in later time steps, whereas the encoder features in earlier time steps exhibit substantial variations compared later ones. Based analysis as shown in Section 3.2, we determine the key time-step as kkey = {50, 49, 48, 47, 45, 40, 35, 25, 15) for the 50-step Stable Diffusion model. As general case, we also applied this key time-step configuration to the ControlNet [10], Text2 Video-zero [4], VideoFusion [5], Dreambooth [7] and Custom Diffusion [8] models based the 50-step SD models. That will not impose any additional searching time for the key time steps for these downstream application scenarios.
Existing methods such as OMS-DPM [54], AutoDiffusion [55]. and DDSM [56] utilize reinforcement learning or machine learning algorithms to search for optimal schedules, time steps, and model sizes. Each search iteration involves frequent image generation and FID calculations. Executing these search algorithms is time-consuming, often exceeding the training time, as noted in DDSM. For instance, the NSGA-II search algorithm used in DDSM incurs search cost approximately 1.1 to 1.2 times that of training a standard diffusion model. We plan to incorporate the NSGA-II algorithm for automatically searching (key time steps as enhancement to FasterDiffusion in our future research.
H
Impact Statements
Diffusion models generate realistic fake images, which assist individuals in their creative endeavors. However, for those lacking the ability to discern whether the images are generated by model, may affect their judgment.
25

---

Figure 18: Additional results of DiT (top) and this method in conjunction with our proposed approach (bottom).
Equipped with our technique, using DMs on the edge devices becomes feasible. Also, importantly, proposed acceleration of DMs does not require access enormous compute, which is required to the methods based on knowledge distillation [18, 17, 19]. This makes our technique also useful for the many smaller actors in the generative AI field.
26

---

cat wcaring sunglesses
An estronaut riding horse on Mars
colorful eulumn leeves scattered on pethway
A dew-covered spider web morning sunlight
snow-cuvered pine tree moonlit forest
dog sitting beach
DDIN
DOIM ours
man glespes sitting in front of
passenger
suchi
sitting
benana
fresh
pulling up to the side of street
with more behind
top of Lable with flowers in
chocolate frosted
green apples are kept
loptop computer
donut siting in baqgie
DPW Selver
DP% DP%- Solver Ours
Yoshua Bengio with beard
A black and white cat relaxing inside laptop
Plate conteining bread covered in some cooked broccoli
A floral centerpeece in the dining room table setting
French bread on a plate with bacon
A lone sailboat dritting on calm waters
DPH. Soberr--+
DPY DPn--Soleee Swwver----e--... Ours
Figure 19: Additional results of text-to-image generation combining SD with DDIM [13), DPM-
Solver [14], DPM-Solver++ [47], and these methods in conjunction with our proposed approach.
27

---

Waterfall surounnded
Burger and fries
woman with glasses standing the
blue bird perched top
A elephant under
A frog sitting flald
DOSA Toko
DDIN TOMi IN Ours
Figure 20: Additional results of text-to to-image generation combining SD with DDIM+ToMe [35], and this method conjunction with our proposed approach.
kangoroo holdding says "very deep leaming"
pair hands dough
crackling flames
through canopy
vintage turntable
book
DOPK
VERY PLEARNNN
DDPM Ours
VEEP LEARNING
Vintage bicycle
A bench sitting on to of a field of tall grass near water
airplene comiing in for landing with full
A big purple public bus called south tyne
bleck and yellow fire hydrent
A busy highway is being viewed from distance.
leaning against
street
DPM
DPA Solvar Curs
Figure 21: Additional results of text-to- tomeeei-imeeeee- generation combining DeepFloyd-IF with DDPM [2] and DPM-Solver++ [47], and these methods in conjunction with proposed approach.
28

---

ratting bahind
10r SOA Jo Of COE
set water balling stove
The cerrng 101
Figure 22: Additional results of Video-zdoo [4] both independently and when combined with our proposed method.
29

---

Treacdo Noom in the right day
The dce in coming the sundy beach
01066
buttersst ratuio Ehe tterstoo enutttochtuttoooooo chuttors ShOto Bruterre Uiterr 2
SOURO shutter shudere ShO BIS.
chuttesteck
Figure 23: Additional results of VideoFusion [5] both independently and when combined with our proposed method.
30

---

Condition
ControlNet
ControlNet Ours
A chair in the middle of garden
A chair next to a cozy fireplace
motorbike the ocean
racer motorbike at competition
Custom Diffusion
Custom Diffusion w/ Dreambooth
Dreambooth
Figure 24: Additional results of ControlNet [10] (top) and personalized tasks [7, 8] (bottom) obtained both independently and conjunction with our proposed method..
31

---

<R> stane <R> "painted on
cat <R> cat <Rs baak bark."
spiderman <R> stone <R> painted on'
neversion
ReVerssio CLUTs
painting squirrel eating burger
painting cating burger
Zoom photo flowers
Zoom photo
sitiigg
PZP Nf Ours
Figure 25: Edited results of Re Version [53] (top) and P2P [27] (bottom) obtained both independently and in conjunction with our proposed method.
32

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
See Abstract and Introduction (Section 1).
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
See Section
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g. if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-texx system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimess..
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
33

---

Justification:
The paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: See Section 4 and Appendix A. We will release our code the public.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
34

---

Answer: [Yes]
Justification:
We will provide our code in supplemental material.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.. cc/ Sewpdidee/oodSummisseeeee
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: /CodeSubmic / //nips. wttpccccpppccccceeececccccccccct /ttp:///ggiiieeeeeeeuuuuiiiiii ssisnnoillll onPolicy))) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
See Section 4 for the experimental setting.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: See Figure 3.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, at least for the experiments that support main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
35

---

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of enrors is not verified.
For asymmetric distributions, the authors should be careful to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative eтror rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of exccution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: See Tables 1, 5, and 9.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: : http///eerrp...ep/ cc cccppblic/eeeeccccccccc ittiectciiiiiieeeeeeeee
Answer: [Yes]
Justification: The rescarch conducted the paper conforms in every respect, with the NeurlPS Code of Ethics https: /tt///nuuisu.e/ http::p:///eeer...ccooop//////// hwtpp///bbl.ccttccccceeeeeeeeeee
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g.. if there is special consid- cration due to laws or regulations their jurisdiction..
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
See Appendix H.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
36

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Question:
Answer: [NA]
Justification: The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We cite papers that we used in Section 4. The models, datasets, and codes used in the paper introduced in detail in Appendix A.1 and Appendix A.6.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
37

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: material.
We provide source code anonymized zip file in the supplemental
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/code/moodll as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [Yes]
Justification: See User Study in Section 4.3.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
38


Reviews:
[{"review_id": "hDIFN8XjOD", "reviewer": "Reviewer_wXBv", "summary": "This method can be applied at inference time without requiring retraining to improve sampling efficiency while maintaining high image quality and can be combined with other approaches to speed up diffusion models. The approach is effective across various conditional diffusion tasks, including text-to-video generation (e.g., Text2Video-zero, VideoFusion), personalized image generation (e.g., Dreambooth), and reference-guided image generation (e.g., ControlNet). Using encoder features from previous time steps as input for decoding multiple later time steps allows for concurrent decoding, further accelerating SD sampling by 41%, DeepFloyd-IF sampling by 24%, and DiT sampling by 34%. Besides, a prior noise injection strategy is introduced to preserve image quality, maintaining texture details in the generated images.", "questions": "Extra Cost to Identify Key Time-Steps: What are the additional costs associated with identifying key time steps in the diffusion process? Is this identification process capable of being learned dynamically, or is it independent of the data? Understanding the computational overhead or resource allocation needed for this identification process is crucial for evaluating the overall practicality of the method.\n\nEnhancing Application Area with Smaller Steps Diffusion Models: Applying this approach to smaller steps diffusion models has the potential to significantly expand its applicability. By effectively handling smaller time steps, the method could be utilized in a wider range of scenarios, enhancing its versatility and impact.", "limitations": "Please refer to the weakness and questions part.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Novelty: The three techniques to design efficient diffusion models are novel.\n\nSignificance: The problem of efficient diffusion transformer inference acceleration is significant in diffusion models.\n\nMethodology: The proposed algorithm is well-formulated and well-explained. \n\nResults: The experimental results show significant improvements over existing methods over SD, DiT, and DeepFloyd-IF and are widely applied to text-to-image, text-to-video generation, and personalization.", "weaknesses": "The visualization results for smaller steps appear suboptimal, potentially due to the absence of consistency models or adversarial distillation models for verification. This raises concerns about the foundational assumption of maintaining high similarity throughout the process. It is crucial to explore whether incorporating consistency models or adversarial distillation methods could enhance the quality of results and ensure the assumption of high similarity holds even in smaller steps. Providing more robust verification methods would strengthen the validity and reliability of the visual outputs generated by the proposed approach.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your feedback and will incorporate the discussions mentioned below to enhance the quality of our paper. Note that we utilize the numerical references to cite sources within the main paper.\n\n**W1\\&Q2: Cooperation with few-step T2I models**\n\nIn the **Limitations** part of the main paper, we present that our method faces challenges in generating quality results when using a smaller number of sampling steps. \nThe underlying reason is that our method FasterDiffusion is accelerating the T2I generative model based on the empirical study over the feature changes at adjacent time-steps in multistep DMs (e.g., Stable Diffusion with 50 steps), as shown in Fig.3. We can observe that the encoder features change minimally, whereas the decoder features exhibit substantial variations across different time steps. This insight motivates us to omit encoder computation at certain adjacent time-steps and reuse encoder features of previous time-steps as input to the decoder in multiple time-steps.\n\nTo check the possibility to combine our method with few-step T2I models, including LCM [A] and SDXL-Torbo [B], we visualize the features for both the encoder and decode of UNet of these models.\nIn 4-step LCM, the features of the encoder and decoder of UNet vary greatly at adjacent time steps (see **Figure.22**). The feature changes of the encoder are not as subtly varied as they are in 50-step Stable Diffusion models, which means that the encoder features cannot be shared in adjacent time-steps  (see **Figure.24**).\n\nThe 4-step SDXL-Turbo [A] shares the same problem as the 4-step LCM, as can be seen in **Figure.23** and  **Figure.25**.\nTherefore, these few-step T2I generative models (LCM and SDXL-Turbo) cannot be combined with FasterDiffusion\n\nHowever, as a future task, we aim to apply these findings to model distillation. We hypothesize that incorporating  the encoder only once and the decoder across multiple time-steps can help capture richer semantic information and accelerate the diffusion model distillation training. \nFor example, in **Figure.26**, we illustrate the comparison between 1-step LCM and 4-step LCM feature visualization, the decoder feature maps are with more diversity with 4 steps. This implies that we can share one encoder by four decoders to capture the semantic information encapsulated in these decoders. By this means, we can achieve faster distillation for the T2I models and quicker parallel T2I generations.\nIn this paper, we focus only on training-free acceleration techniques and will make this training-required distillation method our future work. \n\n**Q1: Additional cost to identify key time steps**\n\nTo identify key time steps, we conducted empirical analysis of feature changes at adjacent time steps in multistep Stable Diffusion models (50-step model as an example).\nThis analysis is based on statistics from the distribution of features across 100 random prompts, which does not impose a high time cost.\n\nBased on analysis as shown in **Section 3.2**, we determine the $\\textit{key}$ time-step as $t^{key}=\\\\{50, 49, 48, 47, 45, 40, 35, 25, 15\\\\}$ for the 50-step Stable Diffusion model. \nAs a general case, we also applied this $\\textit{key}$ time-step configuration to the ControlNet [10], Text2Video-zero [4], VideoFusion [5], Dreambooth [7] and Custom Diffusion [8] models based on the 50-step SD models.\n\nMoreover, once we determine the $\\textit{key}$ time steps through analysis, the T2I diffusion models and their applications using FasterDiffusion always employ the same determined $\\textit{key}$ time steps. \nThis is a **once-for-all** solution, with no additional costs for each application scenario.\n\n---\n\n[A] Latent consistency models: Synthesizing high-resolution images with few-step inference.\n\n[B] Adversarial diffusion distillation.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "B5ZmyQhtAL", "reviewer": "Reviewer_BYFv", "summary": "This paper presents an extensive study of the evolution of internal activations in diffusion U-Nets and uses their findings to motivate a training-free approach for accelerating sampling from diffusion models. The method is demonstrated to successfully speed up inference in a variety of settings, including different architectures (including non-hierarchical DiTs), different samplers, and some modified inference processes.", "questions": "- Can the authors provide examples and evidence of practically relevant settings where DeepCache fails or underperforms significantly compared to the proposed method (such as a combination with DPM-Solver(++), ControlNet, or Dreambooth etc)? These would help address the main method-related weakness mentioned.\n- What do the ellipses in the key time step steps mean? Is every time step between 80 and 25 included for $\\{80,...,25\\}$?\n- Is there a relevant difference between the proposed \"prior noise injection method\" and the various methods that artificially increase the noise level slightly during ODE sampling, such as ``churn'' in (Karras et al., Elucidating the Design Space of Diffusion-Based Generative Models, NeurIPS 2022)?\n- Are all the videofusion samples in the supplementary material just corrupted or is there something to be demonstrated there?\n- Does parallel-batch encoder propagation limit the possible batch size on a single GPU compared to other sampling methods? If so, could the authors provide a performance comparison that individually maxes out the batch size for the various approaches (including competing ones) and then reports the samples/second? As generating multiple variants simultaneously is a standard approach when generating images with diffusion models in practice, this is an important avenue for optimization. Specifically, I am worried that at a maximal batch size, parallel encoder propagation will only offer a negligible speedup compared to encoder propagation (due to larger generation batch sizes being possible with non-parallel encoder propagation, as VRAM usage is substantially different between the two methods [Tab. 8]), at which point this method would provide less than half of the speedup provided by prior art.", "limitations": "The authors have adequately addressed the limitations of their work.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- The method is demonstrated to work well on a wide range of networks & tasks, including U-Net-based LDMs (SD), Imagen-style cascaded pixel-space U-Net diffusion models (IF), and DiTs. Especially DiT is notable, as previous methods do not work on homogeneous transformer architectures out-of-the-box.\n- The extensive appendix provides a lot of useful additional information and experimental results and supplements the empirical study about the evolution of internal features in diffusion models well.\n- The proposed method opens up avenues for further accelerating sampling speed as measured in wall clock delay for single samples by utilizing multiple GPUs to distribute the proposed parallel encoder propagation strategy.\n- The proposed \"prior noise injection method\" helps alleviate artifacts incurred from the optimized sampling process.", "weaknesses": "- No details are provided about the user study, and false claims relating to it are made in the checklist [questions 14 and 15].\n- The main method seems to only be a minor incremental evolution of DeepCache's methodology (encoder features are cached and reused without adaptation of features or network in following steps, optionally multiple times, with the main difference being what is defined as the encoder), and seems to provide a *reduced* speedup when compared in a fair single-GPU setting [Tab. 1, Sec. 4.1 l. 271: 41% speedup vs. 56% speedup, at comparable quality levels].\n- The presentation of the paper needs some work. Font sizes in some figures are excessively small [cf. Figs. 1, 2, 3, 7], Tabs. 2 and 3 are excessively compressed up to a point at which readability is impaired. The paper would also benefit from a thorough re-read and corrections for the camera-ready version. Especially grammar errors that substantially affect the claims made in the paper (e.g., sampling time reduction \"to 41%\" [l. 271] instead of \"by 41%\") and typos that change the meaning of words (e.g., \"rudely\" [l.105]), but also names of methods (e.g., \"DeepFolyd-IF\" [l. 60], \"Stable Diffuison\" [l. 525]).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your feedback and will incorporate the discussions mentioned below to enhance the quality of our paper. Note that we utilize the numerical references to cite sources within the main paper.\n\n**W1. User study details.**\n\nThe study participants were volunteers from our college. The questionnaire consisted of 35 questions, each presenting two images: one from the baseline methods (including ControlNet, SD, DeepFloyd-IF, DPM-Solver++, Dreambooth, Custom Diffusion, Text-to-Video, etc.) and the other from our method FasterDiffusion (one example shown in **Figure.29**). \n\nUsers were required to select the image where the target was more accurately portrayed, or choose “both equally good”. A total of 18 users participated the questionnaire, resulting in totally 630 samples (35 questions × 1 option × 18 users). As the final results shown in **Figure.8**, the chosen percentages for SD, DeepFloyd-IF, DPM-Solver++, Text-to-Video, ControlNet, and Personalize were 48\\%, 44\\%, 51\\%, 52\\%, 46\\%, and 39\\%, respectively. \nThese results show that our method  performs on par with the baseline methods.\n\n**W2\\&Q1. Comparison with DeepCache**\n\nDeepCache is developed based on the observation over the temporal consistency between high-level features.\nIn this paper, we have a more thorough analytical study over the SD model features as shown in **Figure.3**, where we find out the encoder features change less, whereas the decoder features exhibit substantial variations across different time steps. This insight motivates us to omit encoder computation at certain adjacent time-steps and reuse encoder features of previous time-steps as input to the decoder in multiple time-steps. We further determine the key time steps in the T2I inference stage, which helps our method to skip the time steps in a more scientific way. \n\nIt is also important to note that DeepCache is not parallelizable on multiple GPUs since DeepCache needs to use all or part of the encoder and decoder at every time step.\nFasterDiffusion, on the other hand, only uses the encoder at the $\\textit{key}$ time steps, which enables parallel processing at $\\textit{non-key}$ time steps and faster inference time cost. \n\nIt is also worth to notice that FasterDiffusion can be further combined with a wide range of diffusion model based tasks, including Text2Video-zero, Dreambooth, and ControlNet. In this case, DeepCache often shows much slower speed while applied to these tasks. As an example shown in **Table below**, when combined with  ControlNet, DeepCache  is slower by **24\\%** compared to the FasterDiffusion.\n\nMore specifically, since the ControlNet requires an additional encoder, our method FasterDiffusion is able to execute this extra encoder in a parallel manner and reuse it, and that makes the additional time negligible. \nOn the other hand, DeepCache is reusing the decoder feature, which leads the ControlNet to wait for the additional encoder to complete computation at each time-step, resulting in almost no time saved from skipping the encoder in the standard SD models.\nAs shown in **Figure.28**, FasterDiffusion successfully preserves the given structure information and achieves similar results as the original ControlNet.\n\n|          | **Clipscore**$\\uparrow$ | **FID**$\\downarrow$ | **s/image**$\\downarrow$ |\n|--------------------------|-------------------------|---------------------|-------------------------|\n| **ControlNet**           | 0.769                   | 13.78               | 3.20                    |\n| **ControlNet w/ DeepCache** | 0.765                   | 14.18               | 1.89 (1.69x)            |\n| **ControlNet w/ Ours**   | 0.767                   | 14.65               | **1.52 (2.10x)**        |\n\n**Table 14:** When combined with ControlNet (Edge) 50-step DDIM, our inference time shows a significant advantage compared to DeepCache.\n\n---\n\n**W3. Paper presentation**\n\nWe will carefully correct these figures, tables, grammar errors and typos in the future version.\n\n**Q2. Ellipsis meaning in the key time step**\n\nThe ellipsis in $t^{key}$ denotes each time-step between the time-steps on either side of the ellipsis. For example, 80...25 means that every time-step between 80 and 25 is included.\n\n**Q3. Relevant difference**\n\nIn EDM [A] (Karras et al.), they increase the noise level during ODE sampling to improve deterministic sampling, which is referred to as stochastic sampling (i.e., 'churn'). Compared to deterministic sampling, it injects noise into the image at each step, which helps to enhance the quality of the generated images. Song et al. [B] first observed that perturbing data with random noise makes the data distribution more amenable to score-based generative modeling.  Increasing the noise level in EDM is based on Song's paper.\n\nTheir purpose for injecting noise is to perturb the data, whereas our purpose for injecting noise during sampling is to preserve high-frequency details in the image during the denoising process, preventing the diffusion model from removing high-frequency information as noise (see **Figure.6** in the main paper).\nIn addition, our method FasterDiffusion differs from them by only inserting noise in the later stage of the diffusion steps, while they insert noise to all time-steps.\n\n**Q4. Video corruption**\n\nThe files (including the videofusion samples) in the supplementary material are not corrupted. They can be opened correctly using the default video players on Windows or Mac operating systems. \nHowever, on the Linux operating systems with graphical interfaces (e.g., Ubuntu), the default video player cannot open these files normally. You may need to download an alternative video player from the software store (e.g., Celluloid) to read these videos.\n\n**Q5. Parallel or Serial**\n\nPlease refer to our 'global' response (**General Response 1**) for details.\n\n---\n\n[A] Elucidating the design space of diffusion-based generative models\n\n[B] Generative modeling by estimating gradients of the data distribution", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9DTs7zQ66G", "reviewer": "Reviewer_wo3A", "summary": "This paper presents an approach to accelerate diffusion model inference by capitalizing on the minimal change in encoder features across time steps. The proposed encoder propagation strategy reduces encoder computations by reusing encoder features from previous time steps. The proposed method is comparable to existing acceleration methods and demonstrates effective acceleration across diverse tasks.", "questions": "See weakness for details.", "limitations": "The authors have discussed both limitations and potential negative social impacts.", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1.\tThe proposed method offers a promising solution to speed up diffusion models.\n\n2.\tThe experiments are conducted from various tasks, e.g., text to image; text to video; personalized generation and reference-guided generation.", "weaknesses": "1.\tThe evaluation metrics are insufficient, as concerns exist regarding the universality of FID in assessing generative models. It is recommended to validate performance on additional metrics, such as the newly proposed ImageReward [1] and Pick Score [2].\n\n2.\tIt is unclear whether the latency is comparable when fewer sampling steps are used (Table 11 and Table 12). Moreover, the performance improvement is marginal compared to results with fewer sampling steps, and mainly stems from prior noise injection, which is not the core contribution of the paper. I am also curious whether fewer sampling steps, if equipped with prior noise injection, could achieve similar performance as the results presented in this paper.\n\n3.\tComparing multi-GPU results with those single-GPU results from DeepCache seems unfair (Table 1). Besides, the acceleration outcomes on a single GPU are marginally improved when compared with other acceleration methods.\n\n4.\tThe DiT does not inherently include concepts of Encoder and Decoder; thus, categorizing based solely on observations into Encoder and Decoder is somewhat imprecise.\n\n5.\tThe layout of the paper is somewhat disorganized; the order of tables does not align with the sequence of their introduction in the text (e.g., Table 1 and Table 3), making it hard to follow.\n\n[1] Xu J, Liu X, Wu Y, et al. Imagereward: Learning and evaluating human preferences for text-to-image generation. NeurIPS 2023.\n\n[2] Kirstain Y, Polyak A, Singer U, et al. Pick-a-pic: An open dataset of user preferences for text-to-image generation. NeurIPS 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your feedback and will incorporate the discussions mentioned below to enhance the quality of our paper. Note that we utilize the numerical references to cite sources within the main paper.\n\n**W1. Additional metrics**\n\nThanks for your advice. We showcase our experimental results with the ImageReward and PickScore evaluation metrics over the MS-COCO2017 10K dataset, as shown in the **Table below**.\nOur method FasterDiffusion enhances sampling efficiency while preserving the original model performance.\n\n|                           | **ImageReward**$\\uparrow$ | **PickScore**$\\uparrow$ | **s/image** |\n|----------------------------|---------------------------|-------------------------|-------------|\n| **SD (DDIM)**              | 0.149                     | 52.10%                  | 2.42        |\n| **SD (DDIM) w/ Ours**      | 0.162                     | 47.90%                  | 1.42        |\n\n**Table:** Metrics in ImageReward and PickScore.\n\n---\n\n**W2. Few step generation with noise injections**\n\nTo extend the content illustrated in Table 11 and Table 12, \nwe present the full results with fewer sampling steps in **Table below**, specifically with 9 and 25 steps as in Table 11. \n\nWe mainly present the generation results with 9 steps because it is the number of $key$ time-steps for the 50-step SD model determined in our approach. \nWe increase the number of sampling steps (i.e., 25 steps),  and find that the sampling results do not perform as well as DDIM with 50 steps and its variation with FasterDiffusion (Figure.12). \nThis demonstrates that our method is not simply reducing the number of sampling steps.\n\nAs shown on **Table below**,  we conduct\nDDIM scheduler inference time with various steps. \nAlthough FasterDiffusion is slightly longer than DDIM scheduler with 9 steps, our sampling results are much closer the 50-step DDIM generation quality, whereas the 9-step DDIM results are much inferior (see Table.11, Table.12, and Figure.12 for examples).\n\nFor directly applying noise injection to the generation phase, we show image generation examples in **Figure.27** in the rebuttal file. \nFewer sampling steps equipped with prior noise injection result in almost no improvement in image quality. **Table below** further supports this conclusion. \nWhile in our case, FasterDiffusion working without the noise injection will lead to smoother textures, as shown in Figure.6. \n\nThe noise injection technique helps in preserving fidelity in the generated results.\n\n| Sampling method   | T  | **FID** $\\downarrow$ | **Clipscore** $\\uparrow$ | **s/image** $\\downarrow$ |\n|------------------------------|----|-----------------------|--------------------------|--------------------------|\n| **DDIM**                     | 50 | 21.75                 | 0.773                    | 2.42              |\n| **DDIM**                     | 25 | 22.16                 | 0.761                    | 1.54                     |\n| **DDIM w/ noise injection**  | 25 | 21.89                 | 0.761                    | 1.54                     |\n| **DDIM**                     | 9  | 27.58                 | 0.735                    | 0.96                     |\n| **DDIM w/ noise injection**  | 9  | 27.63                 | 0.736                    | 0.96                     |\n| **DDIM w/ ours**             | 50 | **21.08**                 | **0.783**                    | **1.42**               |\n\n**Table:** Quantitative comparison for DDIM with fewer steps.\n\n---\n\n**W3. Comparison with single-GPU methods**\n\nOur advantage over other methods (e.g., DeepCache) is the ability to perform inference on non-key time steps in parallel, therefore enabling parallel processing across multi-GPUs. \nThis characteristic assist our method FasterDiffusion to achieve better speedup performance compared to other methods.\n\nIn this paper, we notice that DeepCache is not parallelizable on multiple GPUs since DeepCache needs to use all or part of the encoder and decoder at every time step.\nFasterDiffusion, on the other hand, only uses the encoder at the $\\textit{key}$ time steps, which enables parallel processing at non-key time steps and faster inference time cost. \nIt is also worth to notice that FasterDiffusion can be further combined with a wide range of diffusion model based tasks, including Text2Video-zero, VideoFusion, Dreambooth, and ControlNet. In this case, DeepCache often shows much slower speed while applied to these tasks. As an example shown in **Table.14 (in 'global' response)**, DeepCache applied to the ControlNet is slower by 24\\% compared to the FasterDiffusion based ControlNet.\n\nMore specifically, since the ControlNet model requires an additional encoder, our method FasterDiffusion is able to execute this extra encoder in a parallel manner and reuse it, and that makes the additional time negligible. \nOn the other hand, DeepCache is reusing the decoder feature, which leads the ControlNet to wait for the additional encoder to complete computation at each time-step, resulting in almost no time saved from skipping the encoder in the standard SD models. \n\nPlease refer to our 'global' response (**General Response 1**) for **more details**.\n\n**W4. Applicability to DiT based models**\n\nThe DiT includes 28 transformer blocks. \nThrough visualization and statistical analysis (see **Figure.30** in the rebuttal PDF and **Figure.11** in main paper),\nwe observe that the features in the first several transformer blocks change minimally, similar to the Encoder in SD, while the features in the remaining transformer blocks exhibit substantial variations, akin to the Decoder in SD. \nFor ease of presentation, we refer to the first several transformer blocks of DiT as the Encoder and the remaining transformer blocks as the Decoder. In **Table 2**, we demonstrate the accelerating performance of our method while applied to DiT-based generation models.\n\n**W5. Paper layout**\n\nWe will carefully design the layout in the future version for better reading experience.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "KPqHNPVOfS", "reviewer": "Reviewer_i4aK", "summary": "The authors analyze the encoder features of diffusion UNet and find that they share a lot of similarities across certain time steps. Based on this observation, the authors propose to reuse the encoder features and do parallel computation in sampling. This significantly fastens the generation process. Additionally, they also introduce a prior noise injection method to improve the generation quality. Extended experiments demonstrate their effectiveness across a wide range of generation tasks, including T2I, T2V, DreamBooth, controlNet. And their method can be seamlessly combined with various noise schedulers.", "questions": "Please see the weakness section.", "limitations": "Yes, they discuss their limitation in section 5.", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "1 This paper is well-written and easy to understand. The analysis in section 1 is insightful. Fig.4 clearly shows the encoder propagation methods. \n2 The empirical evaluation is thorough. The authors validate their method on various datasets, architectures, and use various metrics. They also make a lot of comparison to the SOTA acceleration methods, such as DeepCache.\n3 Sec. A4 shows that the parallel denoising leads to an acceptable memory cost, which addresses my concerns.\n4 This approach can be seamlessly combined with other acceleration methods such as novel noise schedulers and distillations.\n\nOverall, this is an excellent work that significantly reduces diffusion sampling cost in a training-free manner.", "weaknesses": "I think the manual selection of key time-steps may not be optimal. Is it possible to use some automatic strategies to determine the key time-steps in your work? Otherwise when we use a new architecture we would have to redo the encoder feature analysis experiment and manually define the key-time steps again. I think there are several works have explored the selection of key time-steps. Discuss on them will make your work even stronger.\n\n[1] OMS-DPM: Optimizing the Model Schedule for Diffusion Probabilistic Models\n[2] AutoDiffusion: Training-Free Optimization of Time Steps and Architectures for Automated Diffusion Model Acceleration\n[3] Denoising Diffusion Step-aware Models", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your feedback and will incorporate the discussions mentioned below to enhance the quality of our paper. Note that we utilize the numerical references to cite sources within the main paper.\n\n**W1. Automatic selection for key time steps**\n\nThank you for your insightful suggestion. We find that when using uniform $\\textit{key}$ time-steps with the same time interval, the sampling results are acceptable ($t^{key} = \\\\{50, 44, 38, 32, 26, 20, 14, 8, 2\\\\}$), though not as optimal compared to those obtained with non-uniform time-steps (see Table.5 in main paper).\n\nTo identify key time steps, we conducted empirical analysis of feature changes at adjacent time steps in multistep Stable Diffusion models (50-step model as an example).\nThis analysis is based on statistics from the distribution of features across 100 random prompts, which does not impose a high time cost.\nWe found that the encoder features change minimally in later time steps, whereas the encoder features in earlier time steps exhibit substantial variations compared to later ones. \nBased on analysis as shown in Section 3.2, we determine the $\\textit{key}$ time-step as $t^{key}=\\\\{50, 49, 48, 47, 45, 40, 35, 25, 15\\\\}$ for the 50-step Stable Diffusion model. \nAs a general case, we also applied this $\\textit{key}$ time-step configuration to the ControlNet [10], Text2Video-zero [4], VideoFusion [5], Dreambooth [7] and Custom Diffusion [8] models based on the 50-step SD models. That will not impose any additional searching time for the key time steps for these downstream application scenarios.\n\nAs per your suggestions, we explored OMS-DPM [A], AutoDiffusion [B], and DDSM [C]. They are using reinforcement learning or machine learning algorithms to search for the optimal schedule, time steps, and model size, etc. \nEach search iteration involves high-frequent images generation and FID calculation. \nExecuting these search algorithms is extensively time-consuming, often exceeding the training time, as claimed in DDSM.\nFor example, the NSGA-II search algorithm is utilized in DDSM [C]. The search cost for the DDSM network is approximately 1.1 to 1.2 times that of training a standard diffusion model (**see the table below**). \nDue to the short rebuttal period, we have not been able to apply these algorithms to our method, however, we will discuss the use of the NSGA-II algorithm for automatically searching $t^{key}$ time-steps in the final version and include it as an improved version for FasterDiffusion in our future research.\n\n|                 | CIFAR-10 | CelebA |\n|-----------------|----------|--------|\n| **DDPM-train**  | 278      | 435    |\n| **DDSM-train**  | 502      | 1036   |\n| **DDSM-search** | **320**  | **524**|\n\n**Table:** Total GPU hours of DDSM.\n\n---\n\n[A] OMS-DPM: Optimizing the Model Schedule for Diffusion Probabilistic Models\n \n[B] AutoDiffusion: Training-Free Optimization of Time Steps and Architectures for Automated Diffusion Model Acceleration \n\n[C] Denoising Diffusion Step-aware Models", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
**“global” response**

We appreciate all reviewers (**R1=wXBv**, **R2=BYFv**, **R3=wo3A**, **R4=i4aK**) for their positive feedbacks. They note that this paper is well-written (R4) and easy to understand (R4); the technique is novel (R1) and promising (R3); that the proposed algorithm is well-formulated and well-explained (R1); that we present insightful analysis of UNet (R4); that we provide extensive experiments over various network architectures (R1, R2, R4), tasks (R1, R2, R3), and use various metrics (R4); that we can also work on diffusion transformer inference acceleration (R1,R2). Below we respond to general questions raised by reviewers; We use **W** to abbreviate **Weaknesses**, **Q** to represent **Questions** and **L** for **Limitations**.
 Note that we utilize the numerical references to cite sources within the main paper.

**General Response 1. Parallel on multi-GPU or Serial on single-GPU (R2-Q5, R3-W3)**

Our advantage over other methods (e.g., DeepCache) is the ability to perform inference on *non-key* time steps in parallel, therefore enabling parallel processing across multi-GPUs. 
This characteristic assist our method FasterDiffusion to achieve better speedup performance compared to other methods.
It is also noticeable that DeepCache is not parallelizable, since it needs to use all or part of the encoder and decoder at every time step. Due to this reason, DeepCache cannot be further improved by deploying on multiple GPUs.

Deep learning models rely heavily on parallel training and inference across multi-GPUs. Using only a single GPU can result in excessive time consumption while dealing with large models. Therefore， it is a very relevant and impactful contribution that parallelizing existing algorithms and unlocking the speed-up available due to multiple GPU computations. In this paper, our proposed FasterDiffusion parallelizes the processing of multiple time-steps, enhancing processing efficiency. 


In addition, our findings can also be applied to model distillation by using the decoder at multiple time steps to capture richer semantic information, which DeepCache cannot achieve. As a future task, we aim to apply these findings to model distillation. We hypothesize that incorporating the encoder only once and the decoder across multiple time-steps can help capture richer semantic information, and accelerate the diffusion model distillation training. 
For example, in **Figure.26** in rebuttal PDF, we illustrate the comparison between 1-step LCM and 4-step LCM feature visualization, the decoder feature maps are with more diversity with 4 steps. 
This implies that we can share one encoder by 4-step decoder to capture the semantic information encapsulated in 4-step decoder. By this means, we can achieve faster distillation for the T2I models and quicker parallel T2I generations.
In this paper, we focus only on training-free acceleration techniques and will make this training-required distillation method our future work.

|          | **Clipscore**$\uparrow$ | **FID**$\downarrow$ | **s/image**$\downarrow$ |
|--------------------------|-------------------------|---------------------|-------------------------|
| **ControlNet**           | 0.769                   | 13.78               | 3.20                    |
| **ControlNet w/ DeepCache** | 0.765                   | 14.18               | 1.89 (1.69x)            |
| **ControlNet w/ Ours**   | 0.767                   | 14.65               | **1.52 (2.10x)**        |

**Table 14:** When combined with ControlNet (Edge) 50-step DDIM, our inference time shows a significant advantage compared to DeepCache.

Meta-review: The manuscript has been reviewed by four reviewers, all of whom, after the rebuttal, gave positive ratings (including borderline positive).

In general, the reviewers are happy with the novelty, motivation, significance, and experimental validation.

The AC agrees with the consensus, and recommends the manuscript to be accepted. Congrats!
Decision: Accept (poster)