Faster Diffusion: Rethinking the Role of the Encoder for Diffusion Model Inference

Authors: senmao  li
taihang  hu
joost  van de weijer
fahad shahbaz khan
tao  liu
linxuan  li
shiqi  yang
yaxing  wang
ming-ming  cheng
jian  yang
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-04 07:43:43

Abstract: One of the main drawback of diffusion models is the slow inference time for image generation. Among the most successful approaches to addressing this problem are distillation methods. However, these methods require considerable computational resources. In this paper, we take another approach to diffusion model acceleration. We conduct a comprehensive study of the UNet encoder and empirically analyze the encoder features. This provides insights regarding their changes during the inference process. In particular, we find that encoder features change minimally, whereas the decoder features exhibit substantial variations across different time-steps. This insight motivates us to omit encoder computation at certain adjacent time-steps and reuse encoder features of previous time-steps as input to the decoder in multiple time-steps. Importantly, this allows us to perform decoder computation in parallel, further accelerating the denoising process. Additionally,  we introduce a prior noise injection method to improve the texture details in the generated image. Besides the standard text-to-image task, we also validate our approach on other tasks: text-to-video, personalized generation and reference-guided generation. Without utilizing any knowledge distillation technique, our approach accelerates both the Stable Diffusion (SD) and DeepFloyd-IF model sampling by 41$\%$ and 24$\%$ respectively, and DiT model sampling by 34$\%$, while maintaining high-quality generation performance. Our code will be publicly released.

Main paper: 
 introduction: 
 One of the popular paradigms in image generation, Diffusion Models (DMs) [1, 2, 3] have recently achieved significant breakthroughs in various domains, including text-to-video generation [4, 5, 6] , personalized image generation [7, 8, 9] and reference-guided image generation [10, 11, 12] . While diffusion models produce images of exceptional visual quality, their primary drawback lies in the prolonged inference time. The original diffusion model had an inference time several orders of magnitude slower than, for instance, GANs. One of the challenges hindering the acceleration of diffusion models is their inherent sequential denoising process, which limits the possibilities of effective parallelization.To improve inference time speed of diffusion models, several methods have been developed, that can roughly be divided in two sets of approaches. Firstly, involving step reduction, the aim is to reduce the number of sampling steps within diffusion model inference, such as DDIM [13] and DPM-Solver [14] , which have significantly reduced the number of sampling steps. Secondly, in contrast, knowledge distillation progressively distills a slow (many-step) teacher model into a faster (few-step) student model [15, 16] . Some recent works [17, 18, 19] excel at generating high-fidelity images in a few-step sampling scenario but face challenges in maintaining quality and diversity in one-step sampling. The main drawback of the distillation methods is that they require retraining to perform the distillation into faster diffusion models.Figure 2 : Visualising the hierarchical features 1 . We applied PCA to the hierarchical features following PnP [20] and used the top three leading components as an RGB image for visualization. The encoder features change minimally and have similarities at many time-steps (top), while the decoder features exhibit substantial variations across different time-steps (bottom).Orthogonal to these methods, we take a closer look at the sequential nature of the denoising process. We focus on the characteristics of the encoder in pretrained diffusion models (e.g., the SD and the DiT [21] 2 ) Interestingly, based on our analysis presented in Sec 3.2, we discover that encoder features change minimally (Fig. 3a ) and have a high degree of similarity (Fig. 2 (top)), whereas the decoder features exhibit substantial variations across different time-steps (Fig. 3a and Fig. 2 (bottom)). This insight is relevant because it allows us to circumvent the computation of the encoder during multiple time-steps. As a consequence, the decoder computations which are based on the same encoder input can be performed in parallel. Instead, we reuse the computed encoder features computed at one time-step (since these change minimally) as input to adjacent decoders during the following time-steps. More recently, both DeepCache [22] and CacheMe [23] leverage feature similarity to achieve acceleration. However, they rely on sequential denoising, as well as CacheMe requires fine-tuning. Unlike these methods, our approach supports parallel processing, which leads to faster inference (Tab. 2).We show that the proposed propagation scheme accelerates the SD sampling by 24% , DeepFolyd-IF sampling by 18%, and DiT sampling by 27%. Furthermore, since the same encoder features (from previous time-steps) can be used as the input to the decoder of multiple later time-steps, this makes it possible to conduct multiple time-steps decoding concurrently. This parallel procession accelerates SD sampling by 41%, DeepFloyd-IF sampling by 24%, and DiT sampling by 34%. Furthermore, to alleviate the deterioration of the generated quality, we introduce a prior noise injection strategy to preserve the texture details in the generated images. With these contributions, our proposed method achieves improved sampling efficiency while maintaining high image generation quality. Importantly, our method can be combined with several approaches existing to speed up DMs. The main advantage of our method with respect to distillation-based approaches, is that our method can be applied at inference time, and does not require retraining a new faster distillation model; a process that is computationally very demanding and infeasible for actors with limited computational budget. Finally, we evaluate the effectiveness of our approach across a wide range of conditional diffusionbased tasks, including text-to-video generation (e.g., Text2Video-zero [4] and VideoFusion [5] ), personalized image generation (e.g., Dreambooth [7] ) and reference-guided image generation (e.g., ControlNet [10] ).To summarize, we make the following contributions:• We conduct a thorough empirical study of the features of the UNet in the diffusion model showing that encoder features vary minimally (whereas decoder feature vary significantly). • We propose a parallel strategy for diffusion model sampling at adjacent time-steps that significantly accelerates the denoising process. Importantly, our method does not require any training or fine-tuning technique. • Furthermore, we also present a prior noise injection method to improve the image quality (mainly improving the quality of high-frequency textures). • Our method can be combined with existing methods (like DDIM, and DPM-solver) to further accelerate diffusion model inference time. 
related work: 
 Denoising diffusion model. Recently, Text-to-image diffusion models [1, 24, 25, 26] have made significant advancements. Notably, Stable Diffusion and DeepFloyd-IF stand out as two of the most successful diffusion models available within the current open-source community. These models, building upon the UNet architecture, are versatile and can be applied to a wide range of tasks, including image editing [27, 28] , super-resolution [29, 30] , segmentation [31, 32] , and object detection [33, 34] .Given the strong scalability of transformer networks, DiT [21] investigates the transformer backbone for diffusion models. Diffusion model acceleration. Diffusion models use iterative denoising with UNet for image generation, which is time-consuming. There are plenty of works trying to address this issue. One strategy involves employing efficient diffusion model solvers, such as DDIM [13] and DPM-Solver [14] , which have demonstrated notable reductions in sampling steps. Additionally, ToMe [35] exploits token redundancy to minimize the computations necessary for attention operations [36] . Conversely, knowledge distillation methods, exemplified by techniques like progressive simplification by student models [15, 16] , aim to streamline existing models. Some recent studies combine model compression with distillation to achieve faster sampling [37, 38] . Orthogonal to these approaches, we introduce a novel method for enhancing sampling efficiency in DMs inference. We show that our method can be combined with several existing speed-up methods for further acceleration. DeepCache [22] and CacheMe [23] are two recent works that leverage feature similarity to achieve acceleration. DeepCache [22] adopts a strategy of rudely reusing features cached from the previous step, requiring iterative denoising. Furthermore, CacheMe [23] requires additional fine-tuning for Note, at time-step t-1, predicting noise does not require z t-1 (i.e., Eq. 1:z t-2 = αt-2 αt-1 z t-1 + √ α t-2 1 αt-2 -1 - 1 αt-1 -1 • ϵ θ ( X X X z t-1 , t -1, c)). (d) Decoder propagation.The generated images often fail to cover some specific objects in the text prompt. For example, given one prompt case "A man with a beard wearing glasses and a beanie", this method fails to generate the glasses subject. See Appendix F for quantitative evaluation. (e) Applying encoder propagation for nonuniform strategy. By benefiting from our propagation scheme, we are able to perform the decoder in parallel at certain adjacent time-steps. better performance. In contrast, our approach enables parallel processing, leading to considerably faster inference.Question: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes] Justification: See Section 5.Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
method: 
 We first briefly revisit the architecture of the Stable Diffusion (SD) (Sec. 3.1), and then conduct a comprehensive analysis for the hierarchical features of the UNet (Sec. 3.2). Our analysis shows that it is possible to parallelize the diffusion model denoising process partially. Thus, we introduce a novel method to accelerate the diffusion sampling while still largely maintaining the generation quality and fidelity (Sec. 3.3).In the diffusion inference stage, the denoising network ϵ θ takes as input a text embedding c, a latent code z t and a time embedding, predicts noise, resulting in a latent z t-1 using the DDIM scheduler [13] :EQUATIONwhere α t is a predefined scalar function at time-step t (t = T, ..., 1). The typical denoising network uses a UNet-based architecture. It consists of an encoder E, a bottleneck B, and a decoder D, respectively (Fig. 4b ). The hierarchical features extracted from the encoder E are injected into the decoder D by a skip connection (Fig. 4a ). For convenience of description, we divide the UNet network into specific blocks:E = {E(•) s }, B = {B(•) 8 }, and D = {D(•) s }, where s ∈ {8, 16, 32, 64} (see Fig. 4b ). Both E(•) s3 and D(•) s represent the block layers with input resolution s in both encoder and decoder, respectively. Diffusion Transformer (DiT) [21] is a novel architecture for diffusion models. It replaces the UNet backbone with a transformer, which consists of 28 blocks. Based on our observations, we define the first 18 blocks as the encoder, and the remaining 10 blocks as the decoder (see Appendix A.3).In this section, we take the UNet-based diffusion model as example to analyze the properties of the pretrained diffsuion model. We delve into the UNet which consists of the encoder E, the bottleneck B, and the decoder D, for a deeper understanding of the different parts of the UNet. Note the following observed properties also exist in DiT (see Appendix A.3).Feature evolution across time-steps. We experimentally observe that the encoder features exhibit a subtle variation at adjacent time-steps, whereas the decoder features exhibit substantial variations across different time-steps (see Fig. 3a and Fig. 2 ). Specifically, given a pretrained diffusion model, we iteratively produce a latent code z t (see Eq. 1), and the corresponding hierarchical features: {E(z t , c, t) s }, {B(z t , c, t) 8 }, and {D(z t , c, t) s } (s ∈ {8, 16, 32, 64}) 4 , as shown in Fig. 4b . Here, we analyze how the hierarchical features change at adjacent time-steps. To achieve this goal, we quantify the variation of the hierarchical features as follows:EQUATIONwhere d represents the number of channels in E(z t , c, t) s . Similarly, we also compute ∆ B(•)8 and ∆ D(•)s .As illustrated in Fig. 3a , for both the encoder E and the decoder D, the curves exhibit a similar trend: in the wake of an initial increase, the variation reaches a plateau and then decreases, followed by a continuing growth towards the end. However, the extent of change in ∆ E(•)s and ∆ D(•)s is quantitatively markedly different. For example, the maximum value and variance of the ∆ E(•)s are less than 0.4 and 0.05, respectively (Fig. 3a (zoom-in area)), while the corresponding values of the ∆ D(•)s are about 5 and 30, respectively (Fig. 3a ). Furthermore, we find that ∆ D(•)64 , the change of the last layer of the decoder, is close to zero. This is due to the output of the denoising network being similar at adjacent time-steps [39] . In conclusion, the overall feature change ∆ E(•)s is smaller than ∆ D(•)s throughout the inference phase.Feature evolution across layers. We experimentally observe that the feature characteristics are significantly different between the encoder and the decoder across all time-steps. For the encoder E the intensity of the change is slight, whereas it is very drastic for the decoder D. Specifically we calculate the Frobenius norm for hierarchical features E(z t , c, t) s across all time-steps, dubbed asEQUATIONs }. Similarly, we compute F B(•)8 and F D(•)s , respectively.Fig. 3b shows the feature evolution across layers with a boxplot 5 . Specifically, for F E(•)s and F B(•)8 , the box is relatively compact, with a narrow range between their first-quartile and thirdquartile values. For example, the maximum box height (F E (.) 32 ) of these features is less than 5 (see Fig. 3b (zoom-in area)). This indicates that the features from both the encoder E and the bottleneck B slightly change. In contrast, the box heights corresponding to {D(•) s } are relatively large. For example, for the D(.) 64 the box height is over 150 between the first quartile and third quartile values (see Fig. 3b ). Furthermore, we also provide a standard deviation (Fig. 3c ), which exhibits similar phenomena to Fig. 3b . These results show that the encoder features have relatively small discrepancy and a high degree of similarity across all layers. However, the decoder features evolve drastically.Could we omit the Encoder at certain time-steps? As indicated by the previous experimental analysis, we observe that, during the denoising process, the decoder features change drastically, whereas the encoder E features change minimally, and have a high degree of similarities at certain adjacent time-steps. Therefore, as shown in Fig. 4c , We propose to omit the encoder at certain time-steps and use the same encoder features for several decoder steps. This allows us to compute these multiple decoder steps in parallel. Specifically, we delete the encoder at time-step t -1 (t -1 < T ), and the corresponding decoder (including the skip connections) takes as input the hierarchical outputs of the encoder E from the previous time-step t, instead of the ones from the current time-step t-1 like the standard SD sampling (for more detail, see Sec. 3.3).When omitting the encoder at a certain time-step, we are able to generate similar images (Fig. 4c ) like standard SD sampling (Fig. 4a , Tab. 1 (the first and second rows) and additional results in Appendix F). Alternatively, if we use a similar strategy for the decoder (i.e., decoder propagation), we find the generated images often fail to cover some specific objects in the text prompt (Fig. 4d ). For example, when provided with prompt "A man with a beard wearing glasses and a beanie", the SD model fails to synthesize "glasses" when applying decoder propagation. This is due to the fact that the semantics are mainly contained in the features from the decoder rather than the encoder [40] .The encoder propagation, which uses encoder outputs from previous time-step as the input to the current decoder, could speed up the diffusion model sampling at inference time. In the following Sec. 3.3, we give further elaborate on encoder propagation.Diffusion sampling, combining iterative denoising with transformers, is time-consuming. Therefore we propose a novel and practical diffusion sampling acceleration method. During the diffusion sampling process t= {T, ..., 1}, we refer to the time-steps where encoder propagation is deployed, as non-key time-steps denoted as t non-key = t non-key 0 , ..., t non-key N -1. The remaining time-steps are dubbed as t key = t key 0 , t key 1 , ..., t key T -1-N . In other words, we do not use the encoder at time-steps t non-key , and leverage the hierarchical features of the encoder from the time-step t key . Note we utilize the encoder E at the initial time-step (t key 0 = T ). Thus, the diffusion inference time-steps could be reformulated as t key , t non-key , where t key ∪ t non-key = {T, ..., 1} and t key ∩ t non-key = ∅. In the following, we introduce both uniform encoder propagation and non-uniform encoder propagation strategies.As shown in Fig. 3a , The encoder feature change is larger in the initial inference phase compared to the later phases throughout the inference process. Therefore, we select more key time-steps in the initial inference phase, and less key time-steps in later phases. We experimentally define the key time-steps as t key ={50, 49, 48, 47, 45, 40, 35, 25 , 15} for SD model with DDIM, and t key ={100, 99, 98, . . ., 92, 91, 90, 85, 80, . . ., 25, 20, 15, 14, 13, . . ., 2, 1}6 , {50, 49, . . . , 2, 1} and {75, 73, 70, 66, 61, 55, 48, 40, 31, 21, 10} for three stage of DeepFloyd-IF (see detail key time-steps selection in Appendix F.2). The remaining time-steps are categorized as non-key time-steps. We define this strategy as non-uniform encoder propagation (see Fig. 4e ). As shown in Fig. 4c , we also explore the non-key time-step selection with fix stride (e.g, 2), dubbed as uniform encoder propagation.Note that our method does not reduce the number of sampling steps. During encoder propagation, the decoder computes for all time-steps, necessitating time embedding inputs for each time-step to maintain temporal coherence (see detail in Appendix D).Tab. 5 reports the results of the ablation study, considering various combinations of key and non-key time-steps. These results indicate that the set of non-uniform key time-steps performs better in generating images. Parallel non-uniform encoder propagation.When applying the non-uniform encoder propagation strategy, at time-step t ∈ t non-key the decoder inputs do not rely on the encoder outputs at time-step t (see Fig. 4e ). Instead, it relies on the encoder output at the previous nearest key time-step. This allows us to perform parallel non-uniform encoder propagation at these adjacent time-steps in t non-key . We perform decoding in parallel from t to t -k + 1 time-steps. This technique further improves the inference efficiency since the decoder forward in multiple time-steps could be conducted concurrently. We indicate this as parallel-batch non-key time-steps. As shown in Fig. 5 (right), this further reduces evaluation time by 41% for the SD model.Prior noise injection. Although the encoder propagation could improve the efficiency in the inference phase, we observe that it leads to a slight loss of texture information in the generated results (see Fig. 6 (left, middle)). Inspired by related works [41, 42] , we propose a prior noise injection strategy. It combines the initial latent code z T into the generative process at subsequent time-step (i.e., z t ), following z t = z t + α • z T , if t < τ , where α = 0.003 is the scale parameter to control the impact of z T . And we start to use this injection mechanism from τ = 25 step. This strategic incorporation successfully improves the texture information. Importantly, it demands almost negligible extra computational resources. We observe that the loss of texture information occurs in all frequencies of the frequency domain (see Fig. 6 (right, red, and blue curves)). This approach ensures a close resemblance of generated results in the frequency domain to both SD and z T injection (see Fig. 6 (right, red and green curves)), with the generated images maintaining the desired fidelity (see Fig. 6 (left, bottom)). In our experiments, we assess the speed-up of our method compared to others for inference acceleration. We also explore combining our method with these approaches.We do not directly compare our method with distillation methods, which offer superior results but involve computationally expensive retraining.Datasets and evaluation metrics. We randomly select 10K prompts from the MS-COCO2017 validation dataset [43] and feed them into the text-to-image diffusion model to obtain 10K generated images.For the transformer architecture diffusion model, we randomly generate 50K images from 1000 ImageNet [44] class labels. For other tasks, we use the same settings as baselines (e.g., Text2Video-zero [4] , VideoFusion [5] , Dreambooth [7] and ControlNet [10] ). We use the Fréchet Inception Distance (FID) [45] metric to assess the visual quality of the generated images, and the Clipscore [46] to measure the consistency between image content and text prompt. Furthermore, we report the average values for both the computational workload (GFLOPs/image) and sampling time (s/image) to represent the resource demands for a single image. See more detailed implementation information on Appendix A. We first evaluate the proposed encoder propagation method for the standard text-to-image generation task on both the latent space (i.e., SD) and pixel space (i.e., DeepFloyd-IF) diffusion models. As shown in Tab. 1, we significantly accelerate the diffusion sampling with negligible performance degradation. Specifically, our proposed method decreases the computational burden (GFLOPs) by a large margin (27%) and greatly reduces sampling time to 41% when compared to standard DDIM sampling in SD.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?Answer: [NA] Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 
text-to-image generation: 
 Similarly, in DeepFloyd-IF, the reduction in both computational burden and time reaches 15% and 24%, respectively. Furthermore, our method can be combined with the latest sampling techniques like DPM-Solver [14] , DPM-Solver++ [47] , and ToMe [35] . Our method enhances sampling efficiency while preserving good model performance, with negligible variations of both FID and Clipscore values (Tab. 1 (the third to eighth rows)).Our method achieves good performance across different sampling steps (Fig. 7 and see Appendix D for quantitative results.). Importantly, these results show that our method is orthogonal and compatible with these acceleration techniques. As shown in Fig. 1 , we visualize the generated images with different sampling techniques. Our method still generates high-quality results (see Appendix F for additional results).Our method allows to use multi-GPU to generate one image. With multi-GPU parallel, our proposed method further accelerates the SD sampling by 77%, whereas DeepCache [22] and CacheMe [23] achieve speedups of 56% and 44%, respectively (See Tab. 2). These results indicate that we achieve superior acceleration compared to DeepCache [22] and CacheMe [23] .We also evaluate our approach on DiT. As reported in Tab. 3, we achieve accelerations of about 29% and 34% for DiT sampling with image resolution of 256 and 512, respectively, while preserving high-quality results (see Figs. 1 and 18 ).Besides the standard text-to-image task, we also validate our proposed approach on other tasks: text-to-video generation, personalized generation, and reference-guided image generation.Text-to-video. To evaluate our method, we combine it with both Text2Video-zero [4] and VideoFusion [5] . As reported in Tab. 4 (the second and fourth rows), when combined with our method, the two methods have a reduction of approximately 22% to 33% in both computational burden and generation time. These results indicate that we are able to enhance the efficiency of generative processes in the text-to-video task while preserving video fidelity at the same time (Fig. 1 (left, bottom)). As an example, when generating a video using the prompt "Fireworks bloom in the night sky", the VideoFusion model takes 17.92 seconds with 16 frames for the task (1.12s/frame), when combined with our method it only takes 12.27s (0.76s/frame) to generate a high-quality video (Fig. 1 (left, bottom)). Personalized image generation. Dreambooth [7] and Custom Diffusion [8] are two approaches for customizing tasks by fine-tuning text-to-image diffusion models. As reported in Tab. 4 (the ninth to twelfth rows), our method, combining with the two customization approaches, accelerates image generation and reduces computational demands. Visually, it maintains the ability to generate images with specific contextual relationships based on reference images. (Fig. 1 Reference-guided image generation. ControlNet [10] incorporates a trainable encoder, successfully generates a text-guided image, and preserves similar content with conditional information. Our approach can be applied concurrently to two encoders of ControNet. In this paper, we validate the proposed method with two conditional controls: edge and scribble. Tab. 4 (the fifth to eighth row) reports quantitative results. We observe that it leads to a significant decrease in both generation time and computational burden. Furthermore, Fig. 1 (middle, bottom) qualitatively shows that our method successfully preserves the given structure information and achieves similar results as ControlNet. User study. We conducted a user study, as depicted in Fig. 8 , and asked subjects to select results. We apply pairwise comparisons (forced choice) with 18 users (35 pairs of images or videos/user). The results demonstrate that our method performs equally well as the baseline methods. We ablate the results with different selections of both uniform and non-uniform encoder propagation. Tab. 5 reports that the performance of the non-uniform setting outperforms the uniform one in terms of both FID and Clipscore (see Tab. 5 (the third and eighth rows)). Furthermore, we explore different configurations within the non-uniform strategy. The strategy, using the set of key time-steps we established, yields better results in the generation process (Tab. 5 (the eighth row)). We further present qualitative results stemming from the above choices. As shown in Fig. 9 , given the same number of key time-steps, the appearance of nine-step nonuniform strategy I, II and III settings do not align with the prompt "Fireflies dot the night sky". Although the generated image in the twostep setting exhibits a pleasing visual quality, its sampling efficiency is lower than our chosen setting (see Tab. 5 (the second and eighth rows)). Effectiveness of prior noise injection. We evaluate the effectiveness of injecting initial z T . As reported in Tab. 6, the differences in FID and Clipscores without z T (the third column), when compared to DDIM and Ours (the second and fourth columns), are approximately 0.01%, which can be considered negligible. While this is not the case for the visual expression of the generated image, it is observed that the output contains complete semantic information with smoothing texture (refer to Fig. 6 (left, the second row)).Injecting the z T aids in maintaining fidelity in the generated results during encoding propagation (see Fig. 6 (left, the third row) and Fig. 6 (right, red and green curves)). 
conclusion: 
 In this work, We explore the characteristics of the encoder and decoder in UNet of the text-to-image diffusion model and find that encoder feature variation is minimal for many time-steps, while the decoder plays a significant role across all time-steps. Building upon this finding, we propose encoder propagation for efficient diffusion sampling, reducing time on both the UNet-based and the transformbased diffusion models on a diverse set of generation tasks. We conduct extensive experiments and validate that our approach can achieve improved sampling efficiency while maintaining image quality.Limitations: Although our approach achieves efficient diffusion sampling, it faces challenges in generating quality when using a limited number of sampling steps (e.g., 5). In addition, even though our proposed parallelization can also be applied to network distillation approaches [18, 17, 19] , we have not explored this direction in this paper and leave it to future research. Fig. 12 visualizes the hierarchical features for DiT [21] blocks, which includes 28 transformer blocks. Through visualization and statistical analysis (see Fig. 12 and Fig. 11 ), we observe that the features in the first several transformer blocks change minimally (i.e., the first 18 blocks), similar to the Encoder in SD, while the features in the remaining transformer blocks exhibit substantial variations (i.e., the remaining 10 blocks), akin to the Decoder in SD.For ease of presentation, we refer to the first several transformer blocks of DiT as the Encoder and the remaining transformer blocks as the Decoder. In Tab. 3, we demonstrate the accelerating performance of our method while applied to DiT-based generation models. 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes] Justification: See Appendix H.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.• The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: The paper poses no such risks.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: We cite papers that we used in Section 4. The models, datasets, and codes used in the paper are introduced in detail in Appendix A.1 and Appendix A.6.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.• If this information is not available online, the authors are encouraged to reach out to the asset's creators. 13. New Assets Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets? Answer: [Yes] Justification: We provide our source code as an anonymized zip file in the supplemental material. Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
: 
 Figure 1 : Results of our method for a diverse set of generation tasks. We significantly increase the image generation speed (second/image).One of the main drawback of diffusion models is the slow inference time for image generation. Among the most successful approaches to addressing this problem are distillation methods. However, these methods require considerable computational resources. In this paper, we take another approach to diffusion model acceleration. We conduct a comprehensive study of the UNet encoder and empirically analyze the encoder features. This provides insights regarding their changes during the inference process. In particular, we find that encoder features change minimally, whereas the decoder features exhibit substantial variations across different timesteps. This insight motivates us to omit encoder computation at certain adjacent time-steps and reuse encoder features of previous time-steps as input to the decoder in multiple time-steps. Importantly, this allows us to perform decoder computation in parallel, further accelerating the denoising process. Additionally, we introduce a prior noise injection method to improve the texture details in the generated image. Besides the standard text-to-image task, we also validate our approach on 
 
 Appendix: We report the run time and GPU memory consumption ratios for text-to-image task. As shown in Tab. 8, we significantly accelerate the diffusion sampling with encoder propagation while maintaining a comparable memory demand to the baselines (Tab. 8 (the last two columns)). Specifically, our proposed method reduces the spending time (s/image) by 24% and requires a little additional memory compared to standard DDIM sampling in SD (DDIM vs. Ours: 2.62GB vs. 2.64GB). The increased GPU memory requirement is for caching the features of the encoder from the previous time-step. Though applying parallel encoder propagation results in an increase in memory requirements by 51%, it leads to a more remarkable acceleration of 41% (DDIM vs. Ours: 2.62GB vs. 3.95GB). In conclusion, applying encoder propagation reduces the sampling time, accompanied by a negligible increase in memory requirements. Parallel encoder propagation on text-to-image tasks yields a sampling speed improvement of 20% to 43%, requiring an additional acceptable amount of memory.Besides the standard text-to-image task, we also validate our proposed approach on other tasks: text-to-video generation (i.e., Text2Video-zero [4] and VideoFusion [5] ), personalized generation (i.e., Dreambooth [7] and Custom Diffusion [8] ) and reference-guided image generation (i.e., Con-trolNet [10] ). We present the time and memory consumption ratios for these tasks in Tab. 9.As reported in Tab. 9 (top), when combined with our method, there is a reduction in sampling time by 23% and 32% for Text2Video-zero [4] and VideoFusion [5] , respectively, while the memory requirements increased slightly by 3% (0.2GB) and 0.9% (0.11GB).The time spent by reference-guided image generation (i.e., ControlNet [10] ) is reduced by more than 20% with a negligible increase in memory (1%). When integrated with our parallel encoder propagation, the sampling time in this task can be reduced by more than half (51%) (Tab. 9 (middle)). Dreambooth [7] and Custom Diffusion [8] are two approaches for customizing tasks by fine-tuning text-to-image diffusion models. As reported in Tab. 9 (bottom), our method, working in conjunction with the two customization approaches, accelerates the image generation with an acceptable increase in memory. Note that our method, either utilizing encoder propagation or parallel encoder propagation, improves sampling speed without compromising image quality (Sec. 4.1 and Sec. 4.3). We conducted GPU memory consumption ratios using the official method provided by PyTorch, torch.cuda.max_memory_allocated 10 , which records the peak allocated memory since the start of the program.We use the fvcore 11 library to calculate the GFLOPs required for a single forward pass of the diffusion model. Multiplying this by the number of sampling steps gives us the total computational burden for sampling one image. Additionally, using fvcore, we can determine the computational load required for each layer of the diffusion model. Based on the key time-steps set in our experiments, we subtract the computation we save from the original total computational load, which then represents the GFLOPs required by our method. Similarly, fvcore also supports parameter count statistics.For the comparisons of text-to-image generation in Sec. 4, we use the official implementation of DPM-Solver [14] , DPM-Solver++ [47] 18 . We maintain the original implementations of these baselines and directly integrate code into their existing implementations to implement our method. Clipscore [46] is a metric for computing the consistency between text and image. The formula for computing Clip-score between image embedding v and text embedding c, as presented in the original paper, is as follows: CLIP-S = w * max(cos(c, v), 0), where a re-scaling operation is performed using w, set to 2.5 in the official implementation of Clipscore [46] . The rationale behind the re-scaling operation, as provided by the official paper (excerpted from [46] , Section 3, Footnote 6), is as follows:While the cosine similarity, in theory, can range from [-1, 1] (1) we never observed a negative cosine similarity; and (2) we generally observe values ranging from roughly zero to roughly .4. The particular value of w we advocate for, w = 2.5, attempts to stretch the range of the score distribution to [0, 1].Therefore, adhering to the aforementioned setting, we employed the official implementation of Clipscore for evaluation, yielding Clipscore data distributed around 0.75. Some works [8, 48] employ re-scaling operations during evaluation, yielding Clipscore values around 0.75, while others [10, 4] do not, resulting in Clipscore values around 0.3. In essence, both approaches are equivalent, differing only in whether re-scaling is applied at the end. We evaluate Clipscore using w = 2.5 for re-scaling, hence yielding Clipscore data around 0.75, unless otherwise stated.Several efficient diffusion model solvers, such as DDIM [13] , DPM-Solver [14] and DPM-Slover++ [47] , have significantly reduced sampling steps. Our method does not reduce the number of sampling steps. In the encoder propagation, the decoder needs to compute for all time-steps, and we require time embedding inputs for all time-steps to maintain temporal coherence. as exemplified by the hands in Fig. 13 (the second cloumn). We increase the number of sampling steps (i.e., 25 steps), and find that the sampling results do not perform as well as DDIM with 50 steps and its variation with FasterDiffusion (Fig. 13 and Tab. 11). As shown in Tab. 11, the FID and Clipscore of SD (DDIM) with 25 time-steps on the MS-COCO 2017 10K subset are 22.16 and 0.761, respectively, significantly worse than the results obtained with 50 time-steps and ours (FID↓: 21.75, 21.08; Clipscore↑: 0.773, 0.783). This demonstrates that our method is not simply reducing the number of sampling steps. With fewer steps (T=10), both DPM-Solver and DPM-Solver++ also exhibit worse FID and Clipscore (see Tab. 12). These results indicate that our method achieves better performance compared to simply reducing the sampling steps.As shown on Tab. 11, we conduct DDIM scheduler inference time with various steps. Although FasterDiffusion (the last row) is slightly longer than DDIM scheduler with 9 steps, our sampling results are much closer the 50-step DDIM generation quality, whereas the 9-step DDIM results are much inferior (see Tab. 11, Tab. 12, and Fig. 13 for examples).For directly applying noise injection to the generation phase, we show image generation examples in Fig. 14 in the rebuttal file. Fewer sampling steps equipped with prior noise injection result in almost no improvement in image quality. Tab. 11 further supports this conclusion. While in our case, FasterDiffusion working without the noise injection will lead to smoother textures, as shown in Fig. 6 .The noise injection technique helps in preserving fidelity in the generated results.In EDM [49] (Karras et al.), they increase the noise level during ODE sampling to improve deterministic sampling, which is referred to as stochastic sampling (i.e., "churn"). Compared to deterministic sampling, it injects noise into the image at each step, which helps to enhance the quality of the generated images. Song et al. [50] first observed that perturbing data with random Gaussian noise makes the data distribution more amenable to score-based generative modeling. Increasing the noise level in EDM is based on Song's paper.Their purpose for injecting noise is to perturb the data, whereas our purpose for injecting noise during sampling is to preserve high-frequency details in the image during the denoising process, preventing the diffusion model from removing high-frequency information as noise (see Figure .6 in the main paper). In addition, our method FasterDiffusion differs from them by only inserting noise in the later stage of the diffusion steps, while they insert noise to all time-steps.DeepCache is developed based on the observation over the temporal consistency between high-level features. In this paper, we have a more thorough analytical study over the SD model features as shown in Fig. 3 , where we find out the encoder features change less, whereas the decoder features exhibit substantial variations across different time steps. This insight motivates us to omit encoder computation at certain adjacent time-steps and reuse encoder features of previous time-steps as input to the decoder in multiple time-steps. We further determine the key time steps in the T2I inference stage, which helps our method to skip the time steps in a more scientific way.It is also important to note that DeepCache is not parallelizable on multiple GPUs since DeepCache needs to use all or part of the encoder and decoder at every time step. FasterDiffusion, on the other hand, only uses the encoder at the key time steps, which enables parallel processing at these time steps and faster inference time cost.It is also worth to notice that FasterDiffusion can be further combined with a wide range of diffusion model based tasks, including Text2Video-zero, VideoFusion, Dreambooth, and ControlNet. In this case, DeepCache often shows much slower speed while applied to these tasks. As an example shown in Table . 13, when combined with ControlNet DeepCache is slower by 24% compared to the FasterDiffusion.More specifically, since the ControlNet model requires an additional encoder, our method Faster-Diffusion is able to execute this extra encoder in a parallel manner and reuse it, and that makes the additional time negligible. On the other hand, DeepCache is reusing the decoder feature, which leads the ControlNet to wait for the additional encoder to complete computation at each time-step, resulting in almost no time saved from skipping the encoder in the standard SD models. The T2I generation results are shown in Figure . 15, FasterDiffusion successfully preserves the given structure information and achieves similar results as the original ControlNet model. We utilize 50, 20 and 20 time-steps for DDIM scheduler [13] , DPM-solver [14] and DPM-solver++ [47] , respectively. We follow the official implementation of DeepFloyd-IF, setting the time-steps for the three sampling stages to 100, 50, and 75, respectively.Text-to-image generation. We experimentally define the key time-steps as t key = {50, 49,15} for SD model with DDIM [13] , and t key = {100, 99, 98, . . .,10} for three stages of DeepFloyd-IF with DDPM [2] . For SD with both DPM-Solver [14] and DPM-Solver++ [47] , we experimentally set the key time-steps to t key = {20, 19, 18, 17, 15, 10, 5}.Other tasks with text-guided diffusion model. In addition to standard text-to-image tasks, we further validate our approach on other tasks with text-guided diffusion model (Sec. 4.2). These tasks are all based on the SD implementation of DDIM [13] . Through experiments, we set the key time-steps to t key = {50, 49, 48, 47, 45, 40, 35, 25, 15} for Text2Video-zero [4] , VideoFusion [5] ,and ControlNet [10] . For personalized tasks (i.e., Dreambooth [7] and Custom Diffusion [8] ), we set the key time steps to t key = {50, 49,In Sec. 3.2, we have demonstrated that encoder propagation (Fig. 16c ) can preserve semantic consistency with standard SD sampling (Fig. 16a ). However, images generated through decoder propagation often fail to match certain specific objects mentioned in the text prompt (Fig. 16d and Tab. 14 (the third row)).We extended this strategy to include encoder and decoder propagation (Fig. 16e ) as well as decoder and encoder dropping (Fig. 16f ). Similarly, encoder and decoder propagation often fail to cover specific objects mentioned in the text prompt, leading to a degradation in the quality of the generated results (Fig. 16e and Tab. 14 (the fourth row)). On the other hand, decoder and encoder dropping are unable to completely denoise, resulting in the generation of images with noise (Fig. 16f and Tab. 14 (the fifth row)). Note that decoder and encoder dropping (Fig. 16f ) is different from directly reducing the number of time-steps. Time embedding is required as input for each time-step, even when using only the encoder or decoder at each time-step.The study participants were volunteers from our college. The questionnaire consisted of 35 questions, each presenting two images: one from the baseline methods (including ControlNet, SD, DeepFloyd- ) and the other from our method FasterDiffusion (one example shown in Fig. 17 ).Users were required to select the image where the target was more accurately portrayed, or choose "both equally good". A total of 18 users participated the questionnaire, resulting in totally 630 samples (35 questions × 1 option × 18 users). As the final results shown in Fig. 8 , the chosen percentages for SD, DeepFloyd-IF, DPM-Solver++, Text-to-Video, ControlNet, and Personalize were 48%, 44%, 51%, 52%, 46%, and 39%, respectively. These results show that our method performs on par with the baseline methods and demonstrate that FasterDiffusion retains the T2I generation quality while reducing the inference time.We showcase our experimental results with the ImageReward [51] and PickScore [52] evaluation metrics over the MS-COCO2017 10K dataset, as shown in the Tab. 15. Our method FasterDiffusion enhances sampling efficiency while preserving the original model performance. ReVersion [53] is a relation inversion method that relies on the Stable Diffusion (SD). Our method retains the capacity to generate images with specific relations based on exemplar images, as illustrated in Fig. 25 (top) .P2P [27] is an image editing method guided solely by text. When combined with our approach, it enhances sampling efficiency while preserving the editing effect (Fig. 25 (bottom) ).To identify key time steps, we conducted empirical analysis of feature changes at adjacent time steps in multistep Stable Diffusion models (50-step model as an example). This analysis is based on statistics from the distribution of features across 100 random prompts, which does not impose a high time cost. We found that the encoder features change minimally in later time steps, whereas the encoder features in earlier time steps exhibit substantial variations compared to later ones. Based on analysis as shown in Section 3.2, we determine the key time-step as t key = {50, 49, 48, 47, 45, 40, 35, 25 , 15} for the 50-step Stable Diffusion model. As a general case, we also applied this key time-step configuration to the ControlNet [10] , Text2Video-zero [4] , VideoFusion [5] , Dreambooth [7] and Custom Diffusion [8] models based on the 50-step SD models. That will not impose any additional searching time for the key time steps for these downstream application scenarios.Existing methods such as OMS-DPM [54] , AutoDiffusion [55] , and DDSM [56] utilize reinforcement learning or machine learning algorithms to search for optimal schedules, time steps, and model sizes. Each search iteration involves frequent image generation and FID calculations. Executing these search algorithms is time-consuming, often exceeding the training time, as noted in DDSM. For instance, the NSGA-II search algorithm used in DDSM incurs a search cost approximately 1.1 to 1.2 times that of training a standard diffusion model. We plan to incorporate the NSGA-II algorithm for automatically searching t key time steps as an enhancement to FasterDiffusion in our future research.Diffusion models generate realistic fake images, which assist individuals in their creative endeavors. However, for those lacking the ability to discern whether the images are generated by a model, it may affect their judgment. Equipped with our technique, using DMs on the edge devices becomes feasible. Also, importantly, our proposed acceleration of DMs does not require access to enormous compute, which is required to the methods based on knowledge distillation [18, 17, 19] . This makes our technique also useful for the many smaller actors in the generative AI field. and DPM-Solver++ [47] , and these methods in conjunction with our proposed approach. Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.See the visualisation of the hierarchical features for all blocks in Appendix AWe define the first several transformer blocks of DiT as the encoder, and the remaining ones of the transform blocks as the decoder.Once we replace the • with specific inputs in E(•)s, we define that it represents the feature of E(•)sThe feature resolution is half of the previous one in the encoder and two times in the decoder. Note that the feature resolutions of E(.)8, B(.)8 and D(.)64 do not change in the SD model.Each boxplot contains the minimum (0th percentile), the maximum (100th percentile), the median (50th percentile), the first quartile (25th percentile) and the third quartile (75th percentile) values of the feature Frobenius norm (e.g., {F E(z T ,c,T )s , ..., F E(z 1 ,c,1)s }).The ellipsis in t key denotes each time-step between the time-steps on either side of the ellipsis. For example, 80...25 means that every time-step between 80 and 25 is included.We use the official implementation of Clipscore[46] to obtain around 0.75 but around 0.3. See Appendix C. 


Decision: The manuscript has been reviewed by four reviewers, all of whom, after the rebuttal, gave positive ratings (including borderline positive).

In general, the reviewers are happy with the novelty, motivation, significance, and experimental validation.

The AC agrees with the consensus, and recommends the manuscript to be accepted. Congrats!

Reviews: review: paper_summary: paper_summary: This method can be applied at inference time without requiring retraining to improve sampling efficiency while maintaining high image quality and can be combined with other approaches to speed up diffusion models. The approach is effective across various conditional diffusion tasks, including text-to-video generation (e.g., Text2Video-zero, VideoFusion), personalized image generation (e.g., Dreambooth), and reference-guided image generation (e.g., ControlNet). Using encoder features from previous time steps as input for decoding multiple later time steps allows for concurrent decoding, further accelerating SD sampling by 41%, DeepFloyd-IF sampling by 24%, and DiT sampling by 34%. Besides, a prior noise injection strategy is introduced to preserve image quality, maintaining texture details in the generated images.
strength_weakness: strength_weakness: The visualization results for smaller steps appear suboptimal, potentially due to the absence of consistency models or adversarial distillation models for verification. This raises concerns about the foundational assumption of maintaining high similarity throughout the process. It is crucial to explore whether incorporating consistency models or adversarial distillation methods could enhance the quality of results and ensure the assumption of high similarity holds even in smaller steps. Providing more robust verification methods would strengthen the validity and reliability of the visual outputs generated by the proposed approach.
questions: questions: Extra Cost to Identify Key Time-Steps: What are the additional costs associated with identifying key time steps in the diffusion process? Is this identification process capable of being learned dynamically, or is it independent of the data? Understanding the computational overhead or resource allocation needed for this identification process is crucial for evaluating the overall practicality of the method.

Enhancing Application Area with Smaller Steps Diffusion Models: Applying this approach to smaller steps diffusion models has the potential to significantly expand its applicability. By effectively handling smaller time steps, the method could be utilized in a wider range of scenarios, enhancing its versatility and impact.
limitations: limitations: Please refer to the weakness and questions part.
paper_summary: paper_summary: This paper presents an extensive study of the evolution of internal activations in diffusion U-Nets and uses their findings to motivate a training-free approach for accelerating sampling from diffusion models. The method is demonstrated to successfully speed up inference in a variety of settings, including different architectures (including non-hierarchical DiTs), different samplers, and some modified inference processes.
strength_weakness: strength_weakness: - No details are provided about the user study, and false claims relating to it are made in the checklist [questions 14 and 15].
- The main method seems to only be a minor incremental evolution of DeepCache's methodology (encoder features are cached and reused without adaptation of features or network in following steps, optionally multiple times, with the main difference being what is defined as the encoder), and seems to provide a *reduced* speedup when compared in a fair single-GPU setting [Tab. 1, Sec. 4.1 l. 271: 41% speedup vs. 56% speedup, at comparable quality levels].
- The presentation of the paper needs some work. Font sizes in some figures are excessively small [cf. Figs. 1, 2, 3, 7], Tabs. 2 and 3 are excessively compressed up to a point at which readability is impaired. The paper would also benefit from a thorough re-read and corrections for the camera-ready version. Especially grammar errors that substantially affect the claims made in the paper (e.g., sampling time reduction "to 41%" [l. 271] instead of "by 41%") and typos that change the meaning of words (e.g., "rudely" [l.105]), but also names of methods (e.g., "DeepFolyd-IF" [l. 60], "Stable Diffuison" [l. 525]).
questions: questions: - Can the authors provide examples and evidence of practically relevant settings where DeepCache fails or underperforms significantly compared to the proposed method (such as a combination with DPM-Solver(++), ControlNet, or Dreambooth etc)? These would help address the main method-related weakness mentioned.
- What do the ellipses in the key time step steps mean? Is every time step between 80 and 25 included for $\{80,...,25\}$?
- Is there a relevant difference between the proposed "prior noise injection method" and the various methods that artificially increase the noise level slightly during ODE sampling, such as ``churn'' in (Karras et al., Elucidating the Design Space of Diffusion-Based Generative Models, NeurIPS 2022)?
- Are all the videofusion samples in the supplementary material just corrupted or is there something to be demonstrated there?
- Does parallel-batch encoder propagation limit the possible batch size on a single GPU compared to other sampling methods? If so, could the authors provide a performance comparison that individually maxes out the batch size for the various approaches (including competing ones) and then reports the samples/second? As generating multiple variants simultaneously is a standard approach when generating images with diffusion models in practice, this is an important avenue for optimization. Specifically, I am worried that at a maximal batch size, parallel encoder propagation will only offer a negligible speedup compared to encoder propagation (due to larger generation batch sizes being possible with non-parallel encoder propagation, as VRAM usage is substantially different between the two methods [Tab. 8]), at which point this method would provide less than half of the speedup provided by prior art.
limitations: limitations: The authors have adequately addressed the limitations of their work.
paper_summary: paper_summary: This paper presents an approach to accelerate diffusion model inference by capitalizing on the minimal change in encoder features across time steps. The proposed encoder propagation strategy reduces encoder computations by reusing encoder features from previous time steps. The proposed method is comparable to existing acceleration methods and demonstrates effective acceleration across diverse tasks.
strength_weakness: strength_weakness: 1.	The evaluation metrics are insufficient, as concerns exist regarding the universality of FID in assessing generative models. It is recommended to validate performance on additional metrics, such as the newly proposed ImageReward [1] and Pick Score [2].

2.	It is unclear whether the latency is comparable when fewer sampling steps are used (Table 11 and Table 12). Moreover, the performance improvement is marginal compared to results with fewer sampling steps, and mainly stems from prior noise injection, which is not the core contribution of the paper. I am also curious whether fewer sampling steps, if equipped with prior noise injection, could achieve similar performance as the results presented in this paper.

3.	Comparing multi-GPU results with those single-GPU results from DeepCache seems unfair (Table 1). Besides, the acceleration outcomes on a single GPU are marginally improved when compared with other acceleration methods.

4.	The DiT does not inherently include concepts of Encoder and Decoder; thus, categorizing based solely on observations into Encoder and Decoder is somewhat imprecise.

5.	The layout of the paper is somewhat disorganized; the order of tables does not align with the sequence of their introduction in the text (e.g., Table 1 and Table 3), making it hard to follow.

[1] Xu J, Liu X, Wu Y, et al. Imagereward: Learning and evaluating human preferences for text-to-image generation. NeurIPS 2023.

[2] Kirstain Y, Polyak A, Singer U, et al. Pick-a-pic: An open dataset of user preferences for text-to-image generation. NeurIPS 2023.
questions: questions: See weakness for details.
limitations: limitations: The authors have discussed both limitations and potential negative social impacts.
paper_summary: paper_summary: The authors analyze the encoder features of diffusion UNet and find that they share a lot of similarities across certain time steps. Based on this observation, the authors propose to reuse the encoder features and do parallel computation in sampling. This significantly fastens the generation process. Additionally, they also introduce a prior noise injection method to improve the generation quality. Extended experiments demonstrate their effectiveness across a wide range of generation tasks, including T2I, T2V, DreamBooth, controlNet. And their method can be seamlessly combined with various noise schedulers.
strength_weakness: strength_weakness: I think the manual selection of key time-steps may not be optimal. Is it possible to use some automatic strategies to determine the key time-steps in your work? Otherwise when we use a new architecture we would have to redo the encoder feature analysis experiment and manually define the key-time steps again. I think there are several works have explored the selection of key time-steps. Discuss on them will make your work even stronger.

[1] OMS-DPM: Optimizing the Model Schedule for Diffusion Probabilistic Models
[2] AutoDiffusion: Training-Free Optimization of Time Steps and Architectures for Automated Diffusion Model Acceleration
[3] Denoising Diffusion Step-aware Models
questions: questions: Please see the weakness section.
limitations: limitations: Yes, they discuss their limitation in section 5.

Comments: title: IRB considerations
comment: Thank you for providing the additional evaluation for the max batch size setting. In my opinion, the presented speedup of ~20% (down from the 41% presented in the paper, which was already lower than that of other methods) in this fair evaluation setting is not particularly favorable for the method proposed in this work. Thus, I will retain my initial increased score.
We are grateful for your prompt response, and we are glad to clarify and address your concerns further.

**User study details and its IRB concerns**

Thanks for your attention to the user study, particularly in relation to the IRB rules. In any future version, we are committed to incorporating detailed experimental information on the user study accompanied by a thorough explanation of IRB approvals, which we already addressed in the rebuttal phase. 

**Single-GPU inference time comparison**

To further relieve your concerns over the single-GPU setting, we conducted experiments under the same setup. Our observation indicates that SD1.5 model attains a peak batch size of 30, with an average inference time of 1.903 seconds per image. 
Upon integration with our approach FasterDiffusion, the maximum batch size experiences a minor reduction to 25, but the inference time per image demonstrates a significant decrease to 1.551 seconds. 
These results highlight that, despite the constraints of the maximum batch size per GPU, our method delivers a notable enhancement in inference speed. FasterDiffusion presents a robust solution for navigating time-space trade-offs effectively.

**Video format issue**

We appreciate your valuable feedback and the efforts you made in examining our video examples.  We will investigate this matter thoroughly and guarantee the application of standard encoding in any future submissions. 
We are glad to hear that you were able to review the samples and found them reasonable.
In the future version, we aim to release a project page where all examples, including the Videofusion samples, are easily accessed for convenience.
Thank you to the authors for the detailed response. Most of my concerns have been addressed, so I have raised my score.
Dear Reviewer wo3A:

We are grateful for your expeditious response and we are glad to clarify and address your concerns further.

For the PickScore, we used the calculation method recommended by the official github repository reference:

> For a prompt $x$ and a image $y$，the scoring function $s$ computes a real number by representing $x$ using a transformer text encoder and $y$ using a transformer image encoder as d-dimensional vectors, and returning their inner product:
$$
s(x,y) = E_{txt}(x) · E_{img}(y) · T
$$
Where $T$ is the learned scalar temperature parameter of CLIP.

Following that, assuming the original image generated by SD is $y_1$, and the image generated after combining with Ours is $y_2$, we can calculate the preference distribution vector $p$:
$$
p_i = \frac{\exp  s(x, y_i)}{\sum_{j=1}^2 \exp s(x,y_j)}
$$
PickScore official recommended get probability, if there are multiple images available for selection.

We also calculated the original PickScore scores, as shown in the table below:

|                  | PickScore$\uparrow$ | s/image |
| ---------------- | ------------------- | ------- |
| SD(DDIM)         | 21.43               | 2.42    |
| SD(DDIM) w/ Ours | 21.35               | 1.42    |

Our method FasterDiffusion enhances sampling efficiency while preserving the original model performance.

We hope that these clarifications have addressed your concerns satisfactorily and look forward to any further feedback you may have. We are committed to enhancing our work based on your expert guidance.
Thank you for your response.
I'm still a bit confused, as typically, the PickScore should be a specific numerical value, not a percentage.
After carefully reviewing the authors' rebuttal and review responses, only some of my concerns have been resolved. 

One important remaining concern is comparisons in fair settings.
The authors were able to address my concern about a reduced single-GPU speed compared to the previous DeepCache in the case of using ControlNets, but not in general. More importantly, as far as I could see, the authors still did not provide any results that max out the batch size per GPU, as suggested in my initial review. I think this is an essential comparison, as VRAM usage is one of the main practical limitations for practical deployments, both due to the cost of GPUs and, more importantly, because larger batch sizes typically make inference more efficient. This is relevant even if only single images are generated per user, as any standard inference backend will batch multiple calls for efficiency. The fact that the method proposed in this work shows a substantial VRAM increase would suggest that this method's speed is likely limited if one increases the batch size to the maximum possible on a reasonably sized GPU. Additionally, I disagree that the authors' argument regarding multi-GPU inference is practically relevant. Many practical inference systems generate multiple images at once (at which point one can just split up the samples over multiple GPUs instead of splitting up the inference process of each sample over multiple GPUs). In other cases, [unless one is already using a top-of-the-line available GPU, which is heavily non-standard for diffusion production loads], one can likely increase the inference speed by a similar amount at the same cost by just using one higher-end GPU instead of two GPUs.

I was now also able to open the problematic Videofusion example videos from the supplementary material on a Windows machine using the standard media player. My issues with opening them persist on the two different MacOS-based systems I tried opening them with initially (using the standard viewer as suggested by the authors). If the authors are aware of any uncommon codec choices etc that might have affected this, I'd recommend that they switch to a standard encoding for future submissions to prevent these problems. Most importantly, I have been able to look at the samples now, and they seem reasonable.

As the authors have addressed my major critical concerns about the user study and some minor concerns but left my major concerns about fair comparisons insufficiently addressed, I'm raising my score from 3 to 5. With aforementioned shortcomings, I think the main value this work provides in its current form is the analysis of the network, but I'm sceptical about the practical relevance of the proposed method to exploit them in general use cases.
Given the authors' responses to my concerns about the user study, my main concerns relating to it and the missing material in the paper have been resolved. To the best of my knowledge, the user study performed by the authors qualifies as "research involving human subjects" in most jurisdictions, but this might not be the case in theirs, and I am no ethics reviewer who would have the qualification to judge in a more precise manner. I would still urge the authors to double-check that they followed all the rules set by their institutions, country, and NeurIPS, and expect them to include all relevant details, especially those presented as a part of the rebuttal, in the final version.
I appreciate your emphasis on the significance of IRB approvals for user studies in current computer vision research. It is indeed crucial to ensure that research involving human subjects adheres to ethical standards and guidelines, and obtaining IRB approval is a key step in this process. 

The Institutional Review Board (IRB) is a formalized entity tasked with upholding research ethics by conducting a rigorous assessment of the methodologies proposed for research endeavors that involve human subjects. The primary objective of these reviews is to safeguard the wellbeing of study participants, ensuring that they are not subjected to harm and that their rights and welfare are duly protected. This is achieved through a meticulous examination of research protocols and associated documentation, with the intention of identifying and mitigating potential risks. 

In the context of this research paper, it is noteworthy that the study design does not pose any risks of harm to the participants. Instead, we employ a user study methodology wherein participants are invited to express their opinions regarding the images generated by our approach in comparison to those produced by alternative methods. To further safeguard the participants, we have implemented rigorous safety checks within our image generation pipelines, ensuring that no harmful or inappropriate content is included in the output images. Consequently, the study adheres to ethical principles by minimizing potential risks and prioritizing the protection of participant rights and welfare.

In this context, we address checklist item 15 as outlined by the regulatory body, as our study does not involve research concerning human subjects. Rather, it may be construed as an online questionnaire administered for data collection purposes.

We endeavored to address all your concerns comprehensively to ensure clarity in the explanations provided.  If you have further questions about our research, please feel free to engage with us for further discussion.
> **W1. User study details.**
>
> The study participants were volunteers from our college. The questionnaire consisted of 35 questions, each presenting two images: one from the baseline methods (including ControlNet, SD, DeepFloyd-IF, DPM-Solver++, Dreambooth, Custom Diffusion, Text-to-Video, etc.) and the other from our method FasterDiffusion (one example shown in Figure.29).
>
> Users were required to select the image where the target was more accurately portrayed, or choose “both equally good”. A total of 18 users participated the questionnaire, resulting in totally 630 samples (35 questions × 1 option × 18 users). As the final results shown in Figure.8, the chosen percentages for SD, DeepFloyd-IF, DPM-Solver++, Text-to-Video, ControlNet, and Personalize were 48%, 44%, 51%, 52%, 46%, and 39%, respectively. These results show that our method performs on par with the baseline methods.

Thanks for providing these details and the screenshots in the rebuttal PDF, but this does still not resolve the major concerns about question 15 (IRB board approval, potential risks for participants).