AsyncDiff: Parallelizing Diffusion Models by Asynchronous Denoising

Authors: zigeng  chen
xinyin  ma
gongfan  fang
zhenxiong  tan
xinchao  wang
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-03 09:48:14

Abstract: Diffusion models have garnered significant interest from the community for their great generative ability across various applications. However, their typical multi-step sequential-denoising nature gives rise to high cumulative latency, thereby precluding the possibilities of parallel computation. To address this, we introduce AsyncDiff, a universal and plug-and-play acceleration scheme that enables model parallelism across multiple devices. Our approach divides the cumbersome noise prediction model into multiple components, assigning each to a different device. To break the dependency chain between these components, it transforms the conventional sequential denoising into an asynchronous process by exploiting the high similarity between hidden states in consecutive diffusion steps. Consequently, each component is facilitated to compute in parallel on separate devices. The proposed strategy significantly reduces inference latency while minimally impacting the generative quality. Specifically, for the Stable Diffusion v2.1, AsyncDiff achieves a 2.7x speedup with negligible degradation and a 4.0x speedup with only a slight reduction of 0.38 in CLIP Score, on four NVIDIA A5000 GPUs. Our experiments also demonstrate AsyncDiff can be readily applied to video diffusion models with encouraging performances.

Main paper: 
 introduction: 
 Diffusion models [13] stand out in generative modeling and have significantly advanced various fields including text-to-image [43, 41, 45, 46, 72, 78 ] and text-to-video generation [64, 9, 61, 21, 2] , Figure 2 : By preparing each component's input beforehand, we enable parallel computation of the denoising model, which substantially reduces latency while minimally affecting quality.image translation [49, 56, 23] , audio generation [22, 14, 44] , style transfer [62, 4, 17] , low-level vision tasks [47, 60, 40, 26, 8, 70, 3] , image editing [19, 66, 51, 77] , and 3D model generation [42, 18, 37] , among others. However, their widespread application is hindered by the high latency inherent in their multi-step sequential denoising process. This issue becomes more pronounced as the complexity and size of the models increase to enhance generative quality.In response to these challenges, significant research efforts are directed toward enhancing the efficiency of diffusion models. Notably, training-free acceleration methods have garnered increasing popularity due to their low cost and convenience. Numerous studies [35, 63, 76, 67, 53, 25, 33, 57, 34] improve inference speed by skipping redundant calculations in the denoising process. As computational resources grow rapidly, distributing computations across multiple devices has become a more promising approach. Recent advances [52, 24, 58] demonstrate that using distributed computing to parallelize inference effectively increases the acceleration ratio for diffusion models while maintaining acceptable generative quality. Though these methods succeed in parallelizing the diffusion models, they require iterative refining [52] or displaced patch parallelism [24] , resulting in a larger number of model evaluations or low GPU utilization correspondingly. Thus, we wish to propose a new parallel paradigm for diffusion, akin to the model parallelism in distributed computing [15, 38, 28, 16, 39, 65] , which divides the denoising model into several components to be distributed on different GPUs. The primary challenge lies in the inherent sequential denoising process of diffusion models. Each step in this process depends on the completion of its predecessor, forming a dependency chain that impedes parallelization and significantly increases inference latency. Our approach seeks to disrupt this chain, allowing for the parallel execution of the denoising model while closely approximating the results of the sequential process.In this paper, we introduce AsyncDiff, a universal, distributed acceleration paradigm that innovatively explores model parallelism in diffusion models. As shown in Fig 2,  our method sequentially partitions the heavyweight denoising model ϵ θ into multiple components {ϵ n θ } N n=1 based on computational load, assigning each to a separate device. Our core idea lies in decoupling the dependencies between these cascaded components by leveraging the high similarity in hidden states across consecutive diffusion steps. After the initial warm-up steps, each component takes the output from the previous component's prior step as the approximation of its original input. This transforms the traditional sequential denoising into an asynchronous process, allowing components to predict noise for different time steps in parallel. Additionally, we incorporate stride denoising to skip redundant calculations and reduce the frequency of communication between devices, further enhancing efficiency.Through extensive testing across multiple base models, our method effectively distributes the computational burden across various devices, substantially boosting inference speed while maintaining quality. Specifically, with the text-to-image model Stable Diffusion v2.1 [43] , our method achieves a 1.8x speedup with only a marginal 0.01 drop in CLIP Score [11] , and a 4.0x speedup with a slight 0.38 reduction in CLIP Score on two and four NVIDIA A5000 GPUs, respectively. For video diffusion models, AnimateDiff [9] and Stable Video Diffusion [2] , our approach significantly reduces latency by tens of seconds, effectively preserving video quality.In summary, we present a novel distributed acceleration method for diffusion models that significantly reduces inference latency with minimal impact on generation quality. This is achieved by replacing the sequential denoising process with an asynchronous process, allowing each component of the denoising model to run independently across different devices. Extensive experiments on both image and video diffusion models strongly demonstrate the effectiveness and versatility of our method.2 Related Works Diffusion Models. Diffusion models have attracted significant attention due to their powerful generative capabilities across various tasks. Sohl-Dickstein et al. [54] first proposed diffusion probabilistic models. Ho et al. [13] with the introduction of Denoising Diffusion Probabilistic Models (DDPM), enhancing training efficiency and generation quality. Rombach et al. [43] advanced these models by incorporating latent spaces, enabling high-resolution image generation. Despite these advancements, the high latency of the iterative denoising process remains a limitation.Inference Acceleration. Training-based acceleration methods focus on reducing sampling steps [48, 71, 32, 50, 69] or optimizing model architectures [27, 80, 7, 73, 68, 6] . However, these methods incur high training costs and complexity. Training-free methods are gaining popularity due to their ease of use. Some approaches develop fast solvers for SDE or ODE to improve sampling efficiency [31, 1, 30, 74, 81] . Other works [35, 63, 76, 67, 53, 25, 33, 79] observed special characteristics of diffusion models and skipped the redundant computation within the denoising process.Parallelism. The parallelism strategy presents a promising yet underexplored approach to accelerating diffusion models. ParaDiGMS [52] implements Picard iterations for parallel sampling, yet its practical speed-up ratio is modest, and it struggles to maintain consistency with original outputs. Faster Diffusion [25] introduces encoder propagation but significantly compromises quality, and its parallelization remains theoretical. Distrifusion [24] adopts patch parallelism, dividing highresolution images into sub-patches to facilitate parallel inference on each patch by reusing stale activation maps from each layer. However, this approach lacks flexibility across different data types or tasks, often encountering low resource utilization. Furthermore, its reliance on reusing per-layer activation maps greatly increases GPU memory demands thus introducing additional challenges for realistic applications. In contrast, our method uniquely implements model parallelism through asynchronous denoising, achieving substantial acceleration while maintaining a stable resource usage ratio and minimal impact on quality. 
preliminary: 
 Diffusion models [13] are a dominant class of generative models that transform Gaussian noise into complex data distributions via a Markov process. The forward process is defined by:EQUATIONwhere {β t } progressively increases noise until the data becomes indistinguishable from noise. The reverse process, essential for data reconstruction, involves iterative denoising:EQUATIONwhere µ θ (x t , t) is the predicted mean and σ 2 t is the variance. For DDIMs [55] , the reverse update is deterministic:EQUATIONwhere α t is the cumulative product of (1 -β t ). These processes are computationally intensive, influencing the quality of generated samples and necessitating efficient inference methods for practical applications.Traditional diffusion models employ a sequential and synchronous denoising process. At each time step t, the noise-prediction model ϵ θ estimates the noise ϵ t based on the noisy image x t and the time� � � � GPU 3 (� � 4 ) T � �-1 T T-1 T-1 � � T � �-1 T-1 � � T � �-1 T-1 � �-1 T-2 T-2 T-2 T-2 � �-2 T-3 T-3 T-3 T-3 � �-3 T-4 T-4 T-4 T-4 � �-4 � �-5 Warm-up Parallel Parallel Parallel Parallel GPU 2 (� � 3 ) GPU 1 (� � 2 ) GPU 0 (� � 1 )Communication within device Communication across devices T: Time embedding embedding t. The image for the next step, x t-1 , is then generated using a sampler function S(x t , ϵ t , t). This process is iterative, where the generation of ϵ t at each step is dependent on the completion of the previous denoising step, making the process slow, particularly when ϵ θ is computationally intensive.To address the limitations of high latency in diffusion models, leveraging multiple GPUs for distributed inference is a promising solution. Existing studies primarily focus on patch parallelism [24] , where the input image is divided into patches, each processed on a different GPU. While this strategy efficiently distributes computational loads, it still retains the bottleneck of sequential denoising, as each patch must undergo the complete denoising process iteratively. In contrast, our asynchronous diffusion model innovatively introduces a model parallelism strategy. By approximating the sequential denoising as an asynchronous process, this approach enables parallel inference of the noise prediction model, effectively reducing latency and breaking the constraints of sequential execution. Asynchronous Denoising. Figure 3 illustrates our approach to the asynchronous denoising. For a denoising process consisting of T steps, the initial w steps are designated as a warm-up phase, where w is significantly smaller than T . During this phase, the denoising model ϵ θ operates using standard sequential inference. After warm-up steps, rather than splitting the input image, we partition the denoising model ϵ θ into N sequential components, expressed as ϵ θ = {ϵ 1 θ , ϵ 2 θ , ..., ϵ N θ }. Each component is divided to handle a comparable computational load and assigned to a distinct device. This equitable division aims to equalize the time cost of each component to approximately l(ϵ θ )/N , thus minimizing the overall maximum latency. In this setup, original noise prediction for x t can be represented as a cascading operation through these sub-models, defined mathematically as:EQUATIONAlthough each device can independently compute its assigned component, the dependency chain persists because the input for each component ϵ θ,n is derived from the output from its preceding component ϵ θ,n-1 . Therefore, despite the distribution of model components across multiple devices, full parallelization is constrained by these sequential dependencies.Our principal innovation is to break the dependency between cascaded components by utilizing hidden features from previous steps. Observations indicate that the hidden states of each block in the denoising model always exhibit substantial similarity across adjacent time steps. Leveraging this, each component at time step t can take the output from the preceding component at time step t -1 as the approximation of its original input. Specifically, the n-th component ϵ n θ (, t) receives the output of θ and ϵ 2 θ are skipped at time step t. A single parallel batch results in the completion of denoising for two steps, producing x t-1 and x t-2 .GPU 3(� � 3 ) � �-2 � �-1 � � t-ϵ n-1 θ (•, t -1). This alteration allows the noise prediction for x t to be represented as follows:EQUATIONIn this new framework, noise prediction ϵ t is derived from components executed across N previous time steps. This transforms the denoising process from sequential to asynchronous, as the prediction of noise ϵ t already begins before denoising at step t + 1 is completed. At each time step, the N components are running as parts of the noise prediction model for the next N steps. Specifically, the n-th component ϵ n θ , computed in parallel at time t, contributes to the noise prediction for the future time step t -N + n. Figure 3 depicts this asynchronous process using a U-net model with N set to 4. The strong resemblance of hidden states between consecutive diffusion steps enables the asynchronous process to closely mimic the denoising results of the original sequential process.Model Parallelism. By transitioning to an asynchronous denoising strategy, the dependencies among components within the same time step are eliminated. This adjustment allows each component's input for time step t to be prepared in advance, enabling the N split components to be processed concurrently across multiple devices. Once computed, the outputs from each component must be stored and then broadcasted to other devices to facilitate parallel processing for subsequent time steps. In contrast, in the traditional sequential denoising process, the time cost for each step accumulates as follows:EQUATION) By adopting asynchronous denoising to enable parallel computation of each component, the cost for each time step is now given by:EQUATIONwhere max() represents taking the maximum value, and C(comm.) indicates the communication cost across multiple GPUs. As the model components are equally divided by computational load, their time costs are similar, allowing us to approximate the overall cost of each time step as:EQUATIONSince the communication overhead C(comm.) is generally much lower than the model's execution time, it leads to significant overall cost reductions. Moreover, increasing N further reduces time costs but complicates the accurate approximation of the original denoising process.Stride Denoising. While asynchronous denoising reduces latency by parallelizing the denoising model, it completes only one denoising step at a time. To enhance efficiency, we introduce stride denoising, which completes multiple denoising steps simultaneously through a single parallel computation. The diagram is illustrated in Figure 4 , where we set the stride to 2 for clarity. Unlike the continuous broadcasting of hidden states at each time step, stride denoising broadcasts them every two steps. As depicted, at time step t, we conduct denoising alone, and at time step t -1, we compute and broadcast the hidden states for the next parallel computation round. Consequently, the hidden states from time step t are not required, allowing us to skip the calculations for ϵ 1 θ and ϵ 2 θ at this step. In this stride, only ϵ 3 θ (•, t), ϵ previously broadcast hidden states, enabling their parallel processing. Both ϵ 3 θ (•, t) and ϵ 3 θ (•, t -1) share the same feature from ϵ 2 θ (•, t + 1), so the stride should be kept small to maintain quality. Stride denoising effectively reduces both computational load and communication demands by decreasing the parallel computing rounds needed to complete the process. Compared to the significant improvements it brings in efficiency, the quality sacrifice is minimal and can be entirely compensated for by slightly increasing the warm-up steps. We also illustrate the full schematic of it in Appendix Figure 7 .Multi-Device Communication. Parallel inference of the model necessitates efficient communication between devices, as each component ϵ n θ must access the cached hidden state from the preceding component ϵ n-1 θ , which resides on a different device. Post each parallel computation batch, each device stores the current hidden state needed for the next parallel batch. These states, encompassing all component outputs, are then broadcast to all participating devices before the next parallel computation batch. Although each component ϵ n θ primarily uses the cached output of ϵ n-1 θ for its input, it may require residual features [10] from other components. Therefore, it's crucial to broadcast the stored states from every component across all devices before each round of parallel computation.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?Answer: [NA]Justification: The paper does not include theoretical resultsGuidelines:• The answer NA means that the paper does not include theoretical results.• All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced. • All assumptions should be clearly stated or referenced in the statement of any theorems.• The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition. • Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material. • Theorems and Lemmas that the proof relies upon should be properly referenced. 
implementation details: 
 Base models. We validated the broad applicability of AsyncDiff through extensive testing on several diffusion models. For text-to-image tasks, we experimented with three versions of Stable Diffusion: SD 1.5, SD 2.1 [43] , and Stable Diffusion XL (SDXL) [41] . Additionally, we explored the effectiveness of AsyncDiff on video diffusion models using Stable Video Diffusion (SVD) [2] and AnimateDiff [9] . All models were evaluated using 50 DDIM steps. We facilitated communication across multiple GPUs using the broadcast operation from torch.distributed, powered by the NVIDIA Collective Communication Library (NCCL) backend.Table 1 : Quantitative evaluations of AsyncDiff on three text-to-image diffusion models, showcasing various configurations. 'N' indicates the number of components into which the model is divided, and 'S' represents the denoising stride. MACs quantifies the computational load per device for generating a single image throughout the denoising process. Dataset and Evaluation Metrics. We assess the zero-shot generation capability using the MS-COCO 2017 [29] validation set, which comprises 5,000 images and captions. For image generation, quality is measured by the CLIP Score (on ViT-g/14) [11] and Fréchet Inception Distance (FID) [12] , with LPIPS [75] used to check consistency with original outputs. In video generation, quality is evaluated by averaging the CLIP Score across all frames of a video. We also report MACs per device and latency to gauge efficiency comprehensively. All latency measurements were conducted on NVIDIA A5000 GPUs equipped with NVLINK Bridge.Improvements on Base Models. Table 1 displays our acceleration outcomes for three fundamental image diffusion models under various configurations. In this context, 'N' represents the number of segments into which the denoising model is divided, and 'S' denotes the stride of denoising for each parallel computation batch. Our approach, AsyncDiff, not only significantly accelerates processing but also minimally impacts generative quality. The speedup ratio is almost proportional to the number of devices used, demonstrating efficient resource utilization. Visualization results in Figure 5 (a) illustrate the high generative quality achieved even with substantially reduced latency. Although achieving pixel-level consistency with the original output is challenging at high acceleration ratios, the generated image still effectively conveys the semantic information in the prompt, which is crucial for generative results.Pixel-level Consistency by Warm-up. In Table 2 , we explore the balance between pixel-level consistency and processing speed by adjusting the warm-up steps in the diffusion models. As the initial steps of these models play a crucial role in reconstructing the global structure based on text prompts [76] , a modest increase in warm-up steps can significantly enhance consistency with the Comparison with Acceleration Baselines. We evaluated our AsyncDiff method on SD 2.1 against two other parallel acceleration methods: Faster Diffusion [25] and Distrifusion [24] . Faster Diffusion employs encoder propagation but compromises significantly on generative quality. As its parallelism maintains theoretical and lacks a multi-device implementation, we cannot measure its realistic latency with more than one GPU. Its ideal speed-up on 2 devices is about 1.9x. Distrifusion, on the other hand, uses patch parallelism for distributed acceleration but faces potential issues with low resource utilization and high GPU memory demands.According to Table 3 , our method achieves the same operational speed using only 4 GPUs and 3 GPUs as Distrifusion does with 8 GPUs and 4 GPUs, respectively. Additionally, our method requires almost the same amount of memory as the original setup, whereas Distrifusion significantly increases memory requirements, posing extra challenges for practical applications. In terms of generative quality, AsyncDiff and Distrifusion both mirror the original diffusion model's performance at a 1.6x acceleration ratio. However, at higher speedup ratios of 2.3x and 2.7x, our method demonstrates significantly superior generative quality. Qualitative comparisons in Fig 6 further show that AsyncDiff maintains better pixel-level consistency with the original input compared to Distrifusion.As presented in of our method. Video generation, often constrained by exceptionally high latency and substantial computation load, greatly benefits from our approach. For a 50-step video diffusion model, AsyncDiff significantly reduces latency-by tens or even hundreds of seconds-while preserving the quality of generated content. Qualitative results shown in the Appendix. D further corroborate the effectiveness of our method. AsyncDiff achieves an impressive acceleration ratio of over three times while still producing videos that closely match the prompt descriptions, ensuring the rationality of actions and details. These findings highlight the substantial potential of AsyncDiff in accelerating the inference process of video diffusion models.We introduce stride denoising to further enhance the efficiency of the asynchronous denoising process. Stride denoising completes multiple steps simultaneously through a single parallel computation, reducing the number of parallel rounds and communication frequency across devices. For a diffusion process with T steps and warm-up step W , the number of broadcasts decreases from T -W to (T -W )//2 with a stride of 2. This strategy also reduces the computational load on each device by skipping unnecessary calculations. Table 5 shows the effects of stride denoising in our parallel framework with 3 and 4 devices. Stride denoising significantly lowers overall latency and the proportion of communication time, especially as the number of devices used increases. While stride denoising slightly impacts generation quality, this effect is minimal and can be mitigated by a modest increase in warm-up steps, preserving efficiency and maintaining quality.With the recent rise of advanced sampling algorithms for diffusion models, a key concern is whether the acceleration method can adapt to various samplers. AsyncDiff is a universal method that can be combined with different samplers, such as the DDIM sampler [55] and DPM-Solver [31] . In Table 7 , we present the quantitative evaluation of AsyncDiff on SD 2.1 using the DDIM sampler.Compared to using fewer DDIM steps, our method achieves significantly better generation quality at similar speeds, with the improvement becoming more pronounced as speedup increases. Table 6 presents the quantitative evaluation of AsyncDiff on SD 2.1 with the DPM-Solver sampler. At the same speedup ratio, AsyncDiff significantly enhances generation quality compared to the baseline. Qualitative results are also provided in the Appendix figures, demonstrating that our method achieves considerable acceleration while maintaining high consistency with the original output. Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?Answer: [Yes] Justification: We provide a detailed description of our method along with extensive experimental results. 
efficiency analysis on different devices: 
 As a hardware-friendly and versatile method, our acceleration technique delivers strong performance on a wide range of GPUs. We tested inference speeds on the professional-grade NVIDIA RTX A5000, as well as the consumer-grade NVIDIA RTX 2080 Ti and NVIDIA RTX 3090 GPUs. As shown in Table 8 , our method achieved a high acceleration ratio across all three GPUs. Furthermore, our method can be applied as long as the devices have basic communication capabilities.Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material? Answer: [Yes] Justification: We offer the full code along with relevant instructions. Guidelines:• The answer NA means that paper does not include experiments requiring code. • The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
conclusion: 
 In this paper, we propose a new parallel paradigm, AsyncDiff, to accelerate diffusion models by leveraging model parallelism across multiple devices. We split the denoising model into several components, each assigned to a different device. We transform the conventional sequential denoising into an asynchronous process by exploiting the high similarity of hidden states between consecutive time steps, enabling each component to compute in parallel. Our method has been comprehensively validated on three image diffusion models (SD 2.1, SD 1.5, SDXL) and two video diffusion models (SVD, AnimateDiff). Extensive experiments demonstrate that our approach significantly accelerates inference with only a marginal impact on generative quality. This work investigates the practical application of model parallelism in diffusion models, establishing a new baseline for future research in distributed diffusion models.❈ In this document, we provide materials that extend beyond the scope of the main manuscript, constrained by space limitations. Model Segmentation. In our method, we partition the cumbersome denoising model into multiple components, each assigned to a different device. After successfully parallelizing the computation of each component, the time cost for each time step now corresponds to the maximum latency among these components. To optimize parallel processing efficiency, we partition the model into segments that each carry a roughly equal computational load. This arrangement allows all modules to finish their computations nearly simultaneously, making full use of available computational resources. The segmentation strategy is sequential except for SDXL [41] . For the denoising U-net within the SDXL module, we group its first and last blocks into a single segment and apply sequential splitting to the remaining blocks. This is because SDXL has specific needs for high-frequency details, and res connections typically contain abundant high-frequency information. Time Shifting. We introduce a technique called time shifting. Following the warm-up steps, the time embedding for each step is shifted back by one step. For instance, in a 50-step asynchronous denoising process with a warm-up of 2 steps, the original sequence of time embeddings is {50, 49, 48, 47, ..., 3, 2, 1}. With time shifting, this sequence is adjusted to {50, 49, 49, 48, ..., 3, 2}. In certain extreme cases, asynchronous denoising might leave residual noise in the output. Time shifting addresses this by adjusting the time embeddings backward, enhancing the denoising effect. It's important to note that time shifting is not a standard component of our method but is employed optionally. The quantitative results presented in this paper are achieved without the use of time shifting.Stride Denoising. To further enhance efficiency, we introduce stride denoising, which completes multiple denoising steps simultaneously through a single parallel computation. Figure 7 illustrates the full schematic of applying stride denoising to AsyncDiff. In this depiction, the denoising model ϵ θ is divided into three components ϵ n θ and ϵ 2 θ at these steps. Stride denoising effectively reduces both computational load and communication demands by decreasing the parallel computing rounds needed to complete the process. Compared to the significant improvements it brings in efficiency, the quality sacrifice is minimal and can be entirely compensated for by slightly increasing the warm-up steps. 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)? Answer: [NA] Justification: The paper poses no such risks. Guidelines:• The answer NA means that the paper poses no such risks.• models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: The creators or original owners of assets (e.g., code, data, models) used in the paper are properly credited, and the license and terms of use are explicitly mentioned and properly adhered to.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 
new assets: 
 Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?Answer: [Yes] Justification: New assets introduced in the paper are well documented, and the documentation is provided alongside the assets.Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file. 
crowdsourcing and research with human subjects: 
 Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?Answer: [NA] Justification: The paper not involve crowdsourcing nor research with human subjects. Guidelines:• The NA means that the paper does not involve crowdsourcing nor research with human subjects. • Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: Time cost. In Table 9 , we present the time costs associated with model running and inter-device communication when using AsyncDiff on SD 2.1. Generally, communication expenses constitute only a minor fraction of the total time cost, demonstrating that AsyncDiff is an effective distributed acceleration technique suitable for practical application. It is important to note that as the number of devices increases, the time needed for data broadcasting between devices also rises, thereby increasing the proportion of communication costs. However, employing stride denoising can substantially reduce these costs by decreasing the number of parallel rounds needed to complete the denoising process. Speedup Ratio. We also evaluate the acceleration ratio on SD 2.1 with varying numbers of denoising steps. As indicated in Table 10 , AsyncDiff significantly enhances processing speed, even with a denoising procedure consisting of only 25 steps. When the number of steps extends to 100, our approach achieves a speedup of up to 4.3x, surpassing the ratio of devices employed. C More Quantitative Results.To thoroughly assess the quality of images produced following acceleration, we provide quantitative analyses on three base models (SD 2.1 [43] , SD 1.5 [43] , SDXL [41] ) using four additional metrics: the full reference metric, DISTS [5] , and no-reference metrics including MUSIQ [20] , CLIP-IQA [59] , and NIQE [36] . The experimental results in Table 11 demonstrate that our method significantly reduces inference latency while maintaining a high level of quality in diffusion model-generated images. On SD 1.5, our approach not only accelerates the inference process but also brings the image quality closer to the natural distribution.Table 11 : Quantitative evaluations of AsyncDiff three text-to-image diffusion models using more metrics including DISTS [5] , MUSIQ [20] , CLIP-IQA [59] , and NIQE [36] . Qualitative Results on Image Diffusion Models. As depicted in Figure 8 , we present further qualitative results for SD 2.1 and SDXL under various configurations. The speedup achieved is nearly proportional to the number of devices utilized, indicating efficient resource usage by our method. Moreover, the images generated by our approach closely match the text descriptions and are of high quality.Qualitative Results on Video Diffusion Models. We present qualitative evaluations of AsyncDiff applied to the video diffusion models. Figures 9, 10, and 11 illustrate the generated results using our method on the text-to-video model AnimateDiff [9] . Figure 12 displays results from applying our method to the image-to-video model SVD [2] . For a 50-step video diffusion model, AsyncDiff markedly decreases latency-saving tens or even hundreds of seconds-while maintaining the integrity and quality of the generated videos.As a distributed acceleration framework, AsyncDiff necessitates frequent communication between devices throughout the denoising process. Consequently, if the devices lack the capability to communicate effectively or have subpar communication infrastructure, our method may not perform optimally. Additionally, AsyncDiff operates as a plug-and-play acceleration solution that depends on pre-trained diffusion models. Therefore, if the baseline quality of the original diffusion models is unsatisfactory, achieving high-quality results with our method could be challenging.In this paper, we introduce a universal distributed acceleration approach for diffusion models. This method substantially speeds up the inference phase of diverse diffusion models by fully leveraging computational resources. It holds significant potential for practical applications, particularly in computationally intensive generation tasks like video and speech generation. Guidelines:• The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.• The answer NA means the paper does not include experiments.• If the paper includes a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not. • If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable. • Depending on the contribution, reproducibility can be accomplished in various ways.For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed. • While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. , with an open-source dataset or instructions for how to construct the dataset). (d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility.In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results. 


Decision: The paper introduces AsyncDiff, a novel framework that accelerates diffusion models by enabling asynchronous parallelism across multiple devices, effectively utilizing the high similarity of hidden states between consecutive steps. The method achieves impressive speedups (1.8x to 4x) on various diffusion models, including image and video generation, with minimal impact on performance metrics like FID and CLIP score. Although some reviewers expressed a desire for more detailed analysis of communication costs and the underlying similarity of hidden states, the proposed method's versatility and significant practical benefits make it a strong contribution to the field. Given its potential impact, I recommend accepting the paper.

Reviews: review: paper_summary: paper_summary: This paper proposes AsyncDiff to enable diffusion model parallelism across multiple devices and achieve a very impressive speedup with negligible degradation. Specifically, the denoising model is split into several components, each assigned to a different device. The conventional sequential denoising process is transformed into an asynchronous one by exploiting the high similarity of hidden states between consecutive time steps, enabling each component to compute in parallel.
strength_weakness: strength_weakness: 1. The discussion and comparison with the existing diffusion sampler is not sufficient, whether AsyncDiff can be combined with other diffusion samplers such as DPM-Solver.
2. In the absence of an explanation of the GPU, and AsyncDiff's ablation experiments on different GPU devices, can other GPUs achieve a good acceleration ratio?
questions: questions: I do not have other questions about this paper.
limitations: limitations: As stated by the author, AsyncDiff necessitates frequent communication between devices throughout the denoising process. Therefore, if the devices lack the capability to communicate effectively, AsyncDiff may not perform optimally.
paper_summary: paper_summary: This work introduces an acceleration method for diffusion models by distributing the model blocks across multiple GPUs and running different blocks asynchronously. The core motivation of this paper lies in the observation that the hidden states of a block exhibit high similarity across consecutive diffusion steps. This enables the execution of different blocks based on pre-computed results from earlier steps. The proposed method was evaluated on Stable Diffusion v2.1, v1.5, and XL, achieving a 1.8x to 4x acceleration with minimal performance loss on CLIP score and FID.
strength_weakness: strength_weakness: 1) Is the communication cost huge compared to the inference cost? As shown in Table 3, the number of GPUs used in this paper is primarily 2 or 3. Will there be issues if this is extended to 8 or 16 GPUs? Is this method still competitive when the number of devices exceeds 8, where cross-node communication becomes substantial?

2) The experimental section mainly focuses on quality metrics such as FID and CLIP Score. It would be beneficial if the authors could provide more analytical results about the method, such as the similarity of hidden states.

3) Figure 5: “Quantitative” should be “Qualitative”.
questions: questions: 1) What happens if the total number of timesteps is not divisible by the number of GPUs? Is the proposed method still applicable in this scenario?

2) Is the proposed method still effective on low-end GPUs or edge devices?
limitations: limitations: N/A.
paper_summary: paper_summary: This paper proposes AsyncDiff, a plug-and-play acceleration scheme that enables model parallelism across multiple devices. The core method involves dividing the diffusion model into multiple components and executing the inference in parallel. This is facilitated by the high similarity between hidden states in consecutive diffusion steps. With AsyncDiff, it is claimed that off-the-shelf diffusion models can be effectively accelerated with negligible degradation. Therefore, the key contribution of this paper lies in providing a simple and effective method to accelerate the diffusion process through parallelism.
strength_weakness: strength_weakness: 1.	Some concepts, such as the “dependency chain” in ABS, are not well explained. It would be beneficial if the author could provide a minimal explanation for these concepts.
2.	The paper appears to share a related idea with Distrifusion [1]. Could the author provide a more detailed discussion about the key differences between them? For example, the main difference seems to be the splitting schemes, where Distrifusion splits the data, and this approach splits the model. What is the main advantage of AsyncDiff?
3.	Are there any failure cases for the proposed AsyncDiff? 
[1] DistriFusion: Distributed Parallel Inference for High-Resolution Diffusion Models
questions: questions: Additional Questions:
1.	Is this method helpful for large batch sizes?
2.	Is this method applicable to more GPUs?
limitations: limitations: I do not find any potential negative societal impact of this work.
paper_summary: paper_summary: This paper introduces AsyncDiff, an acceleration framework for diffusion models that transforms the traditional sequential denoising process into an asynchronous process. The key insight is that hidden state features in consecutive sampling steps exhibit high similarity. Therefore, feeding the output of the preceding component at time step t-1 as an approximation of the original input to each U-Net component has a negligible impact on performance. This approach allows the parallel denoising processes of AsyncDiff to operate fully asynchronously. Experiments on various versions of Stable Diffusion demonstrate that AsyncDiff significantly speeds up the denoising process for text-to-image generation. Additionally, AsyncDiff is applicable to video generation models such as AnimateDiff and SVD, further showcasing its versatility.
strength_weakness: strength_weakness: 1. Although the main contribution relies on the observation that the hidden states of consecutive steps are similar, the analysis of this phenomenon lacks details. Several key aspects need clearer explanation:
(a) Can the similarity of the hidden states be quantitatively measured? For instance, does a low MSE between hidden states indicate that the two states are “similar”?
(b) Is this phenomenon specific to the U-Net architecture, or is it agnostic to the backbone of the denoising model (e.g., Diffusion Transformer)?
(c) Is it only applicable to DDIM sampling? Does the phenomenon also hold for other fast ODE solvers, such as the DPM solver [1]?

2. While the experiments were conducted using 50 DDIM steps, a comparison with the original model using a number of DDIM steps that achieve a similar speedup would strengthen the argument for AsyncDiff. For instance, in a setup where AsyncDiff achieves a 2.7x speedup, comparing the FID score with the original model using 50 / 2.7 = 19 DDIM steps would clearly demonstrate the necessity of parallelizing the diffusion model.

[1] DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Steps, Lu et al., NeurIPS 2022
questions: questions: Most questions are included in the “Weaknesses” section.

Would a similar asynchronous denoising approach be applicable in a conditional generation setup as well? For instance, achieving faster inference for diverse conditional image generation models, such as ControlNet [1] and Zero-1-to-3 [2], could be practically useful.

[1] Adding Conditional Control to Text-to-Image Diffusion Models, Zhang et al., ICCV 2023

[2] Zero-1-to-3: Zero-shot One Image to 3D Object, Liu et al., ICCV 2023
limitations: limitations: The authors discuss the limitations of AsyncDiff (communication cost, dependency on the base model) and the potential societal impacts.

Comments: title: Final Rating
comment: Thank you for the insightful and constructive comment. Your expertise and time are greatly appreciated in helping improve my work's quality.
The rebuttal from the author has addressed all my concerns, thus I would like to keep my rating as accept.
Dear reviewers,

Could you please respond to the rebuttal, discuss with authors and finalize your score?
Thanks for your valuable feedback and time in reviewing my work, your insights are greatly appreciated.
After thoroughly reviewing the authors' rebuttal, I am satisfied that my original concerns have been addressed. I particularly appreciate the additional experiments with Diffusion Transformers (SD3), and the clear evidence supporting the similarity of the hidden states.

In light of this, I have decided to raise my rating to "Weak Accept."
We sincerely appreciate your dedicated time and effort in reviewing our submission. Your valuable feedback is greatly appreciated.
The authors have posted an effective rebuttal that resolves my concerns. I still vote for a weak acceptance of this work.
Dear Reviewers, Chairs, 

We sincerely appreciate the time and effort you have spent evaluating our submission, and we look forward to the discussion stage. We will include the review stage results in the appendix of our next version.