AsyncDiff: Parallelizing Diffusion Models by Asynchronous Denoising
Authors: Zigeng Chen
Xinyin Ma
Gongfan Fang
Zhenxiong Tan
Xinchao Wang
Venue: NeurIPS 2024 poster (2024)
Keywords: Diffusion Model
Inference Acceleration
Primary area: diffusion_based_models
TL;DR: We introduce AsyncDiff, a universal and plug-and-play diffusion acceleration scheme that innovatively enables model parallelism across multiple devices.

Abstract: Diffusion models have garnered significant interest from the community for their great generative ability across various applications. However, their typical multi-step sequential-denoising nature gives rise to high cumulative latency, thereby precluding the possibilities of parallel computation. To address this, we introduce AsyncDiff, a universal and plug-and-play acceleration scheme that enables model parallelism across multiple devices. Our approach divides the cumbersome noise prediction model into multiple components, assigning each to a different device. To break the dependency chain between these components, it transforms the conventional sequential denoising into an asynchronous process by exploiting the high similarity between hidden states in consecutive diffusion steps. Consequently, each component is facilitated to compute in parallel on separate devices. The proposed strategy significantly reduces inference latency while minimally impacting the generative quality. Specifically, for the Stable Diffusion v2.1, AsyncDiff achieves a 2.7x speedup with negligible degradation and a 4.0x speedup with only a slight reduction of 0.38 in CLIP Score, on four NVIDIA A5000 GPUs. Our experiments also demonstrate AsyncDiff can be readily applied to video diffusion models with encouraging performances.

AsyncDiff: Parallelizing Diffusion Models by Asynchronous Denoising
Zigeng Chen, Xinyin Ma, Gongfan Fang, Zhenxiong Tan, Xinchao Wang"
National University of Singapore zigeng99Cu. nus edu, xinchao@nus. sg
2.8x Speedup
Figure 1: We introduce new distributed acceleration paradigm that attains 2.8x speed-up on Stable Diffusion XL while maintaining pixel-level consistency, using four NVIDIA A5000 GPUs.
Abstract
Diffusion models have garnered significant interest from the community for their great generative ability across various applications. However, their typical multi- step
nature gives rise to high cumulative latency, thereby precluding the possibilities of parallel computation. To address this, we introduce AsyncDiff, a universal and plug-and-ple accelcration scheme that enables model parallelism across multiple devices. Our approach divides the cumbersome noise prediction model into multiple components, assigning each to different device. To break the dependency chain between these components, transforms the ventional sequential denoising into an asynchronous process by exploiting the high similarity between hidden states consecutive diffusion steps. Consequently, each component is facilitated to compute in parallel on separate devices. The proposed strategy significantly reduces inference latency while minimally impacting the gen- erative quality. Specifically, for the Stable Diffusion 12. AsyrcDiff achieves a 2.7x speedup with negligible degradation and a 4.0x speedup with only a slight reduction of 0.38 in CLIP Score, on four NVIDIA A5000 GPUs. Our experiments also demon- strate AsyncDiff can be readily applied to video diffusion models with encouraging performances. Code is available at https: gi thub. com/ czg 225 AsyncDiff
Introduction
Diffusion models 13 stand out in generative modeling and have significantly advanced various
fields including text-to-image 43
46 72 78 and text-to-video generation [64] 61
"Correspoding Author
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Traditional: Denosing Model running Sequential
GPU
Ours: Denoising Model running in Parallel
GPU I
GPU
GPU
Communication within device
Communication across devices
Figure 2: By preparing each component's 's input beforehand, we enable parallel computation of the denoising model, which substantially reduces latency while minimally affecting quality.
image translation
49
56 23
audio generation 22
14
44 style transfer 62
17, low-level vision
77), and 3D model generation 42 18, (7) among others. However, their widespread application is hindered by the high latency inherent in their multi-step sequential denoising process. This issue becomes more pronounced as the complexity and size of the models increase to enhance generative quality.
tasks [47 660 40
26
8
70
image editing 19 66 5
In response these challenges, significant research efforts are directed toward enhancing the effi ciency of diffusion models. Notably, training- foiningg-ree acceleration methods have garnered increasing popularity due their low cost and convenience. Numerous studies B3 63 76 25 33 57 34
improve inference speed by skipping redundant calculations in the denoising process. As computa- tional resources grow rapidly, distributing computations across multiple devices has become more promising approach. Recent advances [52] 24 58) demonstrate that using distributed computing to parallelize inference effectively increases the acceleration ratio for diffusion models while main- taining acceptable generative quality. Though these methods succeed in parallclizing the diffusion models, they require iterative refining [52] or displaced patch parallelism [24], resulting a larger number of model evaluations or low GPU utilization correspondingly.
Thus, we wish propose a new parallel paradigm for diffusion, akin to the model parallelism in distributed computing [15] 38 28 16 39 65]. which divides the denoising model into several components to be distributed on different GPUs. The primary challenge lies in the inherent sequential denoising process of diffusion models. Each step in this process depends on the completion of its predecessor, forming a dependency chain that impedes parallelization and significantly increases inference latency. Our approach seeks to disrupt this chain, allowing for the parallel execution of the denoising model while closely approximating the results the sequential process.
In this paper, we introduce AsyncDiff. a universal, distributed acceleration paradigm that innovatively explores model parallelism diffusion models. As shown Fig our method sequentially partitions the heavyweight denoising model into multiple components { based on computational load, assigning each to a separate device. Our core idea lies in decoupling the dependencies between these cascaded components by leveraging the high similarity in hidden states across consecutive diffusion steps. After the initial warm-up steps, each component takes the output from the previous component's prior step as the approximation of its original input. This transforms the traditional sequential denoising into an asynchronous process, allowing components to predict noise for different time steps in parallel. Additionally, incorporate stride denoising skip redundant calculations and reduce the frequency of communication between devices, further enhancing efficiency.
Through extensive testing across multiple base models, our method effectively distributes the com- putational burden across various devices, substantially boosting inference speed while maintaining quality. Specifically, with the text-to-image -imgg model Stable Diffusion v2.1 43 our method achieves 1.8x speedup with only a marginal 0.01 drop in CLIP Score [[]] and a 4.0x speedup with a slight 0.38 reduction in CLIP Score two and four NVIDIA A5000 GPUs, respectively. For video diffusion
models, DifmateDiff [9] and Stable Video Diffusion 22, our approach significantly reduces latency by tens of seconds, effectively preserving video quality.

---

In summary, we present a novel distributed acceleration method for diffusion models that significantly reduces inference latency with minimal impact on generation quality. This is achieved by replacing the sequential denoising process with asynchronous process, allowing each component of the denoising model run independently across different devices. Extensive experiments on both image and video diffusion models strongly demonstrate the effectiveness and versatility of our method.
2 Related Works
Diffusion Models.
Diffusion models have attracted significant attention due to their powerful generative capabilities across various tasks. Sohl-Dickstein et al. [54] first proposed diffusion probabilistic models. Ho al. [13] with the introduction of Denoising Diffusion Probabilistic Models (DDPM), enhancing training efficiency and generation quality, Rombach et al. [43] advanced these models by incorporating latent spaces, enabling high-resolution image generation. Despite these advancements, the high latency of the iterative denoising process remains limitation.
taaiiing-basete acceleration methods focus reducing sampling steps [88] [1 32 [50 69 or optimizing model architectures [27] 80] 7 [3 68 6 However, these methods incur high training costs and complexity. Training-free methods are gaining popularity due their ease of use. Some approaches develop fast solvers for SDE or ODE to improve sampling efficiency
Inference Acceleration.
33 79 observed special characteristics of diffusion models and skipped the redundant computation within the denoising process.
[ 30 74 [ Other works [35 63 76
67
53 25
The parallelism strategy presents promising yet underexplored approach to accelerating diffusion models. ParaDiGMS [52 implements Picard iterations for parallel sampling, yet its practical speed-up ratio is modest, and it struggles maintain consistency with original outputs. Faster Diffusion [25] introduces encoder propagation but significantly compromises quality, and its parallelization remains theoretical. Distrifusion 24 adopts patch parallelism, dividing high- resolution images into sub-patches to facilitate parallel inference on each patch by reusing stale activation maps from each layer. However, this approach lacks flexibility across different data types or tasks, often encountering low resource utilization. Furthermore, its reliance on reusing per-layer activation maps greatly increases GPU memory demands thus introducing additional challenges for realistic applications. In contrast, our method uniquely implements model parallelism through asynchronous denoising, achieving substantial acceleration while maintaining a stable resource usage ratio and minimal impact on quality.
Parallelism.
3 Methods
3.1 Preliminary
Diffusion models 13 are a dominant class of generative models that transform Gaussian noise into complex data distributions via Markov process. The forward process is defined by:
(= = N(x;;; /1 3,0,,,,,11,33 B,T),
(1)
where {Be} progressively increases noise until the data becomes indistinguishable from noise. The reverse process, essential for data reconstruction, involves iterative denoising:
(=( = =N( Mo(s,, t), of I),
where μe (2,, t) is the predicted mean and of is the variance. For DDIMs (55), the reverse update is deterministic:
at-1 Fí + /1 - a1-111
at
Tt-1
ca c2(
(3)
at
a4-1
where a is the cumulative product of (1 B,). These processes are computationally intensive, influencing the quality of generated samples and necessitating efficient inference methods for practical applications.
3.2 Asynchronous Diffusion Model
Traditional diffusion models employ a sequential and synchronous denoising process. each time step f, the noise- precccpreiition model estimates the noise e; based the noisy image 2, and the time
3

---

Warm-up
Parallel
Parallel
Parallel
Parallel
1oo0
GPU 3 (et)
r-1
T-2
all
GPU2 2 (e
7-2
7-3
T-4
H T-4
GPU 1 (eū)
T-3
2 1100000 Proil 7-1
GPU (et)
7-2
T-3
7-4
Communication within device
Communication across devices
T: Time embedding
Figure 3: Overview of the asynchronous denoising process. The denoising model co is divided into four components {cā 14 In for clarity. Following the warm-up stage, each component's input is prepared in advance, breaking the dependency chain and facilitating parallel processing.
embedding 4. The image for the next step, 1t-1, is then generated using a sampler function S1 exs l)- This process is iterative, where the generation of at each step is dependent the completion of the previous denoising step, making the process slow, particularly when Eş is computationally intensive.
To address the limitations of high latency in diffusion models, leveraging multiple GPUs for distributed inference is a promising solution. Existing studies primarily focus on patch parallelism (24). where the input image is divided into patches, each processed on different GPU. While this strategy efficiently distributes computational loads, still retains the bottleneck of sequential denoising, as each patch must undergo the complete denoising process iteratively. In contrast, asynchronous diffusion model innovatively introduces model parallelism strategy. By approximating the sequential denoising as asynchronous process, this approach enables parallel inference of the noise prediction model, effectively reducing latency and breaking the constraints of sequential execution.
Asynchronous Denoising, Figure illustrates our approach to the asynchronous denoising. For a denoising process consisting of steps, the initial w steps are designated as a warm-up phase, where w is significantly smaller than T. During this phase, the denoising model operates using standard sequential inference. After warm-up steps, rather than splitting the input image, we partition the denoising model co into N sequential components, expressed as
{cb. eg. Cg N. Each component is divided handle a comparable computational load and assigned to a distinct device. This equitable division aims to equalize the time cost of each component to approximately (<eo//,, thus minimizing the overall maximum latency. In this setup, original noise prediction for ia can be represented cascading operation through these sub-models defined mathematically as:
6 cs ( I) (c)) (c) (e (...,,
().
Although each device can independently compute its assigned component, the dependency chain persists because the input for each component 6ø,, is derived from the output from its preceding component 60 Therefore, despite the distribution of model components across multiple devices, full parallelization is constrained by these sequential dependencies.
Our principal innovation is to break the dependency between cascaded components by utilizing hidden features from previous steps. Observations indicate that the hidden states of each block the denoising model always exhibit substantial similarity across adjacent time steps. Leveraging this, each component at time step can take the output from the preceding component time step as the approximation of its original input. Specifically, the nithh component eă 2, t) receives the output of

---

GPU 3
GPU o(e)))
GPU 1(e
GPU 2(e
Doee
Dob
Parallel
X
X--2
Communication within device
Communication across devices
Time embedding
Figure 4: Illustration of stride denoising. The model Eg is divided into three components { m==1. with stride of 2 for clarity. Components cp and eg are skipped time step A single parallel batch results in the completion of denoising for two steps, producing _T-1 and 2-
(,,
1). This alteration allows the noise prediction for T, be represented follows:
E, = (E ( (e. "6. --1)) N 2) .,, 1),
(5)
In this new framework, noise prediction is derived from components executed across N previous time steps. This transforms the denoising process from sequential to asynchronous, as the prediction of noise 6 already begins before denoising at step t + 1 is completed. At each time step, the N components are running as parts of the noise prediction model for the next N steps. Specifically, the n-th component eg. computed in parallel at time f, contributes to the noise prediction for the future time step i N + n. Figure 3 depicts this asynchronous process using a U-net model with N set to 4. The strong resemblance of hidden states between consecutive diffusion steps enables the asynchronous process to closely mimic the denoising results of the original sequential process.
Model Parallelism. By transitioning to an asynchronous denoising strategy, the dependencies among components within the same time step are eliminated. This adjustment allows each component's input for time step to be prepared in advance, enabling the N split components to be processed concurrently across multiple devices. Once computed, the outputs from each component must be stored and then broadcasted to other devices to facilitate parallel processing for subsequent time steps. In contrast, the traditional sequential denoising process, the time cost for each step accumulates as follows:
Ceeg = C((b)) + C(+²) +... + C(+
By adopting asynchronous denoising to enable parallel computation of each component, the cost for each time step is now given by:
Casy (t) = max) C(e))..., )) C(comm.),
where max() represents taking the maximum value, and C(comm.) indicates the communication cost across multiple GPUs. As the model components are equally divided by computational load, their time costs are similar, allowing us to approximate the overall cost of each time step as:
Oseg(1) N
Casy (t)
C(comm.).
8
Since the communication overhead C(comm.) is generally much lower than the model's execution time, leads to significant overall cost reductions. Moreover, increasing N further reduces time costs but complicates the accurate approximation of the original denoising process.
Stride Denoising. While asynchronous denoising reduces latency by parallelizing the denoising model, it completes only one denoising step time. To enhance efficiency, we introduce stride denoising, which completes multiple denoising steps simultancously through single parallel com- putation. The diagram is illustrated in Figure 4 where we set the stride for clarity. Unlike the continuous broadcasting two steps. As depicted, time step and broadcast the hidden states for the next parallel computation round. Consequently, the hidden states from time step are not required, allowing us to skip the calculations for cg and ég at this step. In this stride, only e t), eb(., 1), e 1). and e 1) need computing, all receiving the
hidden states at each time step, stride denoising broadcasts them every
conduct denoising alone, and time step f 1, we compute
5

---

(a) Qualitative Results on SDXL with different configurations
Original
Oen Speader
(sex Spendap
Spondip
Care Spasda
Oun Spendap
Qualitative
Resulls SDXL with different warm--up steps A-33522
Duston Wann-apet Brekzr
Busicm Warmaspe
Diriiess uped
Desizes Wamn-upet
Figure 5: Qualitative Results. (a) Our method significantly accelerates the denoising process with minimal impact on generative quality. (b) Increasing warm-up steps achieves pixel-Ievel consistency with the original output while maintaining high speed-up ratio.
previously broadcast hidden states, enabling their parallel processing. Both c.(,,)) and c 1)
share the same feature from (- t 1), so the stride should be kept small maintain quality. Stride denoising effectively reduces both computational load and communication demands by decreasing the parallel computing rounds needed to complete the process. Compared to the significant improvements it brings in efficiency, the quality sacrifice is minimal and can be entirely compensated for by slightly increasing the warm-up steps. We also illustrate the full schematic of it in Appendix Figure 7
Multi-Device Communication. Parallel inference the model necessitates efficient communication between devices, each component e must access the cached hidden state from the preceding component e = which resides different device. Post each parallel computation batch, each device stores the current hidden state needed for the next parallel batch. These states, encompassing all component outputs, are then broadcast all participating devices before the next parallel computation batch. Although each component eă primarily uses the cached output of eă ep-1 for its input, it may require residual features [0 from other components. Therefore, it's crucial to broadcast the stored states from every component across all devices before each round of parallel computation.
Experiments
4.1
Implementation Details
Base models. We validated the broad applicability of AsyncDiff through extensive testing on several diffusion models. For text-to-imaae tasks, we experimented with three versions of Stable Diffusion: SD 1.5, SD 2.1 43), and Stable Diffusion XL (SDXL) [П]. Additionally, we explored the effectiveness AsyncDiff on video diffusion models using Stable Video Diffusion (SVD) 22 and AnimateDiff 19. All models were evaluated using 50 DDIM steps. We facilitated communication across multiple GPUs using the broadcast operation from torch. disttriistibuuted powered by the NVIDIA Collective Communication Library (NCCL) backend.

---

Table 1: Quantitative evaluations of AsyncDiff on three text-to-image diffusion models, showcasing various configurations. 'N' indicates the number of components into which the model is divided, and 'S' represents the denoising stride. MACs quantifies the computational load per device for generating single image throughout the denoising process.
Base Model
Configuration
Devices
MACs]
latency
Speed
CLIP Scoret
FID
LPIPS
Original Model
761
5.51s
31.60
27.89
Ours (N-2 S=1)
38T
3.03s
1.8x
31.59
27.79
0.2121
Ours Sw!)
2.41s
31.56
0.2755
(Texi-to Image)
Ours (Ne4 S=1)
19T
2.10s
2,6X
31.40
28.
0.3132
Ours (N=2 S=2)
19T
1.82s
3,0x
31.43
28.55
0.3458
Ours (N=3 Sm2)
13T
1.356
4.0x
31.22
29.41
0.3778
Original
Ours (N=2 Sm1)
I7T
1,526
1.8x
30.62
29.94
0.1988
SD (Texi-to-Imagg)
Ours (N=3 S=1)
11T
1.236
2.2x
30.58
29. 89.87
0.2645
Ours (N-4 Sm()
1.01
2.6x
30.52
30.
0. 0,3073
Ours (N=2 S=2)
VT
0,948
2.9X
30.46
30.98
0.3232
S=2)
0.3811
Original Model
299T
13.818
1
32.33
21.43
Ours (N-2 S-1)
150T
8.00s
1.7x
32.21
27.79
0,2509
SDXL. (Iex--lo-Imaag)
Ours (N=3 S=1)
3
100T
3,846
2
32.05
28.03
0.2940
Ours (N-2 S-2)
75T
4,918
2,84
31.70
28. 23.99
0.3209
S=2)
30.27
0.3556
Table 2: Quantitative evaluations of the effect of increasing warm-up steps. More warm-up steps can achieve pixel-level consistency with the original output while slightly reducing processing speed.
Contiguration
SD
1.5
SDXL
Speedupt
CLIIP*
I.PPSS
Specdup
CLIPT
LPIPS|
Speedupt
CLIPI
I.PIPS.
Original Model
31,60
1.0x
30.63
32.33
=arm-up=
3.5x
31,26
0.3289
3.3x
30.16
0,3676
3.8x
31.40
0.3556
Warm-up 5
3.1x
31.27
0.2769
3.0x
30.14
0.3304
3.4k
31.60
0,2993
Warm-up
2.9%
31.32
0.2309
2.7x
30.10
0.2839
3.0x
31.77
0,2521
Warm-up =9
2.7x
31.40
0.1940
2.5x
30.17
0.2354
2.8x
31.92
0,2095
Warm-up
30.22
Dataset and Evaluation Metries. We assess the zero-shot generation capability using the MS-COCO 2017 29 validation set, which comprises 5,000 images and captions. For image generation, quality measured by the CLIP Score (on ViT-g/14) [TT] and Fréchet Inception Distance (FID) [1). with LPIPS (75) used check consistency with original outputs. In video generation, quality is evaluated by averaging the CLIP Score across all frames of a video. We also report MACs per device and latency to gauge efficiency comprchensively. All latency measurements were conducted on NVIDIA A5000 GPUs equipped with NVLINK Bridge.
4.2 Experimental Results on Image Diffusion Models
Improvements on Base Models. Table our acceleration outcomes for three fundamental image diffusion models under various configurations. In this context, 'N' represents the number of segments into which denoising model is divided, and 'S' denotes the stride of denoising for each parallel computation batch. Our approach, AsyncDiff, not only significantly accelerates processing but also minimally impacts generative quality. The speedup ratio is almost proportional to the number of devices used, demonstrating efficient resource utilization. Visualization results in Figure 5 (a) illustrate the high generative quality achieved even with substantially reduced latency. Although achieving pixel-level consistency with the original output is challenging at high acceleration ratios, the generated image still effectively conveys the semantic information in the prompt, which is crucial for generative results.
Pixel-level Consistency by Warm-up. In Table 2 we explore the balance between pixel-level consistency and processing speed by adjusting the warm-up steps in the diffusion models. As the initial steps of these models play a crucial role reconstructing the global structure based on text prompts (76), modest increase in warm-up steps significantly enhance consistency with the

---

Table 3: Quantitative comparison with other parallel acceleration methods. To ensure fair compari- son with Distrifusion, we increased the warm-up steps in our method to match the speedup ratio of Distrifusion, allowing us to fairly compare generation quality and resource costs.
Method
Speed up?
Devices
MACsi
Memory
CLIP Scoret
FID1
LPIPS,
Original Model
L.kx
761
5240MB
31.60
27.87
Faster Dittusion
16x
57T
9692MB
30.84
29.95
0.3477
6538MB
Ours S=1)
1.6x
44T
5450MB
31.59
27.79
0.0944
Distrifusion
2.3x
191
7086MB
31.43
27.97
0.2710
Ours (N=2 S=2)
2.3x
20T
5516MB
31.49
27.71
0,2117
Distrifusion
2.7x
10T
7280MR
31.31
28.12
0.2934
Ours (N=3 S=2)
2.7x
5580MB
31.40
Oigiaa Devize
Specdip Deci
Syodip Desicw
De kp
1 Inviss
Bevio
4 Dviin
Figure 6: Qualitative Comparison with Distrifusion on SD2.1. At the same acceleration ratio, AsyncDiff outperforms in generating higher quality and more consistent images with the original.
original images. Figure (]b) illustrates this trend with qualitative comparisons of generative results SDXL using gradually increasing warm-up steps. Increasing the warm-up steps to 9 achieves visual indistinguishabiliyy from the original output while maintaining an impressive 2. acceleration ratio.
Comparison with Acceleration Baselines. We evaluated our AsyncDiff method on SD 2.1 against
two other parallel acceleration methods: Faster Diffusion (25 and Distrifusion [24). Faster Diffusion employs encoder propagation but compromises significantly on generative quality. As its parallelism maintains theoretical and lacks multi-device implementation, we cannot measure its realistic latency with more than one GPU. Its ideal speed-up on 2 devices about 1.9x. Distrifusion, on the other hand, uses patch parallelism for distributed acceleration but faces potential issues with low resource utilization and high GPU memory demands.
According to Table our method achieves the same operational speed using only GPUs and GPUs as Distrifusion does with 8 GPUs and 4 GPUs, respectively. Additionally, our method requires almost the same amount of memory the original setup, whereas Distrifusion significantly increases memory requirements, posing extra challenges for practical applications. quality, AsyncDiff and Distrifusion both mirror the original diffusion model's performance at 1.6x acceleration ratio. However, at higher speedup ratios of 2.3x and 2.7x, our method demonstrates significantly superior generative quality. Qualitative comparisons. in Fig further show that AsyncDiff maintains better pixel-level consistency with the original input compared to Distrifusion.
terms generative
4.3 Experimental Results on Video Diffusion Models
As presented in Table 4 we conducted experiments with different configurations on two video
diffusion models: SVD 2 (25 frames), and AnimentDiff' [9] (16 frames), to demonstrate the efficacy

---

Table 4: Quantitative evaluations of AsyncDijf on text-to-video and image-to-vide videeo diffusion models. We present the results with various configurations.
Base Model
Configuration
Devices
MACs.
Intency.)
Speed up?
CLIP Score
Original Model
786T
43.5s
1.0x
30.65
AnfmatDDiff
Ours (N-2 Sm!)
393T
24.5s
30,65
(Rext-to-Videe)
Ours S=1)
262T
Ours (N=2 Sw2)
197T
14.2s
3.0x
30.32
Ours (Nm3 Sm2)
4
131T
11.5s
3.8x
30.20
Original Model
3221T
184s
1.0x
26.88
Ours S=1)
SVD (Image-to-Videns
Ours (N=3 Sml)
1074T
80s
2.3x
26.56
Ours (No4 S=1)
805T
686
2.7x
26.19
Table 5: Effect of stride denoising on SD 2.1. Stride denoising significantly lowers overall latency and the communication cost while only slightly compromising the generative quality
Communication
Configuration
MACS.
Latency!
Speedupt
CLIP Score?
Nums.
Latency
AsyncDiff devices)
denoising
2.41s
Faster
times
0.23x(9.558)
AsyncDiff (3 devices) stride denoising
1.82s
Faster
times
0.12x(6.6%%
Aiff (4 devices) w/a stride denoising
19T
2. 2.10s
2.6x Faster
49 times
0.40x(19.0%)
31.40
AsyncDiff (4 devices) w/ stride denoising
13T
1.35s
4.0x Fnster
25 times
0.10s(7.4%)
31,22
of our method. Video generation, often constrained by exceptionally high latency and substantial computation load, greatly benefits from our approach. For a 50-step video diffusion model, AsyncDiff significantly reduces latency-by tens or even hundreds of seconds-while preserving the quality of generated content. Qualitative results shown in the Appendix. D further corroborate the effectiveness of our method. AsyncDiff achieves an impressive acceleration ratio of over three times while still producing videos that closely match the prompt descriptions, ensuring the rationality of actions and details. These findings highlight the substantial potential of AsyncDiff in accelerating the inference process of video diffusion models.
4.4 Effect of Stride Denoising
We introduce stride denoising to further enhance the efficiency of the asynchronous denoising process. Stride denoising completes multiple steps simultaneously through a single parallel computation, reducing the number of parallel rounds and communication frequency across devices. For diffusion process with T steps and warm-up step W, the number of broadcasts decreases from T W to (T W///2 with stride of 2. This strategy also reduces the computational load on each device by skipping unnecessary calculations. Table shows the effects of stride denoising in our parallel framework with and devices. Stride denoising significantly lowers overall latency and the proportion of communication time, especially as the number of devices used increases. While stride denoising slightly impacts generation quality, this effect is minimal and can be mitigated by a modest increase in warm-up steps. preserving efficiency and maintaining quality.
4.5 Compatibility with Various Samplers
With the recent rise of advanced sampling algorithms for diffusion models, key concern is whether the acceleration method can adapt to various samplers. AsyncDiff is a universal method that can be combined with different samplers, such the DDIM sampler (55) and DPM-Solver [I.. In Table 7 we present the quantitative evaluation of AsyncDiff on SD 2.1 using the DDIM sampler. Compared to using fewer DDIM steps, our method achieves significantly better generation quality at similar speeds, with the improvement becoming more pronounced as speedup increases. Table 6 presents the quantitative evaluation of AsyncDiff on SD 2.1 with the DPM-Solver sampler. At the same speedup ratio, AiyncDiff significantly enhances generation quality compared to the baseline. Qualitative results are also provided in the Appendix figures, demonstrating that our method achieves considerable acceleration while maintaining high consistency with the original output.

---

Table 6: Quantitative evaluations of AsyncDiff using DPM-Solver sampler on SD 2.1
Method
Speed up
MACs 4
CLIP Score
Solver 25steps
1.0x
76T
31.57
28.37
DPM-Solver 15steps Ours (N=2 S=1)
1.6x
461
31.52
28.89
1.6%
38T
31.58
27.71
DPM-Solver 10steps
2.2x
30T
31.29
29.28
Ours (N=3 S=1)
2.2x
251
31.36
28.20
Table 7: Quantitative evaluations of AsyncDiff using DDIM sampler on SD 2.1
Method
Speed
MACs
CLII Score
FTD
Original
1.0x
27.89
DDIM 27steps
1.8x
41T
31.53
28.43
Our AsyneDiff (N-2 Sw1)
27.79
DDIM stsps
2.3x
32T
31.46
29.09
AsyneDiff S=1)
DDIM 15steps
3.0x
23T
31.26
30.12
Our AsyneDiff (N=2 S=2)
3.0x
31.43
28.55
DDMM steps
4.0x
17T
30.99
32.25
Our AsyneDiff S=2)
4.0x
31.22
29.41
Table 8: Accelcration Ratio and Latency on Different GPUs
GPU
FP16 Compute
Original
N=2 S-1
N=3
N=2
N=3
NVIDIA RIX A5000
TELOPS 71 TFLOPS
LOx15.513)
18x(3.03s)
2.3x(2.41s)
3.0x(1.82s)
4.0x(1.35s)
NVIDIA RIX 3090
1.0x5.615)
188(33200)
2.1x(2.655)
2.9x(1991)
3.5x(1.605)
1.0x(8200)
1.7x(4.91)
2.0x(4.08))
2.8x(2.94k)
3.5x(2.35.)
5 Efficiency Analysis on Different Devices
As a hardware-friendly and versatile method, our acceleration technique delivers strong performance on wide range of GPUs. We tested inference speeds the professional-gres -gade NVIDIA RTX A5000, as well as the consumer-grade NVIDIA RTX 2080 Ti and NVIDIA RTX 3090 GPUs. As shown in Table g our method achieved high acceleration ratio across all three GPUs. Furthermore, method can be applied as long as the devices have basic communication capabilities.
Conclusion
In this paper, we propose a new parallel paradigm, AsyncDiff. accelerate diffusion models by leveraging model parallelism across multiple devices. We split the denoising model into several components, each assigned to different device. We transform the conventional sequential denoising into an asynchronous process by exploiting the high similarity of hidden states between consecutive time steps, enabling each component to compute in parallel. Our method has been comprehensively validated on three image diffusion models (SD 2. SD 1.5, SDXL) and two video diffusion models (SVD, AnimateDiff). Extensive experiments demonstrate that our approach significantly accelerates inference with only marginal impact generative quality. This work investigates the practical application of model parallelism in diffusion models, establishing a new baseline for future research in distributed diffusion models.
Acknowledgement
This project supported by the Ministry of Education, Singapore, under its Academic Research Fund Tier 2 (Award Number: MOE-T2EP20122-00065
10

---

References
[1] Fan Bao, Chongxuan Li, Jun Zhu, and Bo Zhang. Analytic-dpm: an analytic estimate of the optimal reverse variance diffusion probabilistic models. arXiv preprint arXiv:2201.0633 06503, 2022.
[2] Andreas Blattmann, Tim Dockhorn, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Dominik Lorenz, Yam Levi, Zion English, Vikram Voleti, Adam Letts, et al. Stable vidco diffusion: Scaling latent vidco diffusion models to large datasets. arXiv preprint or erXiv:2311.
2023.
[3] Zigeng Chen, Chaowei Liu, Yuan Yuan, Michael Bi Mi, and Xinchao Wang. Metaisp: Efficient raw-to-srgb mappings with merely Im parameters. In Kate Larson, editor, Proceedings of the Thirty- International Joint Conference on Artificial Intelligence, IJCAI-24, pages 086-694, International Joint Conferences on Artificial Intelligence Organization, 8 2024, Main Track.
[4] Jiwoo Chung, Sangeek Hyun, and Jae-Pil Heo. Style injection in diffusion: A training-free approach for adapting large-scale diffusion models for style transfer. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8795 8805, 2024.
[5] Keyan Ding, Kede Ma, Shiqi Wang, and Fero Simoncelli. Image quality assessment: Unifying structure and texture similarity. IEFE transactions on pattern analysis and machine intelligence, 44(5):2577-2811, 2020.
[6] Gongtan Fang, Xinyin Ma, Mingli Song, Michael Bi Mi, and Xinchao Wang. Depgraph: Towards any structural pruning. In Proceedings of the IEEE/CVE conference on computer vision and pattern recognition, pages 16091-16101, 2023.
[7] Gongfan Fang, Xinyin Ma, and Xinchao Wang. Structural pruning for diffusion models. Advances in neural information processing systems, 36, 2024.
[8] Langing Guo, Chong Wang, Wenhan Yang, Siyu Huang. Yufci Wang. Hanspeter Pfister, and Bihan Wen. Shadowdiffusion: When degradation prior meets difffusion model for shadow removal. In Proceedings the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 14049-14058, 2023.
[9] Yuwei Guo, Ceyuan Yang, Anyi Rao, Yaohui Wang, Yu Qiao, Dahua Lin, and Bo Dai. Animated- iff: Animate your personalized text-to-image diffusion models without specific tuning. arXiv preprint arXiv: ar7iv::337.004725 2023.
[10] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
[11] Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. Clipscore: reference-free evaluation metric for image captioning. arXiv preprint arXiv:2104.087 2021.
[12] Martin Heusel. Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by two time-scale update rule converge to a local nash equilibrium. Advances in neural information processing systems, 30, 2017.
[13] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840 6851, 2020.
[14] Rongjie Huang, Jiawei Huang, Dongchao Yang, Yi Ren, Luping Liu, Mingze Li, Zhenhui Ye, Jinglin Liu, Xiang Yin, and Zhou Zhao. Make-an-audio: Text-to-audio generation with prompt-enhanced diffusion models. In International Conference Machine Learning, pages 13916-13932. PMLR, 2023.
[15] Yanping Huang, Youlong Cheng, Ankur Bapna, Orhan Firat, Dehao Chen, Mia Chen, HyoukJoong Lec, Jiquan Ngiam, Quoe V Lc, Yonghui Wu, et al. Gpipe: Efficient training of giant neural networks using pipeline parallelism. Advances in neural information processing systems, 32, 2019.
[16] Zhihao Jia, Matei Zaharia, and Alex Aiken, Beyond data and model parallelism for deep neural networks. Proceedings of Machine Learning and Systems, 1:1-13, 2019.
[17] Yongeheng Jing, Yining Mao, Yiding Yang. Yibing Zhan, Mingli Song, Xinchao Wang, and Dacheng Tao. Learning graph neural networks for image style transfer. ECCV, 2022.
[18] Animesh Karnewar, Andrea Vedaldi, David Novotny, and Niloy J Mitra. Holodiffusion: Training a 3d diffusion model using 24 images. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages
2023.

---

[19] Bahjat Kawar. Shiran Zada, Oran Lang, Omer Tov, Huiwen Chang. Tali Dekel, Inbar Mosseri, and Michal Irani. Imagic: Text-based real image editing with diffussin models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6007-6017, 2023.
[20] Junjie Ke, Qifei Wang. Yilin Wang. Peyman Milanfar, and Feng Yang. Musiq: Multi-scale image quality transformer. In Proceedings of the IEEE/CVF international conference on computer vision, pages 5148 1158-5157,00 2021.
[21] Levon Khachatryan, Andranik Movsisyan. Vahram Tadevosyan, Roberto Henschel, Zhangyang Wang. Shant Navasardyan, and Humphrey Shi. Text2video-zero: Text-to-image diffusion models are zero-shot video generators. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 19954-15966 2023.
[22] Zhifeng Kong. Wei Ping. Jiaji Huang, Kexin Zhao, and Bryan Catanzaro. Diffwave: A versatile diffusion model for audio synthesis. arXiv preprint arXiv:2009, 2020.
[23] Bo Li, Kaitao Xue, Bin Liu, and Yu-Kun Lai. Bbdm: Image-to-inage translation with brownian bridge diffusion models. In Proceedings the IEEE/CVF conference on computer vision and pattem Recognition, pages 1952-1961, 2023.
[24] Muyang Li, Tianle Cai, Jiaxin Cao, Qinsheng Zhang. Han Cai, Junjie Bai, Yangqing Jia, Ming- Yu Liu, Kai Li, and Song Han. Distrifusion: Distributed parallel inference for high-resolution diffusion models. arXiv preprint 2rX: 19481, 2024.
[25] Senmao Li. Taihang Hu, Fahad Shahbaz Khan, Linxuan Li, Shiqi Yang. Yaxing Wang, Ming-Ming Cheng, and Jian Yang. Faster diffusion: Rethinking the role of unet encoder diffusion models. arXiv preprint 2023.
arXiv:2312.00
[26] Xin Li, Yulin Ren, Xin Jin, Cuiling Lan, Xingrui Wang, Wenjun Zeng Xinchao Wang. and Zhibo Chen. Diffusion models for image restoration and enhancement-a comprehensive survey, arXiv preprint arXiv:2308.09 2023.
[27] Yanyu Li, Huan Wang, Qing Jin, Ju Hu, Pavlo Chemerys. Yun Fu, Yanzhi Wang, Sergey Tulyakov, and Jian Ren. Snapfusion: Text-to-image diffussin model mobile devices within two seconds. Advances in Neural Information Processing Systems, 36, 2024.
[28] Zhuohan Li, Lianmin Zheng, Yinmin Zhong, Vincent Liu, Ying Sheng, Xin Jin, Yanping Huang, Zhifeng Chen, Hao Zhang. Joseph E Gonzalez, et al. (AlpaServe): Statistical multiplexing with model parallelism for deep learning serving. In 17th USENIX Symposium on Operating Systems Design and Implementation (OSDI 23), pages 663-679 2023.
[29] Tsung-Yt Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects context. In Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings. Part V 13, pages 740-755. Springer, 2014.
[30] Luping Liu, Yi Ren, Zhijie Lin, and Zhou Zhao. Pseudo numerical methods for diffusion models on manifolds. arXiv preprint arXiv:2202.0911 097788 2022.
[31] Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. Dpm-solver: A fast ode solver for diffusion probabilistic model sampling in around 10 steps. Advances in Neural Information Processing Systems, 35:5775-5755 2022.
[32] Simian Luo. Yigin Tan, Longbo Huang. Jian Li, and Hang Zhao. Latent consistency models: Synthesizing high-resolution images with few-step inference. arXiv preprint urXiv:2310.04378, 2023.
[33] Zhaoyang Lyu, Xudong Xu, Ceyuan Yang, Dahua Lin, and Bo Dai. Accelerating diffusson models via early stop of the diffuuion process. arXiv preprint arXiv:2205.1 2255, 2022.
[34] Xinyin Ma, Gongfan Fang, Michael Bi Mi, and Xinchao Wang. Leaming-to-caahe: Accelerating diffusion transformer via layer caching. arXiv preprint 0rXiv:2406.0173, 2024.
[35] Xinyin Ma, Gongfan Fang, and Xinchao Wang. Deepeache: Accelcrating diffusion models for free, arXiv preprint
2023.
[36] Anish Mittll, Rajiv Soundararajan, and Alan C Bovik. Making a "completely blind" image quality analyzer. IEEE Signal processing letters, 20(3):209-213 2012.
12

---

[37] Norman Müller. Yawar Siddiqui. Lorenzo Porzi, Samuel Rota Bulo. Peter Kontschieder. and Matthias NieBner. Diffrf: Rendering-guidde 3d radiance field diffuusio.. In Proceedings of the IEEE/CVF Conference on Computer Vixion and Puttern Recognition. pages 4328-4338 2023.
[38] Deepak Narayanan, Aaron Harlap, Amar Phanishayee, Vivek Seshadri, Nikhil R Devanur, Gregory Ganger, Phillip B Gibbons, and Matei Zaharia. Pipedream: generalized pipeline parallelism for dnn training. In Proceedings of the 27th ACM symposium on operating systems principles, pages 1-15, 2019.
[39] Deepak Narayanan, Mohammad Shoeybi, Jared Casper. Patrick LeGresley, Mostofa Patwary, Vijay Korthikanti, Dmitri Vainbrand, Prethvi Kashinkunti, Julie Bernauer, Bryan Catanzaro, et al. Efficient large-scale language model training on gpu clusters using megatron-Im. In Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, pages 1-15, 2021.
[40] Ozan Özdenizci and Robert Legenstein. Restoring vision in adverse weather conditions with bash- denoising diffusion models. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023.
[41] Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion models for high-resolution image synthesis. arXiv preprint arXiv:2307009952 2023.
[42] Ben Poole, Ajay Jain, Jonathan Barron, and Ben Mildenhall, Dreamfusion: Text-to-3d using 2d diffusion. arXiv preprint arXiv:2209.1444 14988, 2022.
[43] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björ Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEL/CVF conference on computer vision and pattern recognition, pages 10684-10695, 2022.
[44] Ludan Ruan, Yiyang Ma, Huan Yang, Huiguo He, Bei Liu, Jianlong Fu, Nicholas Jing Yuan, Qin Jin, and Baining Guo. Mm-diffusion: Learning multi-modal diffusion models for joint audio and video generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition. pages 10219- 10228, 2023.
[45] Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman. Dream- booth: Fine tuning text-to-image diffusion models for subject-driven generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 22500-22510, 2023.
[46] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily I. Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to- image difffusion models with deep language understanding, Advances in neural information processing systems, 35:36479-36498 2022.
[47] Chitwan Saharia, Jonathan Ho, William Chan, Tim Salimans, David Fleet, and Mohammad Norouzi. Image super-resolution via iterative refinement. IEEE transactions on pattern analysis and machine intelligence, 45(4):4713 4726, 2022.
[48] Tim Salimans and Jonathan Ho. Progressive distillation for fast sampling of diffusion models. arXiv preprint arXiv:2202.0055 2022.
[49] Hiroshi Sasaki, Chris G Willcocks, and Toby P Breckon. Unit-ddpm: Unpaired image translation with denoising diffusion probabilistic models. arXiv preprint arXiv:2104.03358 2021.
[50] Axel Sauer, Dominik Lorenz, Andreas Blattmann and Rohin Rombach. Adversarial diffusion distillation. arXiv preprint 2rXiv:231,,17042 2023.
[51] Yujun Shi, Chuhui Xue, Jiachun Pan, Wenqing Zhang, Vincent YF Tan, and Song Bai. Dragdiffusion: Harnessing diffusion models for interactive point-based image editing, arXiv preprint arXiv:2306. 14435, 2023.
[52] Andy Shih, Suneel Belkhale, Stefano Frmon, Dorsa Sadigh, and Nima Anari. Parallel sampling of diffusion models. Advances in Neural Information Processing Systems, 36, 2024.
[53] Junhyuk So, Jungwon Lee, and Eunhycok Park. Frdiff: Feature reuse for exquisite zero-shot acceleration of diffusion models. arXiv preprint arXiv:2112.03517, 2023.
[54] Jascha Sohl- Sohl-DDckkteen. Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International conference on machine learning. pages 2256-2265. PMLR, 2015.
13

---

[55] Jiaming Song. Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models, arXiv preprint arXiv:2010.02502. 2020.
[56] Xuan Su. Jiaming Song. Chenlin Meng, and Stefano Ermon. Dual diffusion implicit bridges for image-to- image translation. arXiv preprint arXiv:2203. 083382 2022.
[57] Zhenxiong Tan, Xinyin Ma, Gongfan Fang, and Xinchao Wang. Litefocus: Accelerated diffusion inference for long audio synthesis. arXiv preprint arXiv:2407 240400440... 2024.
[58] Zhenxiong Tan, Xingyi Yang, Songhua Liu, and Xinchao Wang. Video-infinity; Distributed long video generation. arXiv preprint arXiv:2406.16200 2024.
[59] Jianyi Wang, Kelvin CK Chan, and Chen Change Loy. Exploring clip for assessing the look and feel of images. In Proceedings the AAAI Conference on Artificial Intelligence, volume 37, pages 2023.
[60] Jianyi Wang, Zongsheng Yuc, Shangehen Zhou, Kelvin CK Chan, and Chen Change Loy. Exploiting diffusion prior for real-world image super-resolution arXiv preprint arXiv:2305.07015, 2023.
[61] Jiuniu Wang, Hangjie Yuan, Dayou Chen, Yingya Zhang, Xiang Wang, and Shiwei Zhang. Modelscope text-to-video technical report. arXiv preprint arXiv:2308.0665 2023.
[62] Zhizhong Wang,
Lei Zhao, and Wei Xing. Stylediffusion: Controllable disentangled style transfer via diffusion models. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 7677-7689, 2023.
[63] Felix Wimbauer, Bichen Wu, Edgar Schoenfeld, Xiaoliang Dai, Ji Hou, Zijian He, Artsiom Sanakoyeu, Peizhao Zhang, Sam Tsai, Jonas Kohler, et al. Cache me you can: Accelerating diffusion models through block caching. arXiv preprint wrXiv:2312.03200 2023.
[64] Jay Zhangjic Wu, Yixiao Ge, Xintao Wang. Stan Weixian Lei, Yuchao Gu, Yufei Shi, Wynne Hsu, Ying Shan, Xiaohu Qie, and Mike Zheng Shou. Tune-a-video: One-shot tuning of image diffusion models for text-to-video generation. In Proceedings of the IEEE/CVF International Conference Computer Vision. pages 7623-7633, 2023.
[65] Yuanzhong Xu, HyoukJoong Lee, Dehao Chen, Blake Hechtman, Yanping Huang, Rahul Joshi, Maxim Krikun, Dmitry Lepikhin. Andy Ly. Marcello Maggioni. et al. Gspmd: general and scalable parallelization for ml computation graphs. arXiv preprint 1rX5v:2005.04663, 2021.
[66] Binxin Yang, Shuyang Gu, Bo Zhang. Ting Zhang, Xuejin Chen, Xiaoyan Sun, Dong Chen, and Fang Wen. Paint by example: based image editing with diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18381-18391, 2023.
[67] Xingyi Yang and Xinchao Wang. Hash3d: Training- acceleration for 3d generation. arXiv preprint arXiv:2404. 06691, 2024,
[68] Xingyi Yang, Daquan Zhou, Jiashi Feng, and Xinchao Wang. Diffusion prohabilistic model made slim. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 22552- 22562, 2023.
[69] Tianwei Yin, Michaël Gharbi, Richard Zhang, Eli Shechtman, Fredo Durand, William Freeman, and Taesung Park. One-step diffusion with distribution matching distillation. arXiv preprint arXiv:2311.18828 2023.
[70] Fanghua Yu, Jinjin Gu, Zheyuan Li, Jinfan Hu, Xiangtao Kong, Xintao Wang. Jingwen He, Yu Qiao, and Chao Dong. Scaling up to excellence: Practicing model scaling for photo-realistic image restoration in the wild. arXiv preprint 240i 13627, 2024.
[71] Zongsheng Yue, Jianyi Wang. and Chen Change Loy. Resshift: Efficient diffuusion model for image super-resolution by residual shifting, Advances Neural Information Processing Systems, 36, 2024.
[72] Chenshuang Zhang, Chaoning Zhang, Mengchun Zhang, and In So Kweon. Text-to-image diffusion model in generative ai: A survey. arXiv preprint arXiv:2303.07900 2023.
[73] Dingkun Zhang, Sijia Li. Chen Chen, Qingsong Xie. and Haonan Lu. Laptop-diff: Layer pruning and normalized distillation for compressing diffusion models. arXiv preprint 1.Xx::4404.1199 2024.
[74] Qinsheng Zhang and Yongxin Chen. Fast sampling of diffusion models with exponential integrator. arXiv preprint 2204 2304. 2022.
14

---

[75] Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features perceptual metric. In Proceedings of the IEEE conference computer vision and puttern recognition, pages 586-595, 2018.
[76] Wentian Zhang, Haozhe Liu, Jinheng Xie, Francesco Faccio, Mike Zheng Shou, and Jürgen Schmid- huber. Cross-attention makes inference cumbersome in text-to-image diffusion models. arXiv preprint arXiv:2404.02747, 2024.
[77] Zhixing Zhang, Ligong Han, Armab Ghosh, Dimitris Metaxas, Jian Ren. Sine: Single image editing with text-to-image diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages
2023.
[78] Shihao Zhao, Dongdong Chen, Yen-Chun Chen, Jianmin Bao, Shaozhe Hao, Lu Yuan, and Kwan- Yee K Wong. Uni-controlnet: All-in-one control to text-to-image diffusion models. Advances in Neural Information Processing Systems, 36, 2024.
[79] Xuanlei Zhao, Xiaolong Jin, Kai Wang, and Yang You. Real-time video generation with pyramid attention broadcast. arXiv preprint arXix:240.. /2588, 2024.
[80] Yang Zhao, Yanwu Xu, Zhisheng Xiao, and Tingbo Hou. Mobilediffusion: Subsecond text-to-image generation mobile devices. arXiv preprint arXiv:2311.1656/ 2023.
[81] Kaiwen Zheng, Cheng Lu, Jianfei Chen, and Jun Zhu. Dpm-solver-v3: Improved diffusion ode solver with empirical model statistics, Advances in Neural Information Processing Systems, 36, 2024.
15

---

* In this document, we provide supplementary materials that extend beyond the scope of the main manuscript, constrained by space limitations.
Warm-up
Parallel
Parallel
Parallel
Parallel
|ooo
HooD 7-1
3 (e
7--3
1-5
'-7
|ooo
H
Mose
Moc-I
GPU2 2 (E%)
T-4
T-6
T-8
GPU
Jol 7-2
Do0
(E))
T-4
T-6
T-8
||
GPU 0 (e))
7-6
7-8
Communication within device
Communication across devices
T: Time embedding
Figure 7: Schematic of the asynchronous diffusion model with stride denoising. The model cp is divided into three components {< 33 >= with a stride of for clarity. A single parallel batch results in the completion of denoising for two steps
A More Implementation Details.
Model Segmentation. In our method, we partition the cumbersome denoising model into multiple components, each assigned a different device. After successfully parallelizing the computation of each component, the time cost for each time step now corresponds to the maximum latency among these components. To optimize parallel processing efficiency, we partition the model into segments that each carry roughly equal computational load. This arrangement allows all modules to finish their computations nearly simultaneously, making full use available computational resources. The segmentation strategy is sequential except for SDXL [41. For the denoising U-net within the SDXL module, we group its first and last blocks into single segment and apply sequential splitting to the remaining blocks. This is because SDXL has specific needs for high- figh-freeuency details, and res connections typically contain abundant high-frequency information.
Time Shifting. We introduce a technique called time shifting, Following the warm-up steps, the time embedding for each step is shifted back by one step. For instance, in a 50-step asyn- chronous denoising process with a warm-up of 2 steps, the original sequence of time embeddings is (50,49 49, 48, 47, 3, 2, 1). With time shifting, this sequence adjusted (50, 49, 49, 48. 3. In certain extreme cases, asynchronous denoising might leave residual noise in the output. Time shifting addresses this by adjusting the time embeddings backward, enhancing the denoising effect. It's important to note that time shifting is not a standard component of our method but is employed optionally. The quantitative results presented in this paper are achieved without the use of time shifting
Stride Denoising. To further enhance efficiency, we introduce stride denoising, which completes multiple denoising steps simultaneously through a single parallel computation. Figure 7 illlstrates the full schematic of applying stride denoising to AsyncDiff. is divided into three components <n3 6===1 and for clarity, the stride S is set to 2. Unlike the continuous broadcasting of hidden states at each time step, stride denoising broadcasts them every two steps. As depicted, at time step (T 1,7 3, T 5, T 7]. we conduct denoising alone, and at time step (T 2, T 4, T 6, T - 8). we compute and broadcast the hidden states for the next parallel computation round. Consequently, the hidden states from time step {T 1,T 3, T 5, 7}
this depiction, the denoising model ca
16

---

are not required, allowing us to skip the calculations for cg and cg at these steps. Stride denoising effectively reduces both computational load and communication demands by decreasing the parallel computing rounds needed to complete the process. Compared to the significant improvements it brings efficiency, the quality sacrifice is minimal and can entirely compensated for by slightly increasing the warm-up steps.
B
More Analysis.
Time cost. In Table we present the time costs associated with model running and inter-device communication when using AsyncDiff on SD 2.1. Generally, communication expenses constitute only minor fraction of the total time cost, demonstrating that AsyncDiff is effective distributed acceleration technique suitable for practical application. It is important to note that as the number of devices increases, the time needed for data broadcasting between devices also rises, thereby increasing the proportion of communication costs. However, employing stride denoising can substantially reduce these costs by decreasing the number of parallel rounds needed to complete the denoising process.
Table 9: Time cost comparisons SD) 2.1. 'Ratio' in this table represents the proportion of communication cost to overall latency. All measurements were conducted on NVIDIA A5000 GPUs equipped with NVLINK Bridge
Time Cost
Config
Overall
Running
Comm.
Ratio
N=2 $=1
3.03s
2.90s
0.13s
4.30%
N=3 $=1
2.41s
2.18s
0.23s
9.54%
N-4
2.10s
1.80s
0.30s
14.29%
N=2 S=2
1.82s
1.70s
0.12s
6.59%
N=3 S=2
1.35s
1.25s
0.10s
7.40%
Speedup Ratio. We also evaluate the acceleration ratio on SD 2.1 with varying numbers of denoising steps. As indicated Table [T] AsyncDiff significantly enhances processing speed, even with denoising procedure consisting of only 25 steps. When the number of steps extends to 100, our approach achieves a speedup of up to 4.3x, surpassing the ratio of devices employed.
Table 10: Acceleration ratio on SD 2.1 under different num denoising steps
Speedupt
Config
25steps
50steps
100steps
Origin
1.0x (2.89s)
1.0x (5.51s)
1.0x (10.96s)
N=2 S=1
1.7x (1.70s)
1.8x (3.03s)
1.8x (6.04s)
N=3 S=1
2.1x (1.35s)
2.3x (2.41s)
2.3x (4.71s)
N=4 S=1
2.4x (1.21s)
2.6x (2.10s)
2.7x (4.01s)
N=2 S=2
2.7x (1 (1.05s)
3.0x (1.82s)
3.2x (3.39s)
N=3 8=2
3.4x (0.86s)
4.0x (1.35s)
4.3x (2.52s)
C More Quantitative Results.
To thoroughly assess the quality of images produced following acceleration, we provide quantitative analyses on three base models (SD 2.1 [3]. SD 1.5 433 SDXL (41) using four additional metries: the full reference metric, DISTS [5], and no-reference metrics including MUSIQ (20]. CLIP-IQA [59], and NIQE (36). The experimental results Table demonstrate that our method significantly reduces inference latency while maintaining high level of quality in diffusion model model-generated images. On SD 1.5, our approach not only accelerates the inference process but also brings the image quality closer to the natural distribution.
17

---

Table 11: Quantitative evaluations of AsyncDiff on three text-to-image diffusion models using more
metries including DISTS [5). MUSIQ 20), CLIP-IQA 59 and NIQE [36].
Base Model
Configuration
Devices
DISTS!
MUSIQT
CLIP-IQAT
NIQEL
Original Model
69.95
0.6653
3.9675
Ours (N=2 S=1)
2
0.1041
69.55
0.6539
3.8850
+ Ours (N=3 S=1)
3
0.1280
69.04
0.6441
3,9438
SD 2.1
+ Ours (N=4 S=1)
4
0.1419
68.58
0.6365
3,9724
+ Ours (N=2 $=2)
3
0.1556
68.03
0.6158
3.5761
+ Ours (N=3 S=2)
0.1689
67.13
0,5986
3.6761
Original Model
1
71.98
0.6534
3.5517
+ Ours (N=2 $=1)
2
0.1169
72.21
0.6569
3.7448
+ Ours (N=3 S=1)
3
0.1434
71.73
0.6481
3,8023
SD 1.5
+ Ours (N=4 S=1)
4
0.1599
71.51
0.6442
3.8620
+ Ours (N=2 S=2)
3
0.1668
71.14
0.6323
3.9613
+ Ours (N=3 S=2)
0.1905
69.42
0.6070
4.1047
Original Model
1
71.58
0.6633
4,0743
+ Ours (N=2 S=1)
2
0.1038
70.56
0.6498
4.1139
+ Ours (N=3 $=1)
3
0.1211
69.88
0.6389
4.1585
SDXL
+ Ours (N=4 $=1)
0.1391
67.70
0.6056
4.0927
+ Ours (N=2 S=2)
0.1329
69.56
0.6222
4.1685
Ours (N=3 S=2)
4
0.1527
68.16
0.5955
4.2745
D More Qualitative Results
Qualitative Results on Image Diffusion Models. As depicted in Figure g we present further qualitative results for SD 2.1 and SDXL under various configurations. The speedup achieved is nearly proportional to the number of devices utilized, indicating efficient resource usage by our method. Moreover, the images generated by approach closely match the text descriptions and of high quality.
Qualitative Results on Video Diffusion Models. We present qualitative evaluations of AsyncDiff applied the video diffusion models. Figures TO and illustrate the generated results using our method on the text-to-vide video model Animate Diff [9. Figure [2 displays results from applying our method the image-to-video model SVD 2 For a 50-step video diffusion model, AsyncDiff markedly decreases latency-savine saving tens or even hundreds of seconds seconds-while while maintaining the integrity and quality of the generated videos.
E
Limitations
As distributed acceleration framework, AsyncDiff necessitates frequent communication between devices throughout the denoising process. Consequently, if the devices lack the capability to com- municate effectively have subpar communication infrastructure, optimally. Additionally, AsyncDiff operates as a a aadg-andd-lay acceleration solution that depends on pre-trained diffusion models. Therefore, the baseline quality of the original diffusion models is unsatisfactory, achieving high-quality results with our method could be challenging.
method may not perform
Societal impacts
In this paper, we introduce universal distributed acceleration approach for diffusion models. This method substantially speeds up the inference phase of diverse diffusion models by fully leveraging computational resources. holds significant potential for practical applications, particularly in computationally intensive generation tasks like video and speech generation.
18

---

(a) More Qualitative Results on SDD 2.1 with different configurations
Original
Oun Speeiar (V-2 5-1)
233x Spedaa (NssreesNN-33-))
Oats Speodap Devices (N-4 5-1)
Ouz Sooodap
Oan Specdar Deries (Ni3 S-2)
(N-2 5-2)
(b) More Qualitative Results SDXL with different configurations
Original
2 Desices (N-22
Our 1.7s Spoadap
Currs 244 Speciap Deviess (N-3 So1)
Oura 2n Speciae Devios (Ni44 Si-1)
Oure 2.8 Specakee
Deries (-221
Ous Jás Spoolep
Desices (N-3 S-1)
Figure Qualitative results on SD1.1 and SDXL with different configurations. Our method maintains excellent generation quality even when achieving speedups of up to four times.
19

---

Prompt: Brilliant fireworks the town, Van Gogh style, digital ortwork, ilhattaatvve painterly matte painting, highly detailed, cinematic
Original 43.5s
Ours 23.5a (2 devices)
Ours 11.5a devices)
Figure 9: Qualitative results AnimateDiff (1)
Prompt: panda playing a gaitar, on a boot, in the oceon, high quality Original 43.5s
Ours 23.5s (2 devices)
Ours 11.5% devices)
Figure 10: Qualitative results on AnimateDiff (2)
Prompt book style, Batmon wolking colored, dynamic buckground, full body clean sharp Original 43.5s
Ours 23.5s (2 devices)
devices)
Figure 1I: Qualitative results on AnimateDiff (3)
20

---

Original 184s
Ous 101s 2 Devices
Ous 80s Devices
Ous 64s Devices
Figure 12: Qualitative results on Stable Video Diffusson
21

---

NeurIPS Paper Checklist
The checklist is designed to encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No], or [NA].
[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1-2 sentence) justifiicton right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.
The reviewers of your paper will be asked to use the checklist as one of the factors in their evaluation.
While "[Yes] " is generally preferable to "[No]", ", it is perfectly acceptable to answer "[No] provided proper justification is given (e.g., "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering "[No] or "[NA] is not grounds for rejection. While the questions are phrased in binary way, we acknowledge that the true answer often more nuanced, so please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS paper checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only use the provided macros for your answers.
Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The main claims made in our abstract and introduction accurately reflect the paper's contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes]
22

---

Justification:
We discuss the limitation of our work in the Appendix.E.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approxinations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g. if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification:
The paper does not include theoretical results
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: We provide mental results.
a detailed description of our method along with extensive experi-
Guidelines:
23

---

The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors welcome deseribe the particular way they provide for reproducibilityy In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification:
We offer the full code along with relevant instructions.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.. cc/
wtw.eeuuubissiioooooooooo for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https CodeeSumiiii wwwiiisssooooiiii
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
24

---

Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We provide all the details about the experiment in our paper.
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented in the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer:
[Yes]
Justification:
We provide the details about initialization and dataset split.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
It is OK to report I-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not show in tables figures symmetric error bars that would yield results that are out of range (c.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We provide the details about the computation resources used in the experi- ments.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
25

---

The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethies
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics htttp:://neurij:: http://ppp....pppp.pp..ccaaaaaaa
Answer:
[Yes]
Justification: We strictly adhere to the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g.. if there special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We discuss the societal impacts in the Appendix.F.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Answer: [NA]
Justification: The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses such risks.
26

---

Released models that have high risk for misuse or dual use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (c.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
The creators or original owners of assets (e.g., code, data, models) used in the paper are properly credited, and the license and terms of use are explicitly mentioned and properly adhered to.
Justification:
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, paperswi paperswithcode thcode. com /atasets
has curated licenses for some datascts. Their licensing guide can help determine the license of dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: New assets introduced in the paper are well documented, and the documenta- tion is provided alongside the assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mooel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
27

---

Answer:
[NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Question:
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
28


Reviews:
[{"review_id": "0h62v3URnG", "reviewer": "Reviewer_E7zL", "summary": "This paper proposes AsyncDiff to enable diffusion model parallelism across multiple devices and achieve a very impressive speedup with negligible degradation. Specifically, the denoising model is split into several components, each assigned to a different device. The conventional sequential denoising process is transformed into an asynchronous one by exploiting the high similarity of hidden states between consecutive time steps, enabling each component to compute in parallel.", "questions": "I do not have other questions about this paper.", "limitations": "As stated by the author, AsyncDiff necessitates frequent communication between devices throughout the denoising process. Therefore, if the devices lack the capability to communicate effectively, AsyncDiff may not perform optimally.", "rating": 6, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "1. AsyncDiff is a universal and plug-and-play acceleration scheme and can enable model parallelism across multiple devices. The acceleration that AsyncDiff achieves on image and video diffusion models is impressive\n2. The motivation is clear and the experiments are comprehensive.", "weaknesses": "1. The discussion and comparison with the existing diffusion sampler is not sufficient, whether AsyncDiff can be combined with other diffusion samplers such as DPM-Solver.\n2. In the absence of an explanation of the GPU, and AsyncDiff's ablation experiments on different GPU devices, can other GPUs achieve a good acceleration ratio?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## **Q1: Whether AsyncDiff can be combined with other diffusion samplers such as DPM-Solver**\nThanks for the valuable feedback. AsyncDiff is a universal method that can be used with various samplers, including the DPM-Solver. Table 1 below provides the quantitative evaluation of AsyncDiff on the SD 2.1 with DPM-Solver sampler. With the same speedup ratio, AsyncDiff significantly improves generation quality compared to the baseline. We also provide the qualitative results in **Figure 1 (a)** of our **attached PDF File**. Our method achieves significant acceleration while maintaining higher consistency with the original output. \n\n### Table 1: Quantitative evaluations of AsyncDiff using DPM-Solver sampler\n*Note: We compared the generative quality when using the DPM-Solver sampler and AsyncDiff to achieve the same speedup. The SD 2.1 using a 25-step DPM-Solve sampler is the baseline.*\n\n| **Method**           | **Speed up ↑** | **MACs ↓** | **CLIP Score ↑** | **FID ↓** | \n|----------------------|----------------|------------|------------------|-----------|\n| SD 2.1 DPM-Solver 25steps  | 1.0x     | 76T        | 31.57            | 28.37     | \n| SD 2.1 DPM-Solver 15steps  | 1.6x     |  46T       |  31.52           | 28.89     | \n| **Our AsyncDiff (N=2 S=1)**| 1.6x     |  **38T**   |  **31.58**       | **27.71** | \n| SD 2.1 DPM-Solver 10steps  | 2.2x     | 30T        |  31.29           | 29.28     | \n| **Our AsyncDiff (N=3 S=1)**| 2.2x     | **25T**    | **31.36**        | **28.20** | \n\n\n## **Q2: Can other GPUs achieve a good acceleration ratio?**\nThanks for the valuable comments. We conducted additional experiments to compare the acceleration ratio of AsyncDiff on three different GPUs (NVIDIA RTX A5000, NVIDIA RTX 3090, and NVIDIA RTX 2080 Ti). As shown in Table 2, our method achieved a significant speedup across all tested GPUs.\n\n### Table 2: Acceleration Ratio and Latency on Different GPUs\n*Note:  Acceleration Ratio and Latency are evaluated using CUDA 12.0 and PyTorch 2.1*\n\n| **GPU**     | **FP16 Compute** |Original|N=2 S=1|N=3 S=1|N=2 S=2|N=3 S=2|\n|-------------|------------------|-------------|-------------|-------------|-----------|-----------|\n| NVIDIA RTX A5000   | 117 TFLOPS| 1.0x(5.51s)| 1.8x(3.03s)  | 2.3x(2.41s) | 3.0x(1.82s)| 4.0x(1.35s)|\n| NVIDIA RTX 3090    | 71  TFLOPS| 1.0x(5.61s)| 1.8x(3.20s)  | 2.1x(2.65s) | 2.9x(1.91s)| 3.5x(1.60s)|\n| NVIDIA RTX 2080Ti | 54  TFLOPS| 1.0x(8.20s)| 1.7x(4.91s)  | 2.0x(4.08s) | 2.8x(2.94s)| 3.5x(2.35s)|", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "W1vGkuDw6Z", "reviewer": "Reviewer_iHdZ", "summary": "This work introduces an acceleration method for diffusion models by distributing the model blocks across multiple GPUs and running different blocks asynchronously. The core motivation of this paper lies in the observation that the hidden states of a block exhibit high similarity across consecutive diffusion steps. This enables the execution of different blocks based on pre-computed results from earlier steps. The proposed method was evaluated on Stable Diffusion v2.1, v1.5, and XL, achieving a 1.8x to 4x acceleration with minimal performance loss on CLIP score and FID.", "questions": "1) What happens if the total number of timesteps is not divisible by the number of GPUs? Is the proposed method still applicable in this scenario?\n\n2) Is the proposed method still effective on low-end GPUs or edge devices?", "limitations": "N/A.", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "1) The idea of parallelizing the inference of different blocks is quite interesting. The method is general and can be applied to several diffusion models like Stable Diffusion, Stable Video Diffusion, and AnimateDiff. The method shows robustness to different speed-up ratios as shown in Tables 3, 4, and 5. \n\n2) Extensive experiments were conducted in this paper, encompassing both image generation and video generation. And the results are good: the proposed method as able to achieve significant acceleration with slight performance degradation.", "weaknesses": "1) Is the communication cost huge compared to the inference cost? As shown in Table 3, the number of GPUs used in this paper is primarily 2 or 3. Will there be issues if this is extended to 8 or 16 GPUs? Is this method still competitive when the number of devices exceeds 8, where cross-node communication becomes substantial?\n\n2) The experimental section mainly focuses on quality metrics such as FID and CLIP Score. It would be beneficial if the authors could provide more analytical results about the method, such as the similarity of hidden states.\n\n3) Figure 5: “Quantitative” should be “Qualitative”.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## **Q1: Is the communication cost huge compared to the inference cost? Will there be issues if this is extended to 8 or 16 GPUs? Is this method still competitive when the number of devices exceeds 8, where cross-node communication becomes substantial?**\nThanks for the valuable comment, this is indeed a question that needs further explanation\n\nAs demonstrated in Table 1, the communication cost is minimal compared to the inference cost. \n\nApplying to 8 or 16 GPUs can lead to increased communication costs due to GPU architecture. In a common 8-GPU node, GPUs 0, 1, 2, and 3 form one group, while GPUs 4, 5, 6, and 7 form another. Cross-group communication is about 40% more costly than within a group, the cross-node cost is even higher. This is a shared challenge for all distributed inference frameworks, both in the Diffusion and LLM areas.\n\nOur proposed stride denoising technique effectively alleviates this issue by reducing communication frequency. As shown in Table 2, during a 50-step diffusion process, AsyncDiff requires only 25 communications. This significant reduction means our method is less affected by increased cross-group or cross-node communication costs, keeping it competitive. In the future, we will explore more innovative technologies to further address this general challenge.\n\n### Table 1: Time cost comparisons on SD 2.1\n*Note: 'Overall' represents the overall latency, 'Running' represents the time cost of model running, 'Comm.' represents the communication time cost. 'Ratio' in this table represents the proportion of communication cost to overall latency. All measurements were conducted on NVIDIA A5000 GPUs.*\n| **Configuration** | **Devices** | **Overall** | **Running** | **Comm.** | **Ratio** |\n|------------|-------------|-------------|-------------|-----------|------------|\n| AsyncDiff N=2 S=1    | 2 GPUs| 3.03s       | 2.90s       | 0.13s     | 4.30%     |\n| AsyncDiff N=3 S=1    | 3 GPUs| 2.41s       | 2.18s       | 0.23s     | 9.54%     |\n| AsyncDiff N=4 S=1    | 4 GPUs| 2.10s       | 1.80s       | 0.30s     | 14.29%    |\n| AsyncDiff N=2 S=2    | 3 GPUs| 1.82s       | 1.70s       | 0.12s     | 6.59%     |\n| AsyncDiff N=3 S=2    | 4 GPUs| 1.35s       | 1.25s       | 0.10s     | 7.40%     |\n\n### Table 2: Effect of stride denoising in the 50-step Diffusion Process\n*Note: Stride denoising significantly lowers communication costs by decreasing the communication frequency*\n\n| **Configuration** |Communication Nums↓| Communication Freq↓| \n|------------|-------------|-------------|\n| *AsyncDiff* w/o stride denoising  | 49 times | 0.98/timestep|\n| *AsyncDiff* w/ stride denoising  |  **25 times** | **0.50/timestep**|\n\n## **Q2:  It would be beneficial if the authors could provide more analytical results about the method, such as the similarity of hidden states.**\nThank you for your valuable comment. We provide quantitative analysis and visualization of the hidden state similarities in **Figure 2** and **Figure 3** of the **attached PDF FILE**.\n\n**Figure 2** shows the cosine similarity and MSE for each block's hidden state between adjacent steps. As illustrated, the hidden states are highly similar between most steps, except at the initial stage. This high similarity enables us to use features from the previous step to parallelize calculations. **Figure 3** further visualizes this phenomenon by displaying the hidden states of adjacent steps in the diffusion process. The initial instability demonstrates why slightly increasing the warm-up step can make the AsyncDiff output image pixel-wise consistent with the original output, which is also shown in Table 2 in our submission.\n\n## **Q3: Figure 5: “Quantitative” should be “Qualitative”.**\nThanks for the careful review. We will correct this error in the next version.\n\n## **Q4: What happens if the total number of timesteps is not divisible by the number of GPUs? Is the proposed method still applicable in this scenario?**\nThank you for the valuable comment. In this case, AsyncDiff will not face any issues. Although we split the denoising model into *N* parts, we still obtain one timestep of noise after each parallel batch. However, the runtime for this step in the denoising model is reduced to *1/N*.\n\nThis is supported by the experimental results in Table 1 of our submission. When accelerating a 50-step diffusion model, AsyncDiff performs well, even when using 3 or 4 GPUs.\n\n## **Q5: Is the proposed method still effective on low-end GPUs or edge devices?**\nThank you for the valuable comment. This is a very practical question. \nWe conducted additional experiments to show the acceleration ratio of AsyncDiff on consumer-grade graphics cards (NVIDIA RTX 2080 Ti GPUs). As shown in Table 3, our method still achieved a significant speedup on these GPUs.\n\n### Table 3: Acceleration Ratio and Latency on low-end GPUs\n*Note:  Acceleration Ratio and Latency are evaluated using CUDA 12.0 and PyTorch 2.1*\n\n| **GPU**     | **FP16 Compute** |Original|N=2 S=1|N=3 S=1|N=2 S=2|N=3 S=2|\n|-------------|------------------|-------------|-------------|-------------|-----------|-----------|\n| NVIDIA RTX 2080Ti | 54  TFLOPS| 1.0x(8.20s)| 1.7x(4.91s)  | 2.0x(4.08s) | 2.8x(2.94s)| 3.5x(2.35s)|", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "T3kLInd7zz", "reviewer": "Reviewer_7eQm", "summary": "This paper proposes AsyncDiff, a plug-and-play acceleration scheme that enables model parallelism across multiple devices. The core method involves dividing the diffusion model into multiple components and executing the inference in parallel. This is facilitated by the high similarity between hidden states in consecutive diffusion steps. With AsyncDiff, it is claimed that off-the-shelf diffusion models can be effectively accelerated with negligible degradation. Therefore, the key contribution of this paper lies in providing a simple and effective method to accelerate the diffusion process through parallelism.", "questions": "Additional Questions:\n1.\tIs this method helpful for large batch sizes?\n2.\tIs this method applicable to more GPUs?", "limitations": "I do not find any potential negative societal impact of this work.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "1.\tThe proposed method is well-motivated, and the effectiveness of AsyncDiff has been evaluated on multiple models, making the results compelling.\n2.\tThis method should be useful for video generation, which can take minutes or hours to produce a single video. The paper also supports this with experiments conducted on Stable Video Diffusion.\n3.\tAs shown in Table 5, it seems that the communication cost can be covered by the inference cost, which is a favorable feature for model acceleration.", "weaknesses": "1.\tSome concepts, such as the “dependency chain” in ABS, are not well explained. It would be beneficial if the author could provide a minimal explanation for these concepts.\n2.\tThe paper appears to share a related idea with Distrifusion [1]. Could the author provide a more detailed discussion about the key differences between them? For example, the main difference seems to be the splitting schemes, where Distrifusion splits the data, and this approach splits the model. What is the main advantage of AsyncDiff?\n3.\tAre there any failure cases for the proposed AsyncDiff? \n[1] DistriFusion: Distributed Parallel Inference for High-Resolution Diffusion Models", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## **Q1: Some concepts, such as the “dependency chain” in ABS, are not well explained. It would be beneficial if the author could provide a minimal explanation for these concepts**\nThank you for the valuable feedback. This is indeed a question that requires further explanation. In the diffusion process, there are two \"dependency chains\": one between time steps and the other between model blocks.\n\n**Dependency between time steps:** The denoising process is sequential, meaning that denoising at timestep *t* depends on the completion of denoising at timestep *t-1*. This sequential dependency causes latency accumulation and limits the ability to perform parallel inference.\n\n**Dependency between blocks:** The large denoising model is split into *N* components, each with similar computational loads. The input to the *n*th block depends on the output of the *n-1* th block, which poses challenges for running the entire denoising model in parallel.\n\nOur AsyncDiff seeks to disrupt both the dependency chain between time steps and the dependency chain between blocks by leveraging the hidden state's similarity between adjacent steps, allowing for the parallel execution of the denoising process while closely approximating the results of the sequential process.\n\n## **Q2: Could the author provide a more detailed discussion about the key differences between AsyncDiff and Distrifusion? What is the main advantage of AsyncDiff?** \nThank you for the comment. As you mentioned, the main difference between AsyncDiff and Distrifusion lies in their core approaches. AsyncDiff focuses on model parallelism, enabling the denoising model to run in parallel regardless of the input data. In contrast, Distrifusion emphasizes data parallelism, parallelizing the computation across different patches of the image. \nBased on these differences, the advantages of AsyncDiff can be summarized as follows:\n\n**1. Wider Applicability:**  \nAsyncDiff is a more universal acceleration framework because it focuses on model parallelism, independent of the type of input data or the task of the original model. This makes it suitable for various diffusion-based generation tasks, including text-to-image, text-to-video, image-to-video, super-resolution, ControlNet, speech generation, etc. It can also be used with both Unet-based and DiT-based diffusion models.\n\n**2. Lower Memory Requirements:**  \nDistrifusion requires caching activation maps for each layer, significantly increasing GPU memory demands and complicating practical applications. In contrast, AsyncDiff only needs to cache a few hidden states on each device, keeping memory requirements nearly the same as the original model.\n\n**3. Higher GPU Utilization:**  \nWhen the generated image resolution is not very high, Distrifusion can lead to lower GPU utilization, resulting in underutilization of computing resources. AsyncDiff mitigates this issue by not altering the size of the input data, thus maintaining efficient GPU usage. The experimental results in Table 3 of our submission show that AsyncDiff can achieve the same speedup with four GPUs as Distrifusion does with eight GPUs, while also delivering better generation quality.\n\n### Table 1: Quantitative comparison Distrifusion on SD 2.1\n*To ensure a fair comparison with Distrifusion, we increased the warm-up steps in our method to match the speedup ratio of Distrifusion, allowing us to fairly compare generation quality and resource costs.*\n| **Method**| **Speed up ↑** | **Devices** | **MACs ↓** | **Memory ↓** | **CLIP Score ↑** | **FID ↓** | **LPIPS ↓** |\n|-----|-------|----|---|----|-----|-----|-----|\n| Original Model | 1.0x | 1 | 76T | 5240MB| 31.60| 27.87 | --|\n| **Distrifusion** | 1.6x | **2** | **38T**| 6538MB | **31.59**| 27.89| **0.0178**  |\n| **Ours (N=2 S=1)** | 1.6x | **2** | 44T    | **5450MB** | **31.59**| **27.79** | 0.0944 |\n| **Distrifusion**| 2.3x | 4     | **19T**| 7086MB | 31.43| 27.97 | 0.2710 |\n| **Ours (N=2 S=2)** | 2.3x | **3** | 20T    | **5516MB**| **31.49**| **27.71** | **0.2117**  |\n| **Distrifusion**| 2.7x | 8| **10T**| 7280MB| 31.31 | 28.12     | 0.2934 |\n| **Ours (N=3 S=2)**| 2.7x | **4** |14T| **5580MB**| **31.40**| **28.03** | **0.1940**  |\n\n## **Q3: Are there any failure cases for the proposed AsyncDiff?**\nAs an acceleration method, AsyncDiff depends on the quality of the original base model. If the base model's generation quality is not good, AsyncDiff will also produce unsatisfactory results. While our method can significantly speed up the process and maintain the original generation quality, it typically cannot enhance the base model's original generation quality.\n\n## **Q4:Is this method helpful for large batch sizes?**\nThanks for the comment. As shown in the following Table 2, AsyncDiff will still be helpful for large batch sizes.\n### Table 2: Acceleration and Latency under Different Batchsizes\n*We evaluate the running speed of AsyncDiff on SD 2.1. The latency is evaluated using Nvidia RTX A5000 GPUs, CUDA 12.0, and PyTorch 2.1.*\n| BatchSize|Original|N=2 S=1|N=4 S=1|N=3 S=1|N=2 S=2|N=3 S=2|\n|----|-----|-----|-----|-----|-----|-----|\n| 1| 1.0x(5.51s)| 1.8x(3.03s)  | 2.3x(2.41s)| 2.6x(2.10s)| 3.0x(1.82s)| 4.0x(1.35s)|\n| 4| 1.0x(19.34s)| 1.7x(11.46s)| 2.2x(8.60s)| 2.5x(7.85s) | 2.9x(6.75s)| 4.1x(4.77s)|\n| 8| 1.0x(36.19s)| 1.7x(21.46s)| 2.2x(16.44s)| 2.4x(14.90s) | 2.8x(12.75s)| 4.0x(9.06s)|\n\n## **Q5:Is this method applicable to more GPUs?**\nThanks for the comment. Asyncdiff can be applicable to more GPUs. However, we do not recommend splitting the model into more than four components, as the speed gains are minimal. In practice, combining AsyncDiff with other data parallel methods is more effective.  Asyncdiff is compatible with any other data-parallel solution (batch splitting, patch splitting). Taking advantage of model parallelism and data parallelism at the same time will achieve a more extreme speedup ratio while minimizing the sacrifice of generation quality.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "FJ45JlZcDc", "reviewer": "Reviewer_773c", "summary": "This paper introduces AsyncDiff, an acceleration framework for diffusion models that transforms the traditional sequential denoising process into an asynchronous process. The key insight is that hidden state features in consecutive sampling steps exhibit high similarity. Therefore, feeding the output of the preceding component at time step t-1 as an approximation of the original input to each U-Net component has a negligible impact on performance. This approach allows the parallel denoising processes of AsyncDiff to operate fully asynchronously. Experiments on various versions of Stable Diffusion demonstrate that AsyncDiff significantly speeds up the denoising process for text-to-image generation. Additionally, AsyncDiff is applicable to video generation models such as AnimateDiff and SVD, further showcasing its versatility.", "questions": "Most questions are included in the “Weaknesses” section.\n\nWould a similar asynchronous denoising approach be applicable in a conditional generation setup as well? For instance, achieving faster inference for diverse conditional image generation models, such as ControlNet [1] and Zero-1-to-3 [2], could be practically useful.\n\n[1] Adding Conditional Control to Text-to-Image Diffusion Models, Zhang et al., ICCV 2023\n\n[2] Zero-1-to-3: Zero-shot One Image to 3D Object, Liu et al., ICCV 2023", "limitations": "The authors discuss the limitations of AsyncDiff (communication cost, dependency on the base model) and the potential societal impacts.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. The writing is overall clear and well-structured. The paper effectively explains the limitations of previous methods that use patch parallelism and introduces a novel approach to \"asynchronous\" denoising to address these issues.\n2. The paper provides thorough comparisons with baseline methods, such as Distrifusion, demonstrating clear improvements in both generation quality and efficiency.\n3. The versatility of AsyncDiff is evidenced by experiments conducted on different versions of Stable Diffusion as well as video generation models.\n4. The authors address concerns regarding the overhead of communication costs across multiple GPUs, showing that these costs are significantly lower than the model execution time.", "weaknesses": "1. Although the main contribution relies on the observation that the hidden states of consecutive steps are similar, the analysis of this phenomenon lacks details. Several key aspects need clearer explanation:\n(a) Can the similarity of the hidden states be quantitatively measured? For instance, does a low MSE between hidden states indicate that the two states are “similar”?\n(b) Is this phenomenon specific to the U-Net architecture, or is it agnostic to the backbone of the denoising model (e.g., Diffusion Transformer)?\n(c) Is it only applicable to DDIM sampling? Does the phenomenon also hold for other fast ODE solvers, such as the DPM solver [1]?\n\n2. While the experiments were conducted using 50 DDIM steps, a comparison with the original model using a number of DDIM steps that achieve a similar speedup would strengthen the argument for AsyncDiff. For instance, in a setup where AsyncDiff achieves a 2.7x speedup, comparing the FID score with the original model using 50 / 2.7 = 19 DDIM steps would clearly demonstrate the necessity of parallelizing the diffusion model.\n\n[1] DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Steps, Lu et al., NeurIPS 2022", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## **Q1: Can the similarity of the hidden states be quantitatively measured? For instance, does a low MSE between hidden states indicate that the two states are “similar”?**\nThanks for the valuable comment. We provide quantitative analysis and visualization of the hidden state similarities in **Figure 2** and **Figure 3** of the **attached PDF FILE**.\n\n**Figure 2** presents the cosine similarity and MSE of each block's hidden state between adjacent steps, providing a quantitative measure of similarity. The results show that hidden states are highly similar between most steps, except at the initial stage. This high similarity allows for using features from the previous step to parallelize calculations. **Figure 3** visually illustrates this phenomenon by displaying the hidden states of adjacent steps during the diffusion process. Both quantitative and qualitative analyses support our key insight. The initial instability in the curve explains why slightly increasing the warm-up steps can make the AsyncDiff output more consistent with the original output, as shown in Table 2 of our submission.\n\n## **Q2: Is this phenomenon specific to the U-Net architecture, or is it agnostic to the backbone of the denoising model (Diffusion Transformer)?**\nThanks for the comment. As a universal acceleration method, AsyncDiff is agnostic to the backbone of the denoising model and is compatible with DiT-based diffusion models. As shown in Table 1, our method significantly speeds up the DiT-Based Stable-Diffusion-3-medium model while maintaining high-quality outputs. The qualitative results in **Figure 1 (b)** of the **attached PDF file** demonstrate that AsyncDiff preserves the excellent image quality and accurate text generation of SD 3.\n\n### Table 1: Quantitative evaluations of AsyncDiff on DiT-Based SD 3\n*Note: We compare the generative quality when using the DDIM sampler and AsyncDiff to achieve the same speedup. The SD 3 using a default 28-step DDIM sampler is the baseline.*\n| **Method**           | **Speed up ↑** | **Latency ↓** | **CLIP Score ↑** | **FID ↓** | \n|----------------------|----------------|------------|------------------|-----------|\n| SD 3 Original        | 1.0x     | 10.99s        | 32.26           | 33.99     | \n| SD 3 DDIM=15 steps   | **1.8x**     |  6.10s       |  31.90           | 34.55     | \n| **AsyncDiff (N=2 S=1)**| **1.8x**   |  **6.05s**   |  **32.14**       | **32.28** | \n\n## **Q3: Is it only applicable to DDIM sampling? Does the phenomenon also hold for other fast ODE solvers, such as the DPM solver?**\nThanks for the valuable feedback. AsyncDiff is a universal method that can be used with various samplers, including the DPM-Solver. Table 2 below provides the quantitative evaluation of AsyncDiff on the SD 2.1 with DPM-Solver sampler. With the same speedup ratio, AsyncDiff significantly improves generation quality compared to the baseline. We also provide the qualitative results in **Figure 1 (a)** of our **attached PDF File**. Our method achieves significant acceleration while maintaining higher consistency with the original output. \n\n### Table 2: Quantitative evaluations of AsyncDiff using DPM-Solver sampler\n*Note: We compared the generative quality when using the DPM-Solver sampler and AsyncDiff to achieve the same speedup. The SD 2.1 using a 25-step DPM-Solve sampler is the baseline.* \n\n| **Method**           | **Speed up ↑** | **MACs ↓** | **CLIP Score ↑** | **FID ↓** | \n|----------|---------|------------|----------|-----------|\n| SD 2.1 DPM-Solver 25steps  | 1.0x     | 76T        | 31.57            | 28.37     | \n| SD 2.1 DPM-Solver 15steps  | 1.6x     |  46T       |  31.52           | 28.89     | \n| **Our AsyncDiff (N=2 S=1)**| 1.6x     |  **38T**   |  **31.58**       | **27.71** | \n| SD 2.1 DPM-Solver 10steps  | 2.2x     | 30T        |  31.29           | 29.28     | \n| **Our AsyncDiff (N=3 S=1)**| 2.2x     | **25T**    | **31.36**        | **28.20** | \n\n## **Q4: A comparison with the original model using a number of DDIM steps that achieve a similar speedup would strengthen the argument for AsyncDiff.**\nThanks for the valuable suggestions. In Table 3, we present a more comprehensive quantitative evaluation of AsyncDiff. Our method achieves significantly better generation quality at similar speeds, and the advantage increases with higher speedup.\n\n### Table 3: Additional Quantitative Evaluations of AsyncDiff\n*Note: We compared the generative quality when using the DDIM sampler and AsyncDiff to achieve the same speedup. The default 50-step SD 2.1 is the baseline.* \n| **Method**           | **Speed up ↑** | **MACs ↓** | **CLIP Score ↑** | **FID ↓** | \n|-----------|---------|----------|-----------|-----------|\n| SD 2.1 Original      | 1.0x           | 76T        | 31.60            | 27.89     | \n| SD 2.1 DDIM 27steps  | 1.8x           |  41T       |  31.53           | 28.43     | \n| **Our AsyncDiff (N=2 S=1)**| 1.8x     |  **38T**   |  **31.59**       | **27.79** | \n| SD 2.1 DDIM 21steps  | 2.3x           |  32T       |  31.46           | 29.09     | \n| **Our AsyncDiff (N=3 S=1)**| 2.3x     | **25T**    | **31.56**        | **28.00** | \n| SD 2.1 DDIM 15steps  | 3.0x           |  23T       |  31.26           | 30.12     | \n| **Our AsyncDiff (N=2 S=2)**| 3.0x     |  **19T**   |  **31.43**       | **28.55** | \n| SD 2.1 DDIM 11steps  | 4.0x           |  17T       |  30.99           | 32.25     | \n| **Our AsyncDiff (N=3 S=2)**| 4.0x     | **13T**    | **31.22**        | **29.41** |\n\n## **Q5: Would a similar asynchronous denoising approach be applicable in a conditional generation setup as well?**\nThanks for the valuable comments. AsyncDiff is a universal method that performs well with conditional generation setups. In **Figure 1 (c)** of the **attached PDF file**, we show how AsyncDiff accelerates ControlNet+SDXL. Our method reduces latency from 19.90s to 14.30s using two A5000 GPUs while nearly mirroring the original output.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
Dear Chairs, 

We sincerely appreciate the time and effort you have spent evaluating our submission, and we look forward to the discussion stage. We will include the review stage results in the appendix of our next version.

Meta-review: The paper introduces AsyncDiff, a novel framework that accelerates diffusion models by enabling asynchronous parallelism across multiple devices, effectively utilizing the high similarity of hidden states between consecutive steps. The method achieves impressive speedups (1.8x to 4x) on various diffusion models, including image and video generation, with minimal impact on performance metrics like FID and CLIP score. Although some reviewers expressed a desire for more detailed analysis of communication costs and the underlying similarity of hidden states, the proposed method's versatility and significant practical benefits make it a strong contribution to the field. Given its potential impact, I recommend accepting the paper.
Decision: Accept (poster)