Communication-Efficient Diffusion Denoising Parallelization via Reuse-then-Predict Mechanism
Authors: Kunyun Wang
Bohan Li
Kai Yu
Minyi Guo
Jieru Zhao
Venue: NeurIPS 2025 poster ()
Keywords: Diffusion Models
Parallelism
Efficient Communication
Primary area: infrastructure
TL;DR: We propose ParaStep, a step-wise parallelization method based on a reuse-then-predict mechanism that parallelizes diffusion inference by exploiting adjacent-step similarity, reducing latency with minimal quality loss.

Abstract: Diffusion models have emerged as a powerful class of generative models across various modalities, including image, video, and audio synthesis. However, their deployment is often limited by significant inference latency, primarily due to the inherently sequential nature of the denoising process. While existing parallelization strategies attempt to accelerate inference by distributing computation across multiple devices, they typically incur high communication overhead, hindering deployment on commercial hardware. To address this challenge, we propose $\textbf{ParaStep}$, a novel parallelization method based on a reuse-then-predict mechanism that parallelizes diffusion inference by exploiting similarity between adjacent denoising steps. Unlike prior approaches that rely on layer-wise or stage-wise communication, ParaStep employs lightweight, step-wise communication, substantially reducing overhead. ParaStep achieves end-to-end speedups of up to $\textbf{3.88}$$\times$ on SVD, $\textbf{2.43}$$\times$ on CogVideoX-2b, and $\textbf{6.56}$$\times$ on AudioLDM2-large, while maintaining generation quality. These results highlight ParaStep as a scalable and communication-efficient solution for accelerating diffusion inference, particularly in bandwidth-constrained environments.

Communication-Effiienn Diffusion Denoising Parallelization via Reuse-then-Predict Mechanism
Kunyun Wang" School of Computer Science, Shanghai Jiao Tong University wkykaixingsj tu. edu. cn
Bohan Li"
School of Computer Science, Shanghai Jiao Tong University averlastingnight je ju edu cn
Kai Yu School of Computer Science, Shanghai Jiao Tong University kai, yuos tu. edu. cn
Minyi Guo
School of Computer Science, Shanghai Jiao Tong University guo auo-my@cs.. sjtu. edu. cn
Jieru Zhao
School of Computer Science, Shanghai Jiao Tong University zhao-j jooojjieruaajtu edu.
Abstract
Diffusion models have emerged as a powerful class of generative models across various modalities, including image, video, and audio synthesis. However, their deployment is often limited by significant inference latency, primarily due to the inherently sequential nature of the denoising process. While existing paralleliza- tion strategies attempt to accelerate inference by distributing computation across multiple devices, they typically incur high communication overhead, hindering deployment on commercial hardware. To address this challenge, we propose ParaStep. novel parallelization method based on a seeeethhe-ppediie mecha- nism that parallelizes diffusion inference by exploiting similarity between adjacent denoising steps. Unlike prior approaches that rely on layer-wise stage-wis communication. ParaStep employs lightweight, step-wise communication, sub- stantially reducing overhead. ParaStep achieves end-to-end speedups up to 3.88x SVD, 2.43x on Cog VideoX-o222 and 6.56x on AudioL while maintaining generation quality. These results highlight ParaStep as scalable and communication-rffiiinn solution for accelerating diffusion inference, particularly in handwidth-constranl environments.
Introduction
Benefiting from advances in deep learning and increasingly powerful hardware, diffusion models have demonstrated remarkable performance in image generation 30
16] 19) video generation 11 g 24 25 However, their widespread adoption is
31,
33
38 15 26
2 [40,, and audio generation [35]
severely limited by substantial inference latency. This issue becomes particularly critical in long-form video generation, where producing a few minutes of video may require hours of GPU computation.
'Equal contribution. Corresponding author.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

0.00
MAE.. of Xc
0.10
MAENI MAE
MA of
0.10
MAA. of g:
0.00
1.
MrE of
MAE
MAE
6.05
0.05
0.00
60%
20%
100%
0.00
60%
80%
100%
0.0
0.00
Denoising Processing
Denolsing Processing
Denoising Processing
(a) CogVideoX
(b) HunyuanViden
(c) Latte
Figure I: Relative MAE (as defined in Equation (7) between adjacent denoising steps and / for the noisy sample x and the predicted noise E,. Here, 0% on the x-axis indicates the first step of the denoising process, and 100% indicates the last.
0.150
0.20
Direct reuse X
Reuse-then-predict
MAE of
Xg
0.15
MAE of Ez
0.100
Direct reuse & Reuse-then-predict
1.2
MAE of
MAE of
0.10
0.03
0.075
0.02
0.050
0.01
0.025
0.00
0.0
0.00
0.000
0%
20%
40%
60%
80%
100%
0%
20%
40%
60%
80%
100%
Denoising Processing
Denoising Processing
(a) Latte
(b) HunyuanVides
Figure 2: Dotted lines show the difference in noisy sample between the original model and direct reuse process (stride 2), while solid lines compare the original model with our reuse-then thee-preeicc process (degree = 2). Results for predicted noise follow the same pattern. Reuse-then then-predict process results in smaller deviations from the original model compared to direct reuse process.
The latency primarily stems from two factors: the high computational cost of the noise predic- tor-typically implemented using architectures such as DiT [29] U-Net [32]-and the inherently sequential nature of the denoising process, which invokes the noise predictor repeatedly across dozens or even hundreds of timesteps. Notably, DiT models with 3D attention, this cost scales
quadratically with both spatial resolution and temporal length.
To reduce the inference latency of diffusion models, researchers have proposed variety of solutions. Distillation techniques [27 can reduce the computational and memory overhead of diffusion models. Other works [I8 36 employ post-training quantization (PTQ) compress full-precision models into 8-bit or 4-bit representations without retraining, thereby reducing computation. Caching- based methods have also been explored eliminate redundant computation during the denoising process [37 39 42). While effective, these approaches do not leverage the potential of distributed computing, which has become cornerstone of modern deep learning systems.
To address this, several works have proposed parallelization strategies that leverage distributed infrastructure to accelerate diffusion models [17 3 6 5). These methods perform operation wise,, layer-wise, or stage-wise communication to exchange essential tensors, which incurs substantial communication cost. Despite their effectiveness, the huge communication overhead makes them impractical outside of high-bandwidth data center environments. Therefore, there remains a pressing need for parallel approaches with minimal communication requirements to enable practical deployment on commercial hardware.
We analyze the differences in noisy samples and predicted noise across adjacent denoising steps. As illustrated in Figure except for the initial and final few steps, the predicted noises across neighboring steps exhibit high similarity. This observation suggests that it is possible to reuse the noise generated in the previous step to skip the computation of the current step, as shown in Figure ()). However, such direct reuse leads noticeable degradation in generation quality.
To address this issue, we propose a reuse- reuse-then-pee then--reeicc mechanism. Specifically, the noise from the previous step is reused to generate the noisy sample for the current step, and then a fresh noise prediction is performed based on this sample. Owing to the robustness of noise predictors, the newly predicted noise better approximates the original noise than direct reuse, as evidenced in Figure 2 This procedure can be parallelized through careful scheduling, illustrated Figure 3(c), enabling acceleration without significant loss in generation quality. Communication is required only once per

---

denoising step, and only the samples and noises need to be transmitted. This low communication overhead makes ParaStep highly suitable for bow-bandwidth environments, such as systems connected via PCIe Gen3. A detailed analysis of the communication pattern is provided in Section 4.2
Since the difference between adjacent timesteps is large in the early stages, and errors introduced by reuse can accumulate over multiple steps, it is necessary to perform the original denoising process during this phase before switching to the reuse-then then-predict mechanism. We refer to these initial steps as warm-up steps. The number of warm-up steps is critical hyperparameter: setting it too low may cause significant performance degradation, while setting it too high can limit the achievable speedup. We will analyze the impact of warm-up up step selection in Section 5.5 Although the difference between adjacent timesteps is also large in the final few steps, the errors introduced by reuse only accumulate over small number of steps and do not influence generation performance. Therefore, we continue to apply the reuse-then then-predict mechanism during this phase to maintain acceleration.
For con--comuue-intensiie models, such as audio diffusion models, distributed computing may be unnecessary. In such cases, enabling ParaStep on single device can achieve acceleration with lower computational cost. Based on this insight, we develop single-device variant of ParaStep, called BatchStep, which performs adjacent noise predictions within a single batch rather than distributing them across multiple devices. This design reduces computational overhead while preserving the acceleration benefits of our approach for lightweight models. However, for compute-ineen -nntenive models such as image and video diffusion, BatchStep is not suitable due to their high resource demands. Our code is available at htttps://gituub.oo////// saaijjtu-zho-laee wcw.o-aaaPaaraSeee
We summarize our key contributions as follows:
We propose reuse-then-prediee mechanism that mitigates the quality degradation caused by direct reuse. Building on this mechanism, we introduce a novel distributed sampling method, termed ParaStep, which enables step-wise parallelization across devices for faster inference.
We analyze the communication characteristics of ParaStep and demonstrate that it achieves significant speedup even under www-handwidth settings.
We extend ParaStep to a single-device variant tailored for non-compute intuuttvvetttevve diffusion models, enabling efficient adjacent-step prediction without cross-device communication.
We perform comprehensive experiments in various diffusion-basee models, demonstrating that ParaStep is an effective and generalized method applicable to both vision and audio modalities.
2 Related works
Cache Several works utilize caching mechanisms to reduce the computational cost of the noise predictor [37 M. DeepCache (37) leverages the structural properties of the U-Net architecture, specifically its skip connections between shallow and deep blocks. During less critical timesteps, DeepCache skips the computation of deep blocks by directly forwarding the output of shallow blocks to their corresponding counterparts in deeper layers. In different direction, [T] explores caching based on prompt similarity. It constructs a cache that stores intermediate features of previously pro- cessed prompts, enabling reuse when similar prompts encountered. While both approaches reduce the computation of the noise predictor, neither leverages the benefits of distributed computation.
Parallelism Conventional parallel strategies such data parallelism, pipeline parallelism, and tensor parallelism are generally unsuitable for reducing the latency of diffusion models. Data parallelism and pipeline parallelism are primarily designed to improve throughput and provide little benefit for latency reduction. While tensor parallelism is effective for accelerating large language models (LLMs), it is less suitable for diffusion models due to their large activation sizes.
To fully exploit the computational power of distributed GPUs, several methods have been specifically designed to leverage the intrinsic properties of diffusion models for acceleration [17
Given a parallelism degree of p. DistriFusion [17] partitions the input latent of the noise predictor into p patches and employs all-to-all communication to merge them before each attention and convolution operation. PipeFusion
B o 5
(6] reduces communication costs by reusing one-step stale feature maps in patch-level parallelism. AsyncDiff (3] divides the noise predictor into p stages, assigning each stage to a different device and employing pipeline parallelism to achieve speedup. xDiT [5] further combines PipeFusion with Ring Attention [23] to achieve significant acceleration. However, the high
3

---

communication overhead associated with these methods restricts their effectiveness to high-bandwidth environments, which are typically found only in expensive data center infrastructures. Moreover, except for Ring Attention [23]. all of these approaches rely approximate parallelism methods, which introduce noticeable deviations in the generated results compared the original model.
proposes selective mechanism to reuse the generated noise from the previous timestep, thereby skipping the computation required for the relatively unimportant timesteps. achieves acceleration with only minor degradation in generation quality. The key difference between TeaCache and our approach lies in their focus: TeaCache addresses the question of when to reuse, while our method focuses on how to reuse. Specifically, our reuse-then-prediee mechanism yields smaller performance drop by predicting refined noise estimate rather than directly reusing previous outputs. Morcover, demonstrate that TeaCache be seamlessly combined with method to further improve generation quality while maintaining the same level of accelcration.
Reuse TeaCache 22]
Preliminaries
Diffusion models are class of generative models that progressively transform data from the original distribution into a Gaussian distribution through forward diffusion process, and then reconstruct the original data from Gaussian noise via a reverse (backward) denoising process.
Forward diffusion process Given data sample Xo q(x)) drawn from the original distribution, Gaussian noise is gradually added over T steps to produce the final sample xy, which is a pure Gaussian noise. To avoid simulating the full forward trajectory step by step, x; can be sampled via:
g(x: xo) N (xxi v Vapxo (1 a;,)
Backward diffusion process
Given Xt, the sample x drawn from the conditional distribution:
(x x,) N (x t), ofT)
(2)
where ae is either predefined learned and the mean μa (Xx, t) computed as:
1
aq
X
C7 Et
(3)
az
Denoising computation Given condition a timestep i, and noisy sample x,, the noise predictor
EØ- eg-typically implemented as a U-Net [32] or DiT 29 estimates the noise ez. A scheduler then uses this noise to sample x
= (xe, c), Xt-1 = Scheduler(x,,4,
Training In DDPM [10), model parameters are optimized to minimize the squared error between the true noise and the predicted noise:
V9 ||e- e Eq (Vā,xo VI- āre, +]l
(5)
where or is derived from predefined noise schedule (.²).. At each denoising step t, the model aims to recover the noise originally added to Xo to obtain X₁- Since the differences between adjacent noisy samples are often small, the predicted noise e, tends vary only slightly across timesteps.
generalization of diffusion 77) sampling using the ordinary differential equation. Flow Matching, the generative process transforms sample zo from simple reference distribution into sample z from the target distribution by integrating a learned velocity field va (z, t) over time. This process is discretized as:
Flow Matching Flow Matching 21
Zt+ 2i + At (z
(6)

---

Ncten
Schwekrw
Communccttion
fhesse
Predict Nekke
Original denoising process
Direct process
|
Rr-a
Warmup
(c) Reuse-then-predic process
Figure 3: Illustration of the computation process of diffusion model. (a) The original computation process. (b) Reusing noise cy from the previous timestep to skip the computation of noise prediction in the current timestep T 1. (c) ParaStep: adjacent-step noise prediction is distributed across GPUs using reuse-then- preedic,,,,,,, enabling parallel denoising with minimal communication, Since the computational cost of scheduler operations is negligible, the noise predictor computations GPUO, GPUI, and GPU2 are fully parallelized.
Methods
In conventional diffusion models, the reverse process is implemented as sequential denoising procedure. The noise predictor is invoked repeatedly predict the noise ét, given inputs x,, f, and a conditioning signal c. For simplicity, we omit e the remainder of this paper. The scheduler then uses the predicted noise to compute the denoised sample X Since is usually deep neural network with high computational complexity, whereas the scheduler only performs element-wise additions and multiplications, the dominant source of computational cost and latency in diffusion models lies in the repeated invocation of the noise predictor.
To transform the Gaussian noise Xy into a clean sample the noise predictor must be called sequentially for tens even hundreds of timesteps, resulting significant latency during inference. A naive approach to reduce this cost is to reuse the noise predicted in the previous timestep. However, such direct reuse typically leads to noticeable degradation in generation quality. To mitigate this degradation while preserving efficiency, we propose a truse-then-predic, mechanism, termed ParaStep. Our method first reuses the noise from the previous timestep compute the current noisy sample x,, and then predicts refined noise estimate based on that sample. Through careful design, this procedure allows the prediction of adjacent-step noises to distributed across multiple devices, effectively parallelizing the most expensive part of the computation. Given parallelism degree of p. the overall workload divided such that each device only performs of the noise prediction steps. Before introducing the detailed implementation of ParaStep. we first analyze the similarity of predicted noise across adjacent timesteps, and then describe how previously generated noise be efficiently reused the denoising process.
4.1 Reuse-then-predict mechanism
Reusing noise In each denoising step, the noise predictor aims estimate the total noise that was added to Xo to obtain x,. The predicted noise e; remains highly similar across adjacent timesteps, as shown in Figure We use the Relative Mean Absolute Error (Relative MAE) to quantify the similarity between noisy samples and predicted noise across steps. The metric is defined as:
1 D 2r, Fe+1áá
MAE (x,, x(+1)
12
As illustrated in Figure [ for models such as Cog VideoX MAE between adjacent noise predictions drops below after the first 20% of the total steps. This observation suggests that previously predicted noise be reused to bypass less critical computations.
38 and Hunyuan Video 15] the relative
5

---

T-1
T-1
->Xpp-1
6T-1
8
S
62
6T
2T-2
Er-21
T-1
Warmup
Figure 4: Utilizing batching effect of audio diffusion models, parallel computation of noise predictor can be transformed batching execution on single device.
In Figure aa), timestep T, the noise predictor generates ET using Xy, after which the scheduler computes XT-1 based on cr, Xy, and T. At timestep T - 1, the noise predictor estimates cy-1- which is then used to obtain xy-2- contrast, Figure ())) illustrates a reuse strategy: the predictor first estimates ér from Xy, which is used to obtain Xy-1; then, instead of invoking the predictor again, the model reuses ET at timestep
to estimate the next noisy sample Ny-2- We refer to
this strategy as the direct reuse process.
Suppose the predicted noise is reused for subsequent timesteps. In that case, the number of predictor invocations is reduced by a factor of resulting in a corresponding 1 reduction in latency. However, this aggressive reuse introduces a distributional mismatch from the original denoising trajectory, which can lead to significant performance degradation. To mitigate this, we propose a reuse-then trrse-then-predict mechanism that combines reuse with predictive refinement, effectively reducing the quality drop while retaining computational efficiency.
Parallel denoising via reuse-then-preet preddict mechanism As shown Figure instead of directly reusing the noise predicted in the previous step, more accurate result can be achieved by first using the reused noise to generate the noisy sample for the current step, and then feeding this sample into the noise predictor to predict the refined noise for the next timestep, we call this mechanism ceuse-then-predice Based on this mechanism, we propose a novel distributed sampling method, ParaStep, which parallelizes inference by distributing adjacent-step noise prediction across devices.
As illustrated in FigureBjc), both the noise predictor and the scheduler are replicated on GPUO, GPU1, and GPU2. During the initial warm-up steps, all GPUs follow the standard sequential denoising process. After warm-up. GPUO predicts cT-1 from Xy-1 and timestep T 1, and the scheduler then computes XT-2 accordingly. In parallel, GPU1 skips the computation of c--1 and instead reuses eT from the previous step to compute an approximate noisy sample ŠT It then uses this sample and timestep T - 2 to predict refined noise estimate é- Simultaneously. GPU2 performs the same pren--ricctt process, reusing CT twice to compute X and then predicting Since the scheduler introduces negligible computational overhead,
Er-3 from - and T the noise prediction steps on GPU1 and GPU2 are effectively parallelized with those on GPUO. Once prediction is complete, GPUI and GPU2 transmit their predicted noise values, éT and ēT respectively, to GPUO. GPUO then uses these values to compute the next set of noisy samples, R--a and - Finally, the newly generated sample XT-4 is broadcast back to GPUT and GPU2, and all GPUs advance to timestep T 4. This cycle is repeated for subsequent steps, enabling efficient parallclization of adjacent-step noise prediction across multiple devices. We formalize ParaStep with round-based algorithm, presented in Appendix A
Assuming a parallelism degree of p. ParaStep operates in cycles of length p. Each GPU performs one forward pass of the noise predictor per cycle. Since scheduler operations are negligible in cost, noise prediction across all devices effectively parallelized. Consequently, the total latency of the denoising process is reduced by a factor of y. achieving a theoretical speedup of p.
4.2 Communication analysis
To evaluate the communication overhead in multi-GPU environments, we compare the total com- munication volume of three representative methods: AsyneDiff [3] with stage-wise communication. Ring Attention 2 used in xDiT [5] which performs layer-wise communication, and our proposed ParaStep which operates at the step level. For clarity, we assume that the intermediate feature tensors remain of constant size across layers of the noise predictor, which is typically implemented using DiT U-Net. We also assume the model has L attention layers and the parallelism degree is p.

---

Ring Attention In each layer, every GPU must gather key and value tensors from all other GPUs. The size of the keys/values from other devices is 2=1 M, and the total communication per GPU per layer is therefore 2 M. Across all layers, the total communication CRing per step is:
CRing 2L(p 1)M.
AsyncDiff Each GPU handles pipeline stage, performing broadcast of (p 1)M data per step. Since this occurs for all stages, the total communication CAsyneDiff per step is:
CAsyneDiff p(p 1)M.
ParaStep Each cycle (of length p) involves p send-receive operations and one broadcast. The total communication volume per cycle is (p 1)M (p 1)M 2(p 1)M. Since eycle covers denoising steps, the average communication CraraStep per step is:
2(p 1)M p
CParaStcp
These results suggest that ParaStep significantly reduces communication overhead compared to Ring Attention and AsyncDiff, particularly under limited-bandwidit constraints. A detailed derivation of the communication analysis is provided in Appendix
4.3 Leveraging the batching effect for non-compute-intensive models
Image and video diffusion models are typically computationally intensive, where increasing the batch size leads to a linear increase in latency. In contrast, audio diffusion models exhibit a more favorable batching effect. As shown in Figure increasing the batch size results in only marginal increase in latency. By leveraging this property, we can transform the parallel execution of the noise predictor into batched inference process single device, illustrated in Figure 4 We refer to this single-device variant as BatchStep. The degree of parallelism in ParaStep corresponds to the cycle length in BatchStep. This approach achieves a theoretical reduction in per-step computation of approximately s-1 while completely eliminating the need for multi-device parallelism.
4.4 Dynamic degree of parallelism via TeaCache
TeaCache
22
introduces selective mechanism
for reusing previously generated noise, using a larger stride of reuse in less critical timesteps improve efficiency. We extend ParaStep by inte- grating the computation schedule of TeaCache. Specifically, the degree of parallelism at each timestep is determined based the TeaCache schedule. Since ParaStep has low communication overhead, it can achieve similar speedups Tea- Cache. Moreover, due to the peuee-then-predict mechanism, our method offer superior gener- ation quality.
AudioLDM2
300
CogVideaX
Latency (s)
25c
Latency
200
150
100
Batch Size
Batch Size
(a) AudioLDM2
(b) CogVideoX
Figure Batching effect of AudioLDMM--arre and Cog
Experiments
5.1
Settings
Base models and compared methods Diffusion model is special case of Flow Matching 21 where both perform a step-by-step computation process to reconstruct the target data distribution. To demonstrate the versatility of our method, we evaluate it on both diffusion models and Flow Matching model. For the Flow Matching model, we adopt Stable Diffusion (SD3) as representative text-to-image model. For diffusion models, we evaluate Cog Videox-2222 2b (38 and Latte (26). which are text-to-video models, as well as SVD (2), an image-to-video model. We also include AudioLDM2-large [25] for audio generation. For comparison, include three state-of-the-art
7

---

Efficiency
Visual Quality
Method
Speedup
Latency (s) 4
FID
LPIPS
PSNR
SSIM
SD3 (T = 50)
18.75
AsyncDiff (p 2)
1.61
11.62
7.11
0.2141
16.47
0.7290
XDiT-Pipe (p = 2)
1.50
12.50
6,20
0.1943
16.68
0.7498
ParaStep (p = 2)
1.68
11.16
5.01
0.1362
18.61
0.8157
Table I:
Speedup and generation quality on image model SD3, with resolution of 1440 1440. T is the number of inference steps, and is the degrée of parallelism.
1.70
Method
Latency (8) 4
FAD 4
Speedup
FAD 6.56x
AudioLDM2 (I' 200)
34.80
1.6653
Speedup
1.69
ParaStep (p = 2)
18.86
1.6651
ParaStep (p =
9.86
1.6716
3.53x
1.68
BatchStep (s
17.74
1,6671
2.67x
BatchStep (s
9.45
1.6699
1,85x
1.67
1.00x
Table 3: Generation latency and FAD on AudioLDM2-large p is the degree of paral- lelism, is the cycle length in BatchStep.
1.66
4
8
Figure
FAD under varying degrees of paral- lelism using ParaStep. Model: AMdioLDM2-laage
parallel methods: Aiff (3. PipeFusion 66. and Ring Attention 23 steps for ParaStep is set for AudioLDM2-large 5 for SD3. for SVD, 13 for Cog VideoX-o-22 and 18 for Latte. The number of inference steps is set to 200 for AudioLDM2-large and 50 for all other models.
The number of warm-up
Dataset and evaluation metrics We the MS-COCO 2017 (0) validation for text-to-image
model, VBench [12] for text-to-video and image-to-video models, and AudioCaps (14) for audio model, LudiooDM2-large.. For text-to-image model, we evaluate performance using Peak Signal-to- Noise Ratio (PSNR), Learned Perceptual Image Patch Similarity (LPIPS) [1], Structural Similarity Index Measure (SSIM), and Fréchet Inception Distance (FID) For text-to-video and image-to- video models, we use PSNR, LPIPS, SSIM, and VBench (12v, i2v) score. For audio model, we utilize Fréchet Audio Distance (FAD)
13] which measures the quality of generated audio.
Hardware All experiments are conducted on machine equipped with NVIDIA 4090 GPUs (24GB each)
28 connected via PCIe Gen3.
5.2 Quantitative comparison with baselines
xDiT implements PipeFusion 6
(xDiT-Ring) to accelerate the denoising process. We compare ParaStep with xDiT-Pipe the image model SD3. Since xDiT does not currently support PipeFusion for video generation models, we instead compare ParaStep with xDiT-Ring for video generation. AsyncDiff is another recent parallel method, and we evaluate ParaStep against AsyncDiff on SVD and SD3. Both xDiT-Pipe, AsyncDiff, and ParaStep rely on approximate methods for parallelization, which introduce varying degrees of deviation from the original model. Although xDiT-Ring is theoretically lossless, its computational logic differs from that of the original model, potentially leading to minor discrepancies generated outputs. As shown in Table [ and Table 2] our method achieves state-of-the-ar performance compared to baseline methods on both image and video generation models. For SD3, ParaStep demonstrates the highest speedup while maintaining superior generation quality compared to AsyneDiff and xDiT-Pipe. With parallelism degree of ParaStep achieves 1.68x speedup over the original denoising process, with FID of only 5.01. For video generation models, ParaStep also achieves the highest speedup among all baseline methods while maintaining the best generation quality across different degrees of parallelism on SVD and Cog VideoX-2b These results confirm that our method is both highly efficient and introduces minimal impact on generation quality. is worth noting that, due to the high communication overhead of xDiT-Ring it provides little to speedup in our experimental setup.
(xDiT-Pipe) and Ring Attention 23

---

Efficiency
Visual Quality
Method
Speedup
Latency (8) 4
VBench f
LPIPS
PSNR f SSIM
SVD (14 frames, 1024x576)
SVD 50)
1
$1.04
42.00
AsyncDiff (y 2)
1.37
37.36
41.75
0.0790
25.99
0.8366
AsyneDiff (p 4)
1,80
28.43
41.56
0.1285
23.13
0.7599
ParaStep (p 2)
1.69
30.27
41.90
0.0278
33.35
0.9347
ParaStep
2.49
20.49
41.74
0.0732
26.99
0.8433
Latte (16 frames, 512x512)
Latte (7' = 50)
1
32.56
73.68
xDiT-Ring (p 2)
1.01
32.18
73.87
0.0424
32.79
0.9273
xDiT-Ring (p 4)
1.07
30.50
73.81
0.0431
32.80
0.9266
ParaStep (p 2)
1,43
22.80
73.76
0.0432
32.51
0.9258
ParaStep (p- 4)
1.82
17.91
73.76
0.0533
31,10
0.9115
Cog (45 frames, 512x720)
CogVideoX (T 50)
91.89
76.95
xDiT-Ring (p = 2)
0.86
106.68
76.79
0.0570
32.24
0.9284
XDiT-Ring (p = 4)
0.98
93.44
76.66
0.0817
29.53
0.9028
ParaStep 2)
1.47
62.50
76.97
0.0213
37.57
0.9642
ParaStep (p- 4)
1.93
47.66
76.74
0.0359
34.34
0,9505
Table 2: Comparison of speedup and generation quality across four video generation models using different parallel methods.
Method
Latency (s) |
PSNR 1
2.0
Speedup
SSIM
1.0
TeaCache-slow
52.25
32,25
Speed
1.5
LA3 17 22 100
TeaCache-fast 36.19
24.42
1.0
5
ParaStep-slow 52.61
35.76
0.5
ParaStep-fast 37.27
25.59
0.0
0.6
256
10%
20%
30%
40%
50% 60% 70%
80% 90%100%
Ratio Wammupp Steps (%)
Table 4: Extend ParaStep by integrating the computation schedule of TeaCache achieve superior generation quality.
Figure 7: Impact the number of warm-up steps on the speedup and generation quality of ParaStep. Degree of parallelism: Model: CogVideo
5.3 Effect of BatchStep
We compare the generation latency and FAD of ParaStep and BatchStep on DM2 with the number of warm-up steps achieve acceleration with only minor degradation in generation quality. Notably, since BatchStep transforms the parallel execution of the noise predictor into batched inference process on single device, it eliminates the need for inter-device communication. As result, BatchStep achieves greater acceleration than ParaStep.
1. As shown in Table both ParaStep and BatchStep
5.4 Performance breakdown
To demonstrate that the observed speedup achieved by ParaStep primarily stems from improved communication efficiency, we con- ducted performance breakdown the SVD model. Specifically, we measured both the total latency and the communication-res rellaee latency for AsyncDiff and our proposed ParaStep. As shown in the T ParaStep exhibits significantly lower communication la- tency compared to AifneDiff, which directly contributes to its higher overall speedup.
Method
Total (s)
Comp. (s)
Comm. (s)
SVD
51.04
$1.04
AsyncDiff (p 2)
39.30
30.71
8.59
ParaStep (p 2)
30.52
30.42
AsyncDiff (p 4)
31.30
21.01
10.29
ParaStep (p 4)
20.08
19.94
0.14
Table 5: Comparison of total latency and breakdown into computation and communication times across different parallel settings.

---

5.5 Ablation studies
Scalability our method We evaluate the scalability of ParaStep on AMdioLDM2 with warm-up up step count of 1. As shown in Figure 6 ParaStep achieves 6.56x speedup at parallelism degree of 8, with only a 0.02 increase in FAD, demonstrating that our method can effectively scale to high degrees of parallelism on Audio Additional scalability evaluations on other models are provided in Appendix
Impact of warm-up steps The larger the number of warm-up steps, the fewer the number of parallelizable steps. Due to Amdahl's ' law, the overall speedup of ParaStep limited by: m++-m
where m denotes the ratio of warm-up steps and p represents degree of parallelism.
However, using too few warm-up steps leads to a degradation in generation quality, as the initial denoising steps are more sensitive to prediction errors. As shown in Figure 7 setting the warm- up ratio to 30% offers a favorable trade-off, achieving strong generation performance with only moderate reduction in speedup. Increasing the ratio from 30% 100% yields only marginal improvements in generation quality, but causes substantial drop in acceleration.
Extending ParaStep with TeaCache By integrating the computation schedule of TeaCache into ParaStep, our method achieves superior generation quality compared to TeaCache, with only small overhead from communication, as shown Table Notably, TeaCache-fass adopts a more aggressive reuse schedule than TeaCache-slow Similarly, ParaStep-slow refers to the integration of the TeaCache-slow schedule into ParaStep, while ParaStep-fast extends ParaStep with the more aggressive TeaCache- fast schedule.
6 Conclusion
In this work, we propose ParaStep, a novel parallelization approach for accelerating diffusion model inference. By leveraging reuse-then-predice mechanism, ParaStep significantly reduces communication overhead while preserving high generation quality. Our approach grounded in bypass
the observation that adjacent denoising steps often exhibit strong similarity, allowing costly computations without substantial performance degradation. We further extend this approach with single-device variant, BatchStep, which transforms parallel execution into efficient batched inference for non-compute-intensiie models, such as audio diffusion models. Extensive experimental evaluations confirm that ParaStep delivers superior speedup across diverse modalities, including image, video, and audio generation, while maintaining competitive quality metrics.
7 Acknowledgments
This work is sponsored by the National Natural Science Foundation of China (62472273, 62232015).
10

---

References
[1] Shubham Agarwal, Subrata Mitra, Sarthak Chakraborty, Srikrishna Karanam, Koyel Mukherjee, and Shiv Kumar Saini. Approximate caching for efficiently serving (Text-to-Image) diffusion models. 21st USENIX Symposium on Networked Systems Design and Implementation (NSDI 24), pages 1173-1189, 2024.
[2] Andreas Blattmann, Tim Dockhorn, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Dominik Lorenz, Yam Levi, Zion English, Vikram Voleti, Adam Letts, et al. Stable video diffusion: Scaling latent video diffusion models to large datasets. arXiv preprint arXiv:2311.1517, 2023.
[3] Zigeng Chen, Xinyin Ma, Gongfan Fang, Zhenxiong Tan, and Xinchao Wang. Asynedift: Parallelizing diffusion models by asynchronous denoising. arXiv preprint arXiv:2406. 06911, 2024.
Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, Scaling rectified flow transformers high-resolution image synthesis. In Forty-first international conference on machine learning, 2024.
[5] Jiarui Fang. Jinzhe Pan, Xibo Sun, Aoyu Li, and Jiannan Wang. xdit: an inference engine for diffusion transformers (dits) with massive parallelism. arXiv preprint arXiv:2411.01336 2024.
[6] Jiarui Fang, Jinzhe Pan, Jiannan Wang, Aoyu Li, and Xibo Sun. Pipefusion: Patch-level pipeline parallelism for diffusion transformers inference. arXiv preprint arXiv:2405. 14430, 2024.
[7] Ruiqi Gao, Emiel Hoogeboom, Jonathan Heek, Valentin De Bortoli, Kevin Patrick Murphy, and Tim Salimans. Diffusion models and gaussian flow matching: Two sides of the same coin. In The Fourth Blogpost Track at ICLR 2025, 2025.
[8] Zhifang Guo, Jianguo Mao. Rui Tao, Long Yan, Kazushige Ouchi, Hong Liu, and Xiangdong Wang. Audio generation with multiple conditional diffuusion model. In Proceedings of AAAI Conference on Artificial Intelligence, volume 38, pages 18153-18161, 2024.
[9] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by two time-scale update rule converge to a local nash equilihrium. Advances in neural information processing systems, 30, 2017.
[10] Jomathan Ho, Ajay Jain, and Pieter Abbeel, Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840 6851, 2020.
[11] Rongjie Huang, Jiawei Huang, Dongchao Yang. Yi Ren, Luping Liu, Mingze Li, Zhenhui Ye, Jinglin Liu, Xiang Yin, and Zhou Zhao. Make-an-audio: Text-to-audio generation with prompt-enhanced diffusion models. In International Conference on Machine Learning, pages 13916-13932. PMLR, 2023.
[12] Ziqi Huang, Yinan He, Jiashuo Yu, Fan Zhang, Chenyang Si, Yuming Jiang, Yuanhan Zhang, Tianxing Wu, Qingyang Jin, Nattapol Chanpaisit, et al. Vbench: Comprehensive benchmark suite for video generative models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2024.
[13] Kevin Kilgour, Mauricio Zuluaga, Dominik Roblek, and Matthew Sharifi. Fréchet audio distance: A reference-free metric for evaluating music enhancement algorithms. In 20th Annual Conference of the International Speech Communication Association, Interspeech 2019, Graz, Austria, September 15-19, 2019, pages 2350-2354. ISCA, 2019.
[14] Chris Dongjoo Kim, Byeongchang Kim, Hyunmin Lee, and Gunhee Kim. AudioCaps: Generating captions for audios in the wild. In Jill Burstein, Christy Doran, and Thamar Solorio, editors, Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Volume I (Long and Short Papers), pages 119-132, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics.
[15] Weijie Kong, Qi Tian, Zijian Zhang, Rox Min, Zuozhuo Dai, Jin Zhou, Jiangfeng Xiong, Xin Wu, Jianwei Zhang, et al. Hunyuanvideo: A systematic framework for large video generative models. arXiv preprint 2rXiv:2412.03600 2024.
[16] Black Forest Labs. Flux.
2024.
[17] Muyang Li, Tianle Cai, Jiaxin Cao, Qinsheng Zhang. Han Cai, Junjie Bai, Yangging Jia, Kai Li, and Song Han. Distrifusion: Distributed parallel inference for resolution diffusion models. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 7183-77 11937199,, 2024.
11

---

[18] Xiuyu Li, Yijiang Liu, Long Lian, Huanrui Yang, Zhen Dong, Daniel Kang, Shanghang Zhang, and Kurt Keutzer. Q-diffusion: Quantizing diffusion models. In Proceedings the IEEE/CVF International Conference on Computer Vision, pages 17535-17545, 2023.
[19] Zhimin Li, Jianwei Zhang. Qin Lin, Jiangfeng Xiong, Yanxin Long, Xinchi Deng, Yingfang Zhang, Xingchao Liu, Minhin Huang, Zedong Xiao, et al. Hunyuan-dit: A powerful multi-resolution diffusion transformer with fine-grained chinese understanding. arXiv preprint arXiv:2405. 2024.
[20] Tsung-Y Yi Lin, Michael Maire, Serge Belongic, James Hays, Pictro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitmick. Microsoft coco: Common objects in context. In Computer vision ECCV 2014: 13th European conference, zurich, Switzerland, September 6-12, 2014, proceedings, part 13, pages 740-755. Springer, 2014.
[21] Yaron Lipman, Ricky TQ Chen, Heli Ben-Hamu, Maximilian Nickel, and Matt Le. Flow matching for generative modeling. arXiv preprint urXiv:2210.02747, 2022.
[22] Feng Liu, Shiwei Zhang, Xiaofeng Wang. Yujie Wei, Haonan Qiu, Yuzhong Zhao, Yingya Zhang, Qixiang Ye, and Fang Wan. Timestep embedding tells: It's time to cache for video diffusion model. arXiv preprint MxXiv2241.191088 2024.
[23] Hao Liu, Matei Zaharia, and Pieter Abbeel. Ring attention with blockwise transformers for near-infinite context. arXiv preprint arXiv:2310.01888 2023.
[24] Haohe Liu, Zehua Chen, Yi Yuan, Xinhao Mei, Xubo Liu, Danilo Mandic, Wenwu Wang, and Mark D Plumbley. Audioldm: text-to-audio generation with latent diffusion models. In Proceedings of the 40th International Conference on Machine Learning, pages 14444444474 2023.
[25] Haohe Liu, Yi Yuan, Xubo Liu, Xinhao Mei, Qiuqiang Kong, Qiao Tian, Yuping Wang. Wenwu Wang. Yuxuan Wang, and Mark Plumbley. Audioldm 2: Learning holistic audio generation with self-supervised pretraining. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 2024.
[26] Xin Ma, Yaohui Wang, Gengyun Jia, Xinyuan Chen, Ziwei Liu, Yuan-Fang Li, Cunjian Chen, and Qiao. Latte: Latent diffusion transformer for video generation. arXiv preprint arXiv:2401.03044 2024.
[27] Chenlin Meng, Robin Rombach, Ruiqi Gao, Diederik Kingma, Stefano Ermon, Jonathan Ho,, and Tim Salimans. On distillation of guided diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14297-1434 2023.
[28] NVIDIA Corporation. NVIDIA GeForce RTX 4090 GPU, 2022.
[29] William Peebles and Saining Xie. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF international conference on computer vision, pages 4195 4205, 2023.
[30] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents. arXiv preprint 2rXiv:2244006125 1(2):3, 2022.
[31] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684-10695, 2022.
[32] Olaf Ronneberger, Philipp Fischer, and Thomas Brox, U-net: Convolutional networks for biomedical image segmentation. In Medical image computing and computer-asiisted intervention-MICCA. 2015: 18th international conference, Munich, Germany, October 5-9, 2015, proceedings, part III 18, pages 234-241. Springer, 2015.
[33] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily I. Denton, Kamyar Ghasemipour, Raphacl Gontijo Lopes, Bureu Karagol Ayan, Tim Salimans, al. Photorcalistic text-to- image diffusion models with deep language understanding. Advances in neural information processing systems, 35:36479-36494, 2022.
[34] Axel Sauer, Dominik Lorenz, Andreas Blattmann, and Robin Rombach. Adversarial diffusion distillation. In European Conference on Computer Vision, pages 103. Springer, 2024.
[35] Flavio Schneider. Archisound: Audio generation with diffusionn arXiv preprint urXiv:2301.13267 2023.
[36] Yuzhang Shang, Zhihang Yuan, Bin Xie, Bingzhe Wu, and Yan Yan. Post best-training quantization on diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 1972-1981, 2023.
12

---

[37] Mengwei Xu, Mengze Zhu, Yunxin Liu. Felix Xiaozhu Lin, and Xuanzhe Liu. Deepcache: Principled cache for mobile deep vision. In Proceedings the 24th annual international conference mobile computing and networking, pages 129-144, 2018.
[38] Zhuoyi Yang, Jiayan Teng. Wendi Zheng, Ming Ding, Shiyu Huang. Jiazheng Xu, Yuanming Yang. Wenyi Hong, Xiaohan Zhang, Guanyu Feng, Cogvideox: Text-to-video diffusion models with an expert transformer. arXiv preprint arXiv:2408. 06072, 2024.
[39] Hancheng Ye, Jiakang Yuan, Renqiu Xia, Xiangchao Yan, Tao Chen, Junchi Yan, Botian Shi, and Zhang. Training-fiee free adaptive diffusion with bounded difference approximation strategy. Advances in Neural Information Processing Systems, 37:306- 773,06-332, 2024.
[40] Shenghai Yuan, Jinfa Huang, Xianyi He, Yunyuan Ge, Yujun Shi, Liuhan Chen, Jiebo Luo, and Li Yuan. dlentity-preserving text-to-video generation by frequency decomposition. arXiv preprint 2rXiv:241.. 1.14440 2024.
[41] Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features perceptual metric. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 586-595, 2018.
[42] Chang Zou, Xuyang Liu, Ting Liu, Siteng Huang, and Linfeng Zhang. Accelerating diffusion transformers with token wise feature caching, arXiv preprint arXiv:2410.05317, 2024.
13

---

Appendix
A Round-based implementation of ParaStep
Algorithm 1 Round-based implementation of ParaStep
Require: Number of inference steps T, degree of parallelism p. rank of GPU rank
Ensure:
Final output X0
1: Initialize noisy sample: XT (0, I)
2: Initialize round counter: round t-
3: for = T,T - ,,... do
4: if E warmup_ sermup_steps then Perform the original computation process during warm-up steps
5: Compute noise: 6 to (x+,)
6:
Cache noise: éeache + Et
7:
Update sample: X,-1 Scheduler(x;, f,
8:
else
D Perform the ParaStep computation process non-warm-up steps
9:
if rank round then
10:
Compute noise: ét (x(,x,t) Cache noise: Ecache f- Et
12:
if rank * O then Send E GPU
14:
end if
else
16:
Reuse cached noise: f- Ceuche rank then Receive from GPU round
18:
19:
end if
20:
end if
21:
Update sample: x,-1 Scheduler(x,, 6g) if round = p then
22:
23:
Broadcast x from GPU to all GPUs
24:
end if
25:
Update round counter: round +- (round + 1) mod p
26: end if
27:
end for
28:
return Xo
We formalize ParaStep with a round-based algorithm, presented in Algorithm 1 Each GPU is assigned unique identifier rank, and variable round designates the current master GPU. Assuming a parallelism degree of ParaStep operates in cycles of length The master GPU is the one whose rank matches the current round, and root GPU designated as rank = 0.
In each round, the master GPU invokes the noise predictor eg to estimate ex, while all other GPUs skip this computation. The predicted noise is then sent the root GPU, which it compute the corresponding noisy sample via the scheduler. In the final round of each cycle (i.e., round = 1). the root GPU obtains the last noisy sample and broadcasts it to all GPUs.
Each GPU performs one forward pass the noise predictor per cycle. Since scheduler operations are negligible in cost, noise prediction across all devices is effectively parallelized. Consequently, the total latency of the denoising process is reduced by factor of p, achieving a theoretical speedup of p.
B Detailed derivation of communication analysis
In this section, we provide more detailed derivation of communication volumes for the three parallelism strategies discussed in Section 4.2 Ring Attention, AsyneDiff, and ParaStep. We assume that the intermediate feature tensors remain constant size M across layers of the noise predictor, and that the model contains L attention layers. The parallelism degree is denoted as p.
14

---

B.1 Ring Attention
In Ring Attention 23 the keys and values on each GPU are split into p partitions, each of size y M.M. Before each attention operation, GPUs must gather the keys and values from all other GPUs in ring-style communication pattern to obtain the complete set of keys and values. For each GPU, this process involves:
Gathering p - partitions, each size 1 M, for both keys and values. The total communication volume per GPU per layer is therefore:
p M (keys)
p P-1M M (values)
2(p 1) M.
p
p
p
Since this communication occurs in every attention layer, the total communication volume per GPU step is:
2(p 1) M L.
CRing, per GPU
With GPUs, the total communication volume across all GPUs per step is:
CRing 2L(p 1)M.
B.2 AsyncDiff
In AsyneDiff B], the model is divided into pipeline stages, each assigned to separate GPU. In each step, every stage (or GPU) must broadcast its outputs all other GPUs, resulting communication volume of:
(p- 1)M (per GPU per step).
Since the model contains stages, the total communication volume per step across all GPUs is:
CAsyaeDift = p(p - 1.
B.3 ParaStep
ParaStep adopts a round-based communication pattern, where each cycle contains p steps. In each cycle:
1. Noise transfer: Each GPU (except for GPU 0) sends its predicted noise GPU resulting in:
(p 1) M (noise transfer per cycle).
2. Sample broadcast: In the final step each cycle, GPU 0 broadcasts the predicted noisy sample x to all other GPUs, leading to:
(p 1)M (sample broadcast per cycle).
The total communication volume per cycle is therefore:
2(p 1.
Since one cycle covers p denoising steps, the average communication volume per step is:
CParaStop 2(p 1)M
p
C
Limitations
The main limitations of method as follows:
Firstly, ParaStep is parallelization method, which requires additional computational resources to achieve speedup, effectively trading compute for efficiency.
15

---

Method
Speedup
Latency (s)
VBench
LPIPS
PSNR
SSIM
Hunyuan Video (T=50)
1
27.35
72.93
xDiT-Ring (p 2)
1.08
25.22
72.89
0.0071
37.04
0.9629
xDiT-Ring (p = 4)
1.07
25.45
72.89
0.0071
37.04
0.9629
ParaStep (p = 2)
1.79
15.28
73.00
0.0175
34.24
0.9482
ParaStep (p = 4)
2.67
10.25
72.77
0.0468
30.11
0.9075
Table 6: Speedup and generation quality on MunyuanVVdeeo
Secondly, ParaStep replicates the entire noise predictor across all GPUs, which means it cannot reduce the per-GPU memory consumption of the noise predictor. A typical diffusion pipeline consists of several components, including a text encoder, noise predictor (e.g.. DiT or U-Net), and VAE, among others. Among these, the text encoder is often the primary memory bottleneck. Unlike the noise predictor, the text encoder not replicated; instead, it partitioned into multiple stages distributed across devices, thereby reducing memory usage on each GPU.
D Societal impacts
We propose ParaStep, novel parallelization approach for accelerating diffusion model inference. ParaStep achieves superior generation quality compared to state-of-the-att parallclization methods, while delivering greater speedup through lightweight, step-wise communication.
ParaStep can be applied in commercial settings to accelerate compute-intenus intensive diffusion models, such vision-based models, thereby making diffusion technologies researchers without access to expensive, data center- scale infrastructure. For mnt-commute-inteesse diffusion models, proposed variant BatchStep enables speedup single device, offering nearly free performance gain with minimal hardware cost.
accessible to users and
E
Supplementary experiments
E.1
Using ParaStep to accelerate Hunyuan Video
We compare the speedup and generation quality of ParaStep and xDiT-Ring Hunyuan Video. Since Hunyuan Video requires more GPU memory than the 24GB capacity of an NVIDIA 4090, we apply quantization to reduce memory consumption. In this experiment, each video contains frames with resolution 180x180 180.
As shown in Table 6 ParaStep achieves 2.67x speedup compared to the original model, with PSNR 30.01, indicating that the generated results are highly similar to those of the original model. Notably, when the degree of parallelism is set to 2, the VBench score of ParaStep is even higher than that the original model.
Although xDiT-Ring is theoretically lossless, its computational logic differs from that of the original model, potentially leading to minor discrepancies in generated outputs. As result, the generation quality of xDiT-Ring is generally better than that of ParaStep. However, the significant communication overhead xDiT-Ring severely limits its speedup, achieving only 1. 1.08x improvement, which is substantially lower than that of ParaStep.
16

---

44
Speedup
1.08x
Speedup
VBerch3.888
Speedup
2.75x
10
Speedup
42
VBench
2-
1:6
5
1.69x
1.00x
1.0xx
40
2
1
8
Degree of Parallelism
Degree of Parallelism
Figure 8: Generation quality under varying de- grees of parallelism using ParaStep. Model: SD3.
Figure 9: Generation quality under varying de- grees of parallelism using ParaStep. Model: SVD.
77.5
Speedup
VBench2.43x
80
Speedup
VBenciR.27x
Speedup
2
1.93x
78
VBench
Speedup
75.0
VBench
1-3xx
76
1.00x
72.5
74
70.0
67.5
2
4
1
8
Degree of Parallelism
Degree of Parallelism
Generation quality under varying degrees of parallelism using ParaStep. Model: Cog VideoX-2b..
Figure 10:
Figure 11:
Generation quality under varying
degrees parallelism using ParaStep. Model: Latte
E.2 Scalability of ParaStep on SD3, SVD, Cog Videooo--,,, and Latte
We evaluate the speedup and generation quality of ParaStep under varying degrees of parallelism on SD3, SVD, V2de and Latte. The number of warm-up steps for ParaStep set to 5 for SD3, for SVD, 13 for Cog VideoX-2b, and 18 for Latte, with the number of inference steps fixed at 50. As shown in Figure 8 and Figure 9 with parallelism degree of 8. ParaStep achieves 4.08x speedup on SD3 with a FID of only 11.08, and 3.88x speedup on SVD with minor VBench score drop less than However, for CogVidooX
and Latte, the speedup is less significant, as shown in Figure 10 and Figure II due to the larger number of warm-up steps required for these models.
E.3 Comparison with single-device efficient inference approach
Single-device efficiency methods such DeepCache reduce the latency of diffusion models without requiring additional computational resources. However, directly caching used in DeepCache may cause a significant degradation generation quality. In contrast, our ParaStep, based on the Manssetten-Preedee mechanism, offers balanced trade-off by leveraging additional computational resources achieve higher speedups with minimal quality loss.
To assess the efficiency of ParaStep, we compare it against both DeepCache and AsyncDiff on the SVD model, which adopts the deterministic EulerDiscreteScheduu DeepCache as and the degree of parallelism in AsyncDiff and ParaStep as As shown in Table 7 ParaStep consistently outperforms both baselines in terms of generation quality and latency.
We denote the cache interval in
E.4 Visualization
We visualize the outputs of AsyneDiff, xDiT-Pipe, and ParaStep on SD3. The number of inference steps is set 50, with resolution of 1440x1440. As shown in Figure 12] ParaStep demonstrates superior generation quality compared to AsyncDiff and xDiT-Pipe.
In the first row of Figure 12 compared to the original diffusion model, AsyneDiff generates extra hand, while xDiT-Pipe fails to generate fork and a water cup. In the second row, the logo in the right corner of the bus generated by AsyncDiff differs from that of the original model, and the bag
17

---

Method
Latency
LPIPS
PSNR
SSIM
SVD 50)
$2.23
DeepCache (s 2)
36.42
0.0399
32.7247
0.9155
DeepCache (s 4)
26.90
0,0847
26.4912
0.8415
DeepCache (s 6)
23.75
0.1397
23.1894
0.7728
DeepCache (s 8)
22.24
0,1973
19.9736
0.7160
AsyncDiff (p 2)
39.28
0.0861
24.0769
0.8328
AsyneDiff (p 4)
29.65
0.1306
21.1599
0.7505
ParaStep (p =2)
30.61
0,0283
32.8624
0.9358
ParaStep (p
20.36
0.0751
24.9949
0.8448
Table 7: Comparison of latency and generation quality across different caching and parallelization strategies.
Original
AsyncDiff PSNR=16.47
xDiT-Pipe PSNR=16.68
ParaStep
PSNR=18.61
Figure 12: Comparison of generation quality on SD3 using competing methods. PSNR is used to measure the similarity to the original model, with higher values indicating better quality.
generated by xDiT-Pipe is red, which inconsistent with the original output. In the third row, both AsyncDiff and xDiT-Pipe hallucinate an additional shower that does not appear in the original result.
In contrast, ParaStep achieves higher consistency with the original diffusion model across exam- ples.
18

---

NeurIPS Paper Checklist
1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The abstract and introduction clearly state the claims made, including the contributions, analysis and important experiments results.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
Limitations are discussed in Appendix C
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
19

---

We analyze the theoretical communication volumes of different methods, and providing detailed derivation in Section 4.2
Justification:
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We provide the experimental settings Section 5.1 docode of our methods in Section A
We provide the pseu-
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
20

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification:
The link of Anonymous repository for ParaStep is provided in Section
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.co cc/
Shbllaccguidees/oddeSlbmm.ssss
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/ //nips.co. ccippiccc guwdes/ddeeeee CooeeSumeeeeeeeeeeeeee ssi onyylicy) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data,
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer: [Yes]
Justification: This paper specify the test details in Section 5.1 This paper does not include training.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
The test datasets in our setting is big enough (946 samples for VBench) to perform evaluation, and we control the random seed for all of the models and methods used in our experiments.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
21

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We provide the information on the computer resources in Section 5.1
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics https: /tt////nuripppp hwwpurpp...ppppli///eeeeeeeeeeee htcc/pubic//thhccccdeeeee......d
Answer: [Yes]
Justification:
The research conducted in the paper conform with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- cration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: We discuss that our methods can be scalable and regh-performanc solution for acceleration diffusion model inference in practical environments in Appendix D
Guidelines:
The answer NA means that there is no societal impact of the work performed.
22

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), faimess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: This paper provides novel parallelism method for Diffusion models, does not pose such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging. and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification:
We cite the works used in this paper.
Guidelines:
The answer NA means that the paper does
existing assets. The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include a URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
23

---

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com/dataseta
latasets.papersstt
has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Question:
Answer: [Yes]
Justification: The link of Anonymous repository for ParaStep is provided in Section | This repository includes a README file, which provides instructions on installation, setup, and usage.
Guidelines:
The answer NA means that the paper does release assets.
Researchers should communicate the details the dataset/code/mooel as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsoureing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Question:
Answer: [NA]
Justification:
This paper does involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
24

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if' applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification:
We only use LLM to assist writing.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy ottp:e///earrrpeeccc///oeeeerrre for what should or should not be described.
25


Reviews:
[{"review_id": "l50JqM9dgx", "reviewer": "Reviewer_p6GU", "summary": "The paper tackles the high inference latency of diffusion models, which arises from the sequential nature of the denoising process. The authors identify that existing parallelization strategies often suffer from high communication overhead, limiting their use to expensive, high-bandwidth data center hardware.\n\nTo address this, the paper makes the following key contributions:\n- It introduces the reuse-then-predict mechanism. This is the core novelty. Based on the observation that predicted noise is similar across adjacent timesteps, the method first reuses the noise from a previous step to generate an approximate next latent state, and then predicts a refined noise from this new latent. This avoids the quality degradation of \"direct reuse\".\n- It proposes a distributed algorithm that leverages this mechanism to parallelize the computation of multiple denoising steps across several GPUs. This design requires only lightweight, step-wise communication, making it suitable for commodity hardware.\n- It introduces a single-device variant for non-compute-intensive models (like audio), which transforms the parallel computation into a more efficient batched execution on one GPU.\n- It provides extensive experimental results on state-of-the-art image (SD3), video (SVD, CogVideoX, etc), and audio (AudioLDM2) models, showing significant speedups over baselines with minimal quality degradation.", "questions": "Please refer to the weaknesses.", "limitations": "Please refer to the weaknesses.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths:**\n\n1.  The `reuse-then-predict` idea is novel, and its focus on low communication overhead makes ParaStep a highly practical solution for accelerating diffusion models on widely available hardware.\n2.  he method is shown to be effective across multiple modalities (image, video, audio) and consistently outperforms strong, recent baselines in terms of speedup while maintaining competitive or superior generation quality.\n3.  The inclusion of `BatchStep` for single-GPU acceleration and the demonstration of compatibility with `TeaCache` enhance the paper's contribution.\n\n**Weaknesses:**\n\n1. ParaStep requires replicating the entire model on each GPU. This is a significant limitation that constrains its applicability to models that can fit into the memory of a single GPU. While mentioned briefly in the Appendix, this critical trade-off (trading memory for speed) deserves a more prominent discussion in the main body of the paper.\n2.  As mentioned above, the paper lacks a detailed performance breakdown (computation vs. communication). Such an analysis is crucial to definitively prove that the speedup comes from communication efficiency and to better understand the performance characteristics of both the proposed method and the baselines.\n3. Fig. 3 is slightly complex. It is suggested that the author could hide the technical details and show more of the high level idea.", "was_revised": true, "final_justification": "The rebuttal addressed my concerns, I therefore decided to raise my score.", "rebuttal": "Dear reviewer p6GU,\n\nThank you for providing constructive comments and valuable suggestions. Below, we address each point raised.\n\n**W1: ParaStep requires replicating the entire model on each GPU. This is a significant limitation that constrains its applicability to models that can fit into the memory of a single GPU.** \n\n**Reply:** Thank you for your valuable comment. We apologize for this unclear description in Section Limitations. A typical diffusion pipeline consists of several components, including a text encoder, a noise predictor (e.g., DiT or U-Net), and a VAE, among others. The model as mentioned in the paper denotes the noise predictor, and ParaStep only requires replicating the entire noise predictor across all devices. However, the text encoder, typically the primary memory bottleneck, is not replicated. Instead we **split the text encoder into multiple stages across devices** to reduce per-device memory consumption. We will clarify this point more explicitly in the revised version of the paper. The table below presents the memory breakdown for SD3, CogVideoX-2B, and Latte.\n\n| Pipeline     | Total(MB) | Text encoder(MB) | Noise predictor(MB) |\n| ------------ | --------- | ---------------- | ------------------- |\n| SD3          | 16764     | 11004            | 4024                |\n| CogVdieoX-2b | 14769     | 11004            | 3342                |\n| Latte        | 13188     | 11004            | 2019                |\n\nA natural concern regarding splitting the text encoder is whether this would increase the overall pipeline latency. Since the text encoder is only invoked once to process the input prompt, and splitting it only introduces a single communication step between adjacent stages, the additional latency incurred is minimal. To validate this, we conducted experiments on SD3 to get the latency. In the results, **\"not split\"** refers to using ParaStep without splitting the text encoder, while **\"split\"** refers to using ParaStep with a split text encoder. *P* denotes the degree of parallelism. As shown, splitting the text encoder significantly reduces memory usage, while incurring only a minor increase in latency. By applying both **ParaStep** and **text encoder splitting**, we achieve not only lower latency but also reduced memory usage compared to the baseline.\n\n| SD3            | Latency(s) | Memory usage per GPU(MB) |\n| -------------- | ---------- | ------------------------ |\n| p=1            | 18.81      | 16764                    |\n| p=2, not split | 11.16      | 16764                    |\n| p=2, split     | 11.20      | 11386                    |\n| p=4, not split | 6.79       | 16764                    |\n| p=4, split     | 6.84       | 8698                     |\n\n\n\n**W2: As mentioned above, the paper lacks a detailed performance breakdown (computation vs. communication). Such an analysis is crucial to definitively prove that the speedup comes from communication efficiency and to better understand the performance characteristics of both the proposed method and the baselines.**\n\n**Reply:** Thank you for your valuable suggestion regarding a more detailed performance breakdown. To demonstrate that the observed speedup primarily stems from improved communication efficiency, we conducted experiments on the SVD model. Specifically, we measured both the total latency and the communication-related latency for AsyncDiff and our proposed ParaStep. As shown in the table below, ParaStep exhibits significantly lower communication latency compared to AsyncDiff, which directly contributes to its higher overall speedup.\n\n|                | Total Latency(s) | Latency of computation(s) | Latency of communication(s) |\n| -------------- | ---------------- | ------------------------- | --------------------------- |\n| SVD            | 51.04            | 51.04                     | -                           |\n| AsyncDiff(p=2) | 39.30            | 30.71                     | 8.59                        |\n| ParaStep(p=2)  | 30.52            | 30.42                     | 0.10                        |\n| AsyncDiff(p=4) | 31.30            | 21.01                     | 10.29                       |\n| ParaStep(p=4)  | **20.08**        | 19.94                     | 0.14                        |\n\nWe also provide the performance breakdown on CogVideoX-2B and Latte. Since AsyncDiff does not currently support these models, we report the breakdown results for ParaStep only.\n\n|               | Total Latency(s) | Latency of computation(s) | Latency of communication(s) |\n| :-----------: | ---------------- | ------------------------- | --------------------------- |\n| CogVideoX-2b  | 91.69            | 91.69                     | -                           |\n| ParaStep(p=2) | 51.32            | 50.98                     | 0.34                        |\n| ParaStep(p=4) | 34.85            | 34.22                     | 0.63                        |\n\n|               | Total Latency(s) | Latency of computation(s) | Latency of communication(s) |\n| :-----------: | ---------------- | ------------------------- | --------------------------- |\n|     Latte     | 32.83            | 32.83                     | -                           |\n| ParaStep(p=2) | 20.25            | 20.03                     | 0.22                        |\n| ParaStep(p=4) | 13.38            | 12.92                     | 0.46                        |\n\n\n\n**W3: Fig. 3 is slightly complex. It is suggested that the author could hide the technical details and show more of the high level idea.**\n\n**Reply:** Thank you for your valuable suggestion. We will simplify the Fig.3 by hiding lower-level technical details to highlight the overall design and the high level idea in the revised version of the paper."}, {"review_id": "Lesy4dNl3U", "reviewer": "Reviewer_W87m", "summary": "This paper proposed a novel parallelization method for communication-efficient multi-GPU diffusion, ParaStep, which periodically distributes the dense computation of different diffusion timestep to different GPU devices. In each cycle, the $i$-th device is responsible for the noise prediction of the $i$-th step in the cycle, while the noise predictions at preceding timesteps are reused from the previous cycle, and the noise predictions at subsequent timesteps are either retrieved from the corresponding devices for the first device or skipped for the rest devices. Extensive experiments on different models and datasets demonstrate the effectiveness of the proposed method.", "questions": "See Weaknesses.", "limitations": "yes", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. The paper is well written and easy to understand.\n2. The figures are well plotted to help understand the workload of ParaStep.\n3. The proposed parallelization method looks interesting to me.\n\nWeaknesses:\n1. The main concern is about the theoretical analysis of the proposed method. For example, for different parallelization degrees, what are their upper-bounds of the error distribution induced by reuse-then-prediction?\n2. The workload of ParaStep is based on the similarity of noise predictions at adjacent timesteps, which involves reusing the previously predicted noise to update the next-step noisy sample. That is, the authors assume that some timesteps can directly reuse the previous noise to approximate the current step’s without significant deviation. Then, why not adopt direct reuse at some timesteps that may not need further prediction, which can further speedup the inference and reduce the communication overhead.\n3.\tMost experiments are conducted on the 50-step setting. What about the few-step diffusion setting, e.g., 10-step or less?\n4.\tSome related works are missing. For example, \n\n[1] Training-Free Adaptive Diffusion with Bounded Difference Approximation Strategy, NeurIPS 2024.\n\n[2] Accelerating Diffusion Transformers with Token-wise Feature Caching, ICLR 2025.", "was_revised": true, "final_justification": "The author's rebuttal has satisfactorily addressed my concerns. I believe this work presents a compelling paradigm for seamlessly combining reuse and parallel denoising to advance the trade-off between quality and efficiency. Although the theoretical analysis could be further refined and generalized, I recommend acceptance.", "rebuttal": "Dear reviewer W87m,\n\nThank you for providing constructive comments and valuable suggestions. Below, we address each point raised.\n\n**W1: The main concern is about the theoretical analysis of the proposed method.**\n\n**Reply:** Thank for your suggestion about about the theoretical analysis. We derive the upper bound of the error distribution introduced by the reuse-then-predict mechanism under a parallelism degree of 2. Specifically, we consider two devices, GPU0 and GPU1. We follow the notations and assumptions established in [1], as well as its derivation for the error bound associated with reuse. We appreciate the solid theoretical foundation provided by [1].\n\n[1] Training-Free Adaptive Diffusion with Bounded Difference Approximation Strategy, NeurIPS 2024.\n\n**Assumptions：**\n\n+ $\\epsilon_\\theta(x, t)$ is Lipschitz w.r.t to its parameters $x$ and $t$\n+ The 1st-order difference $\\Delta x_i = x_i - x_{i+1}$ exists and is continuous for $0 \\leq i \\leq T - 1$\n\n**Notation:**\n\n+ Let $x$ denote the accurate (ground-truth) sample.\n+ Let $\\hat{x}$ denote the sample obtained via reuse (referred to as *skip* in [1]).\n+ Let $\\tilde{x}$ denote the sample obtained via the reuse-then-predict mechanism.\n\n**The denoising step at iteration $i$ is defined as:**\n\n$$x_i = f(i) \\cdot x_{i+1} - g(i) \\cdot \\epsilon_\\theta(x_{i+1}, t_{i+1})$$\n\n**The error bound for reusing the sample at iteration $i{-}1$ is derived in [1] as follows: （equation(1)）**\n$$ ||ERR_{i-1}|| = ||\\hat{x} _ {i-1}-x_{i-1}|| \n\\leq  \\mathcal{O}(t_i - t_{i+1}) + \\mathcal{O}(x_i - x_{i+1}).  \\ \\ (1)$$\n\n\nOur method is based on the **reuse-then-predict** strategy. Specifically, we assume that reuse is performed at iteration $i{-}1$, followed by a prediction-based correction at iteration $i{-}2$ (See Figure 3(c) for an Intuitive illustration).\n\n**Step $i$: Standard denoising:**\n\n$$x_i = f(i) \\cdot x_{i+1} - g(i) \\cdot \\epsilon_\\theta(x_{i+1}, t_{i+1})$$\n\n**Step $i{-}1$: Reuse (executed on GPU1):**\n\n$$\\hat{x} _ {i-1} = f(i - 1) \\cdot x_i - g(i - 1) \\cdot \\epsilon_\\theta(x_{i+1}, t_{i+1})$$\n\n**Step $i{-}2$: Predict (executed on GPU1):**\n\n$$\\tilde{x} _ {i-2} = f(i - 2) \\cdot x _ {i-1} - g(i - 2) \\cdot \\epsilon _ \\theta(\\hat{x} _ {i-1}, t _ {i-1})$$\n\n+ The term $f(i{-}2) \\cdot x_{i-1}$ appears here because GPU1 computes $\\epsilon_\\theta(\\hat{x} _ {i-1}, t_{i-1})$ while, **in parallel**, GPU0 has already computed $\\epsilon_\\theta(x_i, t_i)$. As a result, GPU0 can send $x_{i-1}$ to GPU1 in time for the prediction step.\n\n**The error introduced by the prediction step is:**\n\n$$\\tilde{x} _ {i-2}-x _ {i-2}=-g(i-2)*(\\epsilon _ \\theta(\\hat{x} _ {i-1}, t _ {i-1}) - \\epsilon _ \\theta({x} _ {i-1}, t _ {i-1})) \\ \\ (2)$$\n\nBy the Lipschitz continuity of the assumptions,\n\n$$||\\tilde{x} _ {i-2}-x _ {i-2}||<=||g(i-2)*\\mathcal{O}(\\hat{x} _ {i-1}-x _ {i-1})||=\\mathcal{O}(\\hat{x} _ {i-1}-x _ {i-1})$$\n\nAccording to Equation (1) in [1] (error bound for reusing), we can derive the **upper bound of the error distribution introduced by the reuse-then-predict** strategy.\n\n$$||\\tilde{x} _ {i-2}-x _ {i-2}||<=\\mathcal{O}(\\hat{x} _ {i-1}-x _ {i-1}) =\\mathcal{O}(t _ i - t _ {i+1}) + \\mathcal{O}(x _ i - x _ {i+1})$$\n\n**Why reuse-then-predict leads to lower error in our extensive experiments**\n\nAt iteration $i{-}2$, the error introduced by directly applying reuse is:\n\n$$||\\hat{x} _ {i-2}-x _ {i-2}||=||g(i-2) \\cdot [\\epsilon_\\theta(x_{i}, t_{i}) - \\epsilon_\\theta(x_{i-1}, t_{i-1})]||$$\n\nIn contrast, the error introduced by reuse-then-predict is:\n\n$$||\\tilde{x} _ {i-2}-x _ {i-2}||=||g(i-2)*(\\epsilon _ \\theta(\\hat{x} _ {i-1}, t _ {i-1}) - \\epsilon _ \\theta({x} _ {i-1}, t _ {i-1}))||$$\n\nThe model $\\epsilon_\\theta$ exhibits a certain degree of **refinement capability**—it can correct errors in the output that stem from inaccuracies in the input. Specifically, it can produce $\\epsilon _ \\theta(\\hat{x} _ {i-1}, t_{i-1})$ that is close to $\\epsilon _ \\theta(x _ {i-1}, t_{i-1})$, even when $\\hat{x} _ {i-1}$ deviates from $x_{i-1}$.\n\nThis assumption is supported by Figure 2(b) in the main paper: both reuse and reuse-then-predict exhibit a zigzag pattern in error accumulation—errors rise and then fall periodically. However, the error increments for reuse-then-predict are significantly smaller than those of reuse alone, demonstrating the model’s ability to correct for prior inaccuracies. **This error correction capability explains why reuse-then-predict consistently yields lower overall error in our extensive experiments**.\n\n\n\n**W2: Why not adopt direct reuse at some timesteps that may not need further prediction, which can further speedup the inference and reduce the communication overhead.**\n\n**Reply:** Thank you for your insightful suggestion regarding the potential of applying direct reuse at certain timesteps that may not require further prediction. We agree that this strategy could bring additional speedup.\n\nWhile direct reuse can indeed accelerate inference, it often leads to a noticeable drop in generation quality. The core contribution of our ParaStep method lies in the proposed **reuse-then-predict** mechanism, which explicitly refine the quality degradation caused by direct reuse. Since steps in the denoising process **are not equally important**, we believe a hybrid approach—applying ParaStep at critical steps and direct reuse at less important ones—could achieve greater acceleration than ParaStep alone, while maintaining better generation quality than pure direct reuse.\n\nOne challenge of implementing this hybrid strategy lies in its dynamic nature. ParaStep is designed as a parallel method with a fixed degree of parallelism. Introducing dynamic parallelism (as discussed in Section 4.4 of our paper) requires the ability to reallocate computing resources across steps. For instance, if we apply ParaStep with a parallelism degree *P* = 4 for the first 4 steps, and then apply direct reuse at steps 5 and 6, only one device is needed for those steps, leaving the remaining three devices idle. To make such a hybrid approach practical, we would need to design a system that supports dynamic parallelism—allowing idle devices to be reassigned to other requests during execution, thereby maximizing resource utilization. This is an interesting direction that we believe is worth exploring in future work.\n\n\n\n**W3: Most experiments are conducted on the 50-step setting. What about the few-step diffusion setting, e.g., 10-step or less?**\n\n**Reply:** Thank you for your valuable comment. Most of our experiments are conducted under the 50-step setting, as this is the default configuration for models such as CogVideoX-2B and Latte. Setting the number of steps of CogVideoX-2B to 35 results in a significant drop in generation quality.\n\n|                     | Latency(s)↓ | lpips↓     | PSNR↑       | SSIM↑      |\n| ------------------- | ----------- | ---------- | ----------- | ---------- |\n| CogVideoX-2B(T=50)  | 86.92       | -          | -           | -          |\n| CogVideoX-2B(T=40)  | 70.94       | 0.3102     | 18.4771     | 0.6931     |\n| CogVideoX-2B(T=35)  | 64.12       | 0.4033     | 16.2626     | 0.6403     |\n| ParaStep(T-50, p=2) | **58.24**   | **0.0247** | **35.2605** | **0.9588** |\n\nTo evaluate the performance of ParaStep under few-step settings, we additionally conducted experiments on Flux1.dev, which supports few-step generation. We compare the outputs generated by various acceleration methods (e.g., Direct Reuse, reduced-step generation, and ParaStep) against those from the original model (Flux with *T* = 10) to evaluate the generation quality. Here, *P* denotes the degree of parallelism in ParaStep, and *s* represents the stride used in the Direct Reuse method.\n\n|                         | Latency(s)↓ | FID↓       | PSNR↑     | SSIM↑      |\n| ----------------------- | ----------- | ---------- | --------- | ---------- |\n| Flux(T=10)              | 5.79        | -          | -         | -          |\n| **Group 1**             |             |            |           |            |\n| Direct Reuse(T=10, s=2) | 3.33        | 157.34     | 17.56     | 0.6427     |\n| Flux(reduced-step, T=5) | 3.30        | 157.44     | 17.56     | 0.6421     |\n| ParaStep(T=10, p=2)     | 3.47        | **141.28** | **18.72** | **0.6751** |\n| **Group 2**             |             |            |           |            |\n| Direct Reuse(T=10, s=4) | 2.50        | 291.83     | 15.69     | 0.5849     |\n| Flux(reduced-step, T=3) | 2.24        | 268.41     | 15.91     | 0.5877     |\n| ParaStep(T=10, p=4)     | 2.60        | 256.70     | 17.36     | 0.6352     |\n\nThe results in the table demonstrate that ParaStep achieves a clear speedup even under few-step settings, while also outperforming both Direct Reuse and reduced-step generation in terms of output quality.\n\n\n\n**W4: Some related works are missing.**\n\n **Reply:** Thank you for your valuable suggestion on citing [1] and [2], both of which are indeed closely related to our work. We will cite them in the revised version. Specifically, [1] employs a skipping strategy guided by third-order latent differences, aiming to skip as many noise prediction steps as possible. [2] introduces token-wise feature caching, enabling adaptive selection of the most suitable tokens for caching to achieve acceleration. Both of these studies, as well as our work, are inspired by the concept of **reuse**. The key difference is that [1] [2] adopts direct reuse while we propose and adopt a reuse-then-predict mechanism, which effectively mitigates the quality degradation typically caused by direct reuse."}, {"review_id": "Yzvb8vqQTJ", "reviewer": "Reviewer_gLa1", "summary": "This paper introduces ParaStep, a communication-efficient parallelization strategy for accelerating inference in diffusion models. ParaStep exploits the high similarity between noises predicted at adjacent denoising steps quantified by relative MAE to enable efficient reuse of previous noise predictions. Instead of sequentially predicting noise at every step, ParaStep first reuses the noise from the prior step to estimate the current noisy sample, then predicts a refined noise for that sample. This “reuse-then-predict” approach is distributed across multiple GPUs, which greatly reduces communication overhead compared to existing layer-wise or stage-wise parallel diffusion methods. The authors demonstrate that ParaStep achieves significant inference speedups while maintaining generation quality. They also introduce BatchStep, a single-device batching variant for lightweight models, and demonstrate the generality of their approach across image, video, and audio diffusion models through comprehensive experiments.", "questions": "1. Can the selection of key hyperparameters (e.g., warm-up steps, parallelism degree) be automated or made adaptive across different diffusion models?\n\n2. For single-GPU scenarios, how does ParaStep (or BatchStep) compare to step-reduction methods (e.g., DDIM or consistency models) in terms of speedup and quality, do these approaches outperform ParaStep/BatchStep in practice?\n\n3. Can ParaStep be effectively combined with single-device acceleration methods (such as DeepCache or DDIM) for further speedup, and have the authors benchmarked or analyzed such hybrid approaches?", "limitations": "Yes.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strength\n\n1. Substantially reduces multi-GPU communication that makes parallel diffusion inference feasible even on bandwidth-limited hardware.\n\n2. Strong compatibility that works across image, video, and audio diffusion models without retraining.\n\n3. Avoids the generation quality degradation typical in naive distributed inference methods benefited from its reuse-then-predict approach.\n\n4. Achieves significant speedup with minimal output quality loss, validated against multiple state-of-the-art parallel baselines.\n\nWeakness\n\n1. Lacks comparison with single-device efficient inference approaches (e.g., DeepCache, DDIM) or their combinations.\n\n2. Requires manual tuning of hyperparameters (warm-up steps, parallelism degree), with limited discussion of robustness or automation.\n\n3. Needs full model replication on each GPU, limiting scalability for very large models.", "was_revised": true, "final_justification": "I have read the rebuttal and I decided to increase my score from 3 to 4.\nThe paper would benefit from more systematic clarification of the experimental settings and hyper-parameter configurations, but I think the paper propose a pipeline that could be utilized for the deployment of not-that-larger diffusion model series.", "rebuttal": "Dear reviewer gLa1,\n\nThank you for providing constructive comments and valuable suggestions. Below, we address each point raised.\n\n**W1:  Lacks comparison with single-device efficient inference approaches.**\n\n**Reply:** Thank you for your valuable comments. DDIM is a deterministic scheduler that, compared to stochastic schedulers such as DDPM, reduces sampling steps. Since ParaStep is not a scheduler-level work, it is fully compatible with DDIM and other deterministic schedulers. We demonstrate this compatibility in the table below.\n\nTo evaluate the efficiency of ParaStep, we compare it against both DeepCache and AsyncDiff. The experiments are conducted on **SVD**, which uses a **deterministic** scheduler EulerDiscreteScheduler. We denote the cache interval in DeepCache as *s* and the degree of parallelism in AsyncDiff and ParaStep as *p*.\n\n|                | Latency(s)↓ | lpips↓     | PSNR↑       | SSIM↑      |\n| -------------- | ----------- | ---------- | ----------- | ---------- |\n| SVD(T=50)      | 52.23       | -          | -           | -          |\n| DeepCache(s=2) | 36.42       | 0.0399     | 32.7247     | 0.9155     |\n| DeepCache(s=4) | 26.899      | 0.0847     | 26.4912     | 0.8415     |\n| DeepCache(s=6) | 23.745      | 0.1397     | 23.1894     | 0.7728     |\n| DeepCache(s=8) | 22.237      | 0.1973     | 19.9736     | 0.7160     |\n| AsydnDiff(p=2) | 39.278      | 0.0861     | 24.0769     | 0.8328     |\n| AsydnDiff(p=4) | 29.65       | 0.1306     | 21.1599     | 0.7505     |\n| ParaStep(p=2)  | 30.61       | **0.0283** | **32.8624** | **0.9358** |\n| ParaStep(p=4)  | **20.357**  | 0.0751     | 24.9949     | 0.8448     |\n\nFirst, SVD adopts **EulerDiscreteScheduler** as its scheduler. Our method achieves significant acceleration on SVD demonstrates its compatibility with deterministic schedulers. Second, **ParaStep outperforms both DeepCache and AsyncDiff** in terms of generation quality and latency. \n\n**W2: Requires manual tuning of hyperparameters (warm-up steps, parallelism degree), with limited discussion of robustness or automation.**\n\n**Reply:** Thank you for your valuable comment. Regarding the choice of warm-up steps, we find in practice that **10% of the total steps is sufficient for simpler models** such as SVD and SD3, while **30% is a reasonable choice for more complex models** like CogVideoX and Latte. Since generation quality and speedup are both highly sensitive to the number of warm-up steps, but remain stable across different input samples, we can efficiently **profile** a small set of candidates—e.g., [1, 10% × *T*, 20% × *T*, 30% × *T*]—to identify a good trade-off between quality and efficiency.\n\nAs for the degree of parallelism, diffusion models typically use no more than 50 inference steps, making **parallelism degrees below 8** practical for achieving speedup. As shown in Figure 6 of our paper, ParaStep achieves significant acceleration with *p* = 8 while maintaining acceptable generation quality. Therefore, in practice, we can simply set the parallelism degree to the number of available devices on the machine (≤ 8).\n\n**W3: Needs full model replication on each GPU, limiting scalability for very large models.**\n\n**Reply:** Thank you for your valuable comment. We apologize for this unclear description in Section Limitations. A typical diffusion pipeline consists of several components, including a text encoder, a noise predictor (e.g., DiT or U-Net), and a VAE, among others. The model as mentioned in the paper denotes the noise predictor, and ParaStep only requires replicating the entire noise predictor across all devices. However, the text encoder, typically the primary memory bottleneck, is not replicated. Instead we split the text encoder into multiple stages across devices to reduce per-device memory consumption. We will clarify this point more explicitly in the revised version of the paper. The table below presents the memory breakdown for SD3, CogVideoX-2B, and Latte.\n\n| Pipeline     | Total(MB) | Text encoder(MB) | Noise predictor(MB) |\n| ------------ | --------- | ---------------- | ------------------- |\n| SD3          | 16764     | 11004            | 4024                |\n| CogVdieoX-2b | 14769     | 11004            | 3342                |\n| Latte        | 13188     | 11004            | 2019                |\n\nOne may worry that splitting the text encoder increases latency. Since the text encoder is only invoked once to process the prompt, and splitting it only introduces a single communication step between adjacent stages, the **additional latency incurred is minimal**. To validate this, we conducted experiments on SD3. In the results, **\"not split\"** refers to using ParaStep without splitting the text encoder, while **\"split\"** means ParaStep with encoder splitting.\n\n| SD3            | Latency(s) | Memory usage per GPU(MB) |\n| -------------- | ---------- | ------------------------ |\n| p=1            | 18.81      | 16764                    |\n| p=2, not split | 11.16      | 16764                    |\n| p=2, split     | 11.20      | 11386                    |\n| p=4, not split | 6.79       | 16764                    |\n| p=4, split     | 6.84       | 8698                     |\n\n**Q1: Can the selection of key hyperparameters be automated or made adaptive across different diffusion models?**\n\n**Reply:** Thank you for question. We answer this queston in **W2**.\n\n**Q2: For single-GPU scenarios, how does ParaStep (or BatchStep) compare to step-reduction methods?**\n\n**Reply:** Thank you for your question. We will answer this question in two parts, verifying the effects of **ParaStep** and **BatchStep** separately. \n\nAs discussed in **W1**, ParaStep is compatible with deterministic schedulers such as DDIM. Regarding the **consistency model**, it is indeed an effective approach for reducing the number of sampling steps in diffusion models. However, it typically requires **costly and often unstable training**, which limits its practicality in some scenarios. In contrast, our method is **training-free**. This makes it broadly applicable across various diffusion models. \n\nTo evaluate the efficiency of **ParaStep** in comparison with step-reduction methods, we conduct experiments on **Flux1.dev**, a sota diffusion model for image generation. Flux1.dev employs **FlowMatchEulerDiscreteScheduler**, a deterministic scheduler similar to DDIM. In the Direct Reuse method, the stride *s* indicates that *s - 1* steps are skipped for every *s* steps. We also include comparisons with the original Flux model using a reduced number of sampling steps. As shown in the table below, ParaStep consistently achieves higher generation quality than both the corresponding Direct Reuse configurations and the reduced-step method. These results demonstrate the effectiveness of ParaStep relative to step-reduction methods, and further highlight its robustness even in **extreme few-step** settings.\n\n|                         | Latency(s)↓ | FID↓       | PSNR↑     | SSIM↑      |\n| ----------------------- | ----------- | ---------- | --------- | ---------- |\n| Flux(T=10)              | 5.79        | -          | -         | -          |\n| **Group 1**             |             |            |           |            |\n| Direct Reuse(T=10, s=2) | 3.33        | 157.34     | 17.56     | 0.6427     |\n| Flux(reduced-step, T=5) | 3.30        | 157.44     | 17.56     | 0.6421     |\n| ParaStep(T=10, p=2)     | 3.47        | **141.28** | **18.72** | **0.6751** |\n| **Group 2**             |             |            |           |            |\n| Direct Reuse(T=10, s=4) | 2.50        | 291.83     | 15.69     | 0.5849     |\n| Flux(reduced-step, T=3) | 2.24        | 268.41     | 15.91     | 0.5877     |\n| ParaStep(T=10, p=4)     | 2.60        | 256.70     | 17.36     | 0.6352     |\n\nTo evaluate the efficiency of **BatchStep** in comparison with step-reduction method under a single-GPU setting, we conduct experiments on AudioLDM2-base. BatchStep achieves higher generation quality than the step-reduction, demonstrating its effectiveness in single-device scenarios.\n\n|                                | Latency(s)↓ | Speedup↑ | FAD↓   |\n| ------------------------------ | ----------- | -------- | ------ |\n| AudioLDM2(T=200)               | 7.93        | 1        | 1.7765 |\n| AudioLDM2(reduced-step, T=100) | 4.08        | 1.95     | 1.8082 |\n| AudioLDM2(reduced-step, T=50)  | 2.06        | 3.85     | 1.9661 |\n| BatchStep(T=200, s=2)          | 4.30        | 1.84     | 1.7777 |\n| BatchStep(T=200, s=4)          | 2.21        | 3.58     | 1.7805 |\n| BatchStep(T=200, s=8)          | **1.28**    | **6.17** | 1.7897 |\n\n**Q3: Can ParaStep be effectively combined with single-device acceleration methods (such as DeepCache or DDIM) for further speedup?**\n\n**Reply:** Thank you for your suggestion regarding a hybrid approach that combines ParaStep with single-device acceleration methods. Since the steps in the denoising process are **not equally important**, we can apply ParaStep on the more critical steps, and use direct reuse methods (or DeepCache) on the less important ones. We believe this hybrid approach can achieve greater acceleration than ParaStep alone, while maintaining better generation quality than Direct Reuse or DeepCache.\n\nOne challenge of implementing this hybrid approach is the need for **dynamic parallelism** (as discussed in Section 4.4 of our paper). ParaStep is inherently a parallel method. For example, if we execute ParaStep with a parallelism degree *p* = 4 for the first 4 steps, and then switch to DeepCache on steps 5 and 6, only one device is required for those steps—leaving the other three devices idle. To make this hybrid method practical and efficient, we would need to develop a system that supports dynamic parallelism, allowing idle devices to serve other incoming requests and thus maximize overall utilization. This is an interesting direction that we believe is worth exploring in future work."}, {"review_id": "K40oiBk2EG", "reviewer": "Reviewer_eKuX", "summary": "This paper proposes ParaStep, a method to parallelize diffusion model inference by exploiting similarities between adjacent denoising steps. It adopts a reuse-then-predict mechanism, enabling multiple denoising steps to be computed in parallel. For audio diffusion models, where GPU utilization is typically low, the authors instead increase the batch size to process multiple timesteps within a single batch, improving GPU utilization. Experiments on SVD, CogVideoX, and AudioLDM2 demonstrate the effectiveness of the proposed approach.", "questions": "- How does ParaStep perform on FLUX.1, the state-of-the-art image generation model?\n- What is the performance of ParaStep on diffusion models with fewer steps (e.g., 10 or 20)? Current experiments use 50 steps, which is not a typical setting in practice.", "limitations": "Yes.", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "## Strengths\n\n* The topic of distributed inference of diffusion models is underexplored but significant. \n* The idea is simple, and the paper is easy to follow.\n* Using batching to improve GPU utilization for audio diffusion models is interesting.\n* The authors provide code to reproduce their results.\n\n## Weaknesses\n\n* The proposed method has limited scalability. As the degree of parallelism increases, staleness is also increased, which degrades generation quality.\n* The proposed dynamic degree of parallelism seems not useful. In real-world deployments, the number of devices is typically fixed. If the dynamic degree of parallelism is lower than the number of devices, some compute resources are left idle, which is wasteful.\n* Unlike many distributed inference strategies, ParaStep replicates the full model on each device, failing to reduce per-device memory usage.\n* While ParaStep has lower communication amounts than methods like Ring Attention and AsyncDiff, those methods hide communication latency via asynchrony. This critical aspect is not mentioned in the paper, which is not fair.\n* Leveraging similarities between adjacent denoising steps to accelerate or parallelize diffusion models has been extensively explored in prior work, including DistriFusion, AsyncDiff, DeepCache, and TeaCache. The concept of warmup is also explored in DistriFusion.", "was_revised": true, "final_justification": "I have read the rebuttal, and most of my concerns were addressed. Therefore, I raised my score to 4.", "rebuttal": "Dear reviewer eKuX,\n\nThank you for providing constructive comments. Below, we address each point raised.\n\n**W1: The proposed method has limited scalability. As the degree of parallelism increases, staleness is also increased.**\n\n**Reply:** Thank you for your comment. Similar to AsyncDiff and DeepCache, our ParaStep leverages the similarity across adjacent steps in the diffusion process. As the degree of parallelism increases (in ParaStep or AsyncDiff) or as the cache interval increases (in DeepCache), the generation quality of these methods may degrade. This is a limitation of techniques that rely on inter-step similarity.\n\nHowever, we propose **reuse-then-predict** mechanism to effectively reduces the staleness caused by Direct Reuse. As a result, ParaStep achieves **higher generation quality** than the sota methods, as shown in Table 2 of our paper. Moreover, since diffusion models typically use no more than 50 inference steps, a parallelism degree of 4 or 8 is generally sufficient to achieve meaningful acceleration. As demonstrated in Figure 6, ParaStep delivers significant speedup with *p* = 8, while maintaining an acceptable quality drop. \n\n**W2: The proposed dynamic degree of parallelism seems not useful.  Some compute resources are left idle, which is wasteful.**\n\n**Reply:** Thank you for your comment. This dynamic parallelism is particularly suitable for **diffusion serving systems**, where the goal is to respond to multiple requests efficiently and with low latency. For example, the system may initially assign 8 devices to a single request, allowing ParaStep to perform the first 16 denoising steps with a parallelism degree of 8. The system can then **reduce the degree** to 4 for subsequent steps, freeing up 4 devices to handle other incoming requests. If the parallelism degree is later increased back to 8 for the original request, only intermediate states—such as the noise ($\\epsilon$) and the current sample ($X$)—need to be transferred to the newly assigned devices. The associated communication overhead is minimal.\n\n**W3: Unlike many distributed inference strategies, ParaStep replicates the full model on each device.**\n\n**Reply:** Thank you for your comment. We apologize for this unclear description in Section Limitations. A typical diffusion pipeline consists of several components, including a text encoder, a noise predictor (e.g., DiT or U-Net), and a VAE, among others. The model as mentioned in the paper denotes the noise predictor, and ParaStep only requires replicating the entire noise predictor across all devices. However, the text encoder, typically the primary memory bottleneck, is not replicated. Instead we split the text encoder into multiple stages across devices to reduce per-device memory consumption. We will clarify this point more explicitly in the revised version of the paper. The table below presents the memory breakdown for SD3, CogVideoX-2B, and Latte.\n\n| Pipeline     | Total(MB) | Text encoder(MB) | Noise predictor(MB) |\n| ------------ | --------- | ---------------- | ------------------- |\n| SD3          | 16764     | 11004            | 4024                |\n| CogVdieoX-2b | 14769     | 11004            | 3342                |\n| Latte        | 13188     | 11004            | 2019                |\n\nOne may worry that splitting the text encoder increases latency. Since the text encoder is only invoked once to process the input prompt, and splitting it only introduces a single communication step between adjacent stages, the **additional latency incurred is minimal**. To validate this, we conducted experiments on SD3. In the results, **\"not split\"** refers to using ParaStep without splitting the text encoder, while **\"split\"** means ParaStep with encoder splitting.\n\n| SD3            | Latency(s) | Memory usage per GPU(MB) |\n| -------------- | ---------- | ------------------------ |\n| p=1            | 18.81      | 16764                    |\n| p=2, not split | 11.16      | 16764                    |\n| p=2, split     | 11.20      | 11386                    |\n| p=4, not split | 6.79       | 16764                    |\n| p=4, split     | 6.84       | 8698                     |\n\n**W4: Methods like Ring Attention and AsyncDiff hide communication latency via asynchrony. This critical aspect is not mentioned in the paper, which is not fair.** \n\n**Reply:** Thank you for your comment. For Ring-Attention, it leverages asynchronous communication to overlap communication with adjacent computation. However, due to the large communication volume and limited memory bandwidth, communication cannot be fully overlapped with computation, which significantly limits the speedup achieved by Ring-Attention. As shown in Table 2 of our paper, Ring-Attention achieves only a 1.07× speedup due to this communication bottleneck, whereas our ParaStep attains a 1.82× speedup under the same configuration.\n\nFor AsyncDiff, to the best of our knowledge, AsyncDiff performs synchronous communication. To demonstrate that the speedup achieved by ParaStep primarily stems from improved communication efficiency, we conducted experiments on the SVD model. \n\n|                | Total Latency(s) | Latency of communication(s) |\n| -------------- | ---------------- | --------------------------- |\n| SVD            | 51.04            | -                           |\n| AsyncDiff(p=2) | 39.30            | 8.59                        |\n| ParaStep(p=2)  | 30.52            | 0.10                        |\n| AsyncDiff(p=4) | 31.30            | 10.29                       |\n| ParaStep(p=4)  | **20.08**        | 0.14                    |\n\n**W5: Leveraging similarities between adjacent denoising steps to accelerate or parallelize diffusion models has been extensively explored in prior work, including DistriFusion, AsyncDiff, DeepCache, and TeaCache. The concept of warmup is also explored in DistriFusion.**\n\n**Reply:** Thank you for your comment. The core difference between our ParaStep and the methods you mentioned lies in the **reuse-then-predict** mechanism we propose and adopt. This mechanism is designed to **refine the quality degradation** caused by direct reuse, leading to superior generation quality compared to sota acceleration methods.\n\nTo validate this, we conducted experiments comparing ParaStep with both DeepCache and AsyncDiff. As shown, ParaStep outperforms DeepCache and AsyncDiff. Specifically, with a parallelism degree of *p* = 2, ParaStep achieves higher generation quality than both baselines. Furthermore, ParaStep with *p* = 4 not only achieves lower latency than DeepCache with stride *s* = 8, but also delivers higher generation quality, demonstrating its effectiveness in balancing speed and quality.\n\n|                | Latency(s)↓ | lpips↓     | PSNR↑       | SSIM↑      |\n| -------------- | ----------- | ---------- | ----------- | ---------- |\n| SVD(T=50)      | 52.23       | -          | -           | -          |\n| DeepCache(s=2) | 36.42       | 0.0399     | 32.7247     | 0.9155     |\n| DeepCache(s=4) | 26.899      | 0.0847     | 26.4912     | 0.8415     |\n| DeepCache(s=6) | 23.745      | 0.1397     | 23.1894     | 0.7728     |\n| DeepCache(s=8) | 22.237      | 0.1973     | 19.9736     | 0.7160     |\n| AsydnDiff(p=2) | 39.278      | 0.0861     | 24.0769     | 0.8328     |\n| AsydnDiff(p=4) | 29.65       | 0.1306     | 21.1599     | 0.7505     |\n| ParaStep(p=2)  | 30.61       | **0.0283** | **32.8624** | **0.9358** |\n| ParaStep(p=4)  | **20.357**  | 0.0751     | 24.9949     | 0.8448     |\n\n**Q1: How does ParaStep perform on FLUX.1, the state-of-the-art image generation model?** and\n\n**Q2: What is the performance of ParaStep on diffusion models with fewer steps (e.g., 10 or 20)? Current experiments use 50 steps, which is not a typical setting in practice.**\n\n**Reply:** Thank you for your questions. Most of our experiments are conducted under the 50-step setting, as this is the default configuration for models such as CogVideoX-2B and Latte. Setting the number of steps of CogVideoX-2B to 35 results in a significant drop in generation quality.\n\n|                     | Latency(s)↓ | lpips↓     | PSNR↑       | SSIM↑      |\n| ------------------- | ----------- | ---------- | ----------- | ---------- |\n| CogVideoX-2B(T=50)  | 86.92       | -          | -           | -          |\n| CogVideoX-2B(T=40)  | 70.94       | 0.3102     | 18.4771     | 0.6931     |\n| CogVideoX-2B(T=35)  | 64.12       | 0.4033     | 16.2626     | 0.6403     |\n| ParaStep(T=50, p=2) | **58.24**   | **0.0247** | **35.2605** | **0.9588** |\n\nTo assess the performance of ParaStep in few-step settings, we conducted experiments on **Flux1.dev**, which supports few-step generation. Here, *p* denotes the degree of parallelism in ParaStep, and *s* represents the stride used in the Direct Reuse method. The results in the table demonstrate that ParaStep achieves speedup under few-step settings, while also outperforming both Direct Reuse and reduced-step generation in terms of output quality.\n\n|                         | Latency(s)↓ | FID↓       | PSNR↑     | SSIM↑      |\n| ----------------------- | ----------- | ---------- | --------- | ---------- |\n| Flux(T=10)              | 5.79        | -          | -         | -          |\n| **Group 1**             |             |            |           |            |\n| Direct Reuse(T=10, s=2) | 3.33        | 157.34     | 17.56     | 0.6427     |\n| Flux(reduced-step, T=5) | 3.30        | 157.44     | 17.56     | 0.6421     |\n| ParaStep(T=10, p=2)     | 3.47        | **141.28** | **18.72** | **0.6751** |\n| **Group 2**             |             |            |           |            |\n| Direct Reuse(T=10, s=4) | 2.50        | 291.83     | 15.69     | 0.5849     |\n| Flux(reduced-step, T=3) | 2.24        | 268.41     | 15.91     | 0.5877     |\n| ParaStep(T=10, p=4)     | 2.60        | 256.70     | 17.36     | 0.6352     |"}]

Author rebuttal:

Meta-review: This paper presents a new parallelization strategy for diffusion inference and demonstrates significant speedups across various models. Although its score is still at boaderline, after discussions/rebuttal with the reviewers, several concerns have been addressed, primarily regarding experimentation (e.g., limited time-steps, detailed performance breakdowns). I believe these improvements have strengthened the paper, and I would therefore recommend Acceptance as a poster.
Decision: Accept (poster)