Title: Communication-Efficient Diffusion Denoising Parallelization via Reuse-then-Predict Mechanism
Venue: NeurIPS 2025
Accepted: yes
Keywords: Diffusion Models

Parallelism

Efficient Communication
Abstract: Diffusion models have emerged as a powerful class of generative models across various modalities, including image, video, and audio synthesis. However, their deployment is often limited by significant inference latency, primarily due to the inherently sequential nature of the denoising process. While existing parallelization strategies attempt to accelerate inference by distributing computation across multiple devices, they typically incur high communication overhead, hindering deployment on commercial hardware. To address this challenge, we propose $\textbf{ParaStep}$, a novel parallelization method based on a reuse-then-predict mechanism that parallelizes diffusion inference by exploiting similarity between adjacent denoising steps. Unlike prior approaches that rely on layer-wise or stage-wise communication, ParaStep employs lightweight, step-wise communication, substantially reducing overhead. ParaStep achieves end-to-end speedups of up to $\textbf{3.88}$$\times$ on SVD, $\textbf{2.43}$$\times$ on CogVideoX-2b, and $\textbf{6.56}$$\times$ on AudioLDM2-large, while maintaining generation quality. These results highlight ParaStep as a scalable and communication-efficient solution for accelerating diffusion inference, particularly in bandwidth-constrained environments.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes ParaStep, a method to parallelize diffusion model inference by exploiting similarities between adjacent denoising steps. It adopts a reuse-then-predict mechanism, enabling multiple denoising steps to be computed in parallel. For audio diffusion models, where GPU utilization is typically low, the authors instead increase the batch size to process multiple timesteps within a single batch, improving GPU utilization. Experiments on SVD, CogVideoX, and AudioLDM2 demonstrate the effectiveness of the proposed approach.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths\n\n* The topic of distributed inference of diffusion models is underexplored but significant. \n* The idea is simple, and the paper is easy to follow.\n* Using batching to improve GPU utilization for audio diffusion models is interesting.\n* The authors provide code to reproduce their results.\n\n## Weaknesses\n\n* The proposed method has limited scalability. As the degree of parallelism increases, staleness is also increased, which degrades generation quality.\n* The proposed dynamic degree of parallelism seems not useful. In real-world deployments, the number of devices is typically fixed. If the dynamic degree of parallelism is lower than the number of devices, some compute resources are left idle, which is wasteful.\n* Unlike many distributed inference strategies, ParaStep replicates the full model on each device, failing to reduce per-device memory usage.\n* While ParaStep has lower communication amounts than methods like Ring Attention and AsyncDiff, those methods hide communication latency via asynchrony. This critical aspect is not mentioned in the paper, which is not fair.\n* Leveraging similarities between adjacent denoising steps to accelerate or parallelize diffusion models has been extensively explored in prior work, including DistriFusion, AsyncDiff, DeepCache, and TeaCache. The concept of warmup is also explored in DistriFusion.\n\nQUESTIONS:\n- How does ParaStep perform on FLUX.1, the state-of-the-art image generation model?\n- What is the performance of ParaStep on diffusion models with fewer steps (e.g., 10 or 20)? Current experiments use 50 steps, which is not a typical setting in practice.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces ParaStep, a communication-efficient parallelization strategy for accelerating inference in diffusion models. ParaStep exploits the high similarity between noises predicted at adjacent denoising steps quantified by relative MAE to enable efficient reuse of previous noise predictions. Instead of sequentially predicting noise at every step, ParaStep first reuses the noise from the prior step to estimate the current noisy sample, then predicts a refined noise for that sample. This “reuse-then-predict” approach is distributed across multiple GPUs, which greatly reduces communication overhead compared to existing layer-wise or stage-wise parallel diffusion methods. The authors demonstrate that ParaStep achieves significant inference speedups while maintaining generation quality. They also introduce BatchStep, a single-device batching variant for lightweight models, and demonstrate the generality of their approach across image, video, and audio diffusion models through comprehensive experiments.\n\nSTRENGTHS AND WEAKNESSES:\nStrength\n\n1. Substantially reduces multi-GPU communication that makes parallel diffusion inference feasible even on bandwidth-limited hardware.\n\n2. Strong compatibility that works across image, video, and audio diffusion models without retraining.\n\n3. Avoids the generation quality degradation typical in naive distributed inference methods benefited from its reuse-then-predict approach.\n\n4. Achieves significant speedup with minimal output quality loss, validated against multiple state-of-the-art parallel baselines.\n\nWeakness\n\n1. Lacks comparison with single-device efficient inference approaches (e.g., DeepCache, DDIM) or their combinations.\n\n2. Requires manual tuning of hyperparameters (warm-up steps, parallelism degree), with limited discussion of robustness or automation.\n\n3. Needs full model replication on each GPU, limiting scalability for very large models.\n\nQUESTIONS:\n1. Can the selection of key hyperparameters (e.g., warm-up steps, parallelism degree) be automated or made adaptive across different diffusion models?\n\n2. For single-GPU scenarios, how does ParaStep (or BatchStep) compare to step-reduction methods (e.g., DDIM or consistency models) in terms of speedup and quality, do these approaches outperform ParaStep/BatchStep in practice?\n\n3. Can ParaStep be effectively combined with single-device acceleration methods (such as DeepCache or DDIM) for further speedup, and have the authors benchmarked or analyzed such hybrid approaches?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposed a novel parallelization method for communication-efficient multi-GPU diffusion, ParaStep, which periodically distributes the dense computation of different diffusion timestep to different GPU devices. In each cycle, the $i$-th device is responsible for the noise prediction of the $i$-th step in the cycle, while the noise predictions at preceding timesteps are reused from the previous cycle, and the noise predictions at subsequent timesteps are either retrieved from the corresponding devices for the first device or skipped for the rest devices. Extensive experiments on different models and datasets demonstrate the effectiveness of the proposed method.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. The paper is well written and easy to understand.\n2. The figures are well plotted to help understand the workload of ParaStep.\n3. The proposed parallelization method looks interesting to me.\n\nWeaknesses:\n1. The main concern is about the theoretical analysis of the proposed method. For example, for different parallelization degrees, what are their upper-bounds of the error distribution induced by reuse-then-prediction?\n2. The workload of ParaStep is based on the similarity of noise predictions at adjacent timesteps, which involves reusing the previously predicted noise to update the next-step noisy sample. That is, the authors assume that some timesteps can directly reuse the previous noise to approximate the current step’s without significant deviation. Then, why not adopt direct reuse at some timesteps that may not need further prediction, which can further speedup the inference and reduce the communication overhead.\n3.\tMost experiments are conducted on the 50-step setting. What about the few-step diffusion setting, e.g., 10-step or less?\n4.\tSome related works are missing. For example, \n\n[1] Training-Free Adaptive Diffusion with Bounded Difference Approximation Strategy, NeurIPS 2024.\n\n[2] Accelerating Diffusion Transformers with Token-wise Feature Caching, ICLR 2025.\n\nQUESTIONS:\nSee Weaknesses.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper tackles the high inference latency of diffusion models, which arises from the sequential nature of the denoising process. The authors identify that existing parallelization strategies often suffer from high communication overhead, limiting their use to expensive, high-bandwidth data center hardware.\n\nTo address this, the paper makes the following key contributions:\n- It introduces the reuse-then-predict mechanism. This is the core novelty. Based on the observation that predicted noise is similar across adjacent timesteps, the method first reuses the noise from a previous step to generate an approximate next latent state, and then predicts a refined noise from this new latent. This avoids the quality degradation of \"direct reuse\".\n- It proposes a distributed algorithm that leverages this mechanism to parallelize the computation of multiple denoising steps across several GPUs. This design requires only lightweight, step-wise communication, making it suitable for commodity hardware.\n- It introduces a single-device variant for non-compute-intensive models (like audio), which transforms the parallel computation into a more efficient batched execution on one GPU.\n- It provides extensive experimental results on state-of-the-art image (SD3), video (SVD, CogVideoX, etc), and audio (AudioLDM2) models, showing significant speedups over baselines with minimal quality degradation.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths:**\n\n1.  The `reuse-then-predict` idea is novel, and its focus on low communication overhead makes ParaStep a highly practical solution for accelerating diffusion models on widely available hardware.\n2.  he method is shown to be effective across multiple modalities (image, video, audio) and consistently outperforms strong, recent baselines in terms of speedup while maintaining competitive or superior generation quality.\n3.  The inclusion of `BatchStep` for single-GPU acceleration and the demonstration of compatibility with `TeaCache` enhance the paper's contribution.\n\n**Weaknesses:**\n\n1. ParaStep requires replicating the entire model on each GPU. This is a significant limitation that constrains its applicability to models that can fit into the memory of a single GPU. While mentioned briefly in the Appendix, this critical trade-off (trading memory for speed) deserves a more prominent discussion in the main body of the paper.\n2.  As mentioned above, the paper lacks a detailed performance breakdown (computation vs. communication). Such an analysis is crucial to definitively prove that the speedup comes from communication efficiency and to better understand the performance characteristics of both the proposed method and the baselines.\n3. Fig. 3 is slightly complex. It is suggested that the author could hide the technical details and show more of the high level idea.\n\nQUESTIONS:\nPlease refer to the weaknesses.\n\nLIMITATIONS:\nPlease refer to the weaknesses.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}