Title: Accelerating Parallel Diffusion Model Serving with Residual Compression
Venue: NeurIPS 2025
Accepted: yes
Keywords: diffusion model acceleration

parallel inference
Abstract: Diffusion models produce realistic images and videos but require substantial computational resources, necessitating multi-accelerator parallelism for real-time deployment. However, parallel inference introduces significant communication overhead from exchanging large activations between devices, limiting efficiency and scalability. We present CompactFusion, a compression framework that significantly reduces communication while preserving generation quality. Our key observation is that diffusion activations exhibit strong temporal redundancy—adjacent steps produce highly similar activations, saturating bandwidth with near-duplicate data carrying little new information. To address this inefficiency, we seek a more compact representation that encodes only the essential information. CompactFusion achieves this via Residual Compression that transmits only compressed residuals (step-wise activation differences). Based on empirical analysis and theoretical justification, we show that it effectively removes redundant data, enabling substantial data reduction while maintaining high fidelity. We also integrate lightweight error feedback to prevent error accumulation. CompactFusion establishes a new paradigm for parallel diffusion inference, delivering lower latency and significantly higher generation quality than prior methods. On 4$\times$L20, it achieves $3.0\times$ speedup while greatly improving fidelity. It also uniquely supports communication-heavy strategies like sequence parallelism on slow networks, achieving $6.7\times$ speedup over prior overlap-based method. CompactFusion applies broadly across diffusion models and parallel settings, and integrates easily without requiring pipeline rework. Portable implementation demonstrated on xDiT is publicly available at https://github.com/Cobalt-27/CompactFusion


Peer reviews:
comments: SUMMARY:
Diffusion models are computationally intensive and require multi-accelerator parallelism for real-time deployment, but suffer from high communication overhead due to redundant activation exchanges. This paper propose to addresses this by introducing a Residual Compression technique that transmits only compressed residuals, effectively reducing redundancy. This method maintains high generation quality while significantly lowering latency than prior methods.

STRENGTHS AND WEAKNESSES:
- Strengths
  - The proposed CompactFusion delivers lower latency parallel diffusion inference, while achieves higher generation quality than prior methods.
  - Both empirical analysis and theoretical justification are given to show the effectiveness of the proposed method.
- Weaknesses
  - I do not see obvious weaknesses. Some small issues are mentioned in Questions.

QUESTIONS:
- It would be beneficial to include an additional ablation study comparing the model's performance with and without quantization errors. This analysis would help isolate the impact of compression-induced errors and provide a clearer understanding of how quantization affects the overall performance. It may also help understand the upper bound of the effectiveness of the proposed error feedback.
- Has the influence of content complexity been examined? It would be helpful to analyze whether the proposed method performs differently on inputs with complex patterns versus simpler ones. 
- The proposed method introduces compression errors. Will it affect the temporal stability of the generated videos. Has this aspect been evaluated?

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-06-27

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nTo reduce communication overhead in patch parallel inference, this work proposes a novel Fresh Activation strategy. The method yields a modest speedup while preserving model performance through an error feedback mechanism.\n\nSTRENGTHS AND WEAKNESSES:\n[Strengthen]\n1.The proposed method improves the efficiency of parallel inference by reusing activations through the Fresh mechanism.\n2.The authors provide a theoretical analysis demonstrating that the Fresh activation reuse effectively reduces the approximation error.\n3.The method is integrated with existing toolchains, showing potential for practical deployment in real-world applications.\n\n[Weakness]\n1. The motivation of the work lacks clarity. Since communication and computation are overlapped in practice, compressing activations is typically aimed at reducing communication latency. This motivation implicitly assumes that communication latency completely dominates computation time. The paperdoes not provide any empirical evaluation or profiling to justify this assumption.\n2. The effectiveness of activation compression in improving inference speed remains questionable. DistriFusion adopts a longer warm-up process, while CompactFusion only uses a one-step warm-up, which inherently gives it an advantage in speed. However, as shown in Tables 1 and 2, CompactFusion only yields marginal improvements over DistriFusion, especially when low-rank compression is applied. This raises concerns that transmitting compressed activations may not meaningfully accelerate inference.\n3. The implementation details of patch parallelism in CompactFusion are unclear. Does it appear to be built upon the parallelism strategy of DistriFusion, with additional transmission of the residual for the current patch? The paperlacks a clear description of this strategy, leading to confusion. For example, in Figure 3, what does “base” refer to? Why does the error feedback correct the sender’s “base” instead of that of the receiver? How are all layers of the model executed during parallel inference? A more explicit explanation of the overall parallel procedure is necessary.\n4. The paperclaims that CompactFusion generalizes better than DistriFusion across model architectures and inference strategies. However, the underlying reasons for this improved generalization are not clearly articulated. A detailed analysis of the compatibility mechanisms employed by CompactFusion is needed to substantiate this claim.\n\nQUESTIONS:\nplease see the weekness\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 2, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work aims to accelerate parallel diffusion model inference by mitigating communication bottlenecks. The authors focus on the problem of excessive communication overhead caused by exchanging large activation data between devices during distributed training. They address limitations in prior approaches, such as quality degradation from stale activation reuse and insufficient reduction of data volume. The authors propose a compression method that transmits compressed residuals (differences between consecutive steps) instead of full activations, combined with lightweight error feedback to maintain fidelity. The proposed method integrates optimized GPU kernels and system co-design to overlap compression with communication. Evaluations on show that the proposed method brings inference speedups on PCIe/Ethernet with small quality loss and perceptual degradation.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths:**  \n\nThe authors tackle the communication bottleneck in parallel diffusion model inference. Previous methods, such as reusing stale activations, reduce data volume but degrade generation quality. This paper has pointed out a fundamental challenge of balancing communication efficiency and perceptual quality in diffusion modeling.\n\nThe proposed method leverages residual compression, encoding only the differences between consecutive steps, instead of transmitting full activations between steps. This idea is simple but effective and has been validated by the experiments.\n\n\n**Weaknesses:**  \n\nMostly, I think this paper is interesting. Here are some aspects that could be strengthened. \n\nThe authors transmit compressed residuals rather than full activations and leverage temporal redundancy in diffusion steps. So, the compression operations may introduce additional latency. Although it can be mitigated by overlapping with communication and computations, I’m interested in how to balance this compression-latency problem.\n\nQUESTIONS:\nPlease clarify my questions in the weakness part.\n\nLIMITATIONS:\nThis paper has some merits and I have summarized my major concerns in the weakness part.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nExisting sequence parallelism-based methods enable reducing the sample latency of diffusion models by distributing computation across multiple devices. However, sequence parallelism introduces significant communication overhead, limiting its efficiency. This paper proposes CompactFusion, a framework that compresses the residuals of activations between adjacent denoising steps and introduces error feedback to mitigate cumulative global error. By transmitting compressed residuals instead of the raw activations, the communication cost is significantly reduced. Experiments on image and video generation demonstrate the effectiveness of the proposed method.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- The idea of compressing the residuals of activations is supported by convincing evidence and is easy to follow.\n- Compared with baseline methods, the proposed method exhibits superior performance on both speedup and sample quality.\n- The proposed method can interact with existing sequence parallel methods easily, especially those communication-heavy methods.\n\nWeaknesses:\n- The writing of the preliminary part is overly simplified and could be improved. The proposed techniques, such as residual compression and error feedback, are applied to Sequence Parallel. However, the exact inference procedures are not introduced in the main content or the supplementary materials, making it challenging to understand the proposed method. Including more details of Sequence Parallel in the preliminary section, or including a more illustrative figure/algorithm to show the entire sampling process, could make the paper more readable.\n- Though enjoying extremely small communication volume, the proposed Compact-Lowrank suffers from limited speedup brought by low-rank approximation. At the same time, its performance does not have an advantage compared to baseline methods, as well as Compact-1bit and Compact-2bit. Which scenarios is Compact-Lowrank suitable for?\n\nQUESTIONS:\n- In Section 3.2, it is concluded that the error bound of residual compression is significantly smaller than that of naive compression, once we have $\\delta>1-\\frac{1-L^2}{L^2+1}$. Why is this a \"mild condition\"? Is there any quantitative result to support this statement for each compressor (1-bit quantization, 2-bit quantization, and low-rank approximation)?\n- When generating a large number of samples (e.g., for FID evaluation) with several GPUs, will the proposed method be faster than naive sampling (i.e., each GPU generates a portion of samples without any parallel)? \n- Could the authors provide a total execution time of compression, decompression and communication in one single sampling process?\n- Typo: in line 141, ovear -> over; in line 45, paralleli -> parallelism.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}