Title: ClusterFusion: Expanding Operator Fusion Scope for LLM Inference via Cluster-Level Collective Primitive
Venue: NeurIPS 2025
Accepted: yes
Keywords: Operator Fusion

LLM Inference

Inter-core Interconnect
Abstract: Large language model (LLM) decoding suffers from high latency due to fragmented execution across operators and heavy reliance on off-chip memory for data exchange and reduction. 
This execution model limits opportunities for fusion and incurs significant memory traffic and kernel launch overhead.
While modern architectures such as NVIDIA Hopper provide distributed shared memory and low-latency intra-cluster interconnects, they expose only low-level data movement instructions, lacking structured abstractions for collective on-chip communication.
To bridge this software-hardware gap, we introduce two cluster-level communication primitives, ClusterReduce and ClusterGather, which abstract common communication patterns and enable structured, high-speed data exchange and reduction between thread blocks within a cluster, allowing intermediate results to be on-chip without involving off-chip memory.
Building on these abstractions, we design ClusterFusion, an execution framework that schedules communication and computation jointly to expand operator fusion scope by composing decoding stages such as QKV Projection, Attention, and Output Projection into a single fused kernels.
Evaluations on H100 GPUs show that ClusterFusion outperforms state-of-the-art inference frameworks by $1.61\times$ on average in end-to-end latency across different models and configurations.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper applied the idea of \"collective communication\" or tensor parallellism/sharding between multiple devices to the sharding/on-chip collectives between threading blocks/Streaming Processors within a single device, and demonstrates it enables kernel fusion of the entire self-Attention module and performance gain over SOTA inference kernels/engines.\n\nSTRENGTHS AND WEAKNESSES:\nStrength:\n* The motivaiton is intuitive and novel on GPU programming: modern LLM decoding on GPU is bottlenecked by off-chip memory bandwidth and kernel launch/synchronization overhead. And the onchip collective communication between SMs/Thread-blocks and reduce off-chip traffic and promote kernel fusions\n\n* The author abstract and implemented two effective collective communications among a thread block cluster: All-Reduce and All-Gather, which serves the fundamental to perform tensor/operator sharding onchip\n\n* The authors demonstrated significant performance gain e.g., 50% over SOTA inference engine/kernels with BS=1\n\nWeakness:\n* While relative new among GPU, onchip datalfow/collectives have been widely explored by dataflow architecture accelerators like cited GraphCore/Cerebras etc, the associated aggressive kernel fusion has also been studied by https://arxiv.org/pdf/2410.23668 which extends kernel fusion to the entire transformer model. Recently there is also research about mega kernel fusion on GPU: https://hazyresearch.stanford.edu/blog/2025-05-27-no-bubbles, which should be included in discussion\n\n- The authors didn't discuss enough limitations of this approach, detailed below\n\nQUESTIONS:\n1. It is a bit confusing to use the term \"Cluster\" as it typically refers to distributed multi-device/node system. While the core ideas of collectives are the same with a cluster of devices and a cluster of threadblocks, it may be a bit misleading for readers. Similarly the authors called the two primitives Cluster-Reduce/Gather, aren't they just Block/SM level all-reduce/gather?\n\n2. The collectives are implemented via tree pattern, have you considered ring pattern?\n\n3. How do you define the bandwidth between a cluster of SMs? is it defined by peer-to-peer bw if so I am not sure why it is affected by cluster size or is it deifned by bipartite or all-reduce bw etc?\n\n4. Why would the active SMs descrease with size of clusters?\n\n5. Line 256 mentioned the kernel launch overhead is reduced by an order, however based on Figure 7, the kernel is a fusion of 4 kernels of Figure 3, how is it able to reduce kernel launch overhead by an order (or 10 times)?\n\nLIMITATIONS:\n* While included in appendix, I think the authors should discuss the efficacy of the method on higher batchsize > 1 in the main body. e.g., why does higher batch size see a much less performance gain over baselines when no weight is involved (unlike MLPs) so off-chip traffic is purely proportional to batchsize\n\n* The onchip collectives rely on creating buffers with onchip shared-memory, what if the model/kernel is too large so it runs out of the pretious onchip memory?", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-15"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nOn NVIDIA Hopper GPUs, distributed shared memory is introduced to allow two thread blocks within the same thread block cluster to access each other’s shared memory. This feature provides a more efficient method for inter-block communication compared to using global memory. This paper proposes two cluster-level collective primitives that leverage this low-level feature. Based on the proposed primitives, the authors also implement a fused attention mechanism (including QKV projections, self-attention, and output projection) in a single kernel. Experiments are conducted to demonstrate the effectiveness of the proposed primitives and the fused kernel.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n\n1. The proposed techniques target important workloads (i.e., efficient LLM inference) on relevant hardware (i.e., recent NVIDIA GPUs).\n2. The results are promising.\n3. The paper is well written and easy to follow.\n\n**Weaknesses**\n\n1. The proposed primitives themselves do not present much novelty. Reduce and gather are standard collective primitives.\n2. More evidence or experiments are needed to justify the necessity of using the proposed primitives for the fusion.\n\nQUESTIONS:\nThank you to the authors for submitting to NeurIPS 2025. The paper is well written and easy to follow. Overall, I am positive about the submission. I have two concerns that I hope can help improve the paper.\n\n1. **The proposed primitives themselves do not present much novelty. Reduce and gather are standard collective primitives.**\n   While reduce and gather are standard primitives, building them using low-level PTX instructions may be a novel aspect of this work. Therefore, I recommend clarifying which low-level instructions are used (e.g., `ld.shared::cluster.u32`), possibly in a table, and explaining why they are difficult to use directly.\n\n2. **More evidence or experiments are needed to justify the necessity of the proposed primitives for the fusion.**\n   Figure 7 illustrates the fused attention operation using the proposed collective primitives. However, I did not see a fundamental limitation preventing similar kernel without using cluster-level primitives. For example, could the same thread block handle the tasks of both TB0 and TB1 in Figure 7 by reducing the tile size? If so, additional experiments comparing the block cluster approach with a single thread block would be helpful. If not, it would be beneficial to explain why, e.g., due to insufficient shared memory, with concrete numbers.\n\nLIMITATIONS:\nNo", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-20"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes ClusterFusion, a new system to make large language models (LLMs) run faster during inference (when generating tokens) by fusing multiple steps—like QKV projection, attention, and output projection—into a single, optimized computation block.\n\nThe key problem is that current LLM systems run these steps separately and move data to and from off-chip memory, which slows things down. Modern GPUs like NVIDIA Hopper offer on-chip memory and fast interconnects within thread block clusters, but these are hard to use efficiently due to low-level programming.\n\nTo solve this, the authors design two new primitives:\n- ClusterReduce: for reducing values (e.g., sum or max) across thread blocks.\n- ClusterGather: for sharing data between thread blocks.\nThese allow on-chip communication and help fuse multiple operators together in a more coordinated way. The authors implement these ideas into an inference framework called ClusterFusion, which they test on Llama2-7B and DeepSeek-V2-Lite using an NVIDIA H100 GPU. Compared to popular systems like vLLM, SGLang, and TensorRT-LLM, ClusterFusion achieves an average speedup of ~1.6× in decoding time, especially on long sequences.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths \n\n1. Originality: Proposing structured, cluster-level primitives for intra-block GPU communication is non-trivial and novel; Extending fusion beyond single kernels via coordinated scheduling of communication and computation is a clear step forward for system-level optimization in LLM serving.\n2. Significance: This is not just a toy speedup: the ~1.6× end-to-end decoding speedup is meaningful and relevant to real-world LLM deployment at scale; The ideas are hardware-aware and reflect a strong grasp of actual GPU architecture, making them useful beyond just academic interest.\n3. Clarity: The paper is mostly clear and readable. Diagrams (especially Fig. 6 and 7) help explain the concepts well; The structure (problem → primitives → fused dataflow → results) is logical and easy to follow.\n\nWeaknesses \n\n1. Over-reliance on NVIDIA Hopper (H100):\nThe entire framework depends heavily on low-level Hopper-only features (e.g., DSMEM, thread block clusters).\n The generalization to other GPUs (e.g., A100, non-NVIDIA) is not discussed at all. This makes the solution less broadly applicable, and the paper may quickly become obsolete as hardware evolves. \n2. Fusion scope is still bounded by hardware (16-thread-block limit):\nWhile the paper claims to \"expand\" fusion, it’s still limited by the current hardware cluster size.\nThere is no strong roadmap for scaling this beyond what Hopper already allows. \n3. The method is system-level but lacks co-design with compiler/runtime stack:\nThere is no integration or discussion of compiler-level scheduling, which is critical for making this approach portable or usable at scale. How would a user actually integrate this into their LLM stack? Would HuggingFace or TensorRT need to be rewritten?\n4. Ablation and analysis are shallow in parts:\nThe ablation on DSMEM (e.g., Fig. 13) is helpful but too limited. What happens to performance if you scale to batch sizes >1, or mix prompt lengths? A stronger analysis of latency breakdowns, kernel utilization, or memory pressure would help explain where the 1.6× speedup comes from in more detail.\n\nQUESTIONS:\nQuestions & Suggestions for the Authors\n\n1. How portable is ClusterFusion beyond NVIDIA Hopper?\nYour method relies on Hopper-specific features, such as DSMEM and thread block clusters. Can your approach be adapted to other GPUs, e.g., A100?\n2. What does integration with real-world LLM systems look like?\nYou claim speedups over SOTA frameworks like vLLM and TensorRT-LLM—but how would someone actually adopt ClusterFusion in production? Do you require kernel-level rewrites, custom graph compilers, or modified CUDA runtimes? \n3. Why is fusion limited to three modules (QKV, Attention, Output)?\nYou call it “ClusterFusion,” but the fusion scope still ends after Output Projection. Why not include FFN, RMSNorm, or RotaryEmbeddings? Are there hardware barriers or software trade-offs you can explain more clearly? This would clarify how general and extensible the method really is.\n4. Do you expect ClusterFusion to scale to batch sizes >1 or mixed-length prompts?\nAll your evaluations use batch size = 1. That’s common in interactive inference, but real serving systems also deal with batching and variable-length requests. Does your method still deliver speedup in those conditions? If not, could you clarify when it's useful vs. not useful?\n5. How do you recommend users tune cluster size and layout in practice?\nYou provide detailed analysis of cluster size trade-offs, but no guidance on how a practitioner should make these decisions automatically. Could you expose this as a tunable parameter? Or integrate it into a profiling-based auto-tuner?\n\nLIMITATIONS:\n1. Highly hardware-specific design\nThe entire framework is tightly coupled to NVIDIA Hopper GPUs. There is no discussion about compatibility with previous-generation GPUs (e.g., A100) or non-NVIDIA hardware (e.g., AMD, Intel).\nThe method risks becoming obsolete or irrelevant outside the Hopper ecosystem. Suggestion: Clarify how general the design is, and whether a fallback mode or alternative backend is possible.\n2. No discussion on compiler or framework support\nAlthough the paper presents an elegant low-level optimization, there is no clear path to integrating it into real-world ML compilers (like TensorRT, TVM, or PyTorch). Discuss whether a practical implementation could be exposed via kernel templates or integration with tools like Triton or CUTLASS.\n3. Limited task and batch settings\nAll experiments are conducted with batch size = 1 and on specific LLMs (Llama2-7B, DeepSeek-V2-Lite). It’s unclear whether the speedups hold in multi-batch, mixed-length, or multitask environments, which are common in actual deployments. Suggestion: Discuss potential performance changes under more realistic workloads.\n4. Potential for energy-performance tradeoffs is not explored\nWhile the work optimizes latency, it does not address power or energy use, which is critical in datacenter-scale inference. Suggestion: Even if not measured, a brief comment on energy efficiency (e.g., due to reduced off-chip memory traffic) would strengthen the story.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper addresses decoding latency in LLM inference caused by fragmented kernels and frequent off-chip memory sync. Although NVIDIA Hopper GPUs offer low-level DSMEM, they lack high-level communication primitives.  \n\n**ClusterFusion** fills this gap by introducing two on-chip collective operations:  \n- **ClusterReduce**  \n- **ClusterGather**  \n\nThese enable QKV, attention, and output projections to remain on-chip, cutting off-chip transfers and boosting performance. On H100 GPUs with Llama2-7B and DeepSeek-V2-Lite, ClusterFusion achieves a **1.61×** speedup over SGLang, vLLM, TensorRT-LLM, and MLC-LLM.\n\nThe paper contributes:\n1. **Profiling** of decoding dataflow inefficiencies.  \n2. **High-level primitives** for structured inter-block communication on Hopper.  \n3. **ClusterFusion framework** that expands operator fusion and reduces inference latency.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths\n\n1. **Quality**  \n   - The paper delivers a rigorous motivation analysis to algorithm design. I do love the diagrams, pseudocode and smooth storyline of paper that walk me through on-chip end2end design.\n   - Empirical measurements are comprehensive—covering per-kernel breakdowns, DSMEM bandwidth vs. global memory, and SM-to-SM link utilization.\n\n2. **Clarity**  \n   - **ClusterReduce** and **ClusterGather** are crisply specified: their APIs map directly to common data-parallel patterns (all-reduce, broadcast) and hide DSMEM complexities.  \n\n3. **Significance of Results**  \n   - The approach first proposes such primitive and directly targets production-critical workloads. On real hardware (NVIDIA H100), ClusterFusion yields a consistent **1.61×** end-to-end speedup over four competing inference frameworks.\n\n---\n\n## Weaknesses\n\n1. **Originality vs. Engineering Effort**  \n   - The primitives rest on NVIDIA PTX’s DSMEM features; while the abstraction is valuable, it does not introduce a fundamentally new communication algorithm or theoretical model. The novelty is primarily in the high-level API and scheduler integration rather than in core algorithms.\n\n2. **Generality Across Architectures**  \n   - The paper mentioned different architecture may suffer from these issues, but all experiments target Hopper GPUs. It remains unclear how ClusterReduce/Gather would perform—or even map—to architectures without hardware DSMEM (e.g., AMD ROCm, TPUs). A discussion or prototype on alternate hardware would bolster claims of broad applicability.\n\n3. **Scalability to Very Large Models and Contexts**  \n   - Evaluation caps at 7 B-parameter models and 16 K token contexts. It is left open whether on-chip buffers suffice for, say, 70 B models or 100 K-token sequences without falling back to global memory, which would erode benefits.\n\n4. **Depth of Algorithmic Detail**  \n   - The paper’s pseudocode for collective setup omits corner-case handling (e.g., SM load imbalance, DSMEM bank conflicts). A deeper dive into these implementation subtleties would aid reproducibility.  \n   - Latency models for DSMEM vs. global memory are presented qualitatively; a formal performance model (e.g., roofline-style) could clarify when the method breaks even.\n\n5. **Clarity in Presentation of Limitations**  \n   - While the authors briefly note DSMEM capacity limits, they do not quantify how often fusion must be “split” across multiple passes or how that impacts latency.\n\nQUESTIONS:\n1. **Cluster Size (k) Selection**  \n   - **Issue:** The paper lacks concrete guidance on choosing the cluster size \\(k\\).  \n   - **Request:** Provide an analytical or discussion for \\(k\\) selection and for figure 11, could you deliver a comprehensive for more different models (e.g., reasoning model)\n\n2. **Experiment Clarify**\n   - **Issue:** It's unclear for FlashInfer [45], Triton [40] and TVM usage.\n   - **Request:** Could you clearly tell me for vLLM, SGlang, etc, what's the actual kernel we are using for each framework. For example, do we use triton-based kernel or we do use torch.compile generated kernel?\n\n3. **FFN analysis**\n   - **Issue** I am confused why author also use these two primitives for MLP.\n   - **Request** Could you give ma an analysis or discussion for using these two primitives for MLP? And what's the potential benefits?\n\n4. **Evaluation on Long Contexts and Scalability to More Models**  \n   - **Issue:** Claims of benefits for long-sequence decoding are not validated on truly large contexts and larger / multi-modality model.  \n   - **Request:** Report latency and accuracy on models handling > 100 K token contexts (e.g., DeepSeek R1, Qwen-Long, Llama-4) to substantiate your scalability claims.  \n    - **Request:** Provide empirical results or detailed performance projections for multi modality or larger model (e.g., llama 3.3 70 B), and discuss any emerging bottlenecks or diminishing returns.  \n   \n5. **Cross-Architecture Portability**  \n   - **Issue:** The approach is demonstrated only on NVIDIA Hopper GPUs.  \n   - **Request:** Discuss architectural assumptions and outline how ClusterReduce/ClusterGather could be implemented on other platforms (e.g., AMD/Intel GPUs or TPUs). If possible, include a small-scale result on one alternative accelerator.\n\nLIMITATIONS:\nYes. There is no potential negative societal impact of this work.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-06"}