Title: ZeCO: Zero-Communication Overhead Sequence Parallelism for Linear Attention
Venue: NeurIPS 2025
Accepted: yes
Keywords: sequence parallelism

context parallelism

linear attention

long context training
Abstract: Linear attention mechanisms deliver significant advantages for Large Language Models (LLMs) by providing linear computational complexity, enabling efficient processing of ultra-long sequences (e.g., 1M context). However, existing Sequence Parallelism (SP) methods, essential for distributing these workloads across devices, become the primary performance bottleneck due to substantial communication overhead. In this paper, we introduce ZeCO (Zero Communication Overhead) sequence parallelism for linear attention models, a new SP method designed to overcome these limitations and achieve practically end-to-end near-linear scalability for long sequence training. For example, training a model with a 1M sequence length across 64 devices using ZeCO takes roughly the same time as training with an 16k sequence on a single device. At the heart of ZeCO lies All-Scan, a novel collective communication primitive. All-Scan provides each SP rank with precisely the initial operator state it requires while maintaining a minimal communication footprint, effectively eliminating communication overhead. Theoretically, we prove the optimaity of ZeCO, showing that it introduces only negligible time and space overhead. Empirically, we compare the communication costs of different sequence parallelism strategies and demonstrate that All-Scan achieves the fastest communication in SP scenarios. Specifically, on 256 GPUs with an 8M sequence length, ZeCO achieves a 60\% speedup compared to the current state-of-the-art (SOTA) SP method. We believe ZeCO establishes a clear path toward efficiently training next-generation LLMs on previously intractable sequence lengths.


Peer reviews:
comments: SUMMARY:
In this paper, the authors propose a new sequence parallel method for linear attention. While there are a lot of sequence parallel methods for full attention, linear attention has less computation and therefore communication will be a bottleneck. The proposed ZeCO algorithm specifically optimizes for communications. The method is proved both in theory as well as in practice.

STRENGTHS AND WEAKNESSES:
Strengths:
1. The paper is well written.
2. The comparison w/ LASP and LASP2 shows advantage of the proposal method

Weaknesses:
1. Could you provide a loss comparison before and after applying ZeCO to show that there is no numerical issue when applying the proposal method?
2. Could you provide a profile to show how the overlapping of intra-device computation and inter-device communication look like in practive? Sometimes, the overhead of the overlapping itself may be larger than the benefit of overlapping, so would like to double check to make sure the communication and computation takes similar time and therefore overlapping makes sense.

QUESTIONS:
1. What is the DP performance (i.e. fsdp, zero) in figure 1?
2. What is a full attention performance (i.e.  Ulysse) in figure 1?
3. Is the proposal method easily applied to VLM (vision language model)?

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-06-30

comments: SUMMARY:
The paper introduces a novel method called ZeCO to address the communication bottlenecks in sequence parallelism (SP) for linear attention models, particularly relevant for training large language models (LLMs) on ultra-long sequences (e.g., 1M tokens). Contributions: 1. Enables minimal communication. 2. achieves zero communication overhead and minimal extra computation/I/O.

STRENGTHS AND WEAKNESSES:
Strengths:
1.  The paper rigorously proves the communication optimality of the proposed ZeCO approach, showing it achieves the theoretical lower bound in communication volume and latency.
2. Experiments are run at scale and compare ZeCO against strong baseline.

Weaknesses:
1. it's unclear how well the performance improvements generalize to other linear attention variants.
2. The method doesn’t generalize to full attention or hybrid approaches, Besides the accuracy of linear attention is typically bad.

QUESTIONS:
1. My concern is that linear attention typically will lead to obvious accuracy drop. So can authors explain the impacts on model accuracy?   
2. Can authors explain how well the performance improvements when this method is generalized to other linear attention variants?

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 3
IMPACT: 2
ORIGINALITY: 2
DATE: 2025-07-01

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces ZeCO, a sequence parallelism (SP) method that sharply reduces communication bottlenecks in training linear attention models on long input sequences. ZeCO is built around a new communication primitive called All-Scan, which uses pipelined communication to reduce latency when sending state information from a predecessor to a successor devices. They show—empirically and theoretically—that this achieves a near-optimal communication volume that hides communication latency by overlapping with local computation. They demonstrate strong empirical runtime performance on 1B-scale linear attention models in compared to other methods.\n\nSTRENGTHS AND WEAKNESSES:\n# Strengths\n\nA key concern about linear attention models is their amenability to modern hardware. This work represents a novel systems-level contribution resolves a key communication bottleneck, making it easier to scale such models to longer contexts. \n\nThe paper is well-structured, and ideas are presented in a clear and concise manner. Figure 2 is a great visualization of the approach and its advantages. \n\nThe empirical results are strong and demonstrate compelling scaling behavior, benchmarked against alternative approaches. \n\n# Weaknesses\n\nThe approach is limited in scope to linear attention and focuses on GLA models in particular. Can something similar to ZeCO be adapted to quadratic attention models? Are there common subquadratic transformers that cannot be implemented using ZeCO?  \n\nCode is not included in the submission. What is the timeline for open-sourcing the code? \n\nIn the optimality analysis, I would recommend including stating your assumptions (e.g. which costs are assumed to be negligible) more directly at the start of the analysis. \n\nMinor points:\n- L15: “optimaity” —> “optimality”\n- Algorithm 1: “comunication” —> “communication”\n\nQUESTIONS:\nAre there other applications for All-Scan beyond this paper?\n\nLIMITATIONS:\nI would recommend that the authors add an explicit \"Limitations\" section where they can summarize their assumptions and hardware constraints.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 1, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces ZeCO (Zero Communication Overhead), a sequence parallelism (SP) method designed to overcome communication bottlenecks in training large language models (LLMs) with linear attention mechanisms for very long sequences. Existing SP methods for linear attention suffer from substantial communication overhead, hindering efficient scaling across multiple devices. To address thiz challenge, the authors propose \"All-Scan,\" a new collective communication primitive that provides each SP rank with the necessary initial operator state while maintaining a minimal communication footprint, effectively eliminating communication overhead. By leveraging All-Scan, ZeCO provides nearly optimal performance by introducing negligible time and space overhead. By using experimental evaluation, the authors show that ZeCO achieves a 60% speedup over state-of-the-art SP methods on 256 GPUs for an 8M sequence length, and exhibiting near-linear scalability for long sequence training.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1) ZeCO algorithm is intuitive, leverages the characteristic of computation to effectively mask communication time and, importantly, shows superior scaling to long sequences as compared to existing LASP* methods up to 256 GPUs. \n\nWeaknesses:\n1) The paper focusses on linear attention models which have not been very successful for most NLP tasks. This limits the impact of this work. \n\n2) I am confused about the \"optimality\" claim:\n2.1)  In order for ZeCO to be \"optimal\" one would have to prove that any algorithm taking less time than ZeCO would not be able to achieve the required computation (lower-bound proofs). The presented proof doesn't prove this. You might want to reconsider the use of term \"optimal\" here.\n2.2) Even if optimality is used more loosely, I am still not sure about the presented arguments. More specifically \n- Line 191. T_{ideal-SP) and t_{overlaped_comp} are key parameters of the algorithm. How does ZeCO make sure that it provides the best choice for overlaped_comp? This is not clear from the writing.\n- Line 203-205. when K becomes sufficiently large, the overhead approaches zero. I am not sure about this; since K is upper bounded by $d_k$, in the best case T_{ALL_SCAN} is T_{ALL_SCAN} = \\tau(d_k \\times d_v) + (p-1) \\tau(d_v). For large values of $p$ (which is the focus of this paper is all about), the second term is not negligible; in fact, it can become larger than the first term.\n- Line 212. Saying T_extra_comp&IO is negligible is fine, but  this does not imply that ZeCO has made the best possible choices.\n- One last point: there is an additional (redundant computation) in ALL_SCAN which I did not find being discussed in the paper. \n\n3) Experiments are limited to single sized model. what happens to in larger models (7B, 13B). Do we see similar scaling.\n\nWriting suggestions: \n- Please use standard mathematical notation to improve readability, i.e.., lowercase italic letters for scalars, lowercase bold letters for vectors, and uppercase bold letters for matrices.\n- Please make another editing pass. A few examples:\n* Line 15: optimality spelling error\n* Line 40: this is not paradoxical\n* Line 125: \"G\"lobal\n* Line 134: \"S\"tate\n* Line 236: \"R\"untime\n\nQUESTIONS:\n1) How many GPUs can ZeCO scale to while maintaining near-linear throughput scaling for a 1B model? \n2) Can the global state be divided along both $d_k$ and $d_v$ dimensions ?\n3) How does ZeCO scale with larger models? While the authors state that \"additional number of parameters does not affect how cost is calculated\", the dimensions will differ for larger models. It would be beneficial to see comparable results for larger models to confirm expected scaling behavior.\n4) Why is a comparison with full attention SP methods meaningful, given their inherently different workloads? (Referring to Figure 3) \n5) Can ZeCO be extended to training hybrid attention models, such as Jamba?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-09"}