ZeCO: Zero-Communication Overhead Sequence Parallelism for Linear Attention
Authors: Yuhong Chou
Zehao Liu
Rui-Jie Zhu
Xinyi Wan
Tianjian Li
Congying Chu
Qian Liu
Jibin Wu
Zejun MA
Venue: NeurIPS 2025 poster ()
Keywords: sequence parallelism
context parallelism
linear attention
long context training
Primary area: infrastructure
TL;DR: A SoTA sequence parallelism for linear attention with a brand new collective communication.

Abstract: Linear attention mechanisms deliver significant advantages for Large Language Models (LLMs) by providing linear computational complexity, enabling efficient processing of ultra-long sequences (e.g., 1M context). However, existing Sequence Parallelism (SP) methods, essential for distributing these workloads across devices, become the primary performance bottleneck due to substantial communication overhead. In this paper, we introduce ZeCO (Zero Communication Overhead) sequence parallelism for linear attention models, a new SP method designed to overcome these limitations and achieve practically end-to-end near-linear scalability for long sequence training. For example, training a model with a 1M sequence length across 64 devices using ZeCO takes roughly the same time as training with an 16k sequence on a single device. At the heart of ZeCO lies All-Scan, a novel collective communication primitive. All-Scan provides each SP rank with precisely the initial operator state it requires while maintaining a minimal communication footprint, effectively eliminating communication overhead. Theoretically, we prove the optimaity of ZeCO, showing that it introduces only negligible time and space overhead. Empirically, we compare the communication costs of different sequence parallelism strategies and demonstrate that All-Scan achieves the fastest communication in SP scenarios. Specifically, on 256 GPUs with an 8M sequence length, ZeCO achieves a 60\% speedup compared to the current state-of-the-art (SOTA) SP method. We believe ZeCO establishes a clear path toward efficiently training next-generation LLMs on previously intractable sequence lengths.

ZeCO: Zero Communication Overhead Sequence Parallelism for Linear Attention
Yuhong Chou ; Liu' Ruijie Zhu?, Xinyi Wan", Tianjian Li², Congying Chu², Qian Liu?! Jibin Wu't, Zejun Ma?
The Hong Kong Polytechnic University
TTkkTok
UC Santa Cruz
NNaional University of Singapore
Institute of Automation, Chinese Academy of Sciences
Abstract
Linear attention mechanisms deliver significant advantages for Large Language Models (LLMs) by providing linear computational complexity, enabling efficient processing of ultra-long sequences (e.g.. IM context). However, existing Sequence Parallelism (SP) methods, essential for distributing these workloads across devices, become the primary bottleneck due to substantial communication overhead. this paper, we introduce ZeCO (Zero Communication Overhead) sequence paral- lelism for linear attention models, a new SP method designed overcome these limitations and achieve end-to-end near-linear scalability for long sequence train- ing. For example, training model with using ZeCO takes roughly the same time training with an 16k sequence on a single device. At the heart ZeCO lies All-Scan, a new collective communication primitive. All-Scan provides each SP rank with precisely the initial operator state it requires while maintaining a minimal communication footprint, effectively climi- nating communication overhead. Theoretically, showing that introduces only negligible time and space overhead. Empirically, compare the communication costs of different sequence parallelism strategies and demonstrate that All-Scan achieves the fastest communication in SP scenarios. Specifically, 256 devices with 8M sequence length, ZeCO achieves 60% speedup compared to the current state-of-the-ar (SOTA) SP method. We believe ZeCO establishes clear path toward efficiently training next-generation LLMs previously intractable sequence lengths.
sequence length across 64 devices
prove the optimaity of 7eCO,
Introduction
Long-context capabilities are becoming increasingly critical for Large Language Models (LLMs), powering advancements in document-level reasoning, multimodal understanding, and retrieval-
augmented generation where extensive context is paramount [I 2 3 5 6). The trajectory from models like GPT-3.5 (4K context) (7) to Gemini 1.5 Pro [8]
(IM context) highlights this trend. However, pre-training models such vast sequence lengths presents significant computational and communication challenges. Standard self-attention mechanisms exhibit quadratic complexity (O(T.)) with respect sequence length T.. Scaling from 4k to 128k tokens, for instance, inflates the attention FLOPs by over 1000x. The prohibitive computational cost of training on ultra-long sequences restricts the long-context pretraining a specialized adaptation phase (i.e., mid-training) (9] rather
than allowing for extensive training with long sequences from scratch [10 11
15
Equal contribution. Emails: yuhong.chov chou @ connect, polyuu.hk zehaoliu @ polyu.edu.hk. Corresponding authors: qian. tiktok. com; wibin.wu@ppn wwupooly..du....
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Sequence Parallel (SP) Methods: Throughput Scaling
LASP2 (SOTA SP) v. ZeCo (Ours)
1.00
(90TA SF)
1.6X
Throughout ('oeent Seccono)
(Cora)
30.8
40.9K
Liner Scolinng Oloper Bound)
25.4k
3.9X
200g
9.3X
LASPI (Boseine
3.3
Communicction Time (MS)
Runtime Extra (mS)
Throughout / GPU (tokens second;
16
64
256
L
Numbar of CPUs (lgg sca.e)
Figure 1: (Left) ZeCO demonstrates near-linear scaling efficiency when scaling sequence length pro- portionally with device count, approaching the theoretical upper bound. (Right) ZeCO substantially outperforms SOTA SP methods across three performance metrics: communication time, runtime extra cost, and per-GPU throughput. Metrics obtained using 256 GPUs (8M sequences) for communication throughput and 128 GPUs (4M sequences) for runtime extra cost.
Linear attention models [6 offer algorithmic solution by replacing the O(L2) softmax attention with operations linear in sequence length, typically OLL)) where d is the hidden dimension. The efficiency is achieved by compressing the Key-Value (KV) cache, which would otherwise grow with sequence length, into a fixed-size hidden state representation. This method effectively eliminates the computational bottleneck of quadratic complexity and balances the per-token computation across the sequence, enabling efficient processing of ultra-long sequences. While linear attention provides these algorithmic advantages, Sequence Parallelism (SP), essential for distributing such computationally intensive workloads, paradoxically becomes bottleneck that impedes efficient often hampered by issues such as serial execution dependencies or, more significantly, substantial communication overheads that scale poorly. Consequently, despite the inherent efficiency of linear attention, the practical implementation of SP, particularly its communication burden, has become the primary impediment to achieving high throughput and true scalability for long-context training.
scaling across multiple devices. Existing approaches [17 18 T9
In this paper, we introduce ZeCO, novel sequence parallelism strategy designed to overcome the limitations of current methods, particularly for linear attention. ZeCO achieves SOTA scalability through a fundamental redesign of its communication algorithm and catptuuiiiattincompuuuuit scheduling. At the heart ZeCO lies All-Scan, new collective operator executing pipelined receive-scan-send pattern across devices. This new communication primitive achieves the theoret- ically minimal communication volume for SP. Crucially, empirical results confirm that All-Scan consistently delivers the lowest latency among all existing SP communication techniques. To further minimize communication overhead, ZeCO intelligently schedules All-Scan to overlap with local device computation, thereby enabling parallel use of both communication and computation resources. Moreover, ZeCO meticulously optimizes the auxiliary computations inherent in SP, effectively reduc- ing their associated I/O and computational overheads negligible level. In summary, our main contributions can be summarized as follows.
1. We introduce ZeCO, novel sequence parallelism method for linear attention models. ZeCO reformulates sequence parallelism by leveraging our All-Scan collective communication, which employs pipelined communication achieve the theoretically minimum communi- cation volume. This integrated approach enables efficient overlap of communication and computation, incurring minimal extra computational and I/O overhead.
2. We provide a comparative time-cost analysis of different sequence parallelism strategies demonstrates that ZeCO constitutes the minimum required cost, establishing its efficiency.
3. Comprehensive multi-level experiments (collective communication, operator, and model) demonstrate the significant performance gains of ZeCO. As shown in Figure I the All- Scan collective achieves up to 3.9x communication speedup, the fastest existing sequence parallelism method, while the ZeCO sequence parallel operator delivers up to 9.3x overall speedup. At the model level, ZeCO boosts throughput by over 60% and demonstrates near-linear scalability from 256 devices, even with context lengths up to 8M tokens.

---

2 Background & Related Work
In this section, we firstly provide a brief background on Gated Linear Attention (GLA), a general linear attention operator that encompasses a family of linear attention mechanisms [20] 21 22 23
24 25].
Then, we introduce the existing sequence parallelism methods.
We use bold upper-case letters (e.g., Q) to denote matrices, And the same alphabet to represent rows of matrix, such that Q₁ refers the t-th row of Q. Unless otherwise specified, p denotes the number of devices, L denotes the sequence length per device, d denotes the hidden dimension, denotes the number of attention heads, and denotes the chunk length.
2.1 Recurrent and Chunk-wise Form of General Linear Attention
The linear attention mechanism uses the kernel trick to remove the softmax computation in full attention and exchange the calculation order to reduce the attention computational complexity from quadratic linear [16 26). There are many ways implement this mechanism. Sevaral works (27 22 28) has summarized unified form of (diagonal decay) linear model. In this paper, we use gated linear attention (GLA) operator (27), one the generalization forms of linear models, to demonstrate our algorithm. The attention state is updated recurrently as:
S, = (a S + K, V. O, Q.S..
where a, E (0, 1)dh is decay factor: To enable efficient parallelism during training, the sequence is partitioned into N chunks of length C, and the recurrence is reformulated in chunkwise manner. Let chunk include tokens from iC to (i + 1)C - 1, with decay vectors 0; Let S E Raxd be the chunk-level hidden state after processing chunks, i.e., Stj] = Sic- GLA define, Cumulative
decay for the chunk: Mn = DiC+j. Tosen-wise scaling: Tic+j b.c
Ac++j
b.c
where b, IT as. The chunk-level GLA state and output are calculated
S = (7 1) e S + (K() (V)))
(2)
The output of each chunk should be computed as:
[
(4) Sp-11 + [((Q o Ag)) (K (H)) M]
Vij.
Oiseer
[c]
the inter-chunk recurrently updates the global state, while the intra-chunk term handles mask attention computation on the diagonal.
2.2 Sequence Parallelism for Linear Attention Models
LASP1 [17] adopts a chunkwise parallelization strategy by dividing the input sequence into multiple contiguous chunks and evenly distributing them across devices. Each device serially computes the output of each device based on a linear attention formula. For communication, each device receives the state from the previous block and updates it before passing it to the next device. Although this avoids redundant communication volume, it enforces a strict serial order to be executed across devices, causing the total computation time to grow linearly with the number of devices, which severely limits parallel efficiency and throughput.
LASP2 [88 19] follows similar chunkwise computation structure, but replaces the serial state passing with All-Gather communication. Each device must first collect the local states from all other devices and subsequently perform identical scan operation on the same data. This enables devices to do computation parallelism, but introduces substantial communication overhead. The total communication volume grows linearly with the number of devices, each device must collect state tensors from all others.
2.3 Sequence Parallelism for Full Attention
The sequence parallelism in Megatron (29) (also known as Context Parallelism) and Ring Attention (30 achieve global synchronization of KV blocks in either an all-at-once or pipelined manner, based on All Gather and P2P communication respectively, and have been widely adopted (10 [8]. Ulysses (32 distributes the computation of different self-attention heads across devices, which is easy to implement but incompatible with tensor parallelism (TP) and limited by the number of heads.
Ring Self Attention
33 was the earliest method to propose full attention sequence parallelism, but
3

---

Tima
SiM
Sfvj
Six]
S/yv
Device
S/M
8%y
S2
S/n
Device
Device
8%
S/M
57y
9%y
S/x
S
Device
88
%MM
[4
%/x
Device
LASP2
(ours)
.2
92
9g
8/y
s2
S2
P3P Comm
All-Scan-Comm
Al-Gather Comm
Linear-Attn Comp
8):
S2.
S/v)
Device
Algorithm Comparison
53n
Device
8/
9/M
85y]
87y
9/M
86x]
8/1]
All-Scan
(2)
sg
82
Device
Spd)
%
8%
8/
83.
8%
8)
8%
8%
P)o
8
550
8/.
Device
Local State Update
Output Comp
82
1/2
874)
8%y
8/2
8/.
8
Device
Diag Attention
Al-Scan-Comm
(5)
ZeCO Computation
All-Scan Communication
Figure 2: Illustration of 7eCO. 7eCO highlights its strengths in three dimensions: (1) Parallel Scalability: achieving efficiency comparable to DP (sub-figure: Algorithm Comparison); (2) Operator- Level Computation: enabling overlap of communication and local computation for maximal resource utilization (sub-figure: ZeCO Computation); and (3) Communication Pattern: utilizing customized pipelined All-Scan Communication pattern substantially reduce inter-device synchronization delays. (sub-figure: All-Scan Communication)
which limits its applicability. Nevertheless, sequence parallelism for full attention is fundamentally constrained by the self-attention algorithm itself: even disregarding communication, the computational cost becomes prohibitively expensive for ultra-long sequences due the algorithm's inherent complexity.
does not leverage the 1/Oeefficient optimizations of self-attention (34 35 36 37
Method
3.1 ZeCO Sequence Parallel Methods and Communication Requirements
Let each device be assigned sequence of length L, which is partitioned into N = L/C. After projec- tion of imput X IR fxd partitioned into N L/C, we get non- non-overlapping chunks Q [z) Gin] for N. We use S,, E PL denote the Global states, and [n) E N denote local chunk states on each device.
Local State Computation. According to Equation the local states starting from the initial state [0)
2
Within each device, we sequentially compute
O:
Siri (T Sin-1] K
[x].
for L...,N
Compared to GLA, we additionally maintain the cumulative decay vector 7mj- which saves the total multiplicative decay from the first to the nith chunk:
[/]
II Mpq 1=0 for = n= N.
(5)
At the end of this local recurrence, obtain list of local chunk states {S Stj ... ...S
Global State Update. The recurrence for the global state defined as Equation IT. To obtain the series of global states of the device p, the device p must get the last global state 1) from device

---

Algorithm 1 Forward pass for ZeCO with All-Scan comunication
1: Note: The Highlighted part represents the SP adaptation of GLA Algorithm by ZeCO algorithm, and the lower cost of the red part represents the lower extra cost of SP.
2: Input: Q. K, E Rixdd, V E R Lxds G = [=].....aL)) E IR L.xd, chunk size C, num device P. device rank p E {0, 1, ...., 1
3: Divide Q K, G into N E blocks Opjj..Quvv [.. (Gp))..GGvv] of size C dR
each. Divide V into N blocks {V pn... Vin] J of size C' dg each.
4: Initialize $
Rd.
1 E IR SRAM
5: Write 7, 7,S to HBM as T(0) S(o)-
6: for < 0, NN do
7: Load [m]. G (m). V [m] from HBM to SRAM.
8: On chip, compute [n] E IR ,I [(n)] Cxd, and K1n) [in)] 2 (.(j 7 mm)
9:
Write y to HBM [nn.
On chip, compute
(7mm OS
m) [m
11: Write S to HBM S|n]
12: end for
13: In parallel do:
14: parallel stream 1:
15: (p-1yL Spt All-Scan(S(N]] 7(1)
16: parallel stream 2:
17: parfor n < 1, N do
18:
Load [n) Kmjj Giaj E Cxda from HBM to SRAM.
19:
On chip, construct causal mask M E BCxC
20: On chip, compute (n), E Cxde Qp1l Q/1
(n),
Kin)/An
21: On chip, compute P = (Q K [n] O MM RCxC
22: Write P as P11 to HBM.
23:
end parfor
24: stream barrier
for + N do
26:
Load Q [n]. [m) V [n] S (p- I)r,, Sin Mn- 1], P from HBM to SRAM.
27:
On chip, compute [n]
28:
On chip, compute Q(n] ((m) @ [nn
On chip, compute O inneer Q m] 1n-1 + 11 O 1(p) inim PV [n] gxdd,
[n]
30:
On chip, compute O [m) O inter
+ O intra
31: Store Op) to HBM.
32: end for
33: return O (Opp... = I Spi Sp
p- 1. Then update the global State of the current device by applying:
S (5 1) O ((p-1)1 + Sjj]-
(6)
We give a proof of the above global update computation in Appendix A. Since updating each local state jn) its global state S.p 1p independent across chunks, we can first update S/v to the last global state Spr. of device p. and send it to enable the global state update in p + device.
To fulfill this communication requirement, operator in Section 3.2 All-Scan Communication is overlapped with the local computations that do depend on communication; in practice, we compute the diagonal attention scores simultancously as shown in Figure All-Scan allows ZeCO to parallelize both inter-device communication and intra- device computation. In implementation, ZeCO rearranges the standard form of GLA with minimal extra computation and I/O cost, so as to achieve efficient sequential parallel training Algorithm [
propose the All-Scan Collective Communication
3.2 AlI-Scan Collective Communication
To convert local chunk states in each device into globally consistent values, each device p requires the final state S(p-1) from its predecessor. It presents dependency chain between devices, which
5

---

Algorithm 2 All-Scan Algorithm
1: Input: num jen__devieeeee P, device_rank [P]. Local State Siocal, factor 7, the direction tag DIR
2: if DIR = FWD then
3: send_rank = p + 1, reCe rank p for device p, start 0, last = P p-1
4: else if DIR = BWD then
5:
compute send_rank p 1, recu recu_rank for device start = P 1, last = 0 P Single Direction Communication
6: end if
7: Initialize the send state S
8: if p is not O then
Receive state from recv_rand Sreces 10: end if
9:
11: Slice Srece, Sandd, Stocat, alone the first dimension in K blocks.
12: for k <- 0,
do if is start then
13:
14:
Send Stocal to recu_rank else is not then
16:
scad = = + (7 1) x Sk
17:
Send Stocal to recu_rank
else is last then
19:
Stend = St + (N)) 1) x S reov
20:
end if
end for
22:
return Srevv: Syend
will cause a communication latency related to the number of devices. To address this efficiently, we propose an All-Scan Collective Communication strategy to receive, update, and send. Specifically, All-Scan splits large state tensors into smaller segments that can be sequentially transmitted and processed.
Pipelined State Scan. Rather than receive the full state ( 1)z, we partition it along the da dimension into K contiguous blocks send from device p
S (p-11) = C (2) (2)
(())
R)) 1)4
((k) (p-1)L E IR # xd.
Correspondingly, the decay factor is split into aligned segments [N] E 1 Each block of state transmitted pipelined from 1 to p. and immediately applies the update and send:
(i)
(A) s(1) = [N]
1) +
[N]
for h k= = K
(8)
This design enables device + 1 to begin updating its last global state 1(D+ in All-Scan as soon receives the first block of Spt., as shown Algorithm As a Communication Primitive, All-Scan could run independently with other CUDA stream, achieves fine-grained communication-compuaaion overlap, maximizing device utilization and throughput long-context training.
3.3 Efficiency Analysis
We now formally analyze the time cost of sequence parallelism strategies and show that using the All-Scan collective communication algorithm yields the most efficient design for linear attention SP under our analytical framework. We identify two key conditions that characterize such efficiency and demonstrate that ZeCO satisfies both.
1. Zero Communication Overhead: Each device transmits and receives only the minimal essential size of information (data). No redundant communication.
2. Low Extra Cost: Communication is overlapped with other computations much as possible to minimize idle computational resources. Furthermore, the additional computation and I/O overhead introduced by SP is reduced a minimum.

---

Let S E R xd. denote the accumulated state tensor. According to linear attention output Equation this state represents the minimal information that must be communicated between chunks.
Zero Communication Overhead
For sequence distributed across P devices, any SP algorithm must communicate at least the state information across device boundaries. Let V/2co denote the communication volume of the p-th device in ZeCO (All-Scan) Each device sends a last global state to the next device exactly once, resulting in:
V
(x) ZeCO
S
dg x de
This represents the theoretical lower bound. However, existing approaches such as LASP-2 rely all- gather operations, receive local states from all the other P - 1 devices, resulting in a communication volume of (P - 1) x dy x d,, which increases with the number of devices.
Which grows linearly with P. As shown in Figure 3 ZeCO achieves the minimal communication
volume possible for the SP scenario.
Idealized SP strategy Let (T.) denote the total runtime for processing sequence of length I, using P devices with sequence parallelism, and let (PT) represent the runtime for processing a total sequence of length PL on a single device. Let Let Seeat-spPPP,, denote the runtime under ideal conditions, assuming perfect parallelism with zero additional overhead. For an ideal SP, the following properties should be satisfied:
T:1 ideal-sp (PL) = Ti (L) Lideal-sp(PL) (PLL
Thleatop
(10)
P
The implication of Equation (10) is that, in the ideal case. the throughput of sequence parallelism should scale linearly with the number of devices (i.e., the processing time inversely proportional to the throughput). In practice, however, sequence parallelism introduces additional overhead. Therefore, we next analyze the latency under practical scenarios.
In practice, the prerequisite communication latency, computation, and I/O for transferring and synchronizing data between devices introduce additional latency. For ZeCO (and other sequence parallelism), the relation becomes ideal sequence parallelism cost + extra cost, which can be formularized as follow:
(PL) = TV TPdeat-P((P))) + Textr_ cotto_com + (T Slan (overlpeedcoomp
(11)
For ZeCO, the first two components are independent of communication. The last two represent communication latency, accounting for the portion of the All-Scan operator that cannot be overlapped by local diagonal attention (we suppose the worst-case scenario). This can also be viewed as the gap relative the ideal SP. From Equation (II), we can see that Ta spaa-spl (PL) and Toverlnped_com ccmm inherent to the algorithm. Therefore, the key question in SP is to what extent the additional time cost of linear attention, namely Textra_compp eestra_com&&I0 and TAIL Scan, can be reduced. The following presents how ZeCO achieves its efficiency through an analysis of the All-Scan communication strategy, and proves that the additional computation and I/O overhead is negligible.
For ZeCO, the first two components are independent of communication, while the last two correspond saanec--seeifillllly. the portion of the All-Scan operation that cannot be fully
to communication overlapped by local diagonal attention (we consider the worst-case scenario). This latency can also be viewed as the residual gap relative to the ideal SP case. From Equation (T)). Ti PT.) and Teeelupedd comp are intrinsic to the algorithm design. Hence, the central question in SP is to what comp&1/O
extent the additional costs of linear attention, namely T T and T All_Scaan can be minimized. In the following, we show that ZeCO achieves near-optimal efficiency by analyzing the All-Scan communication behavior, and we further demonstrate that cntrr__comptaaion&IJ@@ constitutes negligible portion of the overall system cost.
In All-scan, we partition S into K blocks, each of size R t xd. and transmit these blocks in pipelined fashion as shown in Figure 2 By partitioning the state S into K blocks and updating them pipeline, the effective communication latency could be computed as:
T AlL Sean =(dx x d.)) + (P- l) d.)
(12)
K
7

---

Comm Volume per GPU (MB. sqit scale)
Comp. Speed (lines) (bars) Count
Communication
130
100
Gathen
yoa))
50
4.0-
Time
3.0
Comp. Spoad 11 Roppi
1.5
PURLARD
0.5
16
32
128
256
16
64
256
Number GPUs
Number GPUs Ipl
Figure 3: ZeCO has the lowest communication time while satisfying the lowest communication volume. The left figures show the theoretical values of the algorithm calculation speed and communication volume, and the right figure shows the actual communication time.
where T(-) represents the time required to communicate tensor of the given size. Equation shows the two components of the cost TAIL Scan The first term represents the overhead that can be parallelized by the pipelined approach, which is necessary and corresponds to the minimum communication requirement. The second term accounts for the overhead at the boundaries. As K increases, the boundary overhead decrease, and the degree of overlap improves. Consequently, when K becomes sufficiently large, the boundary overhead approaches zero. Thus, ZeCO with All-Scan achieves the minimal time cost of communication.
The term Textra comp&to consists of two parts: a small number of additional floating point operations, and HBM load and store operations for few auxiliary tensors. In Algorithm the load and store operations for 7/1 in lines 9 and 26 are vector, constituting only 1 of the state tensor. The required d.
additional state can be reused N times, incurring just y overhead. For sequence length of 8192 and a chunk size of 64 (N = 128), which comparable to typical d,. the added overhead less than 1%. The cost of element- wise multiplications is negligible. Hence, Texta_compa comp&10 can be safely ignored in practice. proved that the time of ZeCO Equation
11 should be:
(PL) (PL) Tideat-sp (L) T overriuped_comm + (dk x de) +
(13)
T.1 Sp ( (L) T overlaped comp + (d2 x do),
(14)
where represents negligible computation and I/O cost.
In contrast, existing methods like LASP have strictly serial dependency across devices, resulting in (We assume that the T Textra comp&t/o term other methods can also be optimized to a negligible level. Even so, these methods remain suboptimal. ):
TiAS (PL) = P (Ti + r(dx x do)) TP (PL).
(15)
While LASP-2 improves on LASP with parallel computation, but suffers from higher communication cost:
TeASP Tieat (L 7(de d.) (PL).
(16)
Thus, ZeCO effectively eliminates redundant communication overhead and establishes an efficient
sequence parallelism strategy with minimal extra cost. This efficiency directly translates into superior performance. In F we show the theoretical values communication cost and computational overhead, and the actual values of communication time for different SP algorithms. We also present unified communication and runtime analysis of existing SP algorithm in Appendix A.2
Experiments
We evaluate the efficiency and scalability the proposed 7eCO SP Algorithm and All-Scan Commu- nication Operator IB-GLA models. Our assessment focuses on two aspects: (1) Communication

---

Algorithm Runtime Cost (16k)
Algorithm Runtime Cost (32k)
200
LASP1
LASP1
Time (ms)
LASP2
LASP2
150
ZeCO(Ours)
ZeCO(Ours)
100
DP
128
8
32
64
128
Number of GPUs
Number of GPUs
Throughput (Tokens/sec)
Model throughput (1B-16k)
Model throughput (1B-32k)
107
LASP1
LASP1
LASP2
LASP2
ZeCO\Ours)
ZeCO(Ours)
105
16
128
256
16
128
Number of GPUs
Number of GPUs
Figure 4: Scalability evaluation LASP on SP operator runtime(top half) and Scalability evaluation of LASP Throughput(botton half). In the comparison test of 16k and 32k sequence length per GPU, 7eCO algorithm shows the same stable time as the DP algorithm. In both 16k and 32k, ZeCO exhibits linear scaling curve of throughput growth approach to DP, while the other methods degenerate.
speed of different Collective Communication Operators; (2) The Algorithm-level and model-level scalability under increasing GPU count.
experiments are conducted on a GPU cluster equipped with 256xH100 80GB GPUs. Model is trained in Lingua (38). TTrch-bbase distributed training. We implement the All-Scan com- munication algorithm using the Triton-Dii iirton--ittrbbtedd framework, which integrates OpenSHMEM into the Triton compiler to enable distributed communication within operator implementations 39 40). To ensure a fair comparison with baseline sequence parallelism (SP) methods such as LASPT 17 and LASP2 [88, adapt the chunk-wise gated linear attention operator from the Flash Linear Attention 41 repository for our implementation. The complete experimental setup and data are provided in the Appendix A.3
4.1 Communication Speed
In this experiment, evaluate the communication Runtime of different communication operators under their own communication workload sufficient for correct training. Experiments are conducted with P from 8 to 256 GPUs, and each GPU is assigned 8K sequence length.
We warm up each communication kemeel for 5 rounds, then report the average over 50 runs. More details of communication workloads and protocol differences are discussed in Appendix A.2
As shown in Figure 3 memory-out occurred in the experiments of 128 GPUs and 256 GPUs All- gather (Megatron). For other methods, it should be noted that, for presentation purposes, the upper half of the Y-axis represents the rendering results after taking the log scale. All-Scan significantly outperforms other methods different scales of clusters. Notably, on 256 GPUs, All-Gather (LASP2) is 4x slower than All-Scan.
4.2 SP Algorithm Runtime and Model Throughput
Next, we evaluate both micro-level (algorithm) and macro-level (model training) performance for Linear Attention methods, including LASPI, LASP2, and ZeCO.
SP Algorithm Runtime We measure the forward and backward pass time of each SP operator under the same setting (T. 16K 32K, H 16) and compare against the ideal case of DP operator. The time of the DP operator serves as the theoretical lower bound.

---

Figure 4 demonstrates that in the 128 GPUs experiment (2M and 4M sequence length), ZeCO is only 3 ms slower than the theoretical lower bound for single forward and backward pass, which satisfies our analysis in Section 3.3 and demonstrates the efficiency of our algorithm.
Model Throughput We experimented with IB-GLA models with different sequence parallel
methods to test the training throughput, under the same setting (L 16K 32K, H 16) and compare it against the ideal case a Model that uses DP. The throughput of the GLA model uses DP in training, serves the theoretical upper bound.
For the result shown in Figure as the number of GPUs increases, ZeCO achieves a linear increase in total throughpur, which meets the original intention of sequence parallelism, while other methods experience a serious degradation.
5 Conclusion and Future Works
In this work, we propose ZeCO sequence parallelism for linear attention, achieving SOTA for both theoretical and empirical results. More importantly. our method fully unleashes the algorithmic efficiency of linear models and, for the first time, enables near-linear throughput scaling for sequence parallelism. At the system level, approach introduces the novel All-Scan collective communication primitive, which not only underpins the efficiency of ZeCO but also provides a foundational innovation for advancing distributed computing in the linear model community.
In the future, we plan to pursue three main directions. First, we will further improve the algorithmic implementation of the All-Scan collective communication primitive. For example, tree-like imple- mentation. Second, we aim to generalize the sequence parallelism algorithm for linear attention beyond diagonal decay, extending it to support various forms, including matrix transform structures. Third, we will investigate efficient parallel topologies for sequence parallelism in large-scale models.
Acknowledgments and Disclosure of Funding
This work was supported by the Research Grants Council the Hong Kong SAR (Grant No. C5052- 23G, PolyU15217424, PolyU25216423) and, The Hong Kong Polytechnic University (Project IDs: P0043563).
References
[I] OpenAl et al. GPT-4 Technical Report. 2024. arXiv: 2303. 08774 [cs.CL]
URL: https
//arxiv. abs/ 08774
[2] Hugo Touvron et al. "LLaMA: Open and Efficient Foundation Language Models". In: arXiv preprint arXiv:2302. /397/ (2023).
[3] Hugo Touvron et al. "Llama 2: Open foundation and fine-tuned chat models". In: arXiv preprint arXiv:2307.0m 09288 (2023).
[4] Aaron Grattafiori et al. "The llama 3 herd of models". In: arXiv preprint arXiv:2407.21783 (2024).
[5] Gemma Team al. "Gemma: Open models based gemini research and technology". In: (2024).
arXiv preprint arXiv:2403.
[6] Gemma Team et al. "Gemma 2: Improving open language models at practical size". In: arXiv preprint arXiv:2408.00 00/ 18 (2024).
[7] Tom B Brown et al. "Language models are few-shot learners". In: Advances in neural informa- tion processing systems 33 (2020), pp. 1877-1901.
[8] Gemini Team et al. Gemini 1.5: Unlocking multimodal understanding across millions tokens
URL: / org/ abs org/abs/ 2403 of context. 2024. arXiv: 2403. 05530 [cs.CL]
05530
[9]
Marah I Abdin et al. "Phi-4 Technical Report". In: CoRR abs/2412.08905 (2024), DO1: 10
48550/ ARXIV 2412. 08905 arXiv: 2412. 08905
URL: httts: http///o..rgg/1...44/4 4
arXiv. 2412.2412.88999
[10]
Aaron Grattafiori et al. The Llama 3 Herd of Models. 2024. arXiv: 2407.21783
[cs.AI]
URL: https://arxiv org/ abaaabaaaaa /240 21783
[11]
Qwen et al. Qwen2.5 Technical Report. 2025.
arXiv: 2412. 15115 [cs. CL.
URL.: https
//
15115
10

---

[12] Albert Gu and Tri Dao. Mamba: Linear-Tine Sequence Modeling with Selective State Spaces.
2024. arXiv: 2312. 00752 [cs [G]] URL:
[13] Gemma Team et al. "Gemma 3 technical report". In: arXiv preprint arXiv:2503. (2025).
[14] Aixin Liu et al. "Deepseek-v2: A strong, economical, and efficient mixture-of-expertss language
model". In: arXiv preprint arXiv:2405.04434 (2024).
[15] Aixin Liu al. "Deepseek v3 technical report". In: arXiv preprint arXiv:2412.19437 (2024).
[16] Angelos Katharopoulos et al. Transformers are RNNs: Fast Autoregressive Transformers with
Linear Attention. 2020. arXiv: 2006 16236
[cs LG]
URL:
abs/
2006. 16236
[17] Weigao Sun et al. Linear Attention Sequence Parallelism. 2025. arXiv: 2404. [cs. LG] URL: htt::httss:// //axivv org/ abs/ /204 02882
[18] Weigao Sun et al. LASP-2: Rethinking Sequence Parallelism for Linear Attention and Its Hybrid.
2025, arXiv: 2502. 07563 [cs. LG] URL: httpt:ttttt/stttee .wtg//orgg/ge abs www.ab/////
[19] Aonian Li et al. "Minimax-01: Scaling foundation models with lightning attention". In: arXiv preprint arr.iv:2501.00...
(2025).
[20] Zhen Qin et al. "cosformer: Rethinking softmax in attention". In: arXiv preprint
arXiv:2202.0877 (2022).
[21] Yutao Sun et al. "Retentive network: A successor to transformer for large language models". In: arXiv preprint arXiv:2307.0800 (2023).
[22] Zhen Qin et al. "Hgrn2: Gated linear mns with state expansion". In: arXiv preprint
arXiv:2404.05 07904 (2024).
[23] Tri Dao and Albert Gu. "Transformers are ssms: Generalized models and efficient algorithms through structured state space duality".. In: arXiv preprint arXiv:2405.2106 (2024). [24] Songlin Yang, Jan Kautz, and Ali Hatamizadeh. "Gated Delta Networks: Improving Mamba2
with Delta Rule". In: arXiv preprint arXiv:2412.0644 (2024).
[25] Songlin Yang et al. "Parallelizing linear transformers with the delta rule over sequence length".
In: arXiv preprint arXiv:2406. 06444 (2024).
[26] Krzysztof Choromanski et al. Rethinking Attention with Performers. 2022. arXiv:
200XXXi:20000.00. 2009. 14794
[cs LG] URL: https: http://axivv.v. org
http:///aaaaiii.rrr/bbb/////// http:/aaas///0000.2 14794.
[27] Songlin Yang et al. Gated Linear Attention Transformers with Hardware- Training.
2024. arXiv: 2312. 0663566666666 [ca. LG] URL: htt.:/ttpp://aaaa..oog//bb/////a wbwpa/////2..6.....
[28] Yuhong Chou et al. MetalA: Unified Optimal Linear Approximation to Softmax Attention Map.
[29] Mohammad Shoeybi et al. Megatronn-LM, LM: Training httpps////rxxv.oorr/////111111 htsp/bbb/24111.... 10741
2024. Xiv: 2411 10741 LG] URL: https: //ar:v/.//xivvvva.m wttg/bs.//aae..e
Parameter Language Models
DaraguterrLanuuaee
Using Model Parallelism. 2020. arXiv: ww..999000000 CL] URI.: httpthttts/////iie org/
abs/ / 1909. 08053
[30] Hao Liu, Matei Zaharia, and Pieter Abbeel. "Ring attention with blockwise transformers for near- infininf context". In: arXiv preprint arXiv:2310.014 (2023).
[31] William Brandon et al. "Striped attention: Faster ring attention for causal transformers". In:
arXiv preprint arXiv:2311.0945 (2023).
[32] Sam Ade Jacobs et al. DeepSpeed Ulysses: System Optimizations for Enabling Training of Extreme Long Sequence Transformer Models. 2023. arXiv: 2309. 14509 [cs. LG] URL:
https://arxia: wwwp///g//bbb/ooeeeeeeeeeeeg///e 14509
[33] Shenggui Li et al. "Sequence parallelism: Long sequence training from system perspective".
In: arXiv preprini arXiv: 200. 1 13120 (2021).
[34]
Tri Dao et al. "Fllssattttetnion: Fast and memory-efficient exact attention with io-awareness". In: Advances in neural information processing systems 35 (2022), pp. 16344-16359 16359.
[35] Tri Dao. "Flashattention-2: Faster attention with better parallelism and work partitioning". In:
arXiv preprint arXiv:2307.0866 (2023).
[36] Jay Shah et al. "Flashattention-3: Fast and accurate attention with asynchrony and low- precision". In: Advances in Neural Information Processing Systems 37 (2024), pp. 68658- 68685.
[37] Markus N Rabe and Charles Staats. "Self-attention does not need Onn²) memory", In: arXiv preprint a2X (2021).

---

[38]
Mathurin Videau et al. Meta Lingua: A minimal PyTorch LLM training library. 2024. URL:
https://github. www.coo//aaaaao ccwp//aabbooorreeerr////nneeeeeo
[39] Size Zheng et al. Triton-distributed: Programming Overlapping Kernels on Distributed Al Systems with the Triton Compiler. 2025. arXiv: 2504 19442 [cs. DC]
URL: https : 77
arxiv. org.i..rr/ooooo abs 0bbs/504.... 19442
[40] Size Zheng et al. "Tilelink: Generating efficient compute-communication overlapping kernels using tile-centric primitives". In: arXiv preprint arXiv:2503.22 20313 (2025).
[41] Songlin Yang and Yu Zhang. FLA: Triton-Bused Library for Hardware-Elficien Imple- mentations of Linear Atlention Mechanism. Jan. 2024. URL: https: / //gggttin thub. com fla- org/flash flssh linear- - attention
12

---

NeurIPS Paper Checklist
The checklist is designed to encourage best practices for responsible machine learning research, addressing issues of reproducibility. transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how answer these questions. For each question in the checklist:
You should answer [Yes], [No] [NA].
[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1-2 sentence) justificatiin right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.
The reviewers of your paper will be asked to use the checklist one of the factors in their evaluation. While "[Yes] " generally preferable to "[No]" ", it is perfectly acceptable to answer "[No] " provided a proper justification is given (c.g.. "error bars are not reported because it would be too computationally expensive" " or "we were unable to find the license for the dataset we used"). )) In general, answering "[No] " or "[NA] is not grounds for rejection. While the questions are phrased in binary way, acknowledge that the true answer often more nuanced, so please just use your best judgment and write justification elaborate. All supporting evidence can appear either in the main paper the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS Paper Checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only use the provided macros for your answers.
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We give theoretical proof in Section 3.3 of (1) Zero communication overhead in (2) Analysis of ZeCO as the sota sequential parallel strategy. The experimental data, see in Section proves that 7eCO achieves the fastest running speed, communication efficiency, and throughput,
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should elearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long it is clear that these goals are not attained by the paper.
2. Limitations
13

---

Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [No]
Justification: ZeCO's specifically refers to the sequential parallelism of the linear attention model. The high computational overhead of full attention is not the problem solved by the sequential parallel algorithm, so this limitation is not discussed in the paper. ZeCO's experiments covered in the case of the 1B model, and the experimental limitations not discussed since the additional number of parameters does not affect how the cost is calculated.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors encouraged create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model seeifspeciiiationn asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only lested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed penalize honesty concerning limitations.
3. Theory assumptions and proof's
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification: We present in section3.3 (1) rigorous proof of the Zero Communication Overhead conclusion and (2) an analysis the extra overhead. And Rigorous calculations of the volume of communication and extra computation given in the appendix.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
14

---

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proof's provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
In Section3, we give the complete algorithm and communication strategy, and how to adapt it to specific open-resources model GLA. We provide complete configuration details in Section 4, including model size (IB GLA), input chunk size (C), number of chunks per GPU (N), hardware setup (256xH100), and training framework (Lingua). We specify how throughput and operator latency were measured, including warmup steps, averaging strategy (50 runs), and comparison baselines (c.g., LASP1/2, Ulysses). For fairness, we adopt the same communication primitives and chunk counts used in the original baselines.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that are appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
15

---

5. Open access to data and code
Does the paper provide open access the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [No]
Justification: We plan to open source the complete code the near future, and this article provides all the conditions for reproduction.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (kttps://nips.cm cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
haabblic/guuddess/ddeebbbiissss httpssuimmssissosoooooiooooooo oeeiionoooooooo.l moliicy) for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the proposed method and baselines. only subset of experiments are reproducible, they should state which ones omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer: [Yes]
Justification:
We present the complete experimental setup in section4 and the Appendix
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: Although we conducted multiple runs (e.g., 50 iterations per communication operator) to average performance and reduce variance, we did not report explicit error bars statistical significance intervals in the current version. The primary metrics shown (e.g..
16

---

communication time, throughput) are deterministic in nature and highly stable across runs in our setting.
Guidelines:
The answer NA means that the paper does include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report I-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We present the complete experimental setup in section and in the Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't 't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics http://///eaaeppppppppppptt/ttia
Answer: [Yes]
Justification: This work focuses on algorithmic and systems-level innovations in distributed training for long-sequence models. It does not involve human subjects, sensitive data, or potentially harmful applications. All experiments are conducted using publicly available models and infrastructure. We have reviewed and complied with the NeurIPS Code of Ethics.
17

---

Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g.. if there is special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: This work focuses fundamental algorithmic and systems-level contributions to distributed training of long-sequence models. It does not involve human subjects, user data, deployment systems, or ipplicatioonspeciine models. Therefore, it does not present any direct societal impact.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied partieular applications, let alone deployments. However, if there is direct path any negative applications, the authors should point it out. For example, it is legitimate point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
This work does not involve the release of pretrained models, generative sys- tems, or datasets that pose high risk of misuse. It focuses on communication -atuuiicaatio--effiinttt -fffiiient distributed training algorithms for long-sequence models, and thus does not require addi- tional safeguards.
Justification:
Guidelines:
The answer NA means that the paper poses such risks.
18

---

Released models that have high risk for misuse or dual-use use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (c.g., code, data, models), used in the paper, properly credited and the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: All reused code assets are respected under their respective licenses.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re- re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: There is no new assets.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset used.
At submission time, remember anonymize your assets (if applicable). You either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
19

---

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: This paper does not involve any crowdsourcing research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
The research presented in this paper does not involve human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMS if it is important, original, or non-standard component of the core methods this research? Note that the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Justification: This research does not involve LLMs any important, original, or non- standard components of the core methods.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non non-standard components.
Please refer to our L.LM policy (https: htttee///eerrrrpeccc///eeeeeeeep for what should or should not be described.
20

---

We present A Appendix: Supplementary Proof and Experimental Data
A.1 Global Chunk Update Proof
We prove the correctness of the global correction formula used in Equation of the main text, which expresses the global state at position (p - 1)I + nCC as:
S (p- 1)L++nC
(5 1) @ 1(--1)1 + [(n].
(17)
We begin from the chunkwise recurrence of the Gated Linear Attention (GLA) state update within each device. For any chunk n, the recurrence is:
[n] = (7mml) 1 @ SIn-1| + K [n] [nn
(18)
with initial state Slo) = 0. Unfolding the recurrence, we obtain the closed-form expression of the final local state S|m]
S
x2]
II J=++1
+
1 1is) +
II
+
Sjoj-
(19)
Equation 19 represents the result of local computation, it captures the final local state obtained by starting from zero initial state and considering only the local contribution within the current chunk
The key observation is that the second term I j= 7(b)) @ S vanishes due to the initial condition
Sio = 0, making Snn] completely determined by local information. We now demonstrate the linear decomposition property of global state updates. The key insight is that when non-zero initial state exists, the final global state can be decomposed into two independent linear contributions: attenuated propagation of global computation and the current chunk's 's local contribution. Now suppose instead perform the same recurrence starting from a non-zero initial state S (p--1)z, which is the final global state of the previous device. The updated state at global index (p 1)L + nC becomes:
S(p-1)L-nC =
Σ
7
+ K
(Bl
+
I1 1
@ (0+ S (--)))
(20)
j=it1
(5 1p- 1)2
Sjn]-
(21)
This linear property allows local computation Six) stores only the residual contribution from chunk n, and multiplying the incoming global state ((--)) by the cumulative decay [nj precisely reconstructs the full global state.
A.2 Unified Analysis of Sequence Parallel Methods
In this section,
multi-head attention with head H 32, de dg H
unified analysis of several representative sequence parallel (SP) methods across both full attention and linear attention models. Specifically, compare them from the following three perspectives:
Communication Volume: The total amount of data transferred per device during SP execution.
Computation Cost: The total computation time to process sequence of length PL in parallel.
Additional Computation Overhead: The extra operations introduced due SP-specific logic.
Full Attention Models.
Ulysses: Uses All-to-All communication to exchange Q. K. and Output tensors. Com- munication volume is 4LD per device. Due to full attention's 's quadratic complexity, the computation cost is LP.
Megatron CP: Utilizes All-Gather to collect Q and K from all devices. Communication volume 2PLD. Computation cost is the same as Ulysses, DP
21

---

Linear Attention Models. In linear attention, since each device processes a sequence of length L, the inherent computation per device is LDe (here we ignore the additional lower-order terms introduced by the chse algorithm), which is independent of P.
LASP-1: Employs serial P2P communication. Each device transmits single state tensor S' E _fxeeee, with communication volume De. However, the devices execute sequentially, resulting in an equivalent time overhead (including both communication and computation) as if each device performed P times the workload.
LASP-2:
Uses All-Gather collect all intermediate state tensors across devices. Each device processes all P global states, leading to PDe communication volume and additional computation cost of log(P) + De for sum reduction and state updates.
ZeCO (Ours): Implements pipelined communication via All-Scan. Each device sends/receives only one state S, with communication volume De. It additionally maintains cumulative decay vectors and updates N intermediate states using global recurrence. So, here is an extra computation cost N De + Nd.
In conclusion, for sequence parallelism with full attention, both the computation cost and the number of device parameters are strongly dependent on the number of devices P. which becomes a major efficiency bottleneck. In the case of linear attention, although both the communication and computation costs of LASP-1 and LASP-2 scale with P. the computation cost constitutes only a small fraction of the total overhead. As result, the communication cost's dependence on becomes the primary bottleneck. In contrast, our 7eCO algorithm achieves both communication and computation costs that are independent of the number of devices P.
Table I: Comparison of Sequence Parallel Methods: Communication Volume and Computation Cost (For LASP-1, we consider the sequential execution order)
Method
Communication Volume
Computation Cost
Ulysses (Full)
4LD
L'DP
Megatron CP (Full)
2PLD
T.DP
LASP-1 (Linear)
PDe
PLDe
LASP-2 (Linear)
PDe
LDe + De + NDe
ZcCO (Ours, Linear)
De
NDD Nd
A.3 Experimental Setting and Supplementary Data
In experiment Section 4.1 H is 32, the tensor size of each chunk of segmentation is 16384, the hidden dimension d is 4096, and sequence length per device is 8192. The experimental setup with 5 rounds of warm-up and 50 rounds of experiment was averaged, see in Table
In experiment Section 4.2 In the experiment of algorithm run time, test the GLA-attention algorithm equipped with different SP methods, record the time of iteration of FWD and BWD. H is 16, the tensor size of each chunk of segmentation is 16384, the hidden dimension d is 2048, and sequence length per device is 16384 and 32768. The experimental setup with rounds of warm-up and reported the average of 50 rounds of experiment, see in Table Table 4 In the experiment of Model throughput, we test the GLA-1P Model equipped with different SP methods, and record the throughput in the training stage. H is 32, the tensor size of each chunk of segmentation is 16384, the number of model layers is 20, the hidden dimension is 2048, and the sequence length per device is 16384 and 32768. The experimental setup with 5 rounds of warm-up reported the average of 100 steps of the experiment, see in Table Table 6
A.4 Backward pass for ZeCO with All-Scan comunication
In the backward propagation of the ZeCO algorithm, most of the process is similar to the forward propagation. It is important to note the difference in notation here: 7mn denotes the decay factor for the reverse cumulative product. Furthermore, in the official implementation of gated linear attention, Sin needs to be recomputed during the backward pass. However, since the global initial state has already been obtained during the forward pass, there is no need for all-scan communication when recomputing S
22

---

Algorithm 3 Backward pass for ZeCO with All-Scan comunication
Input: Q. K, G RLxd2 d,d RLxd, chunk size C, num device P. device_rand p
(0. P....P11] O01,...,P-1]
Initialize dS 0 E Rd on SRAM
1: for n + N to 0 do
2: Load ((n) RCxde (n) E RCxd, d0(n) [n] E RCxd from HBM SRAM
3: On chip, compute Ymm) [[m] [n] and Qpn = [n] @ G(n) 7 7 ④ [n)
4: Store 7 in HBM as M[a
5: On chip, compute dS (7 1) dS
d0 d1000pp0
6: Store dS HBM as (S
7: end for
8: In parallel do:
9: parallel stream 1:
dS (S(p-1)L- dSpr. +- All-Scan(dSoo) [10)
parallel stream 2:
12: Load ((--1)z from HBM to SRAM
13: On chip, recompute S|m] with Sin = S n= (0,1,,, N - 1}
14: Store {S E 1,...... 1]]
N
15: for +- 1 to N in parallel do
16: Load Q [m). K(nj (m)) V Vin] dO [n] from HBM to SRAM
17:
Load E R xd. from HBM to SRAM
18:
On chip, construct causal mask
E RBxB
19:
On chip, compute (x)) r
[n] E Cxdx
20:
On chip, compute Q(n] Q(n) @
A
[n].
K
[n)
21:
On chip, compute P [[n] (Q/m
K
22:
On chip, compute dP (dO
[n] OOM M
On chip, compute dK [n] Q
24:
On chip, compute dK [m] jlKjn// [n)
On chip, compute dQ dPKK(n) [n]
26: On chip, compute dQ [] dQ ml] ④ [m]
27: Store P P(x), (n), dK [n] in HBM.
28: end for
29: stream barrier
30: for +- to N in parallel do
31:
Load Pm.j dQ [m]: dk(m
dO
[n]. Qin) Kinj Gla 7n--1; dSpL, S|n-1], from HBM to
SRAM
32:
On chip, compute A [n) r [n] E Cxd4
On chip, compute K [x] Kin) C r [x]
34:
On chip, compute dk [n] = Vini (dS
[-
(1.
o d...))
35:
On chip, compute dkj,,, [n] = dK + dk
④ T
36:
On chip, compute dQ (x) dO(,)) n-11
37:
On chip, compute dQ dQ(u))
+ dQ ④ A mg]
38:
On chip, compute dVV [m] = [m] d0 +
Km dS n
- (7 dsJ.))
39:
Store dk dV (n) in HBM
40: end for
41: Let dQ = (dQ [1] dQ p1 dk = [dK p].... dk M dv = {dV [1) (dV
dV dV
42: Compute dA Q o dQ
K C dK, dG revcum(dA) (dA)
43: return dQ, dK, dV,dG dG
23

---

Table Communication Runtime
Method
GPU Number
All Gather Linear
All-Scan
All Reduce
8
0.37488
0.22578
0.1375
16
0.65769
0.29686
0.22803
32
1.61594
0.44775
0.31073
64
2.54305
0.73899
0,41486
128
4.35
1.27 166
0,50084
256
8.51388
2. 16444
0.60405
Table 3: Algorithm Runtime (16k sequence length)
Method
GPU Number
LASPI
LASP2
ZeCO
GLA (baseline)
8
22.59ms
19.39ms
7.32ms
6.39ms
16
28. 28ms
24.48ms
7.45ms
6,47ms
32
39.03ms
25.72ms
7.65ms
6.44ms
64
68ms
29.17ms
8. 8.04ms
6. 1m1
128
113.71ms
35.72ms
9.88ms
6.55ms
Table Algorithm Runtime (32k sequence length)
Method
GPU Number
LASPI
LASP2
ZeCO
GLA (baseline)
8
41.64ms
27.57ms
12ms
11.76ms
16
52. 14ms
30.80ms
12.41ms
11.74ms
32
72.97ms
33.08ms
12.98ms
11.74ms
64
119.79ma
39.44ms
13.56ms
11.79ms
128
217. ms
42.50ms
15.06ms
11.74ms
Table 5: GLA Model SP Throughput/GPU (tokens/sec) on 1B-16k
Method
GPU Number
LASPI
LASP2
ZeCO
GLA (baseline)
8
26428
37812
44497
47594
16
22244
31415
42328
46786
32
17813
29802
40463
46214
64
12596
27166
39832
45744
128
22386
37955
44847
256
15196
34400
42838
Table GLA Model Throughput/GPU (tokens/sec) on 1B-32k
Method
GPU Number
LASP1
LASP2
ZeCO
GLA (baseline)
8
27014
42946
47369
49633
16
23302
41190
46209
49058
32
18129
39669
45091
47980
64
12268
37485
44468
48230
128
33327
43278
47848
256
25402
40967
46588


Reviews:
[{"review_id": "TPRAtU0Sh9", "reviewer": "Reviewer_8i2q", "summary": "This paper introduces ZeCO (Zero Communication Overhead), a sequence parallelism (SP) method designed to overcome communication bottlenecks in training large language models (LLMs) with linear attention mechanisms for very long sequences. Existing SP methods for linear attention suffer from substantial communication overhead, hindering efficient scaling across multiple devices. To address thiz challenge, the authors propose \"All-Scan,\" a new collective communication primitive that provides each SP rank with the necessary initial operator state while maintaining a minimal communication footprint, effectively eliminating communication overhead. By leveraging All-Scan, ZeCO provides nearly optimal performance by introducing negligible time and space overhead. By using experimental evaluation, the authors show that ZeCO achieves a 60% speedup over state-of-the-art SP methods on 256 GPUs for an 8M sequence length, and exhibiting near-linear scalability for long sequence training.", "questions": "1) How many GPUs can ZeCO scale to while maintaining near-linear throughput scaling for a 1B model? \n2) Can the global state be divided along both $d_k$ and $d_v$ dimensions ?\n3) How does ZeCO scale with larger models? While the authors state that \"additional number of parameters does not affect how cost is calculated\", the dimensions will differ for larger models. It would be beneficial to see comparable results for larger models to confirm expected scaling behavior.\n4) Why is a comparison with full attention SP methods meaningful, given their inherently different workloads? (Referring to Figure 3) \n5) Can ZeCO be extended to training hybrid attention models, such as Jamba?", "limitations": "yes", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1) ZeCO algorithm is intuitive, leverages the characteristic of computation to effectively mask communication time and, importantly, shows superior scaling to long sequences as compared to existing LASP* methods up to 256 GPUs. \n\nWeaknesses:\n1) The paper focusses on linear attention models which have not been very successful for most NLP tasks. This limits the impact of this work. \n\n2) I am confused about the \"optimality\" claim:\n2.1)  In order for ZeCO to be \"optimal\" one would have to prove that any algorithm taking less time than ZeCO would not be able to achieve the required computation (lower-bound proofs). The presented proof doesn't prove this. You might want to reconsider the use of term \"optimal\" here.\n2.2) Even if optimality is used more loosely, I am still not sure about the presented arguments. More specifically \n- Line 191. T_{ideal-SP) and t_{overlaped_comp} are key parameters of the algorithm. How does ZeCO make sure that it provides the best choice for overlaped_comp? This is not clear from the writing.\n- Line 203-205. when K becomes sufficiently large, the overhead approaches zero. I am not sure about this; since K is upper bounded by $d_k$, in the best case T_{ALL_SCAN} is T_{ALL_SCAN} = \\tau(d_k \\times d_v) + (p-1) \\tau(d_v). For large values of $p$ (which is the focus of this paper is all about), the second term is not negligible; in fact, it can become larger than the first term.\n- Line 212. Saying T_extra_comp&IO is negligible is fine, but  this does not imply that ZeCO has made the best possible choices.\n- One last point: there is an additional (redundant computation) in ALL_SCAN which I did not find being discussed in the paper. \n\n3) Experiments are limited to single sized model. what happens to in larger models (7B, 13B). Do we see similar scaling.\n\nWriting suggestions: \n- Please use standard mathematical notation to improve readability, i.e.., lowercase italic letters for scalars, lowercase bold letters for vectors, and uppercase bold letters for matrices.\n- Please make another editing pass. A few examples:\n* Line 15: optimality spelling error\n* Line 40: this is not paradoxical\n* Line 125: \"G\"lobal\n* Line 134: \"S\"tate\n* Line 236: \"R\"untime", "was_revised": false, "final_justification": "", "rebuttal": "> Weakness 1: Limitaion of Pure Linear Model.\n\n**A:**: The reviewer accurately pointed out the current performance limitations of pure linear models. However, we would like to argue that the limitations of pure linear models will not affect our work. \n\n**Hybrid Models and Long Context Problem is Important.** The industry is currently adopting hybrid models, where **linear layers play a dominant role**. Therefore, our method has broad application prospects. Furthermore, the development of fields such as multimodal AI and the recent use of linear models as tokenizers increases the demand for efficient **long-sequence computation**, naturally drawing more attention to sequence parallelism methods for linear models.\n\n> Weakness 2: Reconsider the Optimality Analysis.\n\n**A:** We apologize for the confusion caused by our use of the term \"Optimality\" here. Our original intention was not to present a mathematically rigorous lower bound, but rather to provide a quantitative analysis and design perspective from an engineering standpoint. The analysis method has practical engineering significance, and we will reconsider how to present this part more clearly.\n\n> Weakness 2.1:Definition of overlapable computation.\n\n**A:** We should provide clearer definitions. In the implementation of Gated Linear Attention (GLA), the authors divide the computation into three operators with distinct functions. The first operator is responsible for materializing inter-chunk states. The second operator computes the intra-chunk attention map. The third operator combines the inter-chunk and intra-chunk results to produce the output. We take the GLA operator as our baseline. Our design and the selection of overlaps are based on these operators as fundamental building blocks. Therefore, the overlap naturally corresponds to the computation of the intra-chunk attention map, as this part does not depend on the state.\n\n> Weakness 2.2: Line 203-205, the term related with split number K and device number p.\n\n**A:** This is a very valuable question. The upper bound of \\( K \\) is \\( d_k \\), which is the simplified scenario we presented. However, in practical applications, we also need to consider the number of heads \\( H \\). The states of all \\( H \\) heads need to be transmitted. Typically, \\( d_k \\) and \\( d_v \\) are 128, but \\( H \\times d_k \\) equals \\( D \\), where \\( D \\) is the model dimension. Generally, for models larger than 1B parameters, \\( D \\) is greater than 2048. Therefore, compared to the number of devices \\( p \\) used for sequence parallelism, this is a very large number.\n\n> Weakness 2.3: Extra computation and I/O may not the best choice.\n\n**A:** Our design is based on the perspective of global state updates.  \nFirst, to ensure that the All-Scan communication contains a complete decay for the sequence within a device, there must be storage for the \\(\\gamma\\) vector.  \nSecond, after the global communication, to guarantee that every materialized state undergoes a global update, it is necessary to load the state after All-Scan communication and perform computations with the inter-chunk states; this step cannot be omitted. We will reconsider the way the analysis is presented here.\n\n> Weakness 2.4: additional computation in All-Scan is not discussed\n\n**A:** Thank you for pointing this out. We would like to clarify that the lightweight computation involved in All-Scan communication is included in the time of All-Scan (equation 12) and is inherently negligible to the communication. \n\nThis treatment aligns with standard communication analysis of collective communication like ring all-reduce and other bandwidth analysis [1][2].\n\nWe will revise the final version to clarify that this lightweight accumulation step is a **standard practice in communication**, and like in prior work, its cost is safely ignored in both theoretical and empirical cost models.\n\n> Weakness 3: Scaling Performance on Larger Models\n\n**A:** Conducting experiments on larger-scale models is a constructive suggestion. To alleviate the memory pressure caused by model scaling, we adapted a simple Tensor Parallelism (TP) to our Sequence Parallelism (SP) code framework. We completed comparisons among LASP-2, ZeCO, and DP, scaling up to a maximum of 128 GPUs. In the experiments, we used a sequence length of 32k per SP/DP rank, with TP=8 and activation checkpointing enabled.\n\n**7B model**\n| method / GPUs (total seq len)      | 16 (64K) | 32 (128K) | 64 (256K) | 128 (512K) |\n|------------------|----------|-----------|-----------|-------------|\n| dp               | 3770     | 3685      | 3659      | 3629        |\n| **ZeCO (ours)**             | **3682**     | **3588**      | **3533**      | **3459**        |\n| lasp2            | 3083     | 2595      | 2276      | 1975        |\n\n\n**13B model**\n| method / GPUs (seq len)       | 16 (64K) | 32 (128K) | 64 (256K) | 128 (512K) |\n|------------------|-----------|------------|------------|--------------|\n| dp               | 2277      | 2221       | 2181       | 2181         |\n| **ZeCO (ours)**             | **2242**      | **2168**       | **2130**       | **2095**         |\n| lasp2            | 1622      | 1584       | 1502       | 1389         |\n\n> Question 1: How many GPUs can ZeCO scale to while maintaining near-linear throughput scaling for a 1B model?\n\nFrankly speaking, “near-linear” is an empirical trend and it is difficult to assign a precise numerical value. However, based on our experimental results, at 256 GPUs, the TGS (tokens per GPU per second) metric already shows some degradation (keeps about 87% of the throughput of data parallelism), but it is still significantly better than LASP-2. \n\nWhen we mention the concept of “near-linear,” we aim to draw attention within the linear model community. Currently, the definition of linear complexity mostly focuses on single-GPU scenarios, where time cost grows linearly with sequence length. However, in the context of sequence parallelism, the focus should be on **multi-GPU linear complexity**, meaning that if the number of GPUs scales linearly with sequence length, the sequence processing time should remain approximately constant. Under this definition, **it is difficult to refer to  linear models with current sequence parallelism methods as truly \"linear\".**\n\n> Question 2: Can the global state be divided along both dk and dv dimensions\n\n**A:** This is a very astute observation. **In principle, it is feasible. However, in practice, we usually do not do this.** \n- **In principle**, the decay and add operations for all elements are independent. Therefore, we can naturally partition along both the dk and dv dimensions.\n- **In practice**, there are two reasons why we prefer not to do this. First, computationally, each decay factor \\gamma is responsible for the decay of dv number of scalar values. We aim to fully **leverage the GPU’s parallelism** by processing these dv scalars in batches. Second, in terms of communication, we want each transmitted split chunk to be large enough to ensure **communication efficiency**, which is typically on the order of kilobytes (KB).\n\n> Question 3: Scaling Performance \n\n**A**: As you can see, we demonstrated the performance of the ZeCO method on larger models in Weakness 3.\n\n> Question 4: Why compare full attention SP?\n\n**A**: We present the computation and communication costs of full attention in Figure 3. There are **two purposes for this comparison**. First, from an efficiency perspective, it **demonstrates that our model is the most efficient among all** current sequence parallelism methods. Second, through a brief analysis and comparison, we aim to **help potentially confused readers understand the fundamental workload differences** between full attention and linear attention. Therefore, we do not see the need to further compare with full attention in subsequent experiments.\n\n> Question 5: Can ZeCO be extended to training hybrid attention models, such as Jamba?\n\n**A:** Regarding extension to Hybrid Models, the answer is **Yes**, whereas for Jamba, the answer is no. There is a point that needs to be clarified. Our method applies to all chunk-wise parallelism paradigm models [3], such as GLA, HGRN2, Mamba-2, Lightning Attention, RetNet, and so on. Therefore, **for the vast majority of mainstream hybrid models, including Minimax-01, Nemotron-H and Falcon-H1 [4][5][6], our method is compatible.** However, Jamba uses Mamba-1, which lacks a chunk-wise algorithm — this is an inherent limitation that also affects the applicability of sequence parallelism methods like LASP-2 and ZeCO. Referring to Minimax-01, we only need to apply our method to the linear attention layers, which does not affect the use of other parallelism methods in the full attention layers.\n\n**To concretely address this concern, we conducted preliminary experiments on hybrid models and report the results. Please refer our answer to the weakness 2 from reviewer Nor9**\n\n> Writing Suggestion\n\n**A:** Thank you for your writing suggestions. We will modify them in the revised version.\n\n[1] Andrew Gibiansky. Bringing HPC techniques to deep learning. [Online].\n\n[2] Patarasuk, Pitch, and Xin Yuan. \"Bandwidth optimal all-reduce algorithms for clusters of workstations.\" Journal of Parallel and Distributed Computing 69.2 (2009): 117-124.\n\n[3] Yang, Songlin, et al. \"Gated linear attention transformers with hardware-efficient training.\" arXiv preprint arXiv:2312.06635(2023).\n\n[4] Blakeman, Aaron, et al. \"Nemotron-h: A family of accurate and efficient hybrid mamba-transformer models.\" arXiv preprint arXiv:2504.03624 (2025).\n\n[5] Li, Aonian, et al. \"Minimax-01: Scaling foundation models with lightning attention.\" arXiv preprint arXiv:2501.08313 (2025).\n\n[6] Zuo, Jingwei, et al. \"Falcon-H1: A Family of Hybrid-Head Language Models Redefining Efficiency and Performance.\" arXiv, 2025, arxiv.org/abs/2507.22448."}, {"review_id": "niKQk82aZy", "reviewer": "Reviewer_NCSM", "summary": "The paper introduces ZeCO, a sequence parallelism (SP) method that sharply reduces communication bottlenecks in training linear attention models on long input sequences. ZeCO is built around a new communication primitive called All-Scan, which uses pipelined communication to reduce latency when sending state information from a predecessor to a successor devices. They show—empirically and theoretically—that this achieves a near-optimal communication volume that hides communication latency by overlapping with local computation. They demonstrate strong empirical runtime performance on 1B-scale linear attention models in compared to other methods.", "questions": "Are there other applications for All-Scan beyond this paper?", "limitations": "I would recommend that the authors add an explicit \"Limitations\" section where they can summarize their assumptions and hardware constraints.", "rating": 5, "confidence": 1, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "# Strengths\n\nA key concern about linear attention models is their amenability to modern hardware. This work represents a novel systems-level contribution resolves a key communication bottleneck, making it easier to scale such models to longer contexts. \n\nThe paper is well-structured, and ideas are presented in a clear and concise manner. Figure 2 is a great visualization of the approach and its advantages. \n\nThe empirical results are strong and demonstrate compelling scaling behavior, benchmarked against alternative approaches. \n\n# Weaknesses\n\nThe approach is limited in scope to linear attention and focuses on GLA models in particular. Can something similar to ZeCO be adapted to quadratic attention models? Are there common subquadratic transformers that cannot be implemented using ZeCO?  \n\nCode is not included in the submission. What is the timeline for open-sourcing the code? \n\nIn the optimality analysis, I would recommend including stating your assumptions (e.g. which costs are assumed to be negligible) more directly at the start of the analysis. \n\nMinor points:\n- L15: “optimaity” —> “optimality”\n- Algorithm 1: “comunication” —> “communication”", "was_revised": true, "final_justification": "The author's responses were helpful and clarifying, and did not impact my final score.", "rebuttal": "> Weakness 1: Limited to GLA Model. Can ZeCO-Like Method be adapted to quadratic attention? Any subquadratic attention cannot adapt to ZeCO?\n\n**A:** The reviewer’s question is very insightful. In summary, **first**, the ZeCO method applies to linear models with materialized chunk-wise algorithm and Prefix–Scannable Model [1][2], such as GLA, Mamba-2, HGRN2, and Lightning Attention. Furthermore, it can be used in hybrid models that include these linear attention layers. **Second**, the ZeCO method is designed specifically to fully unleash the performance of linear attention across multiple GPUs. Therefore, unfortunately, we are unable to apply ZeCO-like methods to quadratic attention.\n\n**Which models can use ZeCO?**\n- First, the algorithm needs to follow the chunk-wise parallelism paradigm. For example, the materialized state version of the GLA operator. The algorithm first computes the state for each chunk, which then enables a communication pattern to globally synchronize the initial states across devices.\n- Second, the algorithm must possess the property of being a prefix-scannable model. This allows all devices to perform parallel computations on the states that need to be updated for their local sequences before communication. \n\n**Negative Examples** In the Mamba algorithm, due to the materialization of each chunk’s state consuming excessive GPU memory, only parallel scan is possible, which does not satisfy chunk-wise parallelism. Therefore, ZeCO cannot be applied. For vanilla RNNs, ZeCO cannot be used because they do not possess the prefix-scannable model property.\n\n**Positive Examples** Data-independent attention models such as RetNet and Lightning Attention, as well as data-dependent attention models such as HGRN2, Mamba-2, and GLA, can all be implemented using the GLA operator. Therefore, ZeCO can be applied to all these models.\n\n**Future Works** Models that perform state updates based on matrix transformations, such as DeltaNet and DeltaProduct, satisfy both the materialized chunk-wise algorithm and the prefix-scannable model properties, making them theoretically compatible. However, we still need to implement the matrix transformation All-Scan operation, so adaptation will be carried out in the future.\n\n> Weakness 2: What is the timeline for open-sourcing the code?\n\n**A:** Our communication algorithm is based on nvshmem. We found that Triton-distributed [3][4] enables easier implementation and development. However, this repository is currently undergoing large-scale modifications and adaptations. This month, Triton-distributed has become more stable. We plan to adapt our implementation to the stable version and open-source it for the community’s reference and use. Furthermore, in the future, we will consider integrating this operator into the Triton-distributed project as a fundamental collective communication primitive, contributing to the infrastructure community.\n\n> Weakness 3: I would recommend including stating your assumptions more directly at the start of the analysis.\n\n**A:** This is a very helpful suggestion. We have noticed that another reviewer also raised some questions regarding the analysis of optimality. The reason we included such an analysis in the paper was to provide a quantitative analysis and comparison approach to algorithm design for sequence parallelism in linear models. We acknowledge that the lack of clearly stated assumptions has caused some confusion during reading, and we sincerely apologize for this.\n\nIn future versions, we will make the following revisions:\n- Definitions: We will provide a clear definition of linear model algorithms. The algorithms we adapt and analyze should satisfy (1) materialized chunk-wise algorithm and (2) prefix-scannable model properties.\n- Assumptions: We will state the basic assumptions used in analyzing communication and computational complexity, such as 1)Ignoring lightweight computations within the communication kernel.  2)Computations that can be neglected compared to I/O and operator computation overhead.\n\n> Writing Suggestion\n\n**A:** Thank you for your writing suggestions. We will modify them in the revised version.\n\n> Question 1: Are there other applications for All-Scan beyond this paper?\n\n**A:** In this paper, besides the ZeCO sequence parallelism method, the introduction of the All-Scan collective communication primitive is also one of our contributions. Currently, this novel collective communication is only applied to support the ZeCO sequence parallelism method. However, we will assist the community in improving the All-Scan communication primitive to make it a reusable “building block” for the community. We look forward to this communication primitive playing a greater role in the future.\n\n[1] Yang, Songlin, et al. \"Gated linear attention transformers with hardware-efficient training.\" arXiv preprint arXiv:2312.06635(2023).\n\n[2]Yau, Morris, et al. \"Sequential-Parallel Duality in Prefix Scannable Models.\" arXiv preprint arXiv:2506.10918 (2025).\n\n[3] Zheng, Size, et al. \"Triton-distributed: Programming Overlapping Kernels on Distributed AI Systems with the Triton Compiler.\" arXiv preprint arXiv:2504.19442 (2025).\n\n[4] Zheng, Size, et al. \"Tilelink: Generating efficient compute-communication overlapping kernels using tile-centric primitives.\" arXiv preprint arXiv:2503.20313 (2025)."}, {"review_id": "XibUPjOlZB", "reviewer": "Reviewer_Nor9", "summary": "The paper introduces a novel method called ZeCO to address the communication bottlenecks in sequence parallelism (SP) for linear attention models, particularly relevant for training large language models (LLMs) on ultra-long sequences (e.g., 1M tokens). Contributions: 1. Enables minimal communication. 2. achieves zero communication overhead and minimal extra computation/I/O.", "questions": "1. My concern is that linear attention typically will lead to obvious accuracy drop. So can authors explain the impacts on model accuracy?   \n2. Can authors explain how well the performance improvements when this method is generalized to other linear attention variants?", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n1.  The paper rigorously proves the communication optimality of the proposed ZeCO approach, showing it achieves the theoretical lower bound in communication volume and latency.\n2. Experiments are run at scale and compare ZeCO against strong baseline.\n\nWeaknesses:\n1. it's unclear how well the performance improvements generalize to other linear attention variants.\n2. The method doesn’t generalize to full attention or hybrid approaches, Besides the accuracy of linear attention is typically bad.", "was_revised": false, "final_justification": "", "rebuttal": "> Weakness 1: performance improvements generalize to other linear attention variants.\n\n**A:** When scaling up to 7B and 13B, we did not use the GLA model; instead, we **employed a method similar to HGRN2**, where the data-dependent decay generation is tied with the key vectors. This can be regarded as a “linear model variant.” In terms of performance, ZeCO still guarantees linear scalability, with throughput per GPU nearly approaching that of data parallelism (DP). Therefore, it is evident that our method is applicable to variants of linear attention.\n\n**7B model**\n| method / GPUs (seq len)      | 16 (64K) | 32 (128K) | 64 (256K) | 128 (512K) |\n|------------------|----------|-----------|-----------|-------------|\n| dp               | 3770     | 3685      | 3659      | 3629        |\n| **ZeCO (ours)**             | **3682**     | **3588**      | **3533**      | **3459**        |\n| lasp2            | 3083     | 2595      | 2276      | 1975        |\n\n\n**13B model**\n| method / GPUs (seq len)       | 16 (64K) | 32 (128K) | 64 (256K) | 128 (512K) |\n|------------------|-----------|------------|------------|--------------|\n| dp               | 2277      | 2221       | 2181       | 2181         |\n| **ZeCO (ours)**             | **2242**      | **2168**       | **2130**       | **2095**         |\n| lasp2            | 1622      | 1584       | 1502       | 1389         |\n\n> Weakness 2: The method doesn’t generalize to full attention or hybrid approaches. The accuracy of linear attention is typically bad.\n\n**A:** We also conducted **experiments on 7B and 13B hybrid models**. We want to emphasize that our method applies to the linear attention components within hybrid models, where linear models constitute a dominant portion. \n\nNext are the experimental results for the hybrid model (where we use the Ulysses method for full attention). Although full attention becomes a computational bottleneck at longer sequence lengths, **ZeCO still demonstrates the best performance among sequence parallelism methods.**\n\nAll experiments use TP=8, with SP size = (num GPUs / 8). Sequence length fixed at 32K. Activation checkpointing enabled.\n\n**7B Model**\n\n|GPUs(Seq len) / Method        | Hybrid-dp | **Hybrid-ZeCO (ours)** | Hybrid-lasp2 | Full |\n|----------------|-----------|-------------|---------------|------|\n|32 (128K) | 3552      | **2941**        | 2402          | 1298 |\n\n**13B Model**\n\n|GPUs(Seq len) / Method        | Hybrid-dp | **Hybrid-ZeCO (ours)** | Hybrid-lasp2 | Full |\n|----------------|-----------|-------------|---------------|------|\n|40 (160K)| 2135      | **1692**        | 1496          | 697  |\n\n> Question 1: Impacts on model's accuracy.\n\n**A:** First, we propose a sequence parallelism method that enables training on ultra-long sequences, but **this method is independent of the model’s performance quality**. However, this is a very good question, because pure linear models themselves have limited performance. Therefore, the method needs to serve better-performing models in order to be valuable. We summarize the value of our ZeCO SP as follows:\n- ZeCO can serve hybrid models, which are one of the current trends in large-scale industrial models. In hybrid models, linear layers—where ZeCO demonstrates its value—constitute a dominant proportion.\n- ZeCO is applicable to the vast majority of cutting-edge linear model variants, such as HGRN2, Mamba-2, RWKV-6, and others. Therefore, ZeCO supports a wide range of linear layers.\nThese two points demonstrate that even though pure linear models have performance limitations, the ZeCO method still holds unique value.\n\n> Question 2: Can authors explain how well the performance improvements when this method is generalized to other linear attention variants?\n\n**A:** We understand performance from two aspects: task performance and throughput.\n- **Task performance** as mentioned in Question 1, this method is independent of the model’s performance quality. Therefore, the performance depends on which model ZeCO is applied to.\n- **Throughput** At the current stage, our method is applicable to data-independent decay models such as RetNet, RWKV-5, and Lightning Attention, as well as data-dependent decay models like GLA, Mamba-2, RWKV-6, and HGRN2. Since the underlying operators of these models are all compatible with the GLA operator, there is no difference in throughput among them."}, {"review_id": "5dZ5EaZQ2E", "reviewer": "Reviewer_LbPt", "summary": "In this paper, the authors propose a new sequence parallel method for linear attention. While there are a lot of sequence parallel methods for full attention, linear attention has less computation and therefore communication will be a bottleneck. The proposed ZeCO algorithm specifically optimizes for communications. The method is proved both in theory as well as in practice.", "questions": "1. What is the DP performance (i.e. fsdp, zero) in figure 1?\n2. What is a full attention performance (i.e.  Ulysse) in figure 1?\n3. Is the proposal method easily applied to VLM (vision language model)?", "limitations": "yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. The paper is well written.\n2. The comparison w/ LASP and LASP2 shows advantage of the proposal method\n\nWeaknesses:\n1. Could you provide a loss comparison before and after applying ZeCO to show that there is no numerical issue when applying the proposal method?\n2. Could you provide a profile to show how the overlapping of intra-device computation and inter-device communication look like in practive? Sometimes, the overhead of the overlapping itself may be larger than the benefit of overlapping, so would like to double check to make sure the communication and computation takes similar time and therefore overlapping makes sense.", "was_revised": true, "final_justification": "The rebuttal addressed my concerns.", "rebuttal": "> Weakness1: When applying the ZeCO Method, numerical issues need to be checked.\n\n**A:** The reviewers' requests have made our work more rigorous. To check the numerical issues, we conducted thorough checks ranging from the numerical values of decay and multiplication in the *All-Scan*, the numerical values of *Operator*, to the overall training loss of the *Model*. The results indicate that our method demonstrates good numerical stability.\n- **All-Scan:** We compared the results of All-Scan with those of All-Gather + Prefix Sum in the LASP-2 method. The numerical difference between the two is on the order of 1e-6.\n- **Operator:** We compared the output and gradient numerical values between the ZeCO method and the full-sequence single-GPU computation, using the ratio-error calculation method from flash-linear-attention. For the operators, the output and gradients of q, k, and v have zero error, while the gradient error of gk is on the order of 1e-6.\n- **Model:** We ran a 1B model using three methods: Ulysses sequence parallelism, data parallelism, and ZeCO sequence parallelism. The model was trained on 50B tokens in SlimPajama. The losses were 2.38, 2.32, and 2.31, respectively. Therefore, the overall training of the model is numerically stable.\n\n> Weakness2: Double Check for Overlapping.\n\n**A:** We are honored to receive such professional advice. In fact, we have also considered this point. Following your suggestion, we have obtained the profiler trace. However, due to rebuttal guidelines, we provide a textual description instead.\n\n- **Profiler Analysis:**\nWe provide a timing breakdown comparing the overlapped and non-overlapped execution modes of ZeCO. The table below shows the communication and computation durations extracted from the torch profiler:\n\n| Mode        | Comm Start (ms) | Comm End (ms) | Comp End (ms) |\n|-------------|------------------|----------------|----------------|\n| Non-Overlap | 2769.915         | 2770.020       | 2770.476       |\n| Overlap     | 2542.205         | 2542.384       | 2542.675       |\n\n| Mode        | Comm Duration (ms) | Comp Duration (ms) | Total Duration (ms) |\n|-------------|----------------|----------------|-------------|\n| Non-Overlap | 0.105          | 0.456          | 0.561       |\n| Overlap     | 0.179          | 0.470          | 0.470       |\n\nAs seen above, the total step latency is reduced from 0.561 ms to 0.470 ms. This confirms that communication and computation are indeed overlapped effectively, leading to a reduction in latency.\n\n- **Theoretical Analysis:** In practice, the number of thread blocks launched by All-Scan is very small (around 8 to 16), which is far fewer than the number of SMs available on the GPU. Based on this, we aim to fully utilize the SMs and therefore consider it feasible to partition streams to overlap communication and computation. To ensure that communication completes as quickly as possible and to minimize contention and waiting, we assign a higher priority to the Communication Stream.\n\n> Question 1 and 2: DP performance and full attention performance.\n\n**A:** Thank you for your valuable questions. To address your concerns, we have added the experimental results for both the GLA-DP baseline and the Full Attention (Ulysses) model in the table below:\n\n| Methods\\ GPU number | 8 (256K) | 16 (512K) | 32 (1M) | 64 (2M) | 128 (4M) | 256 (8M) |\n|---------------------|----------|-----------|---------|---------|----------|----------|\n| LASP1               | 27014    | 23302     | 18129   | 12268   |          |          |\n| LASP2               | 42946    | 41190     | 39669   | 37485   | 33327    | 25402    |\n| **ZeCO (ours)**               | **47369**    | **46209**     | **45091**   | **44468** | **43278**  | **40967**  |\n| GLA-DP       | 49633    | 49058     | 47980   | 48230   | 47848    | 46588    |\n| Full Attn (Ulysses) | 3724     | 1885      | 952     |         |          |          |\n\nIt should be noted that LASP1 is not scalable beyond 64 GPUs in our current setup due to its high communication overhead — it becomes prohibitively slow to complete under practical time constraints.\nRegarding Full Attention (Ulysses), since 1B model has a hidden width of 2048, the head number is set to 16 or 32, the parallelism of Ulysses is limited by the number of heads (The maximum value of SP is 32).\n\nWe hope this addition provides a more complete comparison across baselines and attention mechanisms.\n\n> Question 3: Is the proposal method easily applied to VLM (vision language model)?\n\n**A:** Yes, we are pleased to inform you that our method can be easily applied to hybrid models in vision-language models (VLMs). Currently, there is considerable research in the industry on deploying linear hybrid models for multimodal training. For example, Nemotron-H and Minimax-01 [1][2] both have versions of vision-language models. Their scales range from 8B to over 400 billion parameters, demonstrating that these are becoming mature solutions in the industry. Among them, Nemotron uses Mamba-2 for its linear layers, while Minimax-01 uses Lightning Attention. Our ZeCO method is compatible with both types of attention.\n\n[1] Blakeman, Aaron, et al. \"Nemotron-h: A family of accurate and efficient hybrid mamba-transformer models.\" arXiv preprint arXiv:2504.03624 (2025).\n\n[2] Li, Aonian, et al. \"Minimax-01: Scaling foundation models with lightning attention.\" arXiv preprint arXiv:2501.08313 (2025)."}]

Author rebuttal:

Meta-review: This paper proposes an efficient sequence-parallel algorithm for training (certain types of) linear attention models that reduces communication. This involves introducing a new "All scan" communication primitive, which makes it possible to efficiently scale linear attention to long (1M) sequences

On the plus side, the approach is sensible, the paper is well-written, and the empirical experiments support the main claims of the paper. The authors moreover plan to open source their code, which will be helpful to the community.

On the negative side, there were some concerns with regard to: the approach only being relevant for particular types of linear attention models; whether the use of "optimal" is theoretically sound; whether the approach would scale to larger models/GPUs. The authors addressed some of these issues during the rebuttal. I especially encourage the authors to consider the presentation of section 3.3 in light of feedback from reviewer 8i2q.

Overall, I think this is a valuable contribution to the community.
Decision: Accept (poster)