ZeroS: Zero‑Sum Linear Attention for Efficient Transformers
Authors: Jiecheng Lu
Xu Han
Yan Sun
Viresh Pati
Yubin Kim
Siddhartha Somani
Shihao Yang
Venue: NeurIPS 2025 spotlight ()
Keywords: Attention Mechanisms
Transformer
Sequence Modeling
Primary area: deep_learning

Abstract: Linear attention methods offer Transformers $O(N)$ complexity but typically underperform standard softmax attention. We identify two fundamental limitations affecting these approaches: the restriction to convex combinations that only permits additive information blending, and uniform accumulated weight bias that dilutes attention in long contexts. We propose Zero-Sum Linear Attention (ZeroS), which addresses these limitations by removing the constant zero-order term $1/t$ and reweighting the remaining zero-sum softmax residuals. This modification creates mathematically stable weights, enabling both positive and negative values and allowing a single attention layer to perform contrastive operations. While maintaining $O(N)$ complexity, ZeroS theoretically expands the set of representable functions compared to convex combinations. Empirically, it matches or exceeds standard softmax attention across various sequence modeling benchmarks.

ZeroS: Zero-Sum Linear Attention for Efficient Transformers
Jiecheng Lu Xu Han", Yan Sun', Viresh Pati', Yubin Kim' Siddhartha Somani', Shihao Yang": Georgia Institute of Technology', Amazon Web Services? jlu4148gatech, seiittech.edd,,,, shihao. shihao. yaag@isseegateede edu+
Abstract
Linear attention methods offer Transformers ( complexity but typically un- derperform standard softmax attention. We identify two fundamental limitations affecting these approaches: the restriction to convex combinations that only per- mits additive information blending, and uniform accumulated weight bias that dilutes attention in long contexts. We propose Zero-Sum Linear Attention (ZeroS), which addresses these limitations by removing the constant zero-order term 1/t and reweighting the remaining pero-sum softmax residuals. This modification creates mathematically stable weights, enabling both positive and negative values and allowing a single attention layer to perform contrastive operations. While maintaining (NN) complexity, ZeroS theoretically expands the set of representable functions compared to convex combinations. Empirically, matches or exceeds standard softmax attention across various sequence modeling benchmarks. The code implementation is available at this link
Introduction
The Transformer architecture has revolutionized sequence modeling across NLP, vision, speech, and reinforcement learning flexibility, the quadratic O(N² complexity in both time and memory with sequence length N limits its efficient implementation to long-context scenarios 8 [9. Researchers have developed numerous linear-time attention mechanisms
While its self-attention mechanism offers exceptional modeling
g 10 that preserve Transformer's strengths while scaling to
longer sequences. Approaches include sparse attention patterns 15) kernel methods [ 13, 14).
low-rank approximation [18 19), and efficient factorizations 20 Despite reducing from O(N))
to O(N). these variants often underperform standard softmax attention, raising the question: Why do linear approximations save computation but sacrifice accuracy? Recent efforts to bridge this gap typically: 1) hybridize linear attention with local quadratic windows [22] 23], 2) learn softmax matrix low-rank projections [99 24 3) sharpen linear kernels through normalization and gating While offer incremental gains, these approaches often compromise O(N) efficiency, rely on task mask-specifft hyperparameters, introduce instabilities, limiting their practical use.
11 12 25
In this paper, we identify two fundamental limitations affecting linear and even softmax attention: 1) Bottleneck of convex combination 28 softmax attention produces convex combinations of value vectors, with linear attention also aiming to achieve this primarily for numerical stability. However, these combinations can only blend information additively, unable express subtractive contrastive operations directly, forcing models to use multiple layers even for simple differencing tasks. 2) Uniform weight bias and attention dilution [T] 29] In long contexts, attention mechanisms incorporate a roughly uniform N component in their weight expansion, introducing a persistent averaging effect that weakens focused attention and limits modeling of complex patterns. These limitations stem from the Taylor expansion exp(q k) 1 + (q, k) + (q. k)² + where the constant zero-order term enforces non non- negattiviyy for stability but creates an average-pooling bias that diminishes high-order token interactions. Rather than designing complex kernels to approximate
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

softmax while preserving the constant term, we propose simpler solution: remove it. Subtracting the uniform component creates naturally zero-sum weights that permit both positive and negative values, enabling contrastive updates and sharper attention distributions while maintaining stability.
From this insight, introduce ZeroS (Zero-Sum linear attention), achieving linear complexity while matching or exceeding quadratic softmax attention performance through three key elements: 1) Zero-order subtraction: removing the uniform 1/t term from each softmax row to create stable zero-sum weights; 2) Radial-angular decoupling: separating magnitude from direction by applying learned gates to first-order (linear) and higher-order moon-linear) softmax residuals, then reintroducing signed cos 9 terms to restore directional effects; 3) Linear-time implementation: using separable logits and gating for the reweighted aero-sum softmax, combined with linearizable angular computations via prefix sums, maintaining dfN runtime and (dd)) memory.
Our contributions include: 1) Identifying why the uniform zero-order softmax term limits attention mechanisms and demonstrating that its removal is safe and beneficial. 2) Developing Zero-Sum Linear Attention (ZeroS), a linear-time attention supporting negative weights with theoretical sta- bility independent of sequence length. 3) Proving ZeroS offers greater expressivity than convex combinations while maintaining numerical stability. 4) Demonstrating that ZeroS matches or exceeds standard softmax attention on various benchmarks while maintaining linear time complexity.
Background
2.1 Preliminaries: Attention Mechanisms
We consider an input token sequence of length N, represented by the feature matrix X E R where each row z; E Rixé is the embedding at time step t. With Q = XW,, w. K = XWW, h. V XW,, u. an autoregressive (causal) single-head attention layer can be written in its matrix form as
Attn(X) = a (M O (QKT) V W.,
X < X + (LN((),
where W 9. WR. W., W. E dRd are learned projections, LN(() denotes layer normalization, and M E RNN is the causal mask with M.; = 1(i 2 j} - x ii < j), ensuring each position attends only to itself and the past. When a is the row-wise softmax with a 1/vd factor, this represents standard self-attention with (N22) complexity; replacing o by the linearized kernels yields the linear attention variants that can be computed in O(N) [8 [4]. Omitting the causal mask M reverts this to encoder-only attention, attending to all pairs of positions.
Recurrent Form Attention admits an equivalent step-by-step formulation. At time f, let q: [l..k))) where 2e, W k,- = x, ky Vr - x. W. o. Then the output o, E R is o, Σi k.)
(qq. k) = exp(qk / /dd) for vanilla attention. By choosing a kernel feature map Φ(-) such that a (q, k;) = ((q:) dokk)) T, the summations can be rearranged to maintain only the d x hidden state Σi=1 (k;) U, avoiding the full N x N matrix QKT. This yields the linear attention formulation: Og v(g,) E d(k:)
olq:) اis o(k,) Replacing the summation limit & with N converts this from the oncoder-only autoregression into a encoder-only global recurrence, summing over all positions.
2.2 The intuition from existing linear attention research
We begin with insights from previous research on linear attention to introduce two key elements of our ZeroS structure: 1) radial-angular decoupling, and 2) zero-sum reweighted softmax. In softmax attention, each value vector U; is assigned weight Ei=: oxp(g.ks)) forming convex combination that ensures numerical stability by keeping outputs within the convex hull of {w:}
exp(q.k;)
26 27 30 Linear attention variants attempt to approximate this using weights of linearized kernel form
((kk))
8 [4. However, without constraining the sign of ((q.))(k:) this reduces
> ((q:)o(k.)
to affine combination that lacks the sability-ensurinn bounds of convexity. While researchers have addressed this using non-negative feature maps like 1+ELU and ReLU [2, 31 32]. these stability-ensuring modifications still underperform compared to standard softmax attention
Coupling Interaction Between Radial and Angular Components In softmax attention, the core weight term exp(Ile ||| [.]] cos 8) is controlled by both vector magnitudes and their angle 0. Crucially,

---

when cosine flips from positive to negative, large positive values transform into very small ones, with step and i highly coupled within the exponential. In contrast, linear attention applies nonlinear mappings <f.) to query and key g yield only positive values, angles between vectors become restricted to less than 90 degrees, and the angular representation loses its flipping
12 33
calculating ([ ( (k;)]l cos 0'. Since these mappings
values merely serve smooth gating signals between (0, 1). Previous research shows minimal performance changes when replacing softmax with sigmoid, ReLU, or similar functions [34/39], indicating that softmax attention's 's performance derives from modeling coupled angular and magnitude of (t, i) pairs rather than from the exponential property itself. Therefore, when constructing linear attention, should reimplement these complex interactions rather than attempting to approximate softmax or merely mimicking an inner product.
Convexity of Sum-to-One Weights Under this perspective, we revisit the convex combination in softmax attention, which primarily serves numerical stability by preserving norm regardless of sequence length. As weights become more uniform, output norm expectation decreases at approximately 1/vž with sequence length t (assuming zero-mean vectors). However, these strictly positive weights mean input signals v; can only contribute additively outputs. In linear attention, without methods to suppress historical weights, this accumulation leads to attention dilution [1]] 12 29 where uniform signals increasingly dominate sequence length grows. While some approaches address this using local windows or convolutional methods 9 31 40), these represent engineering solutions rather than resolving fundamental limitations of positive weights. Studies [27
show that with softmax weights, a single attention layer cannot express differential or contrastive operations (even with just two tokens). The strictly positive convex combination inherently constrains ability to compress complex operations, limiting parameter efficiency. To enable more flexible parameterization with negative values, we must maintain numerical stability without relying on convex combinations' norm-preserving property while satisfying linear-timg requirements.
Flexible Weighting in Related Works Implementing both the angular flipping effect and expres- siveness requires numerically stable modeling of negative weights. Previous research [28] 44
demonstrated that negative weights improve model performance, while Differential transformer
45
showed benefits from differencing two attention matrices to obtain flexible weights. In linear attention, operations that reduce or delete historical state matrix elements outperform simple accumulation ap- proaches [9 14 31 46 47). In the following sections, we will show that our ZeroS method constructs zero-sum weights based on softmax, improving performance while maintaining numerical stability compared to both standard and linear attention variants. Compared to previous linear attention, ZeroS enables more effective control radial weights and decoupled angular components in (f, pairs from step t information.
3 Methodology
In this section,we demonstrate that using softmax residual terms with zero-sum weights (eliminating zero-order terms) and decoupling radial-angular components in linear attention achieves three key objectives: 1) enabling numerically stable negative weights in a single attention layer for expressing differential and contrastive operations, 2) capturing the essential length-angle interactions in attention weights that allow positive-negatiy flipping effects, and 3) permitting the current step to effectively influence shareable accumulated weights while maintaining linear time complexity. The overall architecture of the final ZeroS block introduced in this section is shown in Fig.
3.1
The Expansion of Softmax Function
Recent research has attempted to approximate softmax using Taylor expansions 448 51]. For input scalars {s:) with 5 Σj= 8j and 8; 5, the second-order Taylor expansion is:
1 de softmax(s:) Pe +
21 8
Σ j=1
03) + (11+113)
t
The zero-order term ensures E, softmax = while first-order terms reflect linear response,
and higher-order terms capture nonlinear interactions and competitive relationships between weights Computing second-order terms based on 8t,; = k would require ((d)) complexity [49], making them impractical. Our approach differs: use logits that depend only on step i, calculate full
3

---

2ns6
Radill
Angular
Value
Simnoda
order Sortmax
Layer Norm
Reweighted sum
||
order Softmax
order
W
W
W
W
W
Deviation Logit
{uu X.
Wa
{u, im x;Waliss
{
(J Caleulation of the deviction lnght
(h) Reweiigted pero-som softrunt block
The ovarall archtecture ZereS (Zoro Surt I Linea Attention
Figure I: Illustration the zero-sum linear attention block, including the computation of deviation logits and the reweighted zero-sum softmax operation
softmax, zero and first-order terms, derive higher-order terms through their differentiation, and employ i-step-dependent gating factors to achieve interaction between (f, pairs different orders.
The zero-order baseline primarily provides accumulated magnitude measurement, contributing - level norm reduction and convexity properties. However, it enables interaction between scores. Eliminating this term creates zero-sum residual weights with both positive and negative values reflecting interaction strength. While full softmax encodes higher-order competitive effects only through positive weight magnitudes, zero-sum residual weights directly express these relationships between vectors based the positive and negative weights, emphasizing contrastive components.
Proposition 3.1 (Convex vs. Zero-Sum Span). Let {wr C. Rd and write C = {E; a,v; 0g 2 0, Σ, 02 = 1}, 2 E W;Y; E, wi 0}, where we denote the (t 11]1--smpples by A fa E R'
0. E aj 1}. Then, letting Vavg Ei Us. {E, a,;; Vavg A,1) & {E w.U; Σ, w; = 0}. i.e. the zero-sum span of {u: - Uavgl strictly contains the deviations achievable by convex weights, with striciness whenever the v; are not all identical.
Corollary 3.2 (Expressive Gain of Zero-Sum Attention). In a residual block 2; + 24 12.+2 + W.Y; softmax weights w; Og yield head deviations in {Σ: C0,00 Vavg a E A.-1); zero-order subtraction wi a; yields head deviations in EE, wuy, E. w; = 0}. Since (E, 0,,00 vsvg) } I {Ei w. Σ, w, = 0}; zero-sum atlention enlarges the set of deviation vectors the head can produce (and hence its expressivity), only without the uniform average direction.
Zero-sum weights can express more complex interactions after removing the zero-order term, with expressivity reduction only in the orthogonal direction of Vavg. This direction typically represents the lowest-cost basis since it requires only average pooling. We can recover this capability through multi- ple attention heads and layer stacking. To strietly ensure this direction is not lost, implementation retains the zero-order term in the first layer, removing it in subsequent layers as described below.
3.2 Reweighted Zero-sum Softmax
We define the reweighted zero-sum softmax operation. For logit input st,5 at step t, we compute: E St.j dt. 8t,i 3t. Subtracting zero-order (1/t) and first-order (6,,/t) terms from softmax yields the residual:
exp(st,5)
8g, 0(11813),
Eet Σ/= exp()r.;)
where E,e,, 0 and E, 8t,/t O. We gate these components using learned scalars of sigmoid(g!) and of sigmoid ) defining zero-sum weights:
S t
Σ
wy, Ew..=0. =0. 0.
Wt, = of
a Et,ir
This form assigns two gating weights: one for the first-order orthogonal direction and another for all directions of second-order and above. For the first attention layer, can optionally preserve

---

the zero-order term using of = tanh(g?). giving w/,,= = = + to fug + Eof though experiments show this has minimal impact across most tasks.
Remark. A key advantage of this formulation for linear-time attention is that even with logits still control interactions of different orders in the final
S1, 8, that independent of i, weights Wi,i through the t-step gating mechanism of across orthogonal directions from softmax expansion. This gate reweighting approach Wr, of é + of', effectively replaces the traditional linearization that decomposes exp( (k:k)) into
( ((::)((k:).
Proposition 3.3 (Preservation of Affine Hull and Expressivity). Let {wili=1 R and write
Vavg Ei= Vss A Vi Vavg. single head with full softmax (or full reweighted softmax with the zero-order term kept) can produce any point in the affine hull
Aff Aff
vi}
Vavg
iml
A single head without the zero-order term (i.e. zero-sum weights) can produce any point in the linear span
A.) Sp 3
A. Em.=o]
Span
Therefore, you use one head (or one layer) that retains the zero-order term and then stack one or more heads (layers) that subtract it, the Minkowski sum of their reachable sets is exactly
(ff + Span(A;} = Af
In other words, after the first full attention layer, it already cover the entire affine hull, and the subsequent zero-sum attentions do not shrink that. The overall network can still express any affine combination of the v,.
Residual Stream Alignment When value vectors {u,} are centered and i.idd., zero-sum attention
produces O, = E Wr,5% with E, Wr = 0 and E[o,] = O. This aligns with decoder-only Transformer's 's residual stream ideology where xa < xt + Attn(x;) should provide pure updates without constant bias. Subtracting the zero-order term naturally centers these residuals, improving training stability.
We now show that the reweighted zero-sum softmax achieves the same level of numerical stability as the original softmax.
Lemma 3.4 (Numerical Stability of Zero-Sum Softmax), Let we, be the reweighted zero-sum softmax weights with E; Wi, = 0, and assume each value vector satisfies [|ulll < B. Then for any step I,
Σ We
max Jweesl Σ |e S
Bt max|w,,)
Moreover, since |wa,sl max L. 18r.5
Jpeal) 21.
and 8, 0(1) under bounded logits, we have OB),, independent of
max; lur.i = (1/Lt) and hence ||Σ, we,
With controllable logits generation methods independent of t. such as the scaled dot- dot-product in original softmax attention, the numerical stability of the above method remains well-controlled. This allows us to employ zero-sum weights that permit negative values while still achieving numerically stable outputs, even without the norm-preserving property of convex combinations.
Proposition 3.5 (Uniform Lipschitz Bound of Zero-Sum Softmax with decay factor 1/ 1/v1). Assume each value vector satisfies ||0,]| S B. each pre-softmax logit $x,((x) [-S, S) Lg -ippschhhi in the imput æ. each residual weight we,s (x)) obeys the scaling wt,;(x:) (x,'(x') I L || a'Il, for some constant Lew depending only S and the sigmoid gates. Let the head outpui be oz(x))
Vi Ei=1 (z) Uis Σ (x) 0. Then for any two inputs x, x,
B L
[ - o.(x')
|
x'll.
Vt
The zero-sum update is -i-sciic in its imputs with constant O(1 Vt), ensuring stable gradients and activations independent of sequence length.
5

---

This proposition introduces 1/vt decay factor that ensures reweighted zero-sum softmax maintains variance reduction similar to convex combinations, promoting training stability. However, since linear attention methods typically apply Layer Normalization to control output variance, LayerNorm effectively supersedes this factor and is sufficient to ensure gradient stability during training.
Reweighted Zero-sum Softmax in Linear Time To achieve linear-time computation, we simplify logits from St,i s, by removing I-dependency. While we could use basic forms like 8; æ, W quadratic forms s; x; W. W x; /d emulate dot-products, we instead propose a design with a more meaningful representation.
We want these logits to express the deviation of step i relative to previous steps. We calculate the negative inner product between each step's 's vector u, = z; W. and its cumulative average. For better assessment of initial steps, we introduce trainable parameters μ R Ixd and E R as smoothing prior, calculating deviation logits 8; as:
e # + Σj 2j=10 Uj
* vd u. u. where ū,
e² +
Let Ti, represent the final computed reweighted softmax result:
de, = 8; 8t, Et,i = exp(s:)
Σ exp(s;) t t
det + of Ei, = Ft,1 = 0,
Σ Expeex((s) (xp(8i) + (o) ot))
of
(8;
8:)
where of = sigmoid( W of = sigmoid x. ) W W E Rdx11 The terms dependent on t and are effectively separated, enabling linear-time computation through prefix sums.
9²
3.3 ZeroS Linear Attention: Interaction Between Radial and Angular Components
The reweighted zero-sum softmax provides strong foundations for linear attention by yielding numerically stable weights (including negative values) with computational simplicity while enabling high-order (t, 2) interactions in token mixing. We leverage linear-time logit inputs that depend only on step i and implement effects on different softmax orders through step / gating.
However, our earlier discussion showed that the angle-flipping effect in softmax attention's s exp(Ilq.ll ||| cos f)) significantly impacts final weights. While reweighted zero-sum softmax ef- fectively models length interactions through i-step logits and t-step gating, lacks control over directional influence when measuring vector differences in (f, pairs. Since zero sum weights provide inherent stability, longer need place cos 0 in the denominator normalizer, we can di- rectly multiply the angular component (cos with the reweighted softmax radial component without positivity constraints. This approach enables seamless integration with rotary positional embedding (RoPE) [52], making the angle term's 's role in measuring relative distance more explicit.
Zero-Sum Linear Attention (ZeroS) We use normalized vectors k; = k and å. / with Ft,i as the radial component and cos the angular component. ZeroS produces the output:
Tt,i cos A Vi, cos 0 q.
With RoPE's dlack-diagonal rotary matrix applied, the angular term becomes cos = c' R Both Tı.i and cos 0 are centered values, preserving zero-sum properties in the weights it, cos O. Though not strictly positive (unlike traditional radial components), Te, captures magnitude effects from step reflecting length-related interactions between (1, pairs.
Linear- Time Scan With logits that depend only on step (e.g. = va ū,ū with u, x; W.
E "),, the radial weight at time can be decomposed into the full softmax term, the Oth-order baseline, and the ost-rrder term:
and ú
1
8
83 Hence the higher-
1=1
Oth
Full
order zero-sum residual is Full Oth 1st. We realize ZeroS by gating the first-order zero-sum and

---

Table 1: Evaluation Results of ZeroS on the MAD benchmark.
Model
Compress
Fuzzy Recall
In-Context Recall
Memorize
Noisy Recall
Selective Copy
Averuge
Hyena
45.2
7.90
81.7
93.1
MultiHead Hyena
44.8
14.4
99.0
89,4
98.6
93.0
73.2
Mamba
52.7
6,70
90.4
89.5
90.1
86.3
69.3
GLA
38.8
6,90
63.3
81.6
88.6
DeltaNet
42.2
35.7
100
52.8
1001
100
71.8
LinAttn
31.1
8,15
91.0
74.9
75.6
93.1
62.3
Transformer
51.6
29.8
94.1
85.2
86.8
99.6
74.5
Zero8
44.0
99.9
88.1
96.1
73.5
ZeroS-SM
45.2
28.0
100
84.3
96.6
98.5
75.4
the higher-order residual with of = sigmoid (z ) and of = (z, W and optionally in the first layer retaining the Oth-order baseline of sigmoid( W (with fixed of = 0 by default). Using normalized directions q: q:/ and k | we maintain the following prefix scans at step f:
E
Pt
S5.
F,=
k
G.
Σ
st s
H.
kj Uz.
==1
:=1
The output is then a gated activation of these scans by the current step's angular vector g::
Or =
Q:
0
Fr
1 G. +
Ps (
H.
+ to,
|G.
P
H,
H.
Fr
t
a, (Full
1st)
at restore)
dg (optional restore)
=
where
tft E.
of of
P: f2
P f 0
of
00
B.
Nt
0
This scan keeps only ( state (F,, Gr, H₂) and updates in O per step, yielding overall O(Nd²) time and (d memory while implementing the zero-sum weighting. Moreover, our reweighted zero-sum approach can also be directly applied standard softmax attention. See section A.1.6 for more details.
4 Experiments
ZeroS's zero-sum formulation enhances the attention layer's expressivity for complex operations, particu- larly evident in in-context learning tasks 37 [33] We evaluate both linear-time ZeroS and quadratic-time ZeroS-SM on recent in-context learning benchmarks, along with experiments on NLP, image, and time se- ries tasks. In all experiments, the multi-head attention module with ZeroS under original benchmark settings. preserving all other com- ponents (MLP/GLU, embeddings, hy byperaarameters) ensure strict alignment with previous standards.
directly replaced
We previously described the prefix-sum computation of autoregressive ZeroS. For the encoder-only ZeroS, the summation simply spans all timesteps. We use the causal version of ZeroS for all datasets except image modeling. We provide more detailed description of the experimental datasets in Appendix A.5 0-th order term addition in our experiments.
Figure 2: Evaluation of ZeroS on RegBench.
For simplicity, we do not apply first-layer
7

---

MAD
We evaluate ZeroS on the MAD benchmark
54 which tests sequence models on in-context tasks, As shown in Table [ ZeroS outperforms other linear- time models (Hyena, Mamba, GLAA, DeltaNet, LinAttn 14 31 (5)), achieving performance closest to Trans- former, while ZeroS-SM further improves upon Trans- former's average score. Task-level analysis shows Ze- roS significantly outperforms LinAttn on In-Context and Noisy Recall tasks, supporting our hypothesis that zero- sum weights enhance algorithmic abilities. However, on tasks like Compress and Memorize that rely less com- plex representations, ZeroS provides minimal gains. Un- like DeltaNet, which actively deletes memory states, Ze- roS maintains strong memorization despite using negative weights, indicating that zero-order modifications pre- serve sequence memory capacity.
Table 2: Evaluation Results on Wexi
Model
PPL (val)
(tc())
Params (M)
FLASH
25.92
26.7
42.17
I+elu
27.44
28.05
44.65
Performer
62.5
63.16
44.65
cosFomee
26.53
27.06
44,65
Syn(D)
31.31
Syn(R)
33,68
34.78
46.75
aMLP
28.08
S4
38.34
39.66
45.69
DSS
39.39
41.07
45.63
GSS
29.61
30.74
43.84
RWKV44
24.31
25.07
46.23
LRU
29,86
31.12
46.75
TNN
23,98
24.67
48.66
Mamba
22.58
23.19
44,99
HGRN2
23.1
23.73
44,66
Tormer
24.78
ZeroS
23.91
24.61
46.31
ZeroS-MM
23.62
24.17
44.69
MQAR We follow the setup of [56] for the MQAR task, which evaluates models' ability to learn induction heads for in-context associative recall. Using the
same hyperparameter sweep, Fig. 3 shows ZeroS performs comparably to vanilla attention across most configurations.
Sequence Length: 64
Sequence
Sequence
Sequence 532
0.8
Heore
0.6
minv
HOHK3
0.2
HGRK2
Zeros
0.0
64
120 Dimrraaiinn
256
532
1.25
205
51.2
3.20
200
332
54
3220 Cimuatsiinn
220
312
Madiul Cinaaroion
Figure 3: Performance evaluation on the MQAR benchmark, illustrating the relationship between model dimension (x-axis) and accuracy (y-axis). ZeroS demonstrates consistent performance advan- tages over other structures across all experimental configurations.
RegBench We evaluate ZeroS on Reg- Bench (57) following the original exper- imental setup (Figure D RegBench tests models' ability to infer regular language structures from examples. ZeroS outper- forms linear-time baselines including GLA, RetNet [58], and RWKV 59
Training Loss
Va Nddiioion Loss
4.3
Uneer
Unee
3.5
Teeakreear
Zerod
Zared
3.4
3.0
10020
20000
50000
40000
50000
0
10000
20002
30000
40000
50000
4.1 Language Modeling
Slep
S0ep
Figure 4: Performance Evaluation of ZeroS on OWT2
WikiText We conduct language modeling on WikiText-100 103 following [660's setup, with results in Table 2 ZeroS outperforms vanilla Transformer at this smaller scale, demonstrating its efficiency. ZeMoS-SM yields further improvements, showing enhanced reasoning capability from the zero-order term removal.
OWT2 We evaluate ZeroS on Open WpenTeee (OWT2) 66 using 12-layer, 768- -osdeimensional GPT-2 architecture with various token layers (see Appendix (..5). Figure4sshowe ZeroS tracks much closer
Table 3: Comparative analy- sis of image classification per- formance ImageNet- lk.
to vanilla Transformer than other linear methods like AFT [62 and GLA, while ZeroS-Sh SM further improves upon vanilla Transformer performance.
DeiT-Tiny
Model
Top-1 Acc
Params (M)
DeTT
72.20
5.7
TNN
72.29
6.4
Image Modeling Following HGRN2 600, wee valuate ZeroS on ImageNet by replacing the DeiT-Tiny architecture's softmax atten-
HGRN1
HIGNN2
75.39
6.1
ZeroS
75.51
6.0
8

---

Table 4: Evaluation of the ZeroS performance on the Time Series Forecasting Benchmark
Models
ZeroS
GLA
AFT
i'Iransformer
PutchTST
DLinear
Metric
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
Weather
0.218
0.265
0.223
0.267
0.220
0.266
0.232
0.274
0.221
0.261
0.233
0.282
Solar
0.192
0.256
0.204
0.266
0.198
0.259
0.219
0.284
0.202
0.254
0.216
0.277
E
0.414
0.433
0,418
0.439
0.433
0.454
0.467
0,431
0.422
0.436
ETTh2
0.341
0.392
0.342
0.390
0.337
0.390
0.374
0.410
0.330
0.379
0.426
0.444
ETTml
0.347
0.387
0.357
0.394
0.348
0.386
0.373
0.401
0.346
0.347
0.376
ETTm2
0.245
0.312
0.250
0.315
0.246
0.311
0.265
0.332
0.247
0.312
0.252
0.326
Table Ablation Study on the MAD benchmark.
Model
Compress
Fuzzy Recall
In-Context Recall
Memorize
Noisy Recall
Selective Copy
Average
ZeroS
44.0
14.9
99.9
88.1
96.1
97.8
73.5
ZeroS 0-th
42.0
10.5
91.4
85.2
90.0
97.1
69.4
ZeroS w/o RWSM
36.3
10.6
91.8
81.7
89.7
95.3
67.6
ZemS w/o Gating
39.7
13.5
96.3
83.0
94.6
97.8
70.8
ZeroS w/o Norm
39.1
12.3
87.0
97.1
69.4
ZemS--S
45.2
28.0
100
84.3
96.6
98.5
75.4
tion with our encoder-only implementation. As shown in Table 3, ZeroS outperforms previous 294 methods including TNN 663 and HGRN1 (64 under comparable parameter budgets.
Time Series Following the setup in 655 evaluate ZeroS on time series forecasting tasks. ZeroS
outperforms both efficient sequence models (GLA, AFT) and domain-specific approaches (iTrans- most datasets.
former [66], PatchTST (67)
4.2 Ablation Studies
We conduct ablation studies MAD and
Table 5: Ablation Study on WikiText-1003
WikiText-103 to analyze key components of
Model
PPL (val)
PPL (test)
Params (M)
ZeroS. Reintroducing the O-th order softmax term reduces performance on In-Context Recall, Noisy Recall, and WikiText, confirming the rep- resentational advantage of zero-sum weights. Replacing the reweighted zero-sum softmax with standard softmax further degrades performance, highlighting the expressive gap between convex combinations and our flexible zero-sum mechanism. Ablating the gating component causes moderate performance drops across most tasks, suggesting it contributes broadly to model flexibility. Finally, removing LayerNorm notably impacts performance on In-Context Recall but not on simpler tasks like Memorize, indicating stable variance is particularly critical for algorithmic reasoning: consistent with normalization's role in linear attention mechanisms. See $A.3 for additional basclines and ablations.
ZeroS
23.91
24.61
46.31
Zero8 0-th
24.05
24.74
46.31
7emo w/o RWSM
24.21
24.97
46.31
ZeroS-SM
23.62
24.17
44.69
5 Conclusion and Limitation
We introduced Zero-Sune Sum Linear Attention (ZeroS), addressing fundamental limitations of linear attention by removing the constant zero-order term from softmax and reweighting the resulting zero-sum residuals. Our approach enables higher-order token interactions while maintaining O(N) complexity, bridging the performance gap between linear and quadratic attention methods. Evalua- tions across diverse tasks show ZeroS matches or exceeds standard softmax attention while offering significant efficiency advantages, challenging the belief that expressivity-efficits tradeoff's in attention mechanisms are inevitable.
As for the limitation, our research prioritizes improving attention's algorithmic expressivity rather than providing engineering optimizations like GPU acceleration implementations found in Mamba or GLA 14). Also, our resource constraints prevented large-scale model training and evaluation on LLM benchmarks, which would involve numerous factors. This focused approach allowed us to precisely identify ZeroS's algorithmic improvements without requiring extensive engineering or computational resources that are typically needed for optimizing large benchmark metrics. Addition- ally, our evaluation of ZeroS primarily focuses on autoregressive tasks. Future work may explore its capabilities non-causal tasks to further extend its applicability.

---

References
[1] A Vaswani. Attention is all you need. Advances in Neural Information Processing Systems, 2017.
[2] Jacob Devlin, Ming-Woi Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training deep bidirectional transformers for language understanding. In Proceedings of the 2019 conference of the North American chapter of the association for computational linguisties: human language technologies, volume I (long and short papers), pages 4171 2019.
[3] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances Neural Information Processing Systems, volume 33, pages 1877-1901. Curran Associates, Inc., 2020.
[4] Alec Radford. Improving language understanding by generative pre-training. OpenAl technical report, 2018.
[5] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x 16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010. / 1929, 2020.
[6] Linhao Dong, Shuang Xu, and Bo Xu. Speech-transformer: a no-recurrence sequence-to- sequence model for speech recognition. In 2018 IEEE international conference on acoustics, speech and signal processing (ICASSP), pages 5884 5888. IEEE, 2018.
[7] Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Misha Laskin, Pieter Abbeel, Aravind Srinivas, and Igor Mordatch. Decision transformer: Reinforcement learning via sequence modeling, Advances in neural information processing systems, 14:55044-15097, 2021.
[8] Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, and François Fleuret. Transformers are mnns: Fast autoregressive transformers with linear attention. In International conference on machine learning, pages 5156-5165. PMLR, 2020.
[9] Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces. arXiv preprini arXiv:23 2023.
[10] Weizhe Hua, Zihang Dai, Hanxiao Liu, and Quoc Le. Transformer quality in linear time. In International conference on machine learning, pages 9099-9117, PMLR, 2022.
[11] Zhen Qin, Xiaodong Han, Weixuan Sun, Dongxu Li, Lingpeng Kong, Nick Barnes, and Yiran Zhong. The devil in linear transformer. arXiv preprint arXiv:2210. 10340, 2022.
[12] Zhen Qin, Weixuan Sun, Hui Deng, Dongxu Li, Yunshen Wei, Baohong Lv, Junjie Yan, Lingpeng Kong, and Yiran Zhong, cosformer: Rethinking softmax in attention. arXiv preprint arXiv:2202.08799, 2022.
[13] Krzysztof Choromanski, Valerii Likhosherstov, David Dohan, Xingyou Song, Andreea Gane, Tamas Sarlos, Peter Hawkins, Jared Davis, Afroz Mohiuddin, Lukasz Kaiser, et al. Rethinking attention with performers. arXiv preprint arXiv:2009. /4794, 2020.
[14] Songlin Yang, Bailin Wang, Yikang Shen, Rameswar Panda, and Yoon Kim. Gated linear attention transformers with hardware-efficient training. In Forty-first International Conference on Machine Learning, 2024.
[15] Iz Beltagy, Matthew E Peters, and Arman Cohan. Longformer: The long-document transformer. arXiv preprint 2555v:2004.05110, 2020.
10

---

[16] Rewon Child, Scott Gray, Alec Radford, and Ilya Sutskever. Generating long sequences with sparse transformers. arXiv preprint arXiv:I904. 10509, 2019.
[17] Manzil Zaheer, Guru Guruganesh, Kumar Avinava Dubey, Joshua Ainslie, Chris Alberti, Santi- ago Ontanon, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, et. Big bird: Transformers for longer sequences. Advances in neural information processing systems, 13:11783-17297, 2020.
[18] Yunyang Xiong, Zhanpeng Zeng, Rudrasis Chakraborty, Mingxing Tan, Glenn Fung, Yin Li, and Vikas Singh. Nyströmformer: A nyström-base based algorithm for approximating self-attention, In Proceedings of the AAAI conference artificial intelligence, volume 35, pages 14138-14148, 2021.
[19] Sinong Wang. Belinda Z Li, Madian Khabsa, Han Fang. and Hao Ma. Linformer: Self-attention with linear complexity. arXiv preprint arXiv:2006. 04768, 2020.
[20] Tri Dao, Dan Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. Flashattention: Fast and memory-efficient exact attention with io-awareness. Advances in neural information processing systems, 35:16344-16352 2022.
[21] Tri Dao. Flashattention-2:: Faster attention with better parallelism and work partitioning, arXiv preprint arXiv:2307.08691, 2023.
[22] Soham De, Samuel L. Smith, Anushan Fernando, Aleksandar Botev, George Cristian-Murand Albert Gu, Ruba Haroun, Leonard Berrada, Yutian Chen, Srivatsan Srinivasan, al. Griffin: Mixing gated linear recurrences with local attention for efficient language models. arXiv preprint arXiv:2402. 19427, 2024.
[23] Zhanghao Wu, Zhijian Liu, Ji Lin, Yujun Lin, and Song Han. Lite transformer with long-short range attention. arXiv preprint arXiv:2004. 1886, 2020.
[24] Yi Tay, Dara Bahri, Donald Metzler, Da-Cheng Juan, Zhe Zhao, and Che Zheng. Synthesizer: Rethinking self-attention for transformer models. In International conference on machine learning, pages 10183-10192. PMLR, 2021.
[25] Bolin Gao and Lacra Pavel. On the properties of the softmax function with application in game theory and reinforcement learning. arXiv preprint arXiv: 1eei11144000000 2017.
[26] Prajit Ramachandran, Niki Parmar, Ashish Vaswani, Irwan Bello, Anselm Levskaya, and Jon Shlens. Stand-alone self-attention in vision models. Advances in neural information processing systems, 32, 2019.
[27] Oliver Richter and Roger Wattenhofer. Normalized attention without probability cage. arXiv preprint arXiv:2005.088 095611 2020.
[28] Pierre Baldi and Roman Vershynin. The quarks of attention: Structure and capacity of neural attention building blocks. Artificial Intelligence, 319:103901, 2023.
[29] Weixuan Sun, Zhen Qin, Hui Deng, Jianyuan Wang. Yi Zhang, Kaihao Zhang, Nick Barnes, Stan Birchfield, Lingpeng Kong, and Yiran Zhong. Vicinity vision transformer. IEEE Transactions 2023.
on Pattern Analysis and Machine Intelligence, 15(10))126355112224
[30] Krzysztof Choromanski, Haoxian Chen, Han Lin, Yuanzhe Ma, Arijit Sehanobish, Deepali Jain, Michael S Ryoo, Jake Varley, Andy Zeng, Valerii Likhosherstov, et al. Hybrid random features. arXiv preprint arXiv:21 10.04367, 2021.
[31] Songlin Yang, Bailin Wang, Yu Zhang, Yikang Shen, and Yoon Kim. Parallelizing linear transformers with the delta rule over sequence length. arXiv preprint arXiv:2406.00 06484, 2024.
[32] Zhen Qin, Dong Li, Weigao Sun, Weixuan Sun, Xuyang Shen, Xiaodong Han, Yunshen Wei, Baohong Lv, Xiao Luo, Yu Qiao, al. Transnormerllm: A faster and better large language model with improved transnormer. arXiv preprint arXiv:2307.14992 2023.
11

---

[33] Krzysztof Marcin Choromanski, Valerii Likhosherstov, David Dohan, Xingyou Song, Andreea Gane, Tamas Sarlos, Peter Hawkins, Jared Quincy Davis, Afroz Mohiuddin, Lukasz Kaiser, David Benjamin Belanger, Lucy Colwell, and Adrian Weller. Rethinking attention with performers. In International Conference on Learning Representations, 2021.
[34] Mitchell Wortsman, Jaehoon Lee, Justin Gilmer, and Simon Kornblith. Replacing softmax with relu in vision transformers. arXiv preprint arXiv: 08586, 2023.
[35] Jason Ramapuram, Federico Danieli, Eeshan Dhekane, Floris Weers, Dan Busbridge, Pierre Ablin, Tatiana Likhomanenko, Jagrit Digani, Zijin Gu, Amitis Shidani, al. Theory, analysis, and best practices for sigmoid self-attention. arXiv preprint arXiv:2409.04433 2024.
[36] Kai Shen, Junliang Guo, Xu Tan, Siliang Tang, Rui Wang, and Jiang Bian. A study on relu and softmax in transformer. arXiv preprint S11i:23002004666 2023.
[37] Yu Bai, Fan Chen, Huan Wang. Caiming Xiong, and Song Mei. Transformers statisticians: Provable in-context learning with in-context algorithm selection. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors, Advances in Neural Information Processing Systems, volume 36, pages 57125-57211. Curran Associates, Inc., 2023.
[38] Hengyu Fu, Tianyu Guo, Yu Bai, and Song Mei. What can a single attention layer learn? a study through the random features lens. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors, Advances in Neural Information Processing Systems, volume 36, pages 11912-11951. Curran Associates, Inc., 2023.
[39] Fanqi Yan, Huy Nguyen, Pedram Akbarian, Nhat Ho, and Alessandro Rinaldo. Sigmoid self- attention is better than softmax self-attention: A mixture-of-experts perspective. arXiv preprint arXiv:250..00281, 2025.
[40] Tri Dao and Albert Gu. Transformers are ssms: through structured state space duality. arXiv preprint 240,,,0000,,,000, 2024.
Generalized models and efficient algorithms
[41] Michael Hahn. Theoretical limitations of self-attention in neural sequence models. Transactions of the Association for Compulational Linguistics, 8: 8:156-171, 01 2020.
[42] Shaked Brody, Uri Alon, and Eran Yahav. On the expressivity role of layernor in transformers' attention, 2023.
[43] Ang Lv, Ruobing Xic, Shuaipeng Li, Jiayi Liao, Xingwu Sun, Zhanhui Kang, Di Wang, and Rui Yan. More expressive attention with negative weights, 2025.
[44] Dongchen Han, Yifan Pu, Zhuofan Xia, Yizeng Han, Xuran Pan, Xiu Li, Jiwen Lu, Shiji Song, and Gao Huang. Bridging the divide: Reconsidering softmax and linear attention. Advances in Neural Information Processing Systems, 37:79221-79245. 2024.
[45] Tianzhu Ye, Li Dong, Yuqing Xia, Yutao Sun, Yi Zhu, Gao Huang, and Furu Wei. Differential transformer. In The Thirteenth International Conference on Learning Representations, 2025.
[46) Yingcong Li, Davoud Ataee Tarzanagh, Ankit Singh Rawat, Maryam Fazel, and Samet Oymak. Gating is weighting: Understanding gated linear attention through in-context learning, 2025.
[47] Bo Peng. Eric Alcaide, Quentin Anthony, Alon Albalak, Samuel Arcadinho, Stella Biderman, Huanqi Cao, Xin Cheng, Michael Chung, Matteo Grella, et al. Rwkv: Reinventing mns for the transformer era. arXiv preprint arXiv:2305. 13048, 2023.
[48] Yixing Xu, Chao Li, Dong Li, Xiao Sheng, Fan Jiang, Lu Tian, and Emad Barsoum. Qt-vit: Improving linear attention in vit with quadratic taylor expansion. In A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomezak, and C. Zhang, editors, Advances in Neural Information Processing Systems, volume 37, pages 83048-83067. Curran Associates, Inc., 2024.
12

---

[49] Tobias Christian Nauen, Sebastian Palacio, and Andreas Dengel. Taylorshift: Shifting the com- plexity of selffattention from squared to linear (and back) using taylor-softmax. In Apostolos Antonacopoulos, Subhasis Chaudhuri, Rama Chellappa, Cheng-Lin Liu, Saumik Bhattacharya, and Umapada Pal, editors, Pattern Recognition, pages 1-16, Cham, 2025. Springer Nature Switzerland.
[50] Simran Arora, Sabri Eyuboglu, Michael Zhang, Aman Timalsina, Silas Alberti, Dylan Zinsley, James Zou, Atri Rudra, and Christopher Ré. Simple linear attention language models balance the recall- -hhouhut tradeoff, 2025.
[51] Yuwei Qiu, Kaihao Zhang, Chenxi Wang, Wenhan Luo, Hongdong Li, and Zhi Jin. Mb- taylorformer: Multi-branch efficient transformer expanded by taylor formula for image dehazing. 2023.
[52] Jianlin Su, Yu Lu, Shengfeng Pan, Ahmed Murtadha, Bo Wen, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding, 2023.
[53] Catherine Olsson, Nelson Elhage, Neel Nanda, Nicholas Joseph, Nova DasSarma, Tom Henighan, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Dawn Drain, Deep Ganguli, Zac Hatfield-Dodde Danny Hernandez, Scott Johnston, Andy Jones, Jackson Kernion, Liane Lovitt, Kamal Ndousse, Dario Amodei, Tom Brown, Jack Clark, Jared Kaplan, Sam McCandlish, and Chris Olah. In-context learning and induction heads, 2022.
[54] Michael Poli, Armin W Thomas, Eric Nguyen, Pragaash Ponnusamy, Björn Deiseroth, Kristian Kersting, Taiji Suzuki, Brian Hie, Stefano Ermon, Christopher Ré, Ce Zhang, and Stefano Massaroli. Mechanistic design and scaling of hybrid architectures, 2024.
[55] Michael Poli, Stefano Massaroli, Eric Nguyen, Daniel Y. Fu, Tri Dao, Stephen Baccus, Yoshua Bengio, Stefano Ermon, and Christopher Ré. Hyena hierarchy: Towards larger convolutional language models, 2023.
[56] Simran Arora, Sabri Eyuboglu, Aman Timalsina, Isys Johnson, Michael Poli, James Zou, Atri Rudra, and Christopher Ré. Zoology: Measuring and improving recall in efficient language models, 2023.
[57] Ekin Akyürek, Bailin Wang, Yoon Kim, and Jacob Andreas. In-context language learning: Architectures and algorithms, 2024.
[58] Yutao Sun, Li Dong, Shaohan Huang, Shuming Ma, Yuging Xia, Jilong Xue, Jianyong Wang, and Furu Wei. Retentive network: A successor to transformer for large language models, 2023.
[59] Bo Peng. Eric Alcaide, Quentin Anthony, Alon Albalak, Samuel Arcadinho. Stella Biderman, Huanqi Cao, Xin Cheng, Michael Chung, Matteo Grella, Kranthi Kiran GV, Xuzheng He, Haowen Hou, Jiaju Lin, Przemyslaw Kazienko, Jan Kocon, Jiaming Kong, Bartlomiej Koptyra, Hayden Lau, Krishna Sri Ipsit Mantri, Ferdinand Mom, Atsushi Saito, Guangyu Song, Xiangru Tang, Bolun Wang, Johan S. Wind, Stanislaw Wozniak, Ruichong Zhang, Zhenyuan Zhang, Qihang Zhao, Peng Zhou, Qinghua Zhou, Jian Zhu, and Rui-Jie Zhu. Rwkv: Reinventing mns for the transformer era, 2023.
[60] Zhen Qin, Songlin Yang, Weixuan Sun, Xuyang Shen, Dong Li, Weigao Sun, and Yiran Zhong. Hgmn2: Gated linear mns with state expansion. arXiv preprint arXiv:2404.07904, 2024.
[61] Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang. Horace He, Anish Thite, Noa Nabeshima, Shawn Presser, and Connor Leahy. The pile: An 800gb dataset of diverse text for language modeling, 2020.
[62] Shuangfei Zhai, Walter Talbott, Nitish Srivastava, Chen Huang, Hanlin Goh, Ruixiang Zhang, and Josh Susskind. An attention free transformer. 2021.
[63] Zhen Qin, Xiaodong Han, Weixuan Sun, Bowen He, Dong Li, Dongxu Li, Yuchao Dai, Lingpeng Kong, and Yiran Zhong. Toeplitz neural network for sequence modeling. The Eleventh International Conference Learning Representations, 2023.
13

---

[64] Zhen Qin, Songlin Yang, Weixuan Sun, Xuyang Shen, Dong Li, Weigao Sun, and Yiran Zhong. Hgm2: Gated linear mns with state expansion, 2024.
[65] Jiecheng Lu and Shihao Yang. Linear transformers as var models: Aligning autoregressive attention mechanisms with autoregressive forecasting, 2025.
[66] Yong Liu, Tengge Hu, Haoran Zhang, Haixu Wu, Shiyu Wang, Lintao Ma, and Mingsheng Long. itransformer: Inverted transformers are effective for time series forecasting. In The Twelfth International Conference Learning Representations, 2024.
[67] Yuqi Nie, Nam H Nguyen, Phanwadee Sinthong, and Jayant Kalagnanam. A time series is worth 64 words: Long-tern forecasting with transformers. In The Eleventh International Conference on Learning Representations, 2022.
[68] Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models, 2016.
[69] Haixu Wu, Jiehui Xu, Jianmin Wang, and Mingsheng Long. Autoformer: Decomposition trans- formers with cnto-correlltion for long-term series forecasting. In Mare' Aurelio Ranzato, Alina Beygelzimer, Yann N. Dauphin, Percy Liang, and Jennifer Wortman Vaughan, editors, Advances in Neural Information Processing Systems 34: Anmual Conference on Neural Information Processing Systems 1021, NeurlPS 2021, December 6-14, 2021, virtual, pages 22419-22433 2021.
[70] Guokun Lai, Wei-Cheng Chang, Yiming Yang, and Hanxiao Liu. Modeling long- and short- term temporal patterns with deep neural networks. In Kevyn Collins-Thomper Qiaozhu Mei, Brian D. Davison, Yiqun Liu, and Emine Yilmaz, editors, The 41st International ACM SIGIR Conference on Research Development Information Retrieval, SIGIR 2018, Ann Arbor, MI, USA, July 08-12, 2018, pages 95-104- ACM, 2018.
[71] Haoyi Zhou, Shanghang Zhang, Jieqi Peng. Shuai Zhang, Jianxin Li, Hui Xiong, and Wancai Zhang. Informer: Beyond efficient transformer for long sequence time-series forecasting. In Thirty-Fifth AAAI Conference Artificial Intelligence, AAAI 2021, Thirty-Thine Conference on Innovative Applications of Artificial Intelligence, IAAI 2021, The Eleventh Symposium on Educational Advances in Artificial Intelligence, EAAI 2021, Virtual Event, February 2-9, 2021, pages 11106-1 166--1111 11115. AAAI Press, 2021.
14

---

A Technical Appendices and Supplementary Material
A.1 Additional Theoretical Discussion
Proposition A.1 (Convex vs. Zero-sum Span). Let Vi...., By E Rd and denote their centroid by
Vavg Σ U/ Write the deviation matrix
AV
[ 11
Uavg.
11 Vavg E Rdxx
Define the convex hull and zero-sum spaces
C:
ag : a, 2 20
-a==1
wi =
Then
C = Vavg + {AV a a E 5 Uavg (AV E RR, 1 0} Vavg + Z, where A,-1 is the (t-1) dimensional probability simplex.
Corollary A.2 (Expressive Capacity). Assume the vs are affinely independent and 2 1 - 1, so that rank AV = 1. Then
dim = t- dim t-
but C is bounded, whereas Z is an unbounded linear subspace of the same dimension. Consequently S Vavg + 2, and zero-sum attention can realise outputs unattainable by any convex-combination
attention.
Proof sketch of Proposition A. and Corollary A.2
Convex representation. For any a A Ee O:2 = Uavg AV a.
2. Zero-sum representation. If w E R satisfies 1 w = 0, then Σ, W:4s = AV w.
3. Striet inclusion. The vector (1. -1,,,....... 0) lies in the hyperplane 1 0 but not in the simplex A,-1; hence Gavg + AV w E Uavg + 2 \. C.
4. Dimensionality. Under affine independence and 2 the matrix AV has rank t 1. Linear images preserve dimension, giving the stated dimensions of C and 2 and proving the corollary.
A.1.1 Proof of Proposition 3.1 (Convex vs. Zero-Sum Span).
Let {we) C Rdd and define
a.0 : aj 20,
Σ a.= a0=1) 2=
W;1i;: :
1c w==
1=1
1=1
1=1
and Bavg 1 E/ For any y E C with weights E A
Uavg Σ
with
O3 - 0.
==1
1=1
Hence the centered convex equals
Caey
w
w; 0.
wi
i=1
which is Z with extra lower bounds on the coefficients. Therefore Caev C Z.
To see the inclusion strict unless all v; are identical, pick # with v, + vn and consider Dj Uk E 2 (take Wi = 1, 1, others 0). If v, E Cdev. we would have
= -1, a, a = 0 (i $ kj, 1, ax
implying Og
for /. 2 2, a contradiction. Thus my 0s & Caevs and Cdev Ç Z whenever the v, are not all equal. If all v, coincide, both sets reduce to {0).
0
15

---

A.1.2 Proof of Corollary 3.2 Expressive Gain of Zero-Sum Attention).
In residual head + + Σ; w the deviation (relative to the average direction) produced by softmax weights
Lav a.u; - -avg Cdev.
Subtracting the zero-order term corresponds to w/ 2. By Proposition 3.1 Cdev ç 2 (for non-degenerate {wr }. so zero-sum attention strictly enlarges the attainable deviation set and therefore the head's ' expressivity. The only removed direction is the uniform average Vavg. which can be recovered across heads or layers.
1 with Σ, w; O, hence deviations lie in 7
A.1.3 Proof of Proposition 3.3
(Preservation of Affine Hull and Expressivity).
Let (uili- C RR, Davg = Σ Vi and A, 8; Vavg.
(i) Full softmax / with zero-order term. A single head with full softmax produces
Riull
a,V; 04 2 0, Σ
13 Conv(u;) S Aff(o:).
(ii) sero-sum (without zero-order term). If If 2im1 W; 0, then
W.U;
wi (Uav + A.) Uavg
A.,
i=
1=1 i=1
i=1
hence
R Rzero--um
u Σ we = 0 -Span (A,, ..... A.}.
Conversely, for any El U4 A E Span A let ū = Ei= i= and define wi = T. Then
E wi 0 and E i=1 w.p. Σ i=1 2 Σ Us Σ
u.(u. Davg) =E:- U, s.
Hence (pan R Reero-sum and thus Reero
(iii) Stacking and Minkowski sum. For any y Aff [b:}
write
y - Uavg + (y Uavg);
where Davg E Conv(v:) and, since E. A; = 0, we have Uarg E Spant Ai). Therefore
Conv(e:) v. + Span Aff
Combining (i) (ii) gives the claimed reachable sets for single heads and their equality to the affine hull when stacked.
A.1.4 Proof of Lemma 3.4 (Numerical Stability of Zero-Sum Softmax).
Assume ||l||| < B for all By the triangle inequality,
wis
2
Ierail ||m,||| S B max we,
==1
By the zero-sum softmax construction (see the main text), under bounded logits we have
<
max
7>
16r,1
JPcal 21
S = =
St.j
1=1
and 61,5. = O(1). Hence max; |wa,l O(1/t), and therefore
Wi, vs
$ Bt ot O(B).
which is independent of t.
16

---

A.1.5 Proof of Proposition 3.5 (Uniform Lipschitz Bound with 1/v2 Decay).
Let (x))) = 1/2 " == (x)) vs with Σi= (),,(x)) = 0 and || S B. For any a, x',
(x,((x)) (),)
B
(x,((x))
we, wr,,(='))
Vi
By the Lipschitz assumption on the weights, (wr,;(ar) wr,((x') S (Lw (t) |1ee x'Il, hence
[|o(()) (æ))ll B
B Lue
| x'll
vl x x'1|-
Thus the head is uniformly Lipschitz with constant B L Vt.
A.1.6 Implementation of ZeroS Softmax Attention (ZeroS-SM)
Zero-sum for Standard Softmax Attention (ZeroS-SM) As shown in Figure 5 our reweighted zero-sum approach can be directly applied to standard softmax attention using logits 8t, = q: kg /vd. with matrix form:
S
Vd QK M, softmax(S), u 2..., N. ; S diag(u) (SIx)
A = diag u) (S S1_1 = A diag(u) (u) 1N 1 A
W (g I A
oe. WV
A.2 Runtime Efficiency of the ZeroS Implementation
In recurrent (scan) form, ZeroS maintains three d x hidden-state bases
k. Vi. 8; k, V.
K.V
and reads them out with query- query-dependent gates; the outputs are summed. While single fused CUDA kernel is not implemented, we obtain practical implementation by invoking existing linear-attention scan three times with different key/value bases:
# prepare reweighted queries q1, q2, q3 from outi run_linattn (q1, k s_i_exp, V, mode= mode='fused sea''ffued__umaaea # s_i
(s_i)
out2 = run_ Lim_linattn (q2, k s_i, out3 run_linattn(33, k, out = outi out2 out3
v, mode= ? fused_ chunk')
We replace the attention layer in a GPT-2 style Transformer (hidden size 7688 12 heads, 12 layers) with various alternatives and evaluate at sequence length 1024. Baselines include implementations from the same library: LinAttn, GatedLinAttn, HGRN2, RWKV6, RWKV7, and softmax attention (naīve and FlashAttention). All runs use a single NVIDIA L40S, batch size 8, FP32. We report mean latency after warm-up. "Fwd" denotes fell-sequence inference (no KV/idden- cache). As shown in Table Under this three-scan implementation, ZeroS attains latency, throughput, and memory usage within the range of established linear-attention variants and close to FlashAttention this setup.
A.3 Additional Baselines and Ablations
Setup. We augment the main results with recent sequence-modeling baselines: Mamba2, Hawk, GatedDeltaNet, and HedgeDog. The evaluation protocol, datasets, and metries follow the main text.
17

---

Model
at
FweSSd
Train Ja
TrainSad
Thr.Ped (Bok2))
The (1lk5)
MemFww (GB)
Irain (GB)
NodimEx Altn (naive)
0.1306
0:0005
013114
120000
027400.7
14.574.8
9.01
10114
RWKV?
0.0876
0.0016
0.262n
0,0010
93,491.00
31,199.35
9.81
10.61
XWXV6
0.0261
0,0335
0.2252
0.0100
107.653.00
36,382992
9:49
9.62
Zeroš
0.3974
113,855.8
41,491
7.48
7.61
HGRNZ
0.0622
0.1480
121,955.16
55,336.55
6.14
6.41
122.919.71
Anto (FlachAmm)
0.3473
125.836.28
55,62077
5,56
GuedLinnAm
0.0000
0,0009
0.1231
0,0008
136.633,08
61,533.88
5.24
3.89
Table 7: Latency. throughput, and peak GPU memory on GPT-2 (768/12/12), sequence length 1024, batch size 8, FP32 on single L40S. "Fwd"" full-sequence inference without caches.
Model
Compress
Fuzzy Recall
In-ConlextRecall
Memnorie
Noisy Recall
SelectiveCopy
Average
ZeroS (Lin)
44.0
14.9
99.9
88.1
96.
97.8
73.5
LinAtto
33.1
91.0
74.9
93.1
62.3
ZeroS (SottmaxAtin)
45.2
28.0
100.0
84.3
96.6
98.5
75.4
SoftmaxAAnn
$1.6
29.8
94.1
85.2
86.8
99.6
74.5
Mamba2
43.6
21.1
96.4
86.9
96.7
93.3
73.0
Hawk
47.7
13.6
93.0
91.3
93.0
77.0
64.5
GatedDeltaNet
45.0
29.8
100.0
80.2
100.0
94.3
74.9
HodgeDog
43.2
17.9
55,9
83.4
46.0
98.4
57.4
Ablation Study
Zero8
44,0
14.9
88.1
96.1
73.5
w/o Angular
39.5
8.5
42,8
54.5
44.8
63.3
42.2
Ang: PosEmb
35.8
73.3
46.2
45.8
46.1
Ang: additive PosEmb
38.1
14.2
94.1
86.6
87.2
93.8
69.0
Rad: (lincar proj)
41.2
15.5
91.9
88.3
86.1
97.2
70.0
Rad: (quad form)
40.9
15.4
92.6
86.6
98.5
70.8
Rad: (2-distance)
40.1
14.8
97.6
82,5
93.6
97.8
71.1
Rad: du (averaging)
41.0
15.0
99.9
93.3
89.6
98.5
73.0
Table 8: Additional baselines and ablations on six MAD tasks; higher is better.
Results. Table reports task accuracies (%). ZeroS attains high scores In-Context Recall and Noisy Recall and yields a strong overall average. Ablations indicate that removing the angular component substantially degrades performance; additive positional embeddings help but do not match RoPE; and the default radial scoring (with negative similarity) achieves the best average among radial variants.
A.4 Illustrative Zero-Sum Construction Examples
Setup. Let [v:)== Rdd A single softmax attention layer produces o E, a,v; with a; 2 and E, a, 1, hence E Conv( (onv((v;)). A single ZeroS layer produce signed, zero-sum combinations o = Σ; w.V; with Σ, W, = 0. Below we list simple sequence-to-sequenee to-seueence mappings that are not representable by single softmax-attentee layer but representable single ZeroS layer.
softmax-attention
Example: Two-token difference. Target = V2. Softmax requires 1, -1 (invalid). ZeroS: W1 = 1, W2 = -1, others 0.
Example: Difference from the mean. Target
ve. Softmax needs negative mass on
:=1
(vihes (invalid). ZeroS: wi = and W1s1 - (so E,
w. 0).
1/2
Example: Alternating differences. For even target o Σ
(v2-11 V2.). Softmax cannot
1=1
realize alternating + weights in one layer. ZeroS: w2
W2 -1 (others 0).
A.5 Additional Dateset Description
A.5.1 MQAR
We adopt the Multi- Multi-Query Associative Recall (MQAR) task introduced by (56) to characterize model's performance repeated, input-dependent lookups over a large vocabulary in single
18

---

forward pass. In the classic associative recall (AR) problem, we store small, static dictionary of key-value pairs and issue single, fixed-position query. MQAR generalizes AR by interleaving multiple key-value pairs, each encoded as two consecutive tokens (hy 12), and allows multiple query tokens anywhere in a sequence of length N. Formally, given
X (To,..,, IN-1),
T; E V
whenever H, k, for some the correct output is
84 = Uj I/+1
and the model must satisfy this for all 1 S < N. By requiring repeated lookups at arbitrary positions, MQAR provides sharp test of dynamic routing and associative recall, directly contrasting these mechanisms with softmax attention's 's flexibility and capacity to handle multiple simultaneous queries.
A.5.2 REGBENCH
RegBench [57 is a synthetic in-context learning benchmark that evaluates a model'ss ability to infer the structure of regular languages from only a few example strings provided in the prompt. Each problem instance presents E [10, 20] example strings (d ...., dx drawn from the same stochastic regular language ( defined by probabilistic finite automaton (PFA). To construct the PFA, RegBench samples a minimal deterministic finite automaton (DFA), the canonical formalization of regular languages. RegBench draws
~ Uniform (4. 12), C~ Uniform(4, 18),
Uniform (1.forrm11.44 (1.4),
and then samples language-specific alphabet Σ of size uniformly without replacement from global symbol set size Cuax = 18. Define the state set = [Si..... Sn L {So) with accepting subset S. {S1,..... S.). For each S; uniformly without replacement select m, symbols xy E E and my target states Sj E S { to form edges (S,, æj, Sj), send all other symbols to So. and minimize via Hopcroft's algorithm to obtain the canonical DFA A'. The PFA inherits A's topology, assigning
1
T(S my
2'S,,z','') = otherwise,
so that Eoer ExeS T(s, a, s') = Vs S.. From this PFA, K strings of length ~ Uniform(1, 50) are sampled from So by simulating (I,, ~ T(S and concatenating 21 -gg Models then perform greedy next-token predictions
i arg max (= ()
# and we report DFA accuracy as in Akyürek et al. (2024)
accuracy (po, L,) NT ΣΣ
1
Ex'
E (x':
Lild' a 0
where NT is the total number of tokens in the test set and L,(x' 2 is the probability of predicting x' following context ( in the language L,. We consider DFA accuracy, the fraction of predictions that correspond to valid transitions the original DFA, as a direct measure how faithfully the model has internalized the underlying regular-language structure.
A.5.3 MAD
We evaluate our proposed architecture using the Mechanistic Architecture Design (MAD) framework. recently developed methodology for cost-effective evaluation of deep learning architectures [54]. MAD consists of a suite of aapabiliyy-argetter benchmarks. including in-context recall, fuzzy recall, selective copying, and compression, that probe fundamental sequence modeling capabilities. This approach has been rigorously validated through extensive experimentation spanning over 500 language models from 70M to 7B parameters, demonstrating a strong correlation between performance on these targeted synthetic tasks and compute-optimal perplexity at scale. Through the employment MAD, which serves as a reliable predictor of large-scale performance, we identify performance advantages without the need for prohibitive computational resources typically associated with architecture validation.
A.5.4 110
19

---

WikiText- 103 [68] is a large-scale language modeling dataset of over 103 million words compiled from 23, 20885 Good and 4,790 Fea- tured Wikipedia articles that have been re- viewed by humans, represent broad cov- erage, and meet common editorial stan- dards. The dataset has long context win- dows, large vocabulary of 267,735 types, and requires preservation of case, punctu- ation, and numerical information so that WikiText- 103 accurately reflects the chal- lenges of real-world text.
Reweighted Zero-sum
Layer Norm
Softmax
Logit
O2
(RoPE)
W
W
W
W k
W
9
{0.)i=1
Figure 5: Block Architecture of The ZeroS-SM Layer
A.5.5 Open Text222
Wee is large-scale, cleaned and deduplicated web-text corpus created open reproduction of OpenAl's WebText dataset: URLs are first extracted from all Reddit submissions with combined greater than 3, then scraped, filtered, and deduplicated at both URL and document levels (using MinHash-LSH) to remove low-quality or redundant content. The resulting "plug-and- play" release comprises 17,103,059 documents (around 65.86 GB uncompressed), covering Reddit submissions from 2005 through April 2020, and serves as a high-diversity, up-to-date pretraining corpus for large language models. We use the code environment provided by nanoGPT to implement this dataset training.
A.5.6 Time Series
We evaluate our module on the time series forecasting benchmark datasets below, following the
experimental setup of [65]. (1) Weather [69 21 metcorological variables (e.g.. temperature
humiditty) collected every 10 minutes in 2020 from a weather station in Germany. (2) Solar (70]
Solar power outputs recorded every 10 minutes in 2006 from 137 photovoltaic plants in the U.S. (3)
ETT Transformer load and temperature data sampled 15-minute (ETTml/ETTm2) and
hourly (ET (TT///ETT)) intervals from July 2016 to July 2018, including 7 key operational features.
A.6 Additional Description of Experimental Settings
Our detailed experimental setup is available the provided code repository. All experiments introduced in this paper can be run on a single Nvidia RTX 4090. For faster training, we parallelize experiments across multiple GPUs. In all benchmarks, we replace the multi-head attention layers with ZeroS layers without modifying any other settings. We do not apply first-layer O-th order term correction or the 1/vL variance scaling in any of our experiments.
A.7 Impact Statement
This paper introduces efficient attention mechanism for transformer-basee models. As fun- damental architectural improvement, ZeroS primarily affects upstream model capabilities rather than specific applications. The positive impacts include potential reductions in computational costs when processing long sequences. Like most foundational ML research, this work could indirectly contribute to both beneficial and potentially harmful applications depending how downstream models implement it. However, as an architectural component rather than a deployed system, ZeroS itself poses minimal direct societal concerns.
http:///www.neellggo////////aaap html
20

---

NeurIPS Paper Checklist
The checklist is designed to encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No], or [NA].
[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1-2 sentence) justifiicton right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.
The reviewers of your paper will be asked to use the checklist as one of the factors in their evaluation.
While "[Yes] " is generally preferable to "[No]", ", it is perfectly acceptable to answer "[No] provided proper justification is given (e.g., "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering "[No] or "[NA] is not grounds for rejection. While the questions are phrased in binary way, we acknowledge that the true answer often more nuanced, so please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS Paper Checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only use the provided macros for your answers.
Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The main claims made in the abstract and introduction accurately reflect the paper's scope, shown in section 2-4.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes]
21

---

Justification:
Yes, as shown in last section of the main text.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model vell-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [Yes]
Justification: Yes, the paper make sure the assumptions and proof are accurate.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: Yes. Please see the provided anonymous code repository.
Guidelines:
The answer NA means that the paper does not include experiments.
22

---

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g.. with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors welcome describe the particular way they provide for reproducibilityy In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: Yes, we have listed all the code environments used in the experiment, which includes an introduction to the data that was applied.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips. cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https //tpps.cc/publicc///cccc
www.deddeee.oeuuummmcecomd ssi ssionyolicy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
23

---

Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e-g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Yes, we fully inherited the code from the benchmark code environment used, without making any modifications to the hyperparameters. This paper, simple extension to the softmax attention and linear attention modules, does not specify any additional hyperparameters.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: We fully rely on the scores generated by the referenced benchmark experimen- tal environments. In experiments such as MAD and RegBench, they conduct large-scale hyperparameter searches and select the optimal results. Error bars are not applicable to these scores.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall
run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the eтror bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (c.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
24

---

Justification: Yes, all tasks used in this paper can be trained on the single Nvidia RTX 4090 GPU that we used. We accelerated the experiments by running multiple tasks in parallel, and we did not have any large-scale tasks that involved using multiple GPUs for distributed training for a single task.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental rums as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code of ethies
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics http::///aarrp...cppaaaaa//aaaaa htcc/uublic/Ethicssuiiddee
Answer:
[Yes]
Justification: Yes, this study fully complies with the Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is special consid- cration due to laws or regulations in their jurisdiction).
Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Yes, we include a impact statement section in the appendix.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (c.g., deploymennt of technologics that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
25

---

11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
This paper poses no such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: Yes, we have provided a complete list of all environments used in the experi- mental section, where all related assets information can be found.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website). the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, paperswi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors encouraged to reach out to the asset's 's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Yes, we have presented the complete code we used in an anonymous code repository.
Justification:
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
26

---

The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/teview based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMSs if it is important, original, or non-standard component of the core methods in this research? Note that the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Justification:
The core method development this research does not involve LLMs.
Guidelines:
The answer NA means that the method development in this research does not involve LLMs as any important, original, or non non-standard components.
Please refer to our LLM policy http://aarrppp.cccooooeeeereee// for what should or should be described.
27


Reviews:
[{"review_id": "nb7PFTP7L2", "reviewer": "Reviewer_WRY9", "summary": "This work identifies key issues in the Linear Attention method proposed in [1] and introduces Zero-Sum Linear Attention to address them. First, it argues that the zeroth-order term in attention dilutes the weights. Removing this term not only resolves that issue but also permits negative weights—naturally contrastive—building on insights from Differential Attention [2]. The authors further show that zero-sum weights can improve expressivity over the convex combination in standard attention and that the zero-sum form is stable under bounded logits. They then discuss the benefits of decoupling direction and magnitude. Finally, they demonstrate that the inherent negative weights and stability of zero-sum attention let them drop the angular term ($\\cos\\theta$) from the denominator, seamlessly integrating with RoPE as a by-product. Extensive experiments across diverse benchmarks and domains indicate that Zero-Sum Linear Attention matches standard attention closely while outperforming other linear variants.\n\nReferences:\n\n[1] Fast Autoregressive Transformers with Linear Attention\n\n[2] Differential Transformer", "questions": "**Questions and Suggestion** \n\n- Line 34: “In long contexts, attention mechanisms incorporate a roughly uniform component … introducing a persistent averaging effect …”\n   \nFor standard softmax attention, large query–key dot products often make the distribution very peaky. emperically the attention patterns also tend to be sparse in the upper layers, so the averaging effect may be less pronounced than implied. For linear attention though, the averaging effect tends to be more pronounced. \n\n- Line 98.: “As weights become more uniform …”\n  Assuming this about attention weights, i would suggest to make this explicit.\n\n- Line 227. In the linear recurrent equations, the zeroth-order term appears to be absent. It would help to state this explicitly to remind the readers.\n\n- A short discussion for the non-causal variant would help improve the clarity for the paper.\n\n- Table 1. Any hypothesis for the weaker performance on the Fuzzy Recall task?\n\n- To keep the paper self-contained, please add brief descriptions of the benchmarks—particularly RegBench and MAD in appendix.", "limitations": "None", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 4, "originality": 4, "strengths_and_weaknesses": "**Strengths**\n\n- The paper proposes several novel ideas:\n  - removing the zeroth-order term,\n  - decoupling angular and radial components, and\n  - using effective gating plus higher-order terms via differentiation—together yielding improved accuracy for linear-attention methods.\n- The manuscript is generally clear, with only minor writing issues.\n- The contributions are well motivated, supported by sound theoretical arguments and strong empirical results.\n\n**Weaknesses**\n- The major weakness is the absence of an efficient implementation for the recurrent equations. Although the authors note this in the “Limitations” section, it remains a significant limitation for practical adoption. At minimum, please report time- and memory-usage comparisons against standard attention implementations such as FlashAttention-2.\n\n- Typos / notation:\n  - Line 146: “$\\sum_i w_i$ = …” (missing RHS).\n  - Line 196: “output be$o_t$” → “output be $o_t$”.\n  - Equation above line 208: should the index be $\\varepsilon_{ti}$, not $\\varepsilon_{i}$?\n  - Line 208: should this be $\\sigma_t^1 = \\operatorname{sigmoid}(x_t W_g^1)$ ? and similarly for $\\sigma_t^h$.", "was_revised": true, "final_justification": "I have read the reviews and rebuttal and am satisfied with the response. Thus I have kept my original rating of acceptance", "rebuttal": "```\nthe absence of an efficient implementation for the recurrent equations...\n```\n\nThank you very much for your recognition of our paper and for your insightful questions. We will include a concise version of the following explanation in the main paper and provide an extended version in the appendix during the revision.\n\nIn recurrence, ZeroS performs associative scans over hidden states corresponding to 3 different $d \\times d$ bases: $e^{s_i} \\hat{\\mathbf{k}}_i^\\top\\mathbf{v}_i$, $s_i \\hat{\\mathbf{k}}_i^\\top \\mathbf{v}_i$, and $\\hat{\\mathbf{k}}_i^\\top \\mathbf{v}_i$. The final output is computed by reading out the scanned results using queries with different gating weights and summing them. Although we have not yet implemented a fused CUDA kernel that performs all scans in a single forward, we can leverage existing linear attention scan implementations (flash linear attention [1]) to perform three separate scans with different key and value bases. This provides a partially efficient implementation:\n\n\n```python\n# prepare reweighted query q1, q2, and q3 from q...\nout1 = run_linattn(q1, k*s_i_exp, v, mode='fused_chunk')\nout2 = run_linattn(q2, k*s_i, v, mode='fused_chunk')\nout3 = run_linattn(q3, k, v, mode='fused_chunk')\nout = out1 + out2 + out3\n# ...\n```\n\nTo evaluate the runtime efficiency of ZeroS under this implementation, we benchmarked it by replacing the attention layers in a GPT-2 Transformer architecture (dimension = 768, number of heads = 12, sequence length = 1024, number of layers = 12) with various attention variants. We adopted the Transformer and softmax attention implementations (both flash and naive versions) from NanoGPT \\[2]. In addition to our partially efficient ZeroS implementation (based on the `fused_chunk_linear_attn` function from flash linear attention \\[1]), we also included the following layers provided by the same library as baselines for comparison: \\[LinAttn, GatedLinAttn, HGRN2, RWKV6, RWKV7], using their default configurations aside from the GPT-2 settings above. All experiments were conducted on a single Nvidia L40S GPU, using a batch size of 8 and FP32 precision. We measured performance in both training and inference modes (Fwd; full-sequence inference without KV-cache or hidden state cache). After several warm-up forward passes, we recorded the average forward latency, standard error, and peak memory usage using multiple runs synchronized with `torch.cuda.synchronize()`. The results are summarized below.\n\n\n| Model | FwdLatency(s) | FwdStd(s) | TrainLatency(s) | TrainStd(s) | ThroughputFwd(tok/s) | ThroughputTrain(tok/s) | MemPeakFwd(GB) | MemPeakTrain(GB) |\n|---|---|---|---|---|---|---|---|---|\n| SoftMaxAttn (Naïve) | 0.1306 | 0.0006 | 0.3334 | 0.0009 | 62740.89 | 24574.18 | 9.61 | 10.34 |\n| RWKV7 | 0.0876 | 0.0016 | 0.2626 | 0.0010 | 93491.90 | 31199.35 | 9.81 | 10.61 |\n| RWKV6 | 0.0761 | 0.0005 | 0.2252 | 0.0010 | 107653.00 | 36382.92 | 9.49 | 9.62 |\n| ZeroS | 0.0720 | 0.0008 | 0.1974 | 0.0011 | 113855.38 | 41491.29 | 7.48 | 7.61 |\n| HGRN2 | 0.0672 | 0.0013 | 0.1480 | 0.0009 | 121955.16 | 55336.55 | 6.14 | 6.43 |\n| LinAttn | 0.0666 | 0.0010 | 0.1477 | 0.0009 | 122949.71 | 55447.86 | 5.79 | 5.90 |\n| SoftmaxAttn (FlashAttn) | 0.0651 | 0.0014 | 0.1473 | 0.0008 | 125836.28 | 55620.75 | 5.45 | 5.56 |\n| GatedLinAttn | 0.0600 | 0.0009 | 0.1331 | 0.0008 | 136633.08 | 61533.68 | 5.74 | 5.89 |\n\nCompared to other linear attention variants, ZeroS, even without a fully fused associative scan implementation, achieves efficiency on par with the average level of existing architectures, and is not far behind standard linear attention.\n\n```\nTypos / notation:\n```\n\nThank you very much for your careful review: your feedback is very helpful in improving the quality of our paper. We sincerely apologize for the errors introduced due to space reduction in the theory section during final editing.\n\n* In Line 146, the RHS is missing the zero-sum weights condition, i.e., it should include $= 0$.\n* Line 196 will have a better formatting.\n* For Line 208, we apologize for the confusion. It should read: $\\sigma\\_t^1 = \\text{sigmoid}(\\mathbf{x}\\_t \\mathbf{W}\\_{g}^1), \\quad \\sigma\\_t^h = \\text{sigmoid}(\\mathbf{x}\\_t \\mathbf{W}\\_{g}^h)$, indicating that the gating weights are determined by the input information at step \\$t\\$.\n\n\n```\nLine 34: “In long contexts, attention mechanisms incorporate a roughly uniform component … introducing a persistent averaging effect …”\n```\n\nWe apologize for the confusion. Your interpretation aligns with our intended meaning. Our logic here is that the expressiveness of softmax attention arises from its ability to deviate from the average distribution (e.g., produce peaked patterns). However, softmax inherently includes a zero-order averaging effect, approximately $1/t$, which is input-independent. The higher-order deviations are responsible for offsetting this persistent averaging and producing more expressive distributions like sharp peaks.\n\nBy removing this zero-order constant normalization, we can free the higher-order terms from the burden of canceling out the averaging effect. This allows them to more efficiently model peaked distributions aligned in QK interactions. Furthermore, we show that eliminating the zero-order term (via zero-sum attention weights) does not lead to numerical instability as sequence length increases, while it enhances expressivity by enabling modeling capabilities, such as differential and contrast patterns that a single softmax attention layer cannot represent. We believe this makes zero-sum attention a more effective form.\n\nWe will revise Section 3.1 to better clarify this point.\n\n```\nLine 98.: “As weights become more uniform …” Assuming this about attention weights\n```\n\nWe will revise the sentence to explicitly include the term \"attention weights.\" \n\n```\nLine 227. ...the zeroth-order term appears to be absent. It would help to state this explicitly to remind the readers.\n```\n\nThank you for the suggestion. We will explicitly state around Line 227 that the zero-order term has a fixed gating factor of zero. This makes clear that, unlike higher-order terms with trainable gating, the zero-order normalization is intentionally disabled.\n\n```\nA short discussion for the non-causal variant would help improve the clarity for the paper.\n```\n\nThank you for the suggestion. We will add a discussion on this in the appendix. Similar to prior work on linear attention, our method can be made causal or non-causal depending on how the KV hidden states are handled: using a prefix scan results in a causal variant, while applying a average-pooling hidden state across all output steps yields the non-causal version.\n\n```\nTable 1. Any hypothesis for the weaker performance on the Fuzzy Recall task?\n```\n\nThank you for this insightful question. As shown in Table 1, ZeroS shows the most significant improvements over linear and softmax attention on the *In-Context Recall* and *Noisy Recall* tasks. These tasks focus on accurately retrieving specific earlier information from complex sequences (a core challenge in language modeling). The observed performance gains align with our motivation for ZeroS, which is to enhance the expressiveness of higher-order deviations beyond the averaging effects of standard attention.\n\nBy contrast, the *Fuzzy Recall* task is more like soft averaging over the input context, which can be effectively handled by simpler, lower-cost structures. Modeling precise alignment is less critical here, and models that naturally perform soft averaging may have a structural bias advantage. Even so, ZeroS still shows a clear improvement over the original linear attention baseline, and only slightly underperforms softmax attention, likely because it lacks the same inductive bias toward averaging input tokens, which the model must instead learn explicitly. Further discussion can be found in our response to Reviewer FheY.\n\n```\nTo keep the paper self-contained, please add brief descriptions of the benchmarks—particularly RegBench and MAD in appendix.\n```\n\nThank you for the suggestion. We will include more task-specific descriptions of RegBench and MAD in Appendix A.3 to make the paper more self-contained.\n\n### References\n\n[1] Yang, S., & Zhang, Y. (2024). Fla: A triton-based library for hardware-efficient implementations of linear attention mechanism (https://github.com/fla-org/flash-linear-attention)\n\n[2] Karpathy, A. (2022). NanoGPT. GitHub Repository. https://github.com/karpathy/nanoGPT"}, {"review_id": "pILBVdRAxP", "reviewer": "Reviewer_kvKy", "summary": "The paper addresses the performance gap between efficient linear attention models and standard quadratic softmax attention, which is attributed to the positive convex weights that makes subtraction impossible. The authors propose a zero-sum attention mechanism so that the weights can be both positive and negative, which leads to both quadratic and linear attention variants. Experiments in language modeling, in-context learning, and time series forecasting show that ZeroS matches or exceeds the performance of baselines.", "questions": "The reviewer would love to see the concerns in the above addressed and revise the rating.", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 2, "originality": 3, "strengths_and_weaknesses": "## Significance\n\n1. Metrics: A major metric that is missing from the evaluation is the wall-clock running time.   \n- On the one hand, I understand that the asymptotic complexities have been provided.   \n- On the other hand, there are aspects not captured by such complexities, including how friendly the computations are to caching, the constants in the asymptotics which matter before sufficiently large N and d are reached, etc.   \n- It might be demanding to do it for all baselines, but at least comparing representative linear/quadratic baselines as well as variants in your ablation study can give a better picture. For instance, MultiHead Hyena, DeltaNet, and Transformer \\- how do their running time compare with that of ZeroS variants?   \n2. Baselines: Since efficient transformer architecture is a fast paced direction, it helps make an impact by comparing with recent open-source baselines  \n- (ICLR 24\\) https://openreview.net/forum?id=Eh0Od2BJIM  \n- (ICLR 25\\) https://openreview.net/forum?id=lXRDQsiP2v\n\n## Clarity\n\n1. Figure 1 is a bit complicated and hard to parse. For a novice reader that scans the figures before reading the paper, it is hard to appreciate the method. Specifically:  \n- There is no label of {u\\_i} in figure 1 or its caption. One has to refer to the text.  \n- The authors may consider contrasting the proposed architecture with a representative baseline, so that it is immediately clear which parts are similar, and which parts are different (and novel).   \n2. Figure 2 has very small labels and legends making it difficult to read.  \n3. Line 134-136 has little context, since the proposed approach has not been introduced yet.  \n4. Typos:   \n- Line 146: something is missing after “∑ i w\\_i=”  \n- The equation block below line 207 has duplications", "was_revised": true, "final_justification": "I am happy with the experiments on space and time complexity, as well as the explanation of efficient forward pass. On the other hand, based on the authors response, I am concerned about the impact of experimenting on a benchmark with context length smaller than 4K: modern applications requires LLMs to reason with a large context length, and this is the regime where a method being both effective and efficient can make a difference. Therefore, I maintain my original rating.", "rebuttal": "```\nMetrics: A major metric that is missing from the evaluation is the wall-clock running time.\n```\n\nThank you very much for your valuable question and suggestion. This is extremely helpful for users interested in implementing ZeroS in practice. We will include a concise version of the following explanation in the main paper and provide a more detailed version in the appendix during the revision.\n\n\nIn recurrence, ZeroS performs associative scans over hidden states corresponding to 3 different $d \\times d$ bases: $e^{s_i} \\hat{\\mathbf{k}}_i^\\top\\mathbf{v}_i$, $s_i \\hat{\\mathbf{k}}_i^\\top \\mathbf{v}_i$, and $\\hat{\\mathbf{k}}_i^\\top \\mathbf{v}_i$. The final output is computed by reading out the scanned results using queries with different gating weights and summing them. Although we have not yet implemented a fused CUDA kernel that performs all scans in a single forward, we can leverage existing linear attention scan implementations (flash linear attention [1]) to perform three separate scans with different key and value bases. This provides a partially efficient implementation:\n\n\n```python\n# prepare reweighted query q1, q2, and q3 from q...\nout1 = run_linattn(q1, k*s_i_exp, v, mode='fused_chunk')\nout2 = run_linattn(q2, k*s_i, v, mode='fused_chunk')\nout3 = run_linattn(q3, k, v, mode='fused_chunk')\nout = out1 + out2 + out3\n# ...\n```\n\nTo evaluate the runtime efficiency of ZeroS under this implementation, we benchmarked it by replacing the attention layers in a GPT-2 Transformer architecture (dimension = 768, number of heads = 12, sequence length = 1024, number of layers = 12) with various attention variants. We adopted the Transformer and softmax attention implementations (both flash and naive versions) from NanoGPT \\[2]. In addition to our partially efficient ZeroS implementation (based on the `fused_chunk_linear_attn` function from flash linear attention \\[1]), we also included the following layers provided by the same library as baselines for comparison: \\[LinAttn, GatedLinAttn, HGRN2, RWKV6, RWKV7], using their default configurations aside from the GPT-2 settings above. All experiments were conducted on a single Nvidia L40S GPU, using a batch size of 8 and FP32 precision. We measured performance in both training and inference modes (Fwd; full-sequence inference without KV-cache or hidden state cache). After several warm-up forward passes, we recorded the average forward latency, standard error, and peak memory usage using multiple runs synchronized with `torch.cuda.synchronize()`. The results are summarized below.\n\n\n| Model | FwdLatency(s) | FwdStd(s) | TrainLatency(s) | TrainStd(s) | ThroughputFwd(tok/s) | ThroughputTrain(tok/s) | MemPeakFwd(GB) | MemPeakTrain(GB) |\n|---|---|---|---|---|---|---|---|---|\n| SoftMaxAttn (Naïve) | 0.1306 | 0.0006 | 0.3334 | 0.0009 | 62740.89 | 24574.18 | 9.61 | 10.34 |\n| RWKV7 | 0.0876 | 0.0016 | 0.2626 | 0.0010 | 93491.90 | 31199.35 | 9.81 | 10.61 |\n| RWKV6 | 0.0761 | 0.0005 | 0.2252 | 0.0010 | 107653.00 | 36382.92 | 9.49 | 9.62 |\n| ZeroS | 0.0720 | 0.0008 | 0.1974 | 0.0011 | 113855.38 | 41491.29 | 7.48 | 7.61 |\n| HGRN2 | 0.0672 | 0.0013 | 0.1480 | 0.0009 | 121955.16 | 55336.55 | 6.14 | 6.43 |\n| LinAttn | 0.0666 | 0.0010 | 0.1477 | 0.0009 | 122949.71 | 55447.86 | 5.79 | 5.90 |\n| SoftmaxAttn (FlashAttn) | 0.0651 | 0.0014 | 0.1473 | 0.0008 | 125836.28 | 55620.75 | 5.45 | 5.56 |\n| GatedLinAttn | 0.0600 | 0.0009 | 0.1331 | 0.0008 | 136633.08 | 61533.68 | 5.74 | 5.89 |\n\nCompared to other linear attention variants, ZeroS, even without a fully fused associative scan implementation, achieves efficiency on par with the average level of existing architectures, and is not far behind standard linear attention.\n\n```\nAbout more baselines...\n```\n\nThank you very much for your helpful suggestions regarding the experimental section. Taking into account feedback from all reviewers, we have added several recently developed attention variants as additional baselines for comparison against ZeroS on the MAD benchmark. These include: [HedgeDog, Hawk, Mamba2, xLSTM, Gated DeltaNet].\n\nRegarding the two baselines you specifically mentioned:\n\n1. For HyperAttention, according to its official implementation (https://github.com/insuhan/hyper-attn), the default value for `min_seq_len` (which determines when hyper attention is activated) is set to 4096. If the input sequence length is below this threshold, the model simply defaults to flash attention instead of using hyper attention. Since the sequence lengths in our benchmark tasks are generally below this threshold, the behavior of hyper attention essentially mirrors that of softmax attention in this setting. Therefore, you may interpret the softmax attention results in our baselines as a proxy for HyperAttention performance under these conditions.\n\n2. For ToST [3], the main focus of the paper is on non-causal sequence modeling tasks such as visual tasks and LRA. Causal modeling is only briefly discussed as an extension, and there is no comparison against other models in that context. In contrast, our paper focuses primarily on causal tasks, with the MAD benchmark mainly about causal scenarios. To avoid biased comparisons across different task scopes, we followed the precedent set by other recent works in selecting baselines and did not include ToST in our experiments. \n\nThat said, we will expand the discussion in the related works section of our revision to better contextualize and contrast attention variants designed for different modeling purposes. Thank you again for the suggestion.\n\nWe present the extended comparison with newly added baselines as follows:\n\n| Model | Compress | Fuzzy Recall | In-Context Recall | Memorize | Noisy Recall | Selective Copy | Average |\n|---|---|---|---|---|---|---|---|\n| ZeroS (Softmax Attn) | 45.2 | 28.0 | 100.0 | 84.3 | 96.6 | 98.5 | 75.4 |\n| Softmax Attn | 51.6 | 29.8 | 94.1 | 85.2 | 86.8 | 99.6 | 74.5 |\n| ZeroS (Lin) | 44.0 | 14.9 | 99.9 | 88.1 | 96.1 | 97.8 | 73.5 |\n| xLSTM | 43.4 | 47.6 | 100.0 | 79.8 | 100.0 | 95.4 | 73.2 |\n| Mamba2 | 43.6 | 21.1 | 96.4 | 86.9 | 96.7 | 93.3 | 73.0 |\n| GDeltaNet | 45.0 | 32.7 | 100.0 | 81.7 | 100.0 | 95.7 | 71.0 |\n| Hawk | 47.7 | 13.6 | 93.0 | 91.3 | 93.0 | 77.0 | 64.5 |\n| LinAttn | 33.1 | 8.2 | 91.0 | 74.9 | 75.6 | 93.1 | 62.3 |\n| HedgeDog | 43.2 | 17.9 | 55.9 | 83.4 | 46.0 | 98.4 | 57.4 |\n\nAs shown, ZeroS maintains a consistent advantage in average performance across a variety of task types compared to recent baselines. Further discussion can be found in our response to Reviewer FheY.\n\n```\nFigure 1 is a bit complicated and hard to parse...\n```\n\nThank you very much for your valuable comments and suggestions: they are extremely helpful in improving the clarity and readability of our paper. We will address these issues in the revision.\n\nFor the notation $\\{u_i\\}$, we will add a clearer annotation indicating that it is obtained by applying a linear projection layer $\\mathbf{W}_u$ to the input $\\{x_i\\}$.\n\nRegarding the comment \"contrasting the proposed architecture with a representative baseline...\", we will revise Figure 1(c) to explicitly label the *Angular* and *Value* components as common elements found in existing linear attention variants, while highlighting the *Radial* component as the main focus and novel contribution of our work.\n\n```\nFigure 2 has very small labels and legends making it difficult to read.\n```\n\nThank you for pointing this out. We will redraw Figure 2 to improve readability by enlarging the legends and labels.\n\n```\nLine 134-136 has little context...\n```\n\nThank you for the suggestion. We will revise this section to ensure that the discussion is clearly contextualized within the topic of the softmax function. Any extended discussion about its application in attention will be moved to a more appropriate part of the paper.\n\n```\nLine 146: something is missing after “∑ i w_i=”\n```\n\nWe greatly appreciate your close reading. Apologies for this oversight: it was caused by compression and restructuring during final editing of the mathematical sections. The right-hand side of the equation at Line 146 is indeed missing the zero-sum constraint, i.e., it should be $ \\sum_i w_i = 0 $. We will correct this in the revision. Additionally, in Line 207, we will retain the second line and remove the preceding portion for clarity.\n\n\n### References\n\n[1] Yang, S., & Zhang, Y. (2024). Fla: A triton-based library for hardware-efficient implementations of linear attention mechanism (https://github.com/fla-org/flash-linear-attention)\n\n[2] Karpathy, A. (2022). NanoGPT. GitHub Repository. https://github.com/karpathy/nanoGPT\n\n[3] Wu, Z., Ding, T., Lu, Y., Pai, D., Zhang, J., Wang, W., ... & Haeffele, B. D. Token Statistics Transformer: Linear-Time Attention via Variational Rate Reduction. In The Thirteenth International Conference on Learning Representations."}, {"review_id": "pIP9oXx6jK", "reviewer": "Reviewer_D2cQ", "summary": "This paper introduces Zero-Sum Linear Attention (ZeroS), a novel linear-time attention mechanism that aims to close the performance gap between efficient transformers and standard softmax attention. The authors identify two key limitations in existing attention mechanisms: 1) the restriction to convex combinations, which prevents subtractive or contrastive operations within a single layer, and 2) a uniform weight bias from the constant zero-order term in the softmax Taylor expansion, which dilutes attention in long contexts.", "questions": "- Proposition 3.3 and Corollary 3.2 elegantly show that zero-sum attention expands the set of representable functions beyond a simple convex hull. Can you provide a simple, concrete example of a sequence-to-sequence task that standard attention would struggle with but that zero-sum attention can solve easily in a single layer?\n\n- The zero-sum quadratic version of your model, ZeroS-SM, consistently outperforms the standard Transformer baseline, for instance in Table 1. This suggests that even without the constraint of linear time complexity, removing the zero-order term is beneficial. Do you see any downsides or trade-offs to applying this modification to standard softmax attention in models where quadratic complexity is not a bottleneck?", "limitations": "The authors provide a good discussion of limitations in the conclusion, noting that their work prioritizes algorithmic expressivity over low-level engineering optimizations (like custom GPU kernels) and that evaluation on very large-scale LLMs was beyond their resource constraints. These are reasonable limitations. One other potential limitation is the complexity of the final algorithm (Algorithm 1), which, while linear, involves more steps and states to track than simpler linear attention mechanisms. This might pose a barrier to adoption or require more engineering effort to optimize for hardware, as the authors allude to.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "## Pros\n- The paper provides a compelling and novel diagnosis of a fundamental weakness in softmax and many linear attention mechanisms: the performance-degrading effect of the zero-order uniform term in the softmax expansion. Identifying this term as a source of \"attention dilution\" and linking it to the limitations of convex combinations is a significant conceptual contribution.\n\n-  The proposed solution—to simply remove the zero-order term and reweight the residuals—is both simple and powerful. The authors provide strong theoretical support for this approach, proving that the resulting zero-sum weights increase expressivity (Proposition 3.1) while maintaining numerical stability (Lemma 3.4, Proposition 3.5).\n\n- The ablation studies in Tables 5 and 6 effectively demonstrate the importance of each component of the ZeroS framework. The results clearly show that removing the zero-sum reweighting (w/o RWSM) or the gating mechanism leads to a performance drop, validating their design choices.\n\n## Cons\n- While the core idea is simple, the final implementation of ZeroS is quite complex, involving several components: the calculation of deviation logits with a smoothing prior, the separation and gating of first-order and higher-order softmax residuals, and the reintroduction of an angular component. This complexity could make the method harder to analyze and adopt compared to simpler linear attention mechanisms.\n\n- The method decouples the attention weight into a radial part (r_t, i) and an angular part (cosθ). While the radial part is well-motivated and ablated, the precise performance contribution of the angular part is less clear. The ablation studies do not isolate its effect. Since the radial component already includes interactions from gating and higher-order softmax terms, it's not immediately obvious how much extra performance is gained by multiplying by the cosθ term, especially when RoPE is already used.", "was_revised": false, "final_justification": "", "rebuttal": "```\nThis complexity could make the method harder to analyze...\n\n... work prioritizes algorithmic expressivity over low-level engineering optimizations ...\n```\n\nThank you very much for your recognition of our work, and for understanding our current focus on improving expressivity rather than achieving optimal efficiency. Based on suggestions from other reviewers, we provide here a more detailed discussion and empirical analysis of the computational complexity.\n\nIn terms of model complexity, ZeroS can effectively be viewed as an extended form of linear attention, with hidden states of size $3 \\times d \\times d$. In practice, the implementation only requires maintaining three such hidden state matrices, performing prefix scans on each, and using gated queries to read out the corresponding results, which are then summed to produce the final output. This process is operationally consistent with that of standard linear attention.\n\nIn recurrence, ZeroS performs associative scans over hidden states corresponding to 3 different $d \\times d$ bases: $e^{s_i} \\hat{\\mathbf{k}}_i^\\top\\mathbf{v}_i$, $s_i \\hat{\\mathbf{k}}_i^\\top \\mathbf{v}_i$, and $\\hat{\\mathbf{k}}_i^\\top \\mathbf{v}_i$. The final output is computed by reading out the scanned results using queries with different gating weights and summing them. Although we have not yet implemented a fused CUDA kernel that performs all scans in a single forward, we can leverage existing linear attention scan implementations (flash linear attention [1]) to perform three separate scans with different key and value bases. This provides a partially efficient implementation:\n\n\n```python\n# prepare reweighted query q1, q2, and q3 from q...\nout1 = run_linattn(q1, k*s_i_exp, v, mode='fused_chunk')\nout2 = run_linattn(q2, k*s_i, v, mode='fused_chunk')\nout3 = run_linattn(q3, k, v, mode='fused_chunk')\nout = out1 + out2 + out3\n# ...\n```\n\nTo evaluate the runtime efficiency of ZeroS under this implementation, we benchmarked it by replacing the attention layers in a GPT-2 Transformer architecture (dimension = 768, number of heads = 12, sequence length = 1024, number of layers = 12) with various attention variants. We adopted the Transformer and softmax attention implementations (both flash and naive versions) from NanoGPT \\[2]. In addition to our partially efficient ZeroS implementation (based on the `fused_chunk_linear_attn` function from flash linear attention \\[1]), we also included the following layers provided by the same library as baselines for comparison: \\[LinAttn, GatedLinAttn, HGRN2, RWKV6, RWKV7], using their default configurations aside from the GPT-2 settings above. All experiments were conducted on a single Nvidia L40S GPU, using a batch size of 8 and FP32 precision. We measured performance in both training and inference modes (Fwd; full-sequence inference without KV-cache or hidden state cache). After several warm-up forward passes, we recorded the average forward latency, standard error, and peak memory usage using multiple runs synchronized with `torch.cuda.synchronize()`. The results are summarized below.\n\n\n| Model | FwdLatency(s) | FwdStd(s) | TrainLatency(s) | TrainStd(s) | ThroughputFwd(tok/s) | ThroughputTrain(tok/s) | MemPeakFwd(GB) | MemPeakTrain(GB) |\n|---|---|---|---|---|---|---|---|---|\n| SoftMaxAttn (Naïve) | 0.1306 | 0.0006 | 0.3334 | 0.0009 | 62740.89 | 24574.18 | 9.61 | 10.34 |\n| RWKV7 | 0.0876 | 0.0016 | 0.2626 | 0.0010 | 93491.90 | 31199.35 | 9.81 | 10.61 |\n| RWKV6 | 0.0761 | 0.0005 | 0.2252 | 0.0010 | 107653.00 | 36382.92 | 9.49 | 9.62 |\n| ZeroS | 0.0720 | 0.0008 | 0.1974 | 0.0011 | 113855.38 | 41491.29 | 7.48 | 7.61 |\n| HGRN2 | 0.0672 | 0.0013 | 0.1480 | 0.0009 | 121955.16 | 55336.55 | 6.14 | 6.43 |\n| LinAttn | 0.0666 | 0.0010 | 0.1477 | 0.0009 | 122949.71 | 55447.86 | 5.79 | 5.90 |\n| SoftmaxAttn (FlashAttn) | 0.0651 | 0.0014 | 0.1473 | 0.0008 | 125836.28 | 55620.75 | 5.45 | 5.56 |\n| GatedLinAttn | 0.0600 | 0.0009 | 0.1331 | 0.0008 | 136633.08 | 61533.68 | 5.74 | 5.89 |\n\nCompared to other linear attention variants, ZeroS, even without a fully fused associative scan implementation, achieves efficiency on par with the average level of existing architectures, and is not far behind standard linear attention.\n\n```\nthe precise performance contribution of the angular part is less clear.\n```\n\nThank you for the valuable question. As shown in Figure 1(c), the Angular (and Value) components align closely with standard linear attention and form the core of effective attention modeling. RoPE and L2 normalization help capture meaningful $\\cos \\theta_{t,i}$ signals tied to relative position ($t - i$). In contrast, the Radial component is our main contribution, which is why we emphasize it in Figure 1(a,b) and the ablation studies.\n\nTo clarify the Angular component’s role, we provide extended ablations analyzing both Angular and Radial parts below. \n\n\n|Model|Compress|FuzzyRecall|In-ContextRecall|Memorize|NoisyRecall|SelectiveCopy|Average|\n|---|---|---|---|---|---|---|---|\n|ZeroS|44.0|14.9|99.9|88.1|96.1|97.8|73.5|\n|w/o Angular|39.5|8.5|42.8|54.5|44.8|63.3|42.2|\n|Ang:w/o PosEmb|35.8|9.4|73.3|46.2|66.2|45.8|46.1|\n|Ang:w/ additive PosEmb|38.1|14.2|94.1|86.6|87.2|93.8|69.0|\n|w/o Radial|35.9|9.6|84.8|86.3|86.5|92.3|65.9|\n|Rad:standard SM|36.3|10.6|91.8|81.7|89.7|95.3|67.6|\n|Rad:Gated ZeroS but add 0-th back|42.0|10.5|91.4|85.2|90.0|97.1|69.4|\n|Rad:Zeros w/o Gating|39.7|13.5|96.3|83.0|94.6|97.8|70.8|\n\n\nRemoving the Angular part (w/o Angular) eliminates the bilinear query-key interactions ($x_t \\widetilde{\\mathbf{W}}_q \\widetilde{\\mathbf{W}}_k x_i$), leaving only scalar weight products between steps $t$ and $i$. This reduces the model to a $3 \\times d$ hidden-state RNN scan over values, instead of the full $3 \\times d \\times d$ structure, significantly lowering expressiveness.\n\nReintroducing queries and keys without positional embeddings (Ang: w/o PosEmb) offers only a minor gain, as $\\cos \\theta_{t,i}$ lacks positional meaning. Using additive positional embeddings (Ang: w/ additive PosEmb) improves performance, comparable to strong baselines—but still below ZeroS with RoPE. This highlights the importance of modeling relative positions via $\\cos \\theta_{t,i}$, aligning with prior findings on linear attention.\n\n```\na sequence-to-sequence task that standard attention would struggle with but that zero-sum attention can solve easily in a single layer...\n```\n\nThank you for the insightful question. Standard softmax attention restricts outputs to the convex hull of input values, $\\mathrm{Conv}({\\mathbf{v}_i})$, allowing only non-negative weights. In contrast, ZeroS enables zero-sum combinations, $\\{\\sum_i w_i \\mathbf{v}_i \\mid \\sum_i w_i = 0\\}$, allowing it to express operations like differencing and contrast, crucial for many reasoning tasks.\n\nThis expanded expressiveness comes with proven numerical stability, and is further enhanced by gating for higher-order terms. While discussed limitations of convex weights are not new, our work (see Section 2.2) builds on this to propose ZeroS as an effective and stable solution.\n\nTo illustrate this, we provide examples of differential and contrast tasks (see refs [27, 41–43] of main paper) that ZeroS naturally supports.\n\n- **1. Minimal Counterexample: Two-Token Difference**\n\n|**Task Definition**|**Output:** $\\mathbf{v}_1-\\mathbf{v}_2$|\n|---|---|\n|**Standard Softmax**|$\\mathbf{o}=\\alpha_1\\mathbf{v}_1+\\alpha_2\\mathbf{v}_2,\\ \\alpha_1,\\alpha_2\\ge0,\\ \\alpha_1+\\alpha_2=1$|\n|**Constraint**|$\\Rightarrow\\alpha_1=1,\\ \\alpha_2=-1$ violates non-negativity|\n|**ZeroS Attention**|$\\mathbf{o}=w_1\\mathbf{v}_1+w_2\\mathbf{v}_2,\\ w_1+w_2=0,\\ w_1=1,\\ w_2=-1\\Rightarrow\\mathbf{o}=\\mathbf{v}_1-\\mathbf{v}_2$|\n\n- **2. Two Simple Task Examples**\n\n|**Task**|**Softmax Fail**|**ZeroS Can Achieve**|\n|---|---|---|\n|(a) Difference Between a Token and Mean|$\\mathbf{o}=\\mathbf{v}\\_1-\\frac{1}{t}\\sum_{i=1}^t\\mathbf{v}\\_i$, requires neg weights|$w_1=1-\\frac{1}{t},\\ w_{i>1}=-\\frac{1}{t},\\ \\sum w_i=0$|\n|(b) Alternating Difference Pattern|$\\mathbf{o}=\\sum\\_{i=1}^{t/2}(\\mathbf{v}\\_{2i-1}-\\mathbf{v}\\_{2i})$, requires alternating ± weights|$w_{2i-1}=1,\\ w_{2i}=-1,\\ \\sum w_i=0$|\n\n\n```\nDo you see any downsides or trade-offs to applying this modification to standard softmax attention in models where quadratic complexity is not a bottleneck?\n```\n\nThank you for the thoughtful question. The trade-offs of applying ZeroS to softmax attention are similar to those seen in the linear case.\n\nIn Table 1, ZeroS (Lin) outperforms standard linear attention across all tasks due to the weaker baseline. For ZeroS (SM) vs. softmax, most gains appear in *In-Context Recall and Noisy Recall*, which require precise retrieval and robustness, reflecting key challenges in complex sequence modeling such as precise retrieval of prior information and robustness to noise. This supports our view that ZeroS improves expressiveness for harder tasks.\n\nIn contrast, for tasks like Fuzzy Recall that favor simple averaging, ZeroS may slightly underperform, as it lacks the softmax’s built-in averaging bias. Learning such behavior from scratch may require more layers or parameters, though averaging itself is cheap and doesn't pose a major bottleneck. \n\nIf a task’s complexity or bias is unclear (particularly whether it relies on simple averaging), we recommend using a multi-layer ZeroS as a general-purpose attention architecture.\n\n### References\n\n[1] Yang, S., & Zhang, Y. (2024). Fla: A triton-based library for hardware-efficient implementations of linear attention mechanism (https://github.com/fla-org/flash-linear-attention)\n\n[2] Karpathy, A. (2022). NanoGPT. GitHub Repository. https://github.com/karpathy/nanoGPT"}, {"review_id": "JR43RPmUgH", "reviewer": "Reviewer_FheY", "summary": "The paper proposes a new variant of the self-attention function, both for the linear and the softmax variant. The new self-attention variant is based on a zero-sum attention weights, instead of the traditional convex attention weights used in softmax attention and linear attention variants. The authors show that the resulting zero-sum attention weights are more expressive, in the sense that the sub-space spanned by the zero-sum attention also covers the convex subspace achieved by the convex attention. Furthermore, in the linear attention model, the paper proposes separating the radial and angular components of the self-attention, with the radial component computed through a Taylor-expansion of the softmax attention, and the angular components computed through vanilla linear attention with QK-norm. \n\nThe performance of the proposed models are investigated on multiple benchmarks, including recall tasks, language modeling, time-series forecasting, and image classification. Furthermore, an ablation study is performed on some of the components of the models.", "questions": "- Can the authors provide examples of tasks that can be solved with the proposed zero-sum softmax, but not the convex softmax? This can significantly help with the credibility of the proposed idea or zero-sum attention.\n- Can the authors provide further benchmark results, such as the needle in the haystack, arithmetic, and sequence length benchmarking results to further solidify the superiority of the proposed model?\n- What would happen if instead of the $u_i$s, you used the query/key in the radial component for calculating $\\delta$? Does the model perform worse? If so, by how much? I’m curious because following the logic of the paper, you can see the magnitude of the query and the key vectors as the input of the radial component, and their angle as the input to the angular component. \n- The notation in the final paragraphs of the third section gets a little bit confusing. Is $H_t$ a scalar or a matrix? Are you computing the inner products of the key and value in $F_t$, $G_t$, $H_t$? If so, why change it compared to the linear attention model?\n- Can you perform some ablation studies explaining how much of the performance contribution comes from the angular component and how much of the performance contribution from the radial component?\n- Can you provide the proofs for the theorems in the main body?\n- Also, there is a typo in proposition 3.1.", "limitations": "I think there are two limitations concerning this paper. Firstly, the motivation behind the main contributions of the paper is not well-established. I am not convinced that the zero-sum attention mechanism is necessary, despite the fact that it provably provides a more expressive self-attention model. Secondly, the paper suffers from a lack of an organized benchmarking effort. I believe the authors should focus on the most recent linear attention variants, and compare their proposed method with those models on all tasks.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 3, "strengths_and_weaknesses": "**Strengths.** The paper starts with a convincing argument about the shortcomings of the linear attention model, citing the relevant studies trying to alleviate these issues. The notations used in the earlier sections of the paper are clear and helpful, and the paper provides a helpful summary of the investigated architecture. The related works section is well integrated into the main contributions of the paper. The theoretical claims seem to be correct, although the proofs for the theorems are not provided. The paper provides an illustration of the proposed method, thus helping the reader to understand the new components in places where the notations fail to do so.\n\n**Weaknesses.** I believe the main issue of the paper is a lack of well-established motivation for the architectural innovations that are at the center of the proposed models. Specifically, the following claims are not well-supported:\n- The need for a more expressive attention mechanism is not supported by the paper properly. It is true that the zero-sum attention is more expressive than a convex attention. However, one could also argue that the original idea behind the attention mechanism tries to establish an inductive bias by this choice. The authors may point to the better performance of the model in certain benchmarks (although, the experiment results are more of a mix-bag), but in some of the experiments the model is actually either in the under-parameterized regime or near it. Consequently, having a more complex model usually results in better performance. \n- The importance of having the negative angular signal from the self-attention mechanism is also not supported beyond the “more expressive” argument. Do the authors have experimental results showing that this is a particularly important issue? For instance, can you perform an experiment wherein you limit the angle between the query and the key in a softmax transformer to the positive range and show significant drop in performance?\n- The gating on the softmax components seem to be coming out of nowhere in the paper, with very limited setup or justification. What is your reason behind this architecture decision? How can you support its importance?\n- The contribution corresponding to the normalization factor is undermined by the fact that there’s extensive research on this component in the linear attention literature. In Mamba2 and the DeltaNet family of models, the post-norm component seems to be eliminating the need for any explicit normalization. In fact, you seem to be using this component in your own architecture. \n- The decision behind introducing the $u_i$ factors in the radial component is not clearly established. Why is it necessary for the $\\delta$ elements to provide information about the “deviation of step $i$ relative to the previous steps?” \n\nThe paper also seems to be suffering from a mixed message problem. Initially, the paper appears to be focusing on providing an answer for the long-standing question in the community about shortcomings of the linear attention model. However, the focus shifts multiple times from the softmax function itself (and how to improve it) to the linear attention and back. And ultimately, in the experiments there is a clear shift towards an emphasis on the softmax variant with the zero-sum structure. While interesting as a secondary investigation for the appendix, I see this decision as a distraction, occupying a space in the main body that could be filled with better experiments on the linear model.\n\nFurthermore, the experiment results are generally a mixed-bag. The model doesn’t seem to be providing significant improvement compared to the baselines, with marginal improvements in certain benchmarks. For instance, in the MAD benchmark the model rarely beats the linear attention variants by a significant margin, and considering the outdated list of baselines, it doesn’t paint a promising picture. Furthermore, there is inconsistency in the list of baselines used in the experiments, with some baselines missing from certain experiments without explanation. I think it would be very helpful if the authors could provide the performance of a consistent set of baselines from the most recent variants of linear attention. For instance, use Gated DeltaNet instead of DeltaNet, and Mamba2 instead of Mamba1, and so on. Furthermore, there are certain models like hedgehog that also aim to find the “secret sauce” of the softmax attention instead of going the \"approximation\" way in order to improve linear attention. I believe you should also include these models in your list of baselines.\n\nMoreover, considering the number of extra components introduced in the proposed architecture, I expected a much more thorough ablation study. For example, it would be helpful to have an experiment showing the effect of the angular component. The angular component seems to be vanilla linear attention coupled with QK norm and RoPE.", "was_revised": true, "final_justification": "I think the paper proposes an interesting approach with the zero-sum softmax, which is supported by helpful and intuitive theoretical discussions. During the rebuttal, some of my concerns were addressed. Notably, the authors provided theoretical and empirical evidence supporting the benefits of using the zero-sum softmax attention, an ablation on the components, further clarification on their motivation behind the gating mechanism, and full proof for the theorems. They've also noted that they intend to improve the notations used in the paper in the final version. This will admittedly be a relatively large overhaul of the paper. \n\nThe only concern of mine which was not addressed during the rebuttal is the main intent of the paper. Specifically, I do not understand whether the paper should be taken as a new variant of the softmax attention, or a linear attention model. If the latter is the aim, then some of the results in the paper (such as the normalization theorem) seem to be more window-dressing than helpful and informative results. If the intent is the former, the authors should have leaned more towards comparing themselves with recent variants of softmax attention, which seem to be completely missing from the results. Therefore, I lean towards acceptance, but I do not have a strong opinion about the paper.", "rebuttal": "```\n... motivation for the innovations ...\n```\n\nThank you very much for your valuable suggestions, which help improve the paper's clarity and accessibility. We will expand on the following points:\n\n```\nmore expressive ... inductive bias by this choice ... in the under-parameterized regime\n```\nYes, softmax ensures convexity via a zero-order term ($1/t$), which stabilizes outputs by compressing them into [0,1]. This introduces a bias toward averaging, suitable for simpler tasks like *Fuzzy Recall (FR) and Compression* in MAD. In contrast, complex tasks such as *In-Context Recall (ICR) and Noisy Recall (NR)* require precise retrieval and benefit from higher-order interactions, where ZeroS shows clear gains (see Table 1), aligning with our goal of enhancing expressivity by removing this constant bias.\n\nIncluding or removing the $1/t$ term reflects a trade-off in parameter efficiency: keeping it allows one-layer averaging but requires multiple layers for differencing, while removing it enables one-layer differencing but multiple layer for averaging (see Proposition 3.3). The latter is often preferable, as averaging is simpler to approximate (parameter-free average-pooling is enough), while differencing is harder to learn.\n\nAs Table 1 shows, ZeroS improves significantly on *ICR and NR*, while gains on *FR and Compression* are smaller: supporting our view that this reflects a true shift in modeling capability preferences, not enough to be explained only by under-parameterization.\n\n```\nlimit the angle between the query and the key\n```\n\nOur early experiments and prior works like RetNet and GLA suggest that lifting the non-negativity constraint on queries and keys (i.e., relaxing convexity) improves performance (one of the key motivations behind our design).\n\nAs a test, we applied a softplus function to enforce positive queries and keys, limiting the sign of $\\cos \\theta$ as follows. This led to a clear drop in performance, especially on complex tasks like *ICR and NR*, showing the importance of signed angular interactions for expressivity.\n\n\n|Model|Compress|FuzzyRecall|In-ContextRecall|Memorize|NoisyRecall|SelectiveCopy|Average|\n|---|---|---|---|---|---|---|---|\n|ZeroS|44.0|14.9|99.9|88.1|96.1|97.8|73.5|\n|ZeroS(QK ForcePositive)|41.3|14.5|79.3|65.4|72.3|82.2|59.2|\n|SoftmaxAttn|51.6|29.8|94.1|85.2|86.8|99.6|74.5|\n|SoftmaxAttn(QK ForcePositive)|51.6|29.5|80.3|80.5|74.5|99.1|69.3|\n\n```\ngating coming out of nowhere in the paper\n```\n\nWe apologize for any confusion caused by our modular presentation, intended to help readers navigate key components. Below is a concise, chronological summary of the motivation. Brackets [ ] indicate our design insights; parentheses ( ) refer to prior findings:\n\n---\n\n[Consider the Taylor expansion of softmax]\n\n↓\n\n(Prior works approximate softmax using low-order terms)\n+\n(Others note that convex weights (due to non-negativity) limit expressivity but improve stability)\n\n↓\n\n[The zero-order term ($1/t$) enforces convexity; higher-order terms yield zero-sum weights for richer interactions]\n\n↓\n\n[Removing the zero-order term allows negative weights (more expressivity) while retaining stability via normalized zero-sum structure]\n+\n(Differential Transformer [3] and others may benefit implicitly from this)\n\n↓\n\n[Our experiments confirm that zero-sum softmax (without $1/t$) improves performance]\n\n↓\n\n[**Removing the zero-order term = applying a fixed gating of 0**, suggesting learnable gating on higher-order terms for more expressivity]\n\n↓\n\n[Experiments support this, leading to the final gated ZeroS design]\n\n---\n\n```\nthe normalization...\n```\n\nYes, that's correct. Lemma 3.4 and Proposition 3.5 provide a theoretical guarantee of numerical stability for zero-sum attention, showing that a simple $1/\\sqrt{t}$ normalization ensures stability across sequence lengths.\n\nWhile many linear attention models use output LayerNorm to stabilize training—making $1/\\sqrt{t}$ optional—LayerNorm is an **engineering fix**, whereas our results offer a **theoretical foundation**.\n\nFor instance, standard linear attention may explode under $-\\infty$ weights, while ZeroS weights (e.g., from $s_i$ without gating), which only subtract $1/t$ from softmax, remain stable. Although ZeroS uses prefix scans on  KV that can't actually handle $-\\infty$ due to this **engineering constraints**, this does not weaken its **stronger theoretical robustness** over standard linear attention.\n\n```\nRecent baselines...; Ablation of angular part...; Why $u_i$ factors...\n```\n\nWe summarize the requested results in the table below.\n\nWe added recent baselines including [Mamba2, Hawk, Gated DeltaNet, xLSTM, HedgeDog], using implementations from the authors of [1] and our codes for HedgeDog for a linear approximation of softmax. ZeroS performs strongly on complex tasks like *ICR and NR*, and outperforms models like RWKV and xLSTM that heavily compress or alter hidden states on *Memorize*, leading to a higher average score.\n\nHedgeDog’s results are particularly interesting: it performs well on soft-averaging tasks (*Compress, FR, Memorize*), but drops sharply on *ICR and NR*, likely because it captures mainly the $1/t$ term from softmax while failing to approximate higher-order terms: supporting our core paper motivation.\n\nThe angular part in ZeroS follows standard linear attention. Removing it (w/o Angular) eliminates query-key bilinear interactions, reducing hidden state from $3 \\times d \\times d$ to $3 \\times d$, and thus significantly reducing expressiveness. Reintroducing QK but removing positional embeddings gives only marginal gains, as $\\cos \\theta_{t,i}$ lacks positional cues. Additive PosEmb improve performance but still fall short of RoPE.\n\nAs for computing $s_i$ from $u_i$ in the radial part: since zero-sum weights model higher-order variation, $s_i$ should reflect such structure. It can be derived locally (e.g., linear projection, quadratic form), or from context (e.g., window mean, deviation, or negative similarity with mean). Since ZeroS targets outlier interactions, negative similarity is especially effective.\n\nOur code supports all variants, and experiments confirm that negative similarity yields the best results (see Rad: u_i), as briefly noted in the main text.\n\n|Model|Compress|FuzzyRecall|In-ContextRecall|Memorize|NoisyRecall|SelectiveCopy|Average|\n|---|---|---|---|---|---|---|---|\n|ZeroS(Lin)|44.0|14.9|99.9|88.1|96.1|97.8|73.5|\n|LinAttn|33.1|8.2|91.0|74.9|75.6|93.1|62.3|\n|ZeroS(SoftmaxAttn)|45.2|28.0|100.0|84.3|96.6|98.5|75.4|\n|SoftmaxAttn|51.6|29.8|94.1|85.2|86.8|99.6|74.5|\n|Mamba2|43.6|21.1|96.4|86.9|96.7|93.3|73.0|\n|Hawk|47.7|13.6|93.0|91.3|93.0|77.0|64.5|\n|GatedDeltaNet|45.0|32.7|100.0|81.7|100.0|95.7|71.0|\n|xLSTM|43.4|47.6|100.0|79.8|100.0|95.4|73.2|\n|HedgeDog|43.2|17.9|55.9|83.4|46.0|98.4|57.4|\n|[AblationStudy↓]|-|-|-|-|-|-|-|\n|ZeroS|44.0|14.9|99.9|88.1|96.1|97.8|73.5|\n|w/o Angular|39.5|8.5|42.8|54.5|44.8|63.3|42.2|\n|Ang: w/o PosEmb|35.8|9.4|73.3|46.2|66.2|45.8|46.1|\n|Ang: w/ additive PosEmb|38.1|14.2|94.1|86.6|87.2|93.8|69.0|\n|w/oRadial|35.9|9.6|84.8|86.3|86.5|92.3|65.9|\n|Rad: u_i - linear proj|41.2|15.5|91.9|88.3|86.1|97.2|70.0|\n|Rad: u_i - quad form|40.9|15.4|92.6|86.6|90.8|98.5|70.8|\n|Rad: u_i - 2-distance|40.1|14.8|97.6|82.5|93.6|97.8|71.1|\n|Rad: u_i - averaging|41.0|15.0|99.9|93.3|89.6|98.5|73.0|\n\n\nRegarding the suggestion on additional benchmarks: due to space and time constraints, and given our experimental coverage, we refer you to relevant existing results. Needle-in-a-haystack behaviors are covered in MAD’s *Selective Copy and ICR*. Arithmetic reasoning is addressed in RegBench and MQAR, both of which extend arithmetic tasks. Sequence length is shown over sub-tasks averaging in synthetic datasets like MQAR.\n\n```\ntasks that can be solved with the proposed zero-sum softmax, but not the convex softmax...\n```\n\nPlease refer to our responses to Reviewer D2cQ.\n\n```\nwhat if we use query/key calculation in the radial component...\n```\n\nThe radial component is designed as a product of two scalars (from steps $t$ and $i$) to enable higher-order zero-sum softmax while keeping $O(N)$ complexity. Using QK dot products instead would require $O(N^2)$, losing linearity. Alternatively, applying channel-wise zero-sum softmax over keys could preserve linearity, but would increase the hidden state to a $3 \\times d \\times d \\times d$ tensor ($O(d^3)$ space). Our follow-up work explores this $O(d^n)$ expressivity-efficiency trade-off, and we’re happy to briefly share this direction here.\n\n```\nnotation in the final paragraphs ... confusing\n```\n\nThis refers to the 3 hidden state matrices. In recurrence, ZeroS performs associative scans over states based on $e^{s_i} \\hat{\\mathbf{k}}_i^\\top\\mathbf{v}_i$, $s_i \\hat{\\mathbf{k}}_i^\\top \\mathbf{v}_i$, and $\\hat{\\mathbf{k}}_i^\\top \\mathbf{v}_i$, then reads them out using different gated queries and sums the results. In short, it's a linear scan over a $3 \\times d \\times d$ hidden state. Please see our response to Reviewer WRY9 for more.\n\n```\ncould be filled with better experiments on the linear model\n```\n\nTo show the generality of zero-sum mechanisms, we included quadratic models. In the revision, we’ll add more discussion and results on linear models to balance coverage.\n\n```\nprovide the proofs ... a typo in proposition 3.1 ...\n```\n\nWe apologize for omitting full proofs in the submission version due to space constraints. We presented the propositions in a concise form that retains the core reasoning, as the results are relatively straightforward. Full details will be restored in the appendix in the revision. We're also happy to walk through the arguments during discussion. We’ll correct the missing zero-sum condition ($= 0$) in proposition 3.1. \n\nFor complexity-related questions, please see our replies to other reviewers.\n\n---\n\n\n### References\n\nSee the same references listed under our response to Reviewer kvKy."}]

Author rebuttal:

Meta-review: This paper proposes an improvement to linear attention stemming from a careful analysis of kernel approximation of softmax attention. This results in several modifications, including the removal of the zeroth order term, residual reweighting, and decoupling of the angular/radial components. Empirical experiments demonstrate the effectiveness of this approach across multiple benchmarks, including diagnostic benchmarks (MAD/Regbench) as real-world domains (language/image/time-series modeling).

All reviewers appreciated the fact that the method was well-motivated and moreover backed up by both theory and solid empirical results spanning diverse domains. It is also noteworthy that the approach can also improve softmax attention layers. There were some initial concerns with regard to wallclock efficiency, but these issues were largely addressed during the rebuttal.

The main weakness is scale: language modeling experiments are only conducted at very small scales. However, I think this is ok given the significant experiments in other domains.

Overall, I believe that this work represents an excellent contribution to the architecture community and deserves to be highlighted at the conference.
Decision: Accept (spotlight)