Title: ZeroS: Zero‑Sum Linear Attention for Efficient Transformers
Venue: NeurIPS 2025
Accepted: yes
Keywords: Attention Mechanisms

Transformer

Sequence Modeling
Abstract: Linear attention methods offer Transformers $O(N)$ complexity but typically underperform standard softmax attention. We identify two fundamental limitations affecting these approaches: the restriction to convex combinations that only permits additive information blending, and uniform accumulated weight bias that dilutes attention in long contexts. We propose Zero-Sum Linear Attention (ZeroS), which addresses these limitations by removing the constant zero-order term $1/t$ and reweighting the remaining zero-sum softmax residuals. This modification creates mathematically stable weights, enabling both positive and negative values and allowing a single attention layer to perform contrastive operations. While maintaining $O(N)$ complexity, ZeroS theoretically expands the set of representable functions compared to convex combinations. Empirically, it matches or exceeds standard softmax attention across various sequence modeling benchmarks.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a new variant of the self-attention function, both for the linear and the softmax variant. The new self-attention variant is based on a zero-sum attention weights, instead of the traditional convex attention weights used in softmax attention and linear attention variants. The authors show that the resulting zero-sum attention weights are more expressive, in the sense that the sub-space spanned by the zero-sum attention also covers the convex subspace achieved by the convex attention. Furthermore, in the linear attention model, the paper proposes separating the radial and angular components of the self-attention, with the radial component computed through a Taylor-expansion of the softmax attention, and the angular components computed through vanilla linear attention with QK-norm. \n\nThe performance of the proposed models are investigated on multiple benchmarks, including recall tasks, language modeling, time-series forecasting, and image classification. Furthermore, an ablation study is performed on some of the components of the models.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths.** The paper starts with a convincing argument about the shortcomings of the linear attention model, citing the relevant studies trying to alleviate these issues. The notations used in the earlier sections of the paper are clear and helpful, and the paper provides a helpful summary of the investigated architecture. The related works section is well integrated into the main contributions of the paper. The theoretical claims seem to be correct, although the proofs for the theorems are not provided. The paper provides an illustration of the proposed method, thus helping the reader to understand the new components in places where the notations fail to do so.\n\n**Weaknesses.** I believe the main issue of the paper is a lack of well-established motivation for the architectural innovations that are at the center of the proposed models. Specifically, the following claims are not well-supported:\n- The need for a more expressive attention mechanism is not supported by the paper properly. It is true that the zero-sum attention is more expressive than a convex attention. However, one could also argue that the original idea behind the attention mechanism tries to establish an inductive bias by this choice. The authors may point to the better performance of the model in certain benchmarks (although, the experiment results are more of a mix-bag), but in some of the experiments the model is actually either in the under-parameterized regime or near it. Consequently, having a more complex model usually results in better performance. \n- The importance of having the negative angular signal from the self-attention mechanism is also not supported beyond the “more expressive” argument. Do the authors have experimental results showing that this is a particularly important issue? For instance, can you perform an experiment wherein you limit the angle between the query and the key in a softmax transformer to the positive range and show significant drop in performance?\n- The gating on the softmax components seem to be coming out of nowhere in the paper, with very limited setup or justification. What is your reason behind this architecture decision? How can you support its importance?\n- The contribution corresponding to the normalization factor is undermined by the fact that there’s extensive research on this component in the linear attention literature. In Mamba2 and the DeltaNet family of models, the post-norm component seems to be eliminating the need for any explicit normalization. In fact, you seem to be using this component in your own architecture. \n- The decision behind introducing the $u_i$ factors in the radial component is not clearly established. Why is it necessary for the $\\delta$ elements to provide information about the “deviation of step $i$ relative to the previous steps?” \n\nThe paper also seems to be suffering from a mixed message problem. Initially, the paper appears to be focusing on providing an answer for the long-standing question in the community about shortcomings of the linear attention model. However, the focus shifts multiple times from the softmax function itself (and how to improve it) to the linear attention and back. And ultimately, in the experiments there is a clear shift towards an emphasis on the softmax variant with the zero-sum structure. While interesting as a secondary investigation for the appendix, I see this decision as a distraction, occupying a space in the main body that could be filled with better experiments on the linear model.\n\nFurthermore, the experiment results are generally a mixed-bag. The model doesn’t seem to be providing significant improvement compared to the baselines, with marginal improvements in certain benchmarks. For instance, in the MAD benchmark the model rarely beats the linear attention variants by a significant margin, and considering the outdated list of baselines, it doesn’t paint a promising picture. Furthermore, there is inconsistency in the list of baselines used in the experiments, with some baselines missing from certain experiments without explanation. I think it would be very helpful if the authors could provide the performance of a consistent set of baselines from the most recent variants of linear attention. For instance, use Gated DeltaNet instead of DeltaNet, and Mamba2 instead of Mamba1, and so on. Furthermore, there are certain models like hedgehog that also aim to find the “secret sauce” of the softmax attention instead of going the \"approximation\" way in order to improve linear attention. I believe you should also include these models in your list of baselines.\n\nMoreover, considering the number of extra components introduced in the proposed architecture, I expected a much more thorough ablation study. For example, it would be helpful to have an experiment showing the effect of the angular component. The angular component seems to be vanilla linear attention coupled with QK norm and RoPE.\n\nQUESTIONS:\n- Can the authors provide examples of tasks that can be solved with the proposed zero-sum softmax, but not the convex softmax? This can significantly help with the credibility of the proposed idea or zero-sum attention.\n- Can the authors provide further benchmark results, such as the needle in the haystack, arithmetic, and sequence length benchmarking results to further solidify the superiority of the proposed model?\n- What would happen if instead of the $u_i$s, you used the query/key in the radial component for calculating $\\delta$? Does the model perform worse? If so, by how much? I’m curious because following the logic of the paper, you can see the magnitude of the query and the key vectors as the input of the radial component, and their angle as the input to the angular component. \n- The notation in the final paragraphs of the third section gets a little bit confusing. Is $H_t$ a scalar or a matrix? Are you computing the inner products of the key and value in $F_t$, $G_t$, $H_t$? If so, why change it compared to the linear attention model?\n- Can you perform some ablation studies explaining how much of the performance contribution comes from the angular component and how much of the performance contribution from the radial component?\n- Can you provide the proofs for the theorems in the main body?\n- Also, there is a typo in proposition 3.1.\n\nLIMITATIONS:\nI think there are two limitations concerning this paper. Firstly, the motivation behind the main contributions of the paper is not well-established. I am not convinced that the zero-sum attention mechanism is necessary, despite the fact that it provably provides a more expressive self-attention model. Secondly, the paper suffers from a lack of an organized benchmarking effort. I believe the authors should focus on the most recent linear attention variants, and compare their proposed method with those models on all tasks.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces Zero-Sum Linear Attention (ZeroS), a novel linear-time attention mechanism that aims to close the performance gap between efficient transformers and standard softmax attention. The authors identify two key limitations in existing attention mechanisms: 1) the restriction to convex combinations, which prevents subtractive or contrastive operations within a single layer, and 2) a uniform weight bias from the constant zero-order term in the softmax Taylor expansion, which dilutes attention in long contexts.\n\nSTRENGTHS AND WEAKNESSES:\n## Pros\n- The paper provides a compelling and novel diagnosis of a fundamental weakness in softmax and many linear attention mechanisms: the performance-degrading effect of the zero-order uniform term in the softmax expansion. Identifying this term as a source of \"attention dilution\" and linking it to the limitations of convex combinations is a significant conceptual contribution.\n\n-  The proposed solution—to simply remove the zero-order term and reweight the residuals—is both simple and powerful. The authors provide strong theoretical support for this approach, proving that the resulting zero-sum weights increase expressivity (Proposition 3.1) while maintaining numerical stability (Lemma 3.4, Proposition 3.5).\n\n- The ablation studies in Tables 5 and 6 effectively demonstrate the importance of each component of the ZeroS framework. The results clearly show that removing the zero-sum reweighting (w/o RWSM) or the gating mechanism leads to a performance drop, validating their design choices.\n\n## Cons\n- While the core idea is simple, the final implementation of ZeroS is quite complex, involving several components: the calculation of deviation logits with a smoothing prior, the separation and gating of first-order and higher-order softmax residuals, and the reintroduction of an angular component. This complexity could make the method harder to analyze and adopt compared to simpler linear attention mechanisms.\n\n- The method decouples the attention weight into a radial part (r_t, i) and an angular part (cosθ). While the radial part is well-motivated and ablated, the precise performance contribution of the angular part is less clear. The ablation studies do not isolate its effect. Since the radial component already includes interactions from gating and higher-order softmax terms, it's not immediately obvious how much extra performance is gained by multiplying by the cosθ term, especially when RoPE is already used.\n\nQUESTIONS:\n- Proposition 3.3 and Corollary 3.2 elegantly show that zero-sum attention expands the set of representable functions beyond a simple convex hull. Can you provide a simple, concrete example of a sequence-to-sequence task that standard attention would struggle with but that zero-sum attention can solve easily in a single layer?\n\n- The zero-sum quadratic version of your model, ZeroS-SM, consistently outperforms the standard Transformer baseline, for instance in Table 1. This suggests that even without the constraint of linear time complexity, removing the zero-order term is beneficial. Do you see any downsides or trade-offs to applying this modification to standard softmax attention in models where quadratic complexity is not a bottleneck?\n\nLIMITATIONS:\nThe authors provide a good discussion of limitations in the conclusion, noting that their work prioritizes algorithmic expressivity over low-level engineering optimizations (like custom GPU kernels) and that evaluation on very large-scale LLMs was beyond their resource constraints. These are reasonable limitations. One other potential limitation is the complexity of the final algorithm (Algorithm 1), which, while linear, involves more steps and states to track than simpler linear attention mechanisms. This might pose a barrier to adoption or require more engineering effort to optimize for hardware, as the authors allude to.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper addresses the performance gap between efficient linear attention models and standard quadratic softmax attention, which is attributed to the positive convex weights that makes subtraction impossible. The authors propose a zero-sum attention mechanism so that the weights can be both positive and negative, which leads to both quadratic and linear attention variants. Experiments in language modeling, in-context learning, and time series forecasting show that ZeroS matches or exceeds the performance of baselines.\n\nSTRENGTHS AND WEAKNESSES:\n## Significance\n\n1. Metrics: A major metric that is missing from the evaluation is the wall-clock running time.   \n- On the one hand, I understand that the asymptotic complexities have been provided.   \n- On the other hand, there are aspects not captured by such complexities, including how friendly the computations are to caching, the constants in the asymptotics which matter before sufficiently large N and d are reached, etc.   \n- It might be demanding to do it for all baselines, but at least comparing representative linear/quadratic baselines as well as variants in your ablation study can give a better picture. For instance, MultiHead Hyena, DeltaNet, and Transformer \\- how do their running time compare with that of ZeroS variants?   \n2. Baselines: Since efficient transformer architecture is a fast paced direction, it helps make an impact by comparing with recent open-source baselines  \n- (ICLR 24\\) https://openreview.net/forum?id=Eh0Od2BJIM  \n- (ICLR 25\\) https://openreview.net/forum?id=lXRDQsiP2v\n\n## Clarity\n\n1. Figure 1 is a bit complicated and hard to parse. For a novice reader that scans the figures before reading the paper, it is hard to appreciate the method. Specifically:  \n- There is no label of {u\\_i} in figure 1 or its caption. One has to refer to the text.  \n- The authors may consider contrasting the proposed architecture with a representative baseline, so that it is immediately clear which parts are similar, and which parts are different (and novel).   \n2. Figure 2 has very small labels and legends making it difficult to read.  \n3. Line 134-136 has little context, since the proposed approach has not been introduced yet.  \n4. Typos:   \n- Line 146: something is missing after “∑ i w\\_i=”  \n- The equation block below line 207 has duplications\n\nQUESTIONS:\nThe reviewer would love to see the concerns in the above addressed and revise the rating.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work identifies key issues in the Linear Attention method proposed in [1] and introduces Zero-Sum Linear Attention to address them. First, it argues that the zeroth-order term in attention dilutes the weights. Removing this term not only resolves that issue but also permits negative weights—naturally contrastive—building on insights from Differential Attention [2]. The authors further show that zero-sum weights can improve expressivity over the convex combination in standard attention and that the zero-sum form is stable under bounded logits. They then discuss the benefits of decoupling direction and magnitude. Finally, they demonstrate that the inherent negative weights and stability of zero-sum attention let them drop the angular term ($\\cos\\theta$) from the denominator, seamlessly integrating with RoPE as a by-product. Extensive experiments across diverse benchmarks and domains indicate that Zero-Sum Linear Attention matches standard attention closely while outperforming other linear variants.\n\nReferences:\n\n[1] Fast Autoregressive Transformers with Linear Attention\n\n[2] Differential Transformer\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n\n- The paper proposes several novel ideas:\n  - removing the zeroth-order term,\n  - decoupling angular and radial components, and\n  - using effective gating plus higher-order terms via differentiation—together yielding improved accuracy for linear-attention methods.\n- The manuscript is generally clear, with only minor writing issues.\n- The contributions are well motivated, supported by sound theoretical arguments and strong empirical results.\n\n**Weaknesses**\n- The major weakness is the absence of an efficient implementation for the recurrent equations. Although the authors note this in the “Limitations” section, it remains a significant limitation for practical adoption. At minimum, please report time- and memory-usage comparisons against standard attention implementations such as FlashAttention-2.\n\n- Typos / notation:\n  - Line 146: “$\\sum_i w_i$ = …” (missing RHS).\n  - Line 196: “output be$o_t$” → “output be $o_t$”.\n  - Equation above line 208: should the index be $\\varepsilon_{ti}$, not $\\varepsilon_{i}$?\n  - Line 208: should this be $\\sigma_t^1 = \\operatorname{sigmoid}(x_t W_g^1)$ ? and similarly for $\\sigma_t^h$.\n\nQUESTIONS:\n**Questions and Suggestion** \n\n- Line 34: “In long contexts, attention mechanisms incorporate a roughly uniform component … introducing a persistent averaging effect …”\n   \nFor standard softmax attention, large query–key dot products often make the distribution very peaky. emperically the attention patterns also tend to be sparse in the upper layers, so the averaging effect may be less pronounced than implied. For linear attention though, the averaging effect tends to be more pronounced. \n\n- Line 98.: “As weights become more uniform …”\n  Assuming this about attention weights, i would suggest to make this explicit.\n\n- Line 227. In the linear recurrent equations, the zeroth-order term appears to be absent. It would help to state this explicitly to remind the readers.\n\n- A short discussion for the non-causal variant would help improve the clarity for the paper.\n\n- Table 1. Any hypothesis for the weaker performance on the Fuzzy Recall task?\n\n- To keep the paper self-contained, please add brief descriptions of the benchmarks—particularly RegBench and MAD in appendix.\n\nLIMITATIONS:\nNone", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "CLARITY": 3, "IMPACT": 4, "ORIGINALITY": 4, "DATE": "2025-07-10"}