Title: Delta Attention: Fast and Accurate Sparse Attention Inference by Delta Correction
Venue: NeurIPS 2025
Accepted: yes
Keywords: sparse attention

transformer

LLM

efficiency
Abstract: The attention mechanism of a transformer has a quadratic complexity, leading to high inference costs and latency for long sequences. However, attention matrices are mostly sparse, which implies that many entries may be omitted from computation for efficient inference. Sparse attention inference methods aim to reduce this computational burden; however, they also come with a troublesome performance degradation. We discover that one reason for this degradation is that the sparse calculation induces a distributional shift in the attention outputs. The distributional shift causes decoding-time queries to fail to align well with the appropriate keys from the prefill stage, leading to a drop in performance. We propose a simple, novel, and effective procedure for correcting this distributional shift, bringing the distribution of sparse attention outputs closer to that of quadratic attention. Our method can be applied on top of any sparse attention method, and results in an average 36\%pt performance increase, recovering 88\% of quadratic attention accuracy on the 131K RULER benchmark when applied on top of sliding window attention with sink tokens while only adding a small overhead. Our method can maintain approximately 98.5\% sparsity over full quadratic attention, making our model 32 times faster than Flash Attention 2 when processing 1M token prefills.


Peer reviews:
comments: SUMMARY:
- This work proposes a new sparse attention method targeting the computational complexity of self-attetion during the model inference.
- This method exploits the distribution shift between the full attention and sparse attention, and proposes a delta method to bridge this distribution gap.
- This method is extensively evaluated on several long context tasks, achieving superior performances than existing methods while keeping sparsity.

STRENGTHS AND WEAKNESSES:
# strength
- The paper is well written and the motivation is clear. the analysis about the distribution shift of attention scores shown in fig 3 is insightful.
- The performance gains seem significant, while the increased latency is minimum. the ablations about $\Delta$ and recomputation are comprehensive.
- As reported, this method could be seamlessly integrated with existing sparse methods.
# weakness
- Ablation of $\gamma$ on down-stream tasks are preferred to be reported (now only ppl). i wonder does it requires task-specific tuning for ultra-long contexts. 
- Correction efficacy diminishes in higher network layers (Appendix Figs. 13-15), which may limit the gains in more complex and reasoning tasks.

QUESTIONS:
- i wonder does it requires task-specific tuning for ultra-long contexts. 
- the proposed method seems rely on an assumption that nearby tokens have similar representations, this may not be the case in complex or reasoning tasks, while the proposed method is mostly evaluated on long context benchmarks.

LIMITATIONS:
Please refer to the weakness and questions
TITLE: Official Review
CLARITY: 3
IMPACT: 2
ORIGINALITY: 2
DATE: 2025-06-30

comments: SUMMARY:
This work proposes $\Delta$ Attention, a method for mitigating the bias caused by sparse attention. The proposed method computes the residual between full attention and sparse attention on a subset of queries, and subsequently applies them as residuals for all queries under sparse attention. Empirically the proposed method improves performance for several sparse attention methods on 3 datasets.

STRENGTHS AND WEAKNESSES:
Strengths:
- Interesting new method which improves quality for sparse attention models.
- Extensive empirical results.
- Clear presentation.

Weakness:
- While the experiments span multiple base models and datasets, there is no statistical significance test. Statistical significance tests are useful even for deterministic methods. I'd like to encourage the authors report standard test results such as paired permutation tests.
- The writeup could benefit from analysis of more sophisticated interpolation methods. In Eq 7 the same correction is applied for contiguous $\gamma$ rows. But what if we linearly interpolate between corrections in these $\gamma$ rows, for example?

QUESTIONS:
Please see my interpolation comment. Did the authors try alternate interpolation methods?

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 4
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-07-01

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work aims to bridge the gap between attention scores obtained from sparse attention methods and those from full attention during the prefilling stage. The core idea is to introduce a $\\Delta$ matrix that approximates the difference between sparse and full attention scores. To construct this $\\Delta$ matrix, the authors compute full attention for a small subset of queries (e.g., 1/64) and assume that neighboring queries exhibit similar delta values. Based on this assumption, the $\\Delta$ values from the small subset are repeated across neighboring positions to approximate the full matrix.\n\nThe proposed $\\Delta$ attention mechanism adds negligible latency compared to standard sparse attention methods, while providing clear performance improvements.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n- The idea of using a small number of full attention computations to correct or enhance sparse attention scores is novel and interesting.\n- The experimental results effectively demonstrate the benefits of the approach.\n- The paper is well-written and easy to follow.\n\n**Weaknesses**\n1. A key assumption in the method is that delta values for neighboring queries are similar. However, this assumption is not empirically validated. It would strengthen the paper if the authors provided an analysis of the actual $\\Delta$ matrix, such as measuring correlations between neighboring delta values.\n2. Such an analysis could also inspire better methods for constructing the approximated $\\Delta$ matrix. For instance, instead of simply duplicating delta values, interpolation techniques might yield more accurate approximations. It's unclear how performance scales with the parameter $\\gamma$ (i.e., the fraction of queries used for full attention). At what $\\gamma$ value does the method match the performance of full attention?\n\nQUESTIONS:\n- Do the $\\Delta$ matrices look similar across different inputs? In other words, to what extent is the $\\Delta$ matrix query-dependent versus being a property of the model itself?\n\nLIMITATIONS:\nPlease see Weaknesses.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 4, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses a critical limitation in sparse attention methods: the performance degradation caused by distributional shift in attention outputs. The authors propose ∆ Attention, a post-processing correction method that computes the difference between sparse and full attention for a subset of queries and applies this correction across all positions. The method achieves performance improvements on various tasks while maintaining computational efficiency.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths:**\n1. The proposed ∆ Attention is practical for improving accuracy with minimal computational overhead, which can be applied to various existing sparse attention techniques.\n2. Excellent motivation by identifying the root cause of sparse attention degradation: distributional shift between dense and sparse attention\n3. Evaluation across multiple benchmarks, including RULER, Infinite-Bench, and PG19\n\n**Weaknesses:**\n1. The paper fails to provide convincing theoretical evidence for why computing full attention for only sparse queries (every γ-th position) is sufficient to correct the distributional shift across all positions. The core assumption that (A∆V)i ≈ (A∆V)i+ν is supported merely by empirical cosine similarity measurements (Figure 6b), lacking rigorous theoretical grounding. This fundamental gap undermines our understanding of when and why the method works\n2. The experimental evaluation suffers from two critical issues: (1): When comparing with KV cache eviction methods (e.g., StreamingLLM), the authors fail to explicitly acknowledge that ∆ Attention requires storing complete Key-Value pairs, dramatically increasing memory requirements from ~2K tokens to the full sequence length. This makes the comparison fundamentally unfair, as the methods operate under entirely different memory budgets. (2): For advanced sparse attention techniques with dynamic sparsity (HiP, MInference), the improvements are marginal (4-8%), raising questions about the method's cost-effectiveness when applied to already sophisticated approaches. The paper lacks an analysis of why the method shows such stark performance differences across baseline categories.\n3. Some previous work has already worked on fixing the distribution shift for different kinds of sparse attention at test time, including:\n    \n      [1]. Dong, Harry, et al. \"Get more with less: Synthesizing recurrence with kv cache compression for efficient llm inference.\" arXiv preprint arXiv:2402.09398 (2024).\n\n      [2]. Yao, Jiayi, et al. \"CacheBlend: Fast large language model serving for RAG with cached knowledge fusion.\" Proceedings of the Twentieth European Conference on Computer Systems. 2025.\n\n\n      [3]. Sun Y, Ye T, Dong L, et al. Rectified Sparse Attention[J]. arXiv preprint arXiv:2506.04108, 2025.\n\n      [4]. Yang X, Chen T, Chen B. APE: Faster and Longer Context-Augmented Generation via Adaptive Parallel Encoding[J]. arXiv preprint arXiv:2502.05431, 2025.\n\nThese methods will further refine the distribution at test time. However, the authors do not discuss about these related work.\n\nQUESTIONS:\nSee weaknesses\n\nLIMITATIONS:\nSee weaknesses", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-03"}