Title: SeerAttention: Self-distilled Attention Gating for Efficient Long-context Prefilling
Venue: NeurIPS 2025
Accepted: yes
Keywords: LLM

Sparse Attention

Long-Context LLM
Abstract: Attention is the cornerstone of modern Large Language Models (LLMs). Yet its quadratic complexity hinders efficiency and scalability, especially for long-context processing. A promising approach is to leverage sparsity in attention.  However, existing sparsity-based solutions predominantly rely on predefined patterns or heuristics at the attention head level, struggling to adapt dynamically to different contexts efficiently. We propose SeerAttention, a simple yet effective attention mechanism that directly learns the block-level attention sparsity from the LLM itself. Inspired by the gating mechanism in Mixture of Experts (MoE), SeerAttention augments the conventional attention with a **learnable gate** that **selectively activates important blocks** within the attention map. Specifically, the gate first pools the query (Q) and key (K) tensors along the sequence dimension and processes them through learnable linear layers. The resulting matrices are then multiplied together to produce the gating scores, which are used to predict block-level attention sparsity. Combined with our block-sparse FlashAttention kernel, SeerAttention can achieve significant speedup on GPUs. When applied to pre-trained LLMs, SeerAttention only requires training the gate parameters in a lightweight self-distillation manner, allowing rapid convergence. Our evaluation results demonstrate that SeerAttention achieves better model accuracy and lower latency for long-context pre-filling compared to prior methods. Code is available at: https://github.com/microsoft/SeerAttention.


Peer reviews:
comments: SUMMARY:
This paper introduces SeerAttention, which employs a learnable method to achieve sparsity of attention computations. The paper also introduces self-distillation and kernels to achieve efficient training and inference. Experiments show that the method can achieve better efficiency while preserving the performance.

STRENGTHS AND WEAKNESSES:
Strengths: 
* This paper introduces a new sparse attention, which can be obtained through distillation from the full attention. 
* The authors design kernels to achieve efficiency. 
* Experiments show the performance and efficiency of this method. 

Weaknesses:
* **Lack of differences from other methods**: from the perspective of novelty, the core idea is similar to NSA and MoBA. In addition, I believe these attention variants can also be integrated into existing models via self-distillation rather than training them from scratch, as the authors proposed.
* **Unfair comparison**: Compared with baselines, the trainable parameters are larger, and knowledge distillation is also employed in SeerAttention. Thus, I think the comparisons are not fair. I suggest that the authors can train the models with baselines using similar distillation methods.
* **Lack of ablation studies**:
    * w/o self-distillation
    * Different settings of sparsity and block size.

QUESTIONS:
Seeing Weaknesses

LIMITATIONS:
Yes
TITLE: Official Review
CLARITY: 3
IMPACT: 2
ORIGINALITY: 2
DATE: 2025-06-09

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a dynamic block sparse attention approach to accelerate the prefilling stage of large language models (LLMs). Specifically, it pools query (Q) and key (K) matrices into blocks along the sequence dimension, projecting them via learnable linear layers, and computes inner products to predict blocks with high attention values. Only the blocks estimated to have high attention values are computed explicitly. The authors also propose a corresponding training scheme using KL-divergence to align dense and sparse attention masks.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n\n1. **Effective design:** The method is straightforward and effective. Experimental results demonstrate that SeerAttention outperforms state-of-the-art sparse attention methods across the LongBench and RULER benchmarks.\n2. **Practical speedup:** The proposed custom kernels successfully translate theoretical improvements into practical wall-clock speedups. At 80% sparsity, SeerAttention achieves a 1.41x prefilling speedup across sequence lengths of 4K-128K in the RULER benchmark, showing superior scalability with increasing sparsity.\n\n**Weakness**\n\n1. **Fairness of comparisons:** SeerAttention exclusively uses sparse attention in the prefilling phase, whereas baseline methods (MoA and DuoAttention) apply sparse attention during both prefill and decoding. To ensure a fair comparison, the authors should either:\n   - Evaluate all methods with sparse prefill and dense decoding, isolating the impact of sparse decoding.\n   - Alternatively, discuss the efficiency-performance trade-offs more comprehensively across varying input-output lengths to highlight differences due to sparse decoding.\n\n2. **Clarity of illustrations and explanations:** The presentation can be improved for clarity:\n   - It is unclear whether the parameters $W_q$ and $W_k$ in Equation 1 are newly introduced learnable parameters distinct from the original query/key projection layers. Also it probably is, the author is advised to explicitly clarify that in the paper.\n   - Lines 80-81 conflate the concepts of \"dynamic\" (varying by input) and \"heterogeneous\" (varying across heads and layers).\n   - The specific method used for generating attention masks during evaluation (top-k or threshold-based) is not explicitly stated.\n3. **Novelty relative to prior work:** The relationship to previous works such as MInference, which also uses block-sparse attention with pooling-based block estimation, should be clarified in greater detail. Clearly articulating the differences from existing methods would strengthen the paper’s novelty claims.\n\nQUESTIONS:\n1. Could you provide additional results comparing sparse prefill with dense decoding across MoA, DuoAttention, and SeerAttention to ensure fairness and isolate the effect of sparse prefill?\n2. Could you include a comprehensive figure or table illustrating the performance-efficiency trade-offs (e.g., benchmark scores versus runtime) of various sparse attention methods **across different density levels**? This would effectively demonstrate SeerAttention’s relative strengths across conditions.\n\nLIMITATIONS:\nYes. The authors transparently acknowledge the limitation of exclusively optimizing prefilling rather than decoding. This focus is acceptable, given future work could explore efficient decoding strategies.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-06-23"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces SeerAttention, a post-training attention mechanism aimed at improving the efficiency of long-context inference in LLMs by learning sparse attention patterns through a self-distilled gating mechanism named AttnGate. Inspired by the gating mechanism in Mixture of Experts (MoE), SeerAttention adds a learnable gate to each attention layer that identifies salient Q-K blocks via pooled Q/K representations. The gating module is trained via KL-divergence to mimic block-level 2D max-pooled full attention outputs, using a lightweight self-distillation setup that freezes the base model. At inference time, a threshold or TopK is applied to the gating outputs to control sparsity. \n\nEmpirically, it demonstrates favorable accuracy-speed trade-offs on long-context tasks like LongBench and RULER, with significant speedups (up to 7.3× at 128k sequence length) and minimal accuracy degradation even at high sparsity than existing sparsity based prefilling techniques, such as MInference and DuoAttention.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\t1. The proposed method is well-justified and thoughtfully engineered, with a solid motivation tied to dynamic sparsity in attention. The use of self-distillation from full attention as a ground truth is conceptually sound and computationally efficient due to their custom max-pooling kernel.\n\t2. The work targets a critical bottleneck—long-context inference latency—and achieves consistent improvements across multiple model sizes and benchmarks.\n\t3. The paper is mostly well-written and thorough.\n\nWeaknesses:\n      1. Comparative Cost: The training cost s non-trivial compared to heuristic methods like DuoAttention or MInference. A more detailed comparison of training overhead vs performance gain is missing.\n      2. Lack of training data efficiency analysis, the paper relies on the RedPajama dataset for self-distillation but provides no justification for this choice or analysis of how the choice of training data impacts the learned AttnGate. Especially comparing to MInference and DuoAttention. This is important in practical applications.\n      3. Dynamic sparsity introduces challenging in batch inference.\n\nQUESTIONS:\n1. Comparison to DuoAttention/MInference in Training Cost: How does the training cost of SeerAttention (in GPU hours) compare to  DuoAttention/MInference (lightweight heuristics)? If significantly higher, can the authors provide a cost-benefit analysis?\n\n2. Can ground-truth sparsity (gt) be used directly? In Eq. (2), AttnGate is trained to approximate gt. Why not directly use gt to select top-k blocks (as an accuracy upper bound)? How much accuracy is lost compared to using gt?\n\n3. Is pooling method model-dependent? In section 3.1 and Fig. 2, different pooling strategies yield different performance. Are these consistent across model architectures (e.g., GQA vs MQA heads)? Does the best pooling generalize?\n\n4. Can the authors list and compare tunable parameters required by SeerAttention vs DuoAttention?\n\n5. In Table 1, it is better to show the individual sparsity under different length. For difference context lengths, the sparsity would be difference.\n\nLIMITATIONS:\n1. Training Resource Requirement: Although only the AttnGate is trained, the cost would be higher than heuristic methods. This might limit use in low-resource or deployment-only settings.\n\n2. The paper focuses exclusively on the pre-filling stage and does not discuss how SeerAttention could be extended to the decoding phase. If the proposed gating mechanism or block-sparse attention kernel is not easily compatible with decoding, this could significantly limit the method’s practical applicability, especially in real-time LLM deployments where decoding dominates inference latency.\n\n3. Lack of memory consumption analysis, especially for long context inference.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces SeerAttention—a post-training sparse attention method that augments the attention block with a learnable gating layer, which can identify block-sparsity patterns in attention maps. The gating layer introduced in the paper adds minimal overhead. It does not require from-scratch training, and the authors train it using a self-distillation technique, employing a KL-divergence loss with the ground-truth attention sparsity. They also augment the flash-attention kernel with their pooling scheme to ensure efficient attention during training of the gating layers. Following a block-sparsity pattern, the authors ensure compatibility with the FlashAttention kernel, making SeerAttention practically useful and demonstrating speedup over FA over large sequence lengths. The authors show extensive evaluation over several models and benchmarks and compare against various baselines. Their results show promising results with SeerAttention outperforming SOTA sparsity methods while being more efficient.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths\n- The information in the paper is very well presented. The authors describe their evaluation setup in detail and explain the choices made in their design and training of SeerAttention fairly well.\n- SeerAttention is compatible with FlashAttention, and shows practical speedup over the SOTA attention method for long context inference. This makes it deployable in practical settings.\n- The evaluation is extensive, with SeerAttention showing minimal drop in accuracy over multiple models and benchmarks. The attention maps identified by the gate layer also look promising.\n\n## Weaknesses\n- The choice of threshold in the evaluations is such that the sparsity levels for SeerAttention for both RULER and LongBench hover around 0.5-0.6. These are fairly low sparsity regimes, and attention maps have much higher sparsity. It would be great if the authors could show benchmark accuracy results for a wider range of sparsity with SeerAttention\n- The authors mention show ~7x speedup using SeerAttention, but that occurs at 90% sparsity, and there is no information in the paper to understand the accuracy trade-off with that kind of sparsity. At 50-60% sparsity, the speedups are more in the range of 1.5-1.6, which can be considered good.\n- While the training scheme for the gate layers developed by the authors is efficient and does not require full model retraining, it is still expensive. By the author’s own experiments, training the gate layers of a 7B model can take up to 40 A100 hours. For larger models like Llama 70B or 405B, this might be even more expensive. In this regime, this might be a big trade-off when compared to techniques that require no post-training.\n- SeerAttention only optimises the pre-fill stage and thus will not provide much benefit in decode-heavy workloads\n\nQUESTIONS:\n- Can the authors show the results for LongBench and RULER with different sparsity levels for SeerAttention?\n- In the Short-Context evaluation, MMLU and ARC-c have very low average sparsities - 3.4 and 26. This is not the case for other short-context benchmarks. Have the authors analysed why that is the case? Is it possible that, depending on the dataset used to train the gate layers, it may not detect sparsity in certain downstream tasks?\n- What is the memory overhead of gate parameters added to the model?\n- SeerAttention applies block PE to the non-RoPE Q and K. As far as I know, the keys in KV-Cache are stored after RoPE application. How does SeerAttention handle this?\n\nLIMITATIONS:\nLimited discussion. While the authors do mention future work in their conclusion with potential for improvements to SeerAttention, having a dedicated section on Limitations would be helpful. There are several limitations which can be discussed: applicability to larger models due to post-training requirements, applicability to the decode stage in inference, and memory overhead due to added gating parameters.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-07-01"}