SeerAttention: Self-distilled Attention Gating for Efficient Long-context Prefilling
Authors: Yizhao Gao
Zhichen Zeng
DaYou Du
Shijie Cao
Peiyuan Zhou
Jiaxing Qi
Junjie Lai
Hayden Kwok-Hay So
Ting Cao
Fan Yang
Mao Yang
Venue: NeurIPS 2025 poster ()
Keywords: LLM
Sparse Attention
Long-Context LLM
Primary area: deep_learning

Abstract: Attention is the cornerstone of modern Large Language Models (LLMs). Yet its quadratic complexity hinders efficiency and scalability, especially for long-context processing. A promising approach is to leverage sparsity in attention.  However, existing sparsity-based solutions predominantly rely on predefined patterns or heuristics at the attention head level, struggling to adapt dynamically to different contexts efficiently. We propose SeerAttention, a simple yet effective attention mechanism that directly learns the block-level attention sparsity from the LLM itself. Inspired by the gating mechanism in Mixture of Experts (MoE), SeerAttention augments the conventional attention with a **learnable gate** that **selectively activates important blocks** within the attention map. Specifically, the gate first pools the query (Q) and key (K) tensors along the sequence dimension and processes them through learnable linear layers. The resulting matrices are then multiplied together to produce the gating scores, which are used to predict block-level attention sparsity. Combined with our block-sparse FlashAttention kernel, SeerAttention can achieve significant speedup on GPUs. When applied to pre-trained LLMs, SeerAttention only requires training the gate parameters in a lightweight self-distillation manner, allowing rapid convergence. Our evaluation results demonstrate that SeerAttention achieves better model accuracy and lower latency for long-context pre-filling compared to prior methods. Code is available at: https://github.com/microsoft/SeerAttention.

Self-distilled Attention Gating for Efficient Long-context Prefilling
Yizhao Gao
Zhichen Zeng>*t
Dayou Du
Shijie Cao
Peiyuan Zhou Fan Yang*
Jiaxing Qi Junjie Lai
Hayden Kwok-Hay So'
Ting Cao
Mao Yang'
Universsity of Hong Kong
University of Washington SNVIDIA
University of Edinburgh
*Microsoft Research
Tsinghua University
Abstract
Attention is the cornerstone of modern Large Language Models (LLMs). Yet its quadratic complexity hinders efficiency and scalability, especially for long- context processing. A promising approach is leverage sparsity in attention. However, existing sparsity-based solutions predominantly rely on predefined pat- terns or heuristics at the attention head level, struggling adapt dynamically different contexts efficiently. We propose SeerAttention, a simple yet effective attention mechanism that directly learns the block-level attention sparsity from the LLM itself. Inspired by the gating mechanism in Mixture of Experts (MoE), SeerAttention augments the conventional attention with a learnable gate that se- lectively activates important blocks within the attention map. Specifically, the gate first pools the query (Q) and key (K) tensors along the sequence dimension and processes them through learnable linear layers. The resulting matrices are then multiplied together to produce the gating scores, which are used predict block-level attention sparsity. Combined with our block-sparse FlestAttention kernel, Attention can achieve significant speedup on GPUs. When applied pre-trained LLMs, SecrAttention only requires training the gate parameters in a lightweight self-distillation manner, allowing rapid convergence. Our evaluation results demonstrate that SeerAttention achieves better model accuracy and lower latency for contect pre- filliiiling compared to prior methods. Code is available at: https//ggghaaccooommmmcoottttttt
Introduction
Despite its effectiveness, the quadratic complexity of attention demands substantial computation and memory resources, limiting the scalability and efficiency of LLMs, especially for long-context windows. This challenge has become active research topie in the community. One potential solution replace the quadratic attention with cheaper architectures like linear attention or recurrent networks [30) (2) 40 47] with subquadratic complexity. While these approaches are more efficient, the majority of state-of-the-an large language models (LLMs) continue to use full attention to achieve better performance.
Attention is fundamental mechanism in transformer-based LLMS 51
A promising approach with increasing interests is to leverage sparsity in attention. Sparsity commonly exists in attention maps, and it becomes more prominent in longer contexts. In certain LLM attention heads, the sparsity ratio reach 95% even 99%, posing great opportunities for efficiency improvements. However, prior post-training methods often rely on predefined sparsity patterns or heuristics to approximate the attention mechanism 28
54 The sparsity observed in
'Equal Contribution.
I Work partially done during the internship Microsoft Research. Corresponding author: shijiccao@ microsoft.com.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

attention maps varies significantly across different models, imput contexts and attention heads, making predefined patterns or heuristies insufficient. On the other hand, pre-training sparse attention model from scratch like Native Sparse Attention 58 or MoBA (38 is costly and can not be directly applied to other dense pre-trained models.
In this paper, we introduce SeerAttention a simple yet effective post-training distillation method that brings tangible attention sparsity to any full-attention models through self-distillation without relying on predefined sparsity patterns. To achieve this, SeerAttention augments conventional attention with a learnable gate called ArmGate that selectively activates a small subset of important blocks in the attention map, drawing inspiration from the gating mechanism in MoE [45]. The AttnGates are trained with the 2D block-level sparsity ground truth generated by the original LLMs. Importantly, this distillation process only requires learning the gating parameters, while all other model parameters remain fixed. This leads to fast training process as only the newly added gate weights requires to compute gradient. In this way, without relying on human heuristic observations, users can obtain tailored AttnGates for different models.
Our results demonstrate that ScerAttention surpasses state-of-the-art post-training sparse attention methods like MInference 228], MoA [8] and DuoAAtention [54] in terms of long context model accu- racy and pre-filling latency. SeerAttention achieves highly linear speedup over dense configurations, delivering a 7.3x speedup with 90% sparsity on sequences of 128k. Notably, in contrast to previous methods that require careful calibration of sparse configuration for different heads, SeerAttention offers strong capabilities of adaptation different heads and contexts. Remarkably, on top of block- sparse pattern, SterAttention exhibits the ability to learn more diverse patterns, including A-shape and VSrt, further demonstrating its versatility and performance.
Our contributions can be summarized as follows:
We propose SeerAttention, an innovative learnable attention gating mechanism to enhance efficiency for long-context LLMs.
We have developed a self-distillation training scheme to efficiently train the AttnGate, enabling it to learn the intrinsic sparsity of a pre-trained model.
Experiments show that SeerAttention outperforms previous approaches, offering adaptability to various context lengths and sparsity ratios.
2 Background and Related Works
Powerful but Complex Attention in Transformer. The advent of attention mechanisms, partieu- larly within the Transformer architecture (51), marked a sigmificant advancement in natural language processing. Attention enables improved handling of long-range dependencies and better understand- ing of context by attending each token to every other token in the sequence, resulting in a quadratic memory and time complexity O(n²). where the sequence length. This presents a significant challenge as the community moves towards LLMs that can process increasingly longer contexts. Many studies explore alternative attention mechanisms mitigate this complexity. The Reformer architecture (31 reduces the complexity to O(n log n) and the linear attention mechanism [30 57
further decreases complexity to On)). Recently, there has been a trend of revisiting recurrent neural networks, leading to the proposal of new architectural frameworks such as RWKV 40, RetNet [7] and Mamba [20). Despite their promise of efficiency, these methods struggle to match the performance of full attention mechanisms, particularly with larger models and longer contexts.
Intrinsic but Dynamic Sparsity in Attention. Attention mechanisms inherently exhibit sparsity, which arises from the attention map A generated by Q and K: A = softmax (ettmax(QK"T /Vd). The softmax function often produces multitude of negligible scores that can be treated as zeros without impacting model accuracy [59] 35 52.77 36). Attention sparsity becomes more pronounced with longer contexts, presenting opportumities to optimize inference speed. Unfortunately, this sparsity is dynamic, varying across different context inputs and attention heads, each displaying distinct sparsity locations and ratios. Prior research has attempted to approximate attention sparsity using predefined patterns and heuristics [18 28 for different attention heads. Yet, these methods lack generality and often rely on handcrafted features, struggling to fully capture the sparsity behavior of attention mechanisms. The dynamic and dnput-dependent nature of attention sparsity echoes the principles of
2

---

Block
KT
KT
KF
Pooling
Pooling
Full
AttnMsp
Layear|Layer
Linear
Pooling
LinearLayer Layer
Pooling
Linea Layer
Topk
See
KLdiv Loss
Threshold
MaxPooling
Block Sparse Flash-Attn
AttiCaate Ouput
Binary Mask
Truth
Training AttnGate
Inference with Attn
The AttnGate in SeerAttention first pools the Q and K tensors sequence dimension and passes through learnable linear layers. The AttnGate output are trained to mimic the 2D maxpooled results of the pre-trained model. During inference, it applies TopK or Thresholding on the AttnGate output to locate activate blocks.
Figure 1: Overall of SeerAttention.
Mixture of Experts (MoE) models 45 17 suggesting that sparsity should ideally be learned directly from data within the model itself. This approach would allow models adaptively harness sparsity, improving efficiency while maintaining accuracy.
Long-Context LLM Optimizations. The ability to process long contexts erucial for large language models (LLMs) as it enables them to retain and utilize more extensive information. However, comes with substantial computational and memory costs. Various research efforts have explored different strategies to optimize long-context processing. One major direction is improving prefill efficiency, where techniques such as prompt compression [27 39 and sparse attention [28] [8]
[2).)) Another approach focuses optimizing the decoding phase by introducing sparse loading mechanisms [56] 6]. Additionally, several methods aim to compress the KV cache, including KV
cache sharing 3 5
KV eviction policies [63 34 19]. and KV quantization 37 25 14 61.
3 SeerAttention
SeerAttention adopts a fully learning-basse based approach to adaptively identify attention sparsity in LLMs and leverages the learned sparsity for efficient inference. To ensure efficiency modern hardware like GPUs, we focus on learning block sparsity, which can seamlessly integrate with the tiling computation scheme of FlashAttention [ 12] Figure I illustraats the overall diagram of SeerAttention, which augments conventional attention with a learnable gating module, termed Attention Gate (AttnGate). The AttnGate modules contain learnable parameters (linear layers) and distilled to mimic the 2D-Maxpooled results of the attention maps. At inference time, the AttnGate can predict the block-level sparsity for the subsequent attention computation with block-sparse Flash Attsntittntionn kernel, which significantly enhances performance by reducing I/O and computation overhead.
3.1 Attention Gate Design
The AttnGate module is designed to learn block-wise information with minimal overhead. It takes the original matrices Q and K as inputs and downsamples them using pooling operations along the sequence dimension. As shown in Figure [] for a given attention head, the sizes of the pooled Q and K become [seq/ B, d), where is the kernel and stride size of the pooling operation (non-overlapped blocks). The downsampled Q and K are then processed through linear layer and multiplied together, similar to the standard attention operation. This results each element corresponds to one block in the original full attention map. With typical block size of 64, the output of the AttnGate module is only a096 the size of the original attention map, making super efficient to compute. To its simplest form, the AttnGate output soft score can be expressed as:
matrix of size [seg/ / B, seq/ B), where

---

Q. RoPE W, concat, my p!d) (Q (Qrope))
(la)
K. RoPE Wi concat y p(( (Knope))
(1b)
0 softmax Qg /V/va).
(Ic)
where p(9) and p/(h) represents different pooling operations for Q and K, and is the hidden size of the tensors similar to attention computation. The detailed algorithm will be explained as follows.
Pooling Method Selection. Pooling op- erations downsample tensors and may lead to information loss. To better preserve the characteristies of the attention ten- sors, SeerAttention allows different pool- ing methods to be composed for Q and K. Specifically, consider average, max, and min pooling. When applying multiple pool- ing methods to either Q or K, the resulting pooled tensors are concatenated along the hidden dimension before being fed into the subsequent linear layer. Figure 2 presents the test perplexity on the PG19 42 dataset for the top 15 pooling combinations using the LLaMA-3., 1-8B model. We observe that applying AvgPooling on Q and a com- bination of Max, Min, and AvgPooling on K yields the best perplexity across different sparsity ratios. This trend may relate to prior findings in LLM quantization, where K tensors exhibit more outliers. Incorporating Max and Min pooling thus helps capture these extreme activations, leading to richer feature representations after pooling.
Omax,
Qavg_Kmin
10.2
Qmax Kminagg
Qmin Kmaxminava
giwg Kavg_K
Qavg. Kmaumin
Omin, Kmaxmin
Perplexity
Qmin Kmaxava
Qavg Kminavg
gmar Kmaravy
Qavi Kmanavg
10.1
Omin, Kmnnvg
Oavg, Kmauminavg
Qavg Kima
10.0
Sparsity
Figure 2: Test perplexity of different pooling combi- nations on PG19. The best configuration applies Avg- Pooling on Q and a combination of Max, Min, and AvgPooling on K.
Length Extrapolation of AttnGate using Positional Encoding. Recent state-of-the-ar LLMs typically employ positional encoding (PE) such as RoPE [46] to encode positional information. If the AttnGate relies solely on the original RoPE the model, i.e., feeding the AttnGate with Qrope and Krope. the positial information can possibly be damaged because of the pooling operation. This compromises the length extrapolate ability of ÅttnGate during distillation. To address this issue, we re-apply block-level PE in AttnGate with input wihtout PE, Qnope and K nopee (shown in Equation 9). To represent the block-level information, the RoPE in AttnGate uses a reduced 0' = 0/ B/B, where the original RoPE theta of the L.M..
Train Length 8k
Train Length 32k
Train Length 128k
Perplexity PG19
Without AttnGate-PE
Without AttnGate-PE
Without AttnGate-PE
60
AttnGate-PEE
AttnGate-PE
With AttnGate-PE
40
20
8k
16k
32k
64k
128k
8k
16k
32k
64k
128k
8k
16k
32k
64k
128k
Eval Length
Eval Length
Eval Length
Figure 3: Perplexity Comparison Between two PE setting in AttnGate on PG19 dataset. The block-level RoPE in AttnGate allows it to effectively learn the block-level positional information, resulting better test performance for different context length. Without the AttnGate PE, it fails to deliver reasonable results with data longer than training length.
Figure 3 presents the test perplexity results with and without the block-level RoPE design in AttnGate. The results indicate that without this block-level RoPF design, AttnGate fails perform adequately on evaluation data longer than 8k when trained with 8k length data. Similarly, when trained with 32k

---

length data, it does not perform well on 128k length data. However, with the additional block-level RoPE, AttnGate can extrapolate to different context lengths, significantly enhancing the model performance and training efficiency.
3.2 AttnGate Training
While the introduced SeerAttention architecture is straightforward, training presents challenges. Jointly training the gate and model from scratch, as in MoE, is costly and difficult. Fortunately, unlike MoE, where gating network must learn expert selection from scratch, the AttnGate in SeerAttention has a ground truth from standard attention for distillation.
Obtaining the Ground Truth. We the 2D-MaxPooled attention map from full attention as ground truth to distill AttnGate, as illustrated Figure [ Semantically, means that only when all the attention score in block is small, the 2D-MaxPooled results will be small, which aligned with the block-sparse definition. However, obtaining the max-pooled attention map for training non-trivial especially in long-context scenarios due to quadratic memory consumption of the intermediate QK" T results. To address this challenge, we customize an efficient kernel that directly outputs the MaxPooled attention map ground truth by modifying Attentino kernel but largely reuses its original computation flow. The detailed design are explained in A.1
Loss Function. The Kullback-Leible divergence loss [29 is use to distill the AttnGate. Since AttGaae uses softmax in output similar to full attention computation, the row summation of gating score will always be 1. KL-divergence loss allows the training process to focus mimicking the attention distribution instead absolute magnitude like dean-square-error loss. The overall distillation process can be expressed as:
Prope K KTope
gt MaxPool2D softmax(
(2)
O AttnGatee Q Qnoper Knope);
(3)
loss = DKL (gt |]
(4)
3.3 Inference with SeerAttention
After self-distillation training process, AterAttention utilizes the trained AttnGate generate gating score for each block within the full attention mechanism. These scores are then used to select the final activated sparse blocks. Further combined with our backend Block-sparse FlashAttention kernel, SeerAttention can achieve significant speedup for long-context prefilling while maintaining high accuracy.
Generating Binary Block Mask. SeerAttention provides the fiexibility to convert the floating-point gating scores into a final binary block mask using either the TopK Thresholding methods. If using the Topk method, the & blocks with the highest scores in each row are selected.
bi {
if E TopK(o;, inndex,
(5)
otherwise.
Alternatively, users can activate blocks with score exceeding a threshold. This can further saving the need of sorting the AttnGate output score. threshold
(6)
Notably, once AttnGate is trained, for the inference stges, we can adjust the TopK ratio or threshold- based at test time to achieve various trade-offs.
Block Sparse Flash-Attn Kernel. In designing the Block Sparse Mass--Attention kernel, the block size of AttnGate is aligned with the tiling size used in Tash-AAtentionn, typically 64 or 128. By doing so, we can create a customized block-sparse Flash-Attention kernel that leverages the binary block mask generated by AttnGate to selectively skip the 1/O and computation for unactivated blocks. This approach is highly efficient on modern GPUs, as it optimizes the processing of sparse data at the block level rather than dealing with fine- grainnd element-wise level, leading to significant performance gains.
5

---

Experiments
In this section, we evaluate both the accuracy and efficiency of SeerAttention. In our current experiments, block-size B for the AttnGate and sparse kernel is fixed at 64 and AttnGate solely applies the prefill stage.
Models, Baselines and Tasks. We apply SeerAttention to the pre-trained models Llama-3. 1.tmmaa-.11-- 1--BBB Instruct and Llama-3.1 15.1-70B-Inntruce [15]. as well as Bwen2.5-7B-Instruue Qwen2.5- 105.5-114B-Insstrue and Qwen2.5- -1-2B-Inntttctt [55] in the experiments. We compare SeerAttention with three state- of-the-art sparse attention methods, MoA [8]. MInference [28]. and DuoAttention [54]. It shoule be noted that support/oofiiuration Thus, we only compare with them using Llama-3.1-88 3.11-88-Instruc model. MoA uses offline search scheme to apply static sparse patterns across different attention heads. In our experiment, we adopt their official implementation with "KV Sparsity" in 0.5 which means "Attention Sparsity" in 0.35. MInference dynamically generates sparse indices using heuristic methods for each head based on pre-defined sparse patterns. We used their official configuration for ..amm-33.-888 Instruct model, where all attention heads choose the "Vertical-Slac Slashh" sparsity pattern. DuoAAtention differentiates some attention heads as streaming heads [53] while keep the rest as dense heads, In the following experiment, we adopted their official setup for
is the only model that all the other methods provide official
model with 50% head as streaming heads. We evaluate the performance using on two long context benchmarks: LongBench [ and RULER (26). and 4 short-context task from Open LLM Leaderboard [50] HellaSwag [60], MMLU (24), ARC-challenge 19, GSM8K [10]. For long- cong-context benchmark, we follow similar practice in Star Attention (2] and SCBench B3 that only applies sparsity in context
rather than question in SeerAttention. All the evaluation were run on A 100 GPUs.
Distillation Training Setup. We use the RedPajama [ dataset for AttnGate distillation, which chunked into 64k with BOS and EOS tokens. Our training employs a learning rate of 1e-3 with cosine decay. We set the global batch size to and conduct training for only 500 steps, leveraging A100 GPUs. As only AttnGate parameters
DeepSpeed [43] stage 2 optimization updated, the distillation process can be completed within around 40 A 100 hours for 7B or 8B models. To prevent the quadratic memory explosion that occurs when saving the intermediate attention map for ground truth generation, we customized a FlashAttention kernel. This kernel directly outputs the 2D max-pooled ground truth top of the original attention computation. Further details about this kernel can be found in A.1
learned and
4.1 Accuracy of Evaluation
LongBench Evaluation. LongBench is long-context underrsaddig benchmark. We compare with those of MoA, MInference, and DuoAttention using the L.lam-3..1-8-- 1-8B-Instruct model. DuoAAtention
Table 1: LongBench Results on Llama and Qwen models.
Avg.
Avg.
Model
Method
0-4k
4-8k
8k+
Acc.
Sparsity
Full Attention
55.32
53.98
52.9
54.07
0.0
MInference
55.23
53.78
52.18
53.73
0.31
Llama-3.1-88
MoA
50.74
49.84
51.89
50.82
0.35
DuoAttention
53.77
52.17
51.27
52.40
0.5*
SeerAttention
55.43
54.49
52.69
54.20
0,50
Full Attention
58.32
57.29
57.32
57.64
0.0
1imma-3.--7BB-Inttmee
SeerAttention
57.83
56.07
55.61
56.50
0.62
Full Attention
53.72
50.52
48.21
50.81
0.0
Qwen2.5-7B
SeerAttention
53.94
50.78
48.73
51.80
0.55
Full Attention
54.64
53.16
50.68
52.83
0.0
Qwen2.5-14B
ScerAttention
54.55
52.84
51.21
52.86
0.55
Full Attention
56.29
52.17
51.83
53.43
0.00
Qwen2.5-32B
SeerAttention
56.43
51.93
52.01
53.45
0.56
50% streaming heads, the real sparsity <50%

---

uses 50% of the heads as streaming heads and 50% as dense heads. For streaming heads, the attention only occurs in the attention sink and recent tokens. As result, it is not less than 50% sparsity overall. In this benchmark test, Sternttention employs a threshold of 2e-3 for all AttnGates. With the same threshold, different attention gates can exhibit varying sparsity ratios, and longer context data tends to be sparser. This approach allows for a more adaptive allocation of sparsity. As demonstrated in Table [I SeerAttention consistently outperforms other methods across various test lengths. Notably, in the 0-4k and 4-8k tests, our score surpasses even the dense baseline. This may be attributed to AttnGate filtering out noisy attention certain cases. Furthermore, SeerAttention achieves the highest average score and the highest average sparsity across all tests. For other models except SeerAttention demonstrates similar accuracy performance compared to dense baseline with 50% averaged sparsity. The 8k+ split typically has an sparsity around 70%.
Llama-3.1-4
Table 2: RULER Benchmark Results on Llama-3.
Model.
Average Accuracy
Average
Methods
4k
8k
16k
32k
64k
128k
Speedup
Full Attention 95.53
92.37
92.01
87.63
84.39
76,26
88.01
1.00
MInference
95.53
92.64
91.37
85.71
83.24
67.02
85.92
0.83
DuoAttention
95.64
92.08
90.71
84.75
83.24
75.32
86.96
1.09
SeerAttention 95.53
92.71
92.02
88.49
83.48
73.37
87.60
1.41
RULER Evaluation. RULER is long-contex. LLM evaluation benchmark consisting of 13 challenging sub-tasks. It generates tests with data sizes ranging from 4k to 128k. In this experiment, SeerAttention employs threshold of 5e-4, which allows automatically adapt sparsity from approximately 10% for 4k data around 85% for 128k data. Due to out-of-memory (OOM) issues in some tests, MoA was excluded from this benchmark. provides detailed accuracy results across different evaluation lengths. It is evident that Seer achieves the best accuracy in most tests (8k-64k). For the 128k test, DuoAAtention has less than 50% sparsity, whereas SeerAttention maintains an sparsity higher than 80%, which accounts for the slightly lower performance. SeerAttention also attains the highest average accuracy compared to other models (only 0.41% lower than the dense baseline) while delivering the highest average end-to-end speedup (1.41x) x) in prefilling time.
Table 3: Short Context Tests on Llama-3.1 3.1-8B- 1--8--IInsrue Model
MMLU
HellaS.
ARC-c
GSM-SK
Full Attention
68.1
80.1
60.7
75.7
SecrAttention
67.9
79.8
60.2
75.6
Avg Sparsity
3.4
50.4
26
52.1
Avg Seqlens
118
840
395
872
Short Context Test. For short context input, attention contributes smaller proportion in the total runtime. Consequently, sparse attention does not significantly enhance latency performance. Never- theless, evaluate SeerAttention accuracy performance under very high threshold 3e-2 to achieve high sparsity. The results, as shown 33 indicate that SecrAttention exhibits negligible accuracy loss. For instance, with an average sequence length of 872 in the GSM-8K task, Attenittntion achieves only 0. % degradation in accuracy with 52% averaged sparsity.
4.2 Efficiency Evaluation
We evaluate the efficiency of SeerAttention using our implementation of CUDA kernels. We evaluate the kernel-level as well as end-to-end speedup using a Results are compared to FlashAttention-2 (dense baseline), MoA, MInference and Attenttttion.
on a single A100 GPU.
4.2.1 Kernel evaluation
Negligible Overhead incurred by AttnGate. shows the kernel-level latency breakdown of SeerAttention. It demonstrates that the overhead introduced by the AttnGate during inference minimal. For instance, at context length of 32K and sparsity of 0.5, the AttnGate contributes
7

---

FlashAttention-2 (dense)
Block Sparse Attn
AttnGate
0.9
x.15x
0.9
7.37x
0.9
7.34x
0.8
x
0.8
4.02x
3.78x
Sparsiyy
0.7
2.16x
0.7
2.76x
2.53x
0.6
1.76x
0.6
2.12x
0.6
1.92x
0.5
1.56x
0.5
1.72x
0.5
1.55x
Full
Full
Full
10
30
50
200
400
600
800
Latency (ms)
Latency (ms)
Latency (ms)
Sequence Length 8K
Sequence Length 32K
Sequence Length: 128K
Figure SeerAttention Speedup over FlashAttentione at the Kernel Level. The lateney overhead from AttnGates is minimal. Our block-sparse attention kernel achieves highly linear speedup over dense configurations, delivering a 7.3x speedup with 90% sparsity on sequences 128k. The AttnGate overhead amost diminishes in 128k context length.
Block Sparse (SeerAttention)
A Block (MoA)
Vertical-Slaa (Minference)
15
Speedup
10
5
0.00
0.25 0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75 1.00
Sparsity
Sparsity
Sparsity
Sequence Length:
8K
Sequence Length:
32K
Sequence Length 128K
Figure 5: Kernel-level Speedup Comparison Between Different Works. SeerAttention translates sparsity speedup more effectively.
only 1% to the total latency of an attention layer. In the cases of 128K sequence length, the relative overhead almost diminishes.
Block-sparse FlashAttention Kernel Speedup. Figure allso shows that our kernel exibits linear speedup various sparsity levels. At sequence length of 128K with 90% sparsity, SttrAttention achieves a speedup of 7.3x compared with FlashAttention-2 (full attention) on single A A100 GPU, This demonstrates the effectiveness of the block-level sparsity employed by SeerAttention, which is highly efficient on GPUs and translates into high speedup.
Kernel-level Comparison with Related Works. We compare the kernel-level speedup of SeerAt- tention with MoA and MInference. MInference uses offline calibration to identify a pre-defined sparse pattern for each layer. For Llama-3. 1-88------B- model, MInference consistently uses "Vertical- slash" pattern across all layers. During runtime, MInference will dynamically generate non-zero indices based on their approximation algorithm. On the other hand, MoA uses "A-shape" blocks as their sparse pattern and calibrate the shape parameters offline under given sparsity constraint. DuoAt- tention is omitted kernel-leve comparison as it's a combination between streaming and dense head, whose performance is mixture results of block sparse attention and dense FlashAttention.
Figure 5 shows the sparsity v.s. speedup plots of different methods on 8k, 32k, 128k sequences length, where the speedup baseline is FlashAttention-2. The kernel-level sparsity statistics were collected from PG19 datasets. For MoA, we generated the sparse configurations under their 0.5 overall "KV-sparsity" constraints, which corresponds to average of 0.35 sparsity in attention. The results demonstrates that the block- block-sparse attention kernel used Seer Attentitn outperforms both MoA and MInference in most cases.

---

Minference
MoA
DupAttention
SeerAttention
SeerAttention
DusAttention
MoA
Minference
0.0
0.5
1.0 1.5
0.0
0.5
1.0
1.5
0.0
1.0
2.0
3.0
Speedup
Speedup
Speedup
Sequence Length 8K
Sequence Length: 32K
Sequence Length: 128K
Figure Comparing Prefilling Time Speedup on RULER Test Setting. SeerAttention outperforms related works in most long-context data scenarios (2 16k). For longer context data, the attention mechanism constitutes larger proportion of the total runtime, allowing sparse methods achieve better speedup. Overall, ScerAttention achieves the highest average speedup (1.41x) while maintain- ing the best average accuracy under this RULER benchmark setting.
4.2.2 End-to-end Speedup Comparison.
To assess the end-to-ent speedup of our method, we measured the average prefilling time, or time- to first-tokken (TTFT), using the Llama-3.1 3.1-BBBBBBBB----.. IB-Insttrct model the RULER test discussed above. Since attention takes up more runtime with longer contexts, all methods generally achieve better speedup with longer context lengths. It should be noted that SeerAttention across all tests RULER, automatically adjusting to higher sparsity for longer contexts (ranging from approximately 10% sparsity for 4k around 85% sparsity for 128k). This approach results in an end-tor end-to-end prefilling speedup of up to 2.43x on 128k length. On the other hand, MInference experiences slowdown with data sizes less than 64k due significant overhead in searching for sparse indices during runtime. It is feasible for SeerAttention to adjust to higher sparsity to achieve greater speedup in shorter contexts as tradeoff. Nevertheless, SeerAttention delivered the highest average accuracy (87.6) and the greatest average speedup (1.41x) in this RULER benchmark setting.
identical threshold
4.3 Training Cost and Parameter Overhead.
The additional training cost of SeerAttention is modest. On LalaMAA---B--nsttee it requires about 40 A100 GPU hours, similar to lower than DuoAttention. In comparison, MInference calibrates faster but shows lower accuracy and slower inference. SeerAttention introduces about 101 M trainable parameters, roughly 1.3 % of the model, and adds less than 5 % memory and latency overhead compared with FlashAttention, supported by our customized distillation kernels. The gating module also scales well to larger models, such 503 MB for LLaMA-3. 1-70B and 252 MB for DeepSeek-PPe M---istill-Qwenn deployments.
showing that the method remains lightweight and scalable for large
4.4 Visualization of Learned Attention Maps.
The AttnGate module automatically learns diverse sparse patterns without any prior knowledge or heuristics. Figure shows several example outputs from AttnGate, including (a) "A-shape." or streaming head (b) "Vertical," (c) "Slash" with empty vertical spaces, (d) block sparsity along the diagonal, and (e) random patterns. These patterns not only encompass but also extend beyond those observed in previous works such MoA and MInference, showcasing the versatillty of our learning based methods.
(a)
(d)
(e)
Figure 7: Visualization of the AttnGate's outputs.

---

Conclusion and Future Work
This paper presents SeerAttention, a new attention mechanism that learns and leverages the intrinsic sparsity in attention to boost long-context LLMs. SeerAttention learns the attention sparsity from the LLM itself with a lightweight self-distillation approach. Our experiments demonstrate that SeerAttention outperforms previous approaches in terms of long context model accuracy and pre- filling latency. For future work, there are several promising directions to explore for improving and expanding the capabilities of SecrAttention. One key area is enhancing the training methodologies for SeerAttention, such as applying SeerAttention in long-contex continued pre-training with more training tokens achieve higher sparsity without sacrificing accuracy. Another important avenue is applying SterAttention in the decoding stage, especially for long-CoT.
Acknowledgments
We gratefully acknowledge Prof. Ang Li from University of Washington for his help with the paper revision and valuable feedback.
References
[1] Shantanu Acharya, Fei Jia, and Boris Ginsburg. Star attention: Efficient Ilm inference over long sequences. arXiv preprint arXiv:2411.17116 2024.
[2] Shantanu Acharya, Fei Jia, and Boris Ginsburg. Star attention: Efficient Ilm inference over long sequences. arXiv preprint arXiv:2411.171I6 2024.
[3] Joshua Ainslie, James Lee-Thee Thorp. Michiel de Jong, Yury Zemlyanskiy, Federico Lebrón, and Sumit Sanghai. Gqa: Training generalized multi-query transformer models from multi-head checkpoints. arXiv preprint arXiv:2305.13245 2023.
[4] Yushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu, Jiankai Tang, Zhidian Huang, Zhengxiao Du, Xiao Liu, Aohan Zeng, Lei Hou, Yuxiao Dong, Jie Tang, and Juanzi Li. Longbench: A bilingual, multitask benchmark for long context understanding. arXiv preprint arXiv:2308. 14508, 2023.
[5] William Brandon, Mayank Mishra, Aniruddha Nrusimha, Rameswar Panda, and Jonathan Ragan Kelly. Reducing transformer key-value cache size with cross-layer attention. arXiv preprint arXiv:2405. / 12981, 2024.
[6] Zhuoming Chen, Ranajoy Sadhukhan, Zihao Ye, Yang Zhou, Jianyu Zhang, Niklas Nolte, Yuandong Tian, Matthijs Douze, Leon Bottou, Zhihao Jia, et al. Magicpig: Lsh sampling for efficient Ilm generation. arXiv preprint arXiv:24 10. 16179, 2024.
[7] Rewon Child, Scott Gray, Alec Radford, and Ilya Sutskever. Generating long sequences with sparse transformers. arXiv preprint arXiv: 10509, 2019.
[8] Yu-Neng Chuang, Tianwei Xing, Chia-Yure Yuan Chang, Zirui Liu, Xun Chen, and Xia Hu. Learning to compress prompt in natural language formats. arXiv preprint arXiv:2402. 18700, 2024.
[9] Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:
2018
[10] Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110. 2021.
[11] Together Computer. Redpajama: an open dataset for training large language models, 2023.
http:p////.iiith..ooo///ggterrrr /eedaaa Data
[12] Tri Dao. Flashattention-2:: Faster attention with better parallelism and work partitioning. 2023. URL https:////axx http://iwwwwra/////////a//// http://ss/220000055 08691
10

---

[13] Tri Dao, Dan Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. Flashattention: Fast and memory-efficient exact attention with io-awareness, Advances Neural Information Processing Systems, 35: 1634441663,,, 2022.
[14] Shichen Dong, Wen Cheng, Jiayu Qin, and Wei Wang. Qaq: Quality adaptive quantization for Ilm kv cache. arXiv preprini urXiv:2403.0464 2024.
[15] Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21773, 2024.
[16] Hugging Face. Open rl: A fully open reproduction of deepseek-rl, January 2025. URL https: / thub. cow http:///wuuggiggaac///opeeaa.ee.
[17] William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. Journal of Machine Learning Research, 23 (120):1-39, 2022.
[18] Tianyu Fu, Haofeng Huang, Xuefei Ning. Genghan Zhang, Boju Chen, Tianqi Wu, Hongyi Wang, Zixiao Huang, Shiyao Li, Shengen Yan, et al. Moa: Mixture of sparse attention for automatic large language model compression. arXiv preprint arXiv:2406. 14909, 2024.
[19] Suyu Ge, Yunan Zhang, Liyuan Liu, Minjia Zhang, Jiawei Han, and Jianfeng Gao. Model tells you what to discard: Adaptive kv cache compression for llms. arXiv preprint 10X0022310.01801, 2023.
[20] Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces. arXiv preprint arXiv:2312.0073 2023.
[21] Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang. Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in Ilms via reinforcement learning. arXiv preprint arXiv:2501. 2025.
[22] Insu Han, Rajesh Jayaram, Amin Karbasi, Vahab Mirrokni, David P Woodruff, and Amir Zandieh. Hyperattention: Long-context attention in near-linear time. arXiv preprint arXiv:2310005869, 2023.
[23] Dan Hendrycks. Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.0m 033000 2020.
[24] Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.0m 03300, 2020.
[25] Coleman Hooper, Sehoon Kim, Hiva Mohammadzadeh, Michael W Mahoney, Yakun Sophia Shao, Kurt Keutzer, and Amir Gholami. Kvquant: Towards 10 million context length Ilm inference with kv cache quantization. arXiv preprint arXiv:2401.18073 2024.
[26] Cheng-Ping Hsieh, Simeng Sun, Samuel Kriman, Shantanu Acharya, Dima Rekesh, Fei Jia, Yang Zhang, and Boris Ginsburg. Ruler: What's 's the real context size of your long-context language models? arXiv preprint arXiv:2404.06654 2024.
[27] Huiqiang Jiang, Qianhui Wu, Xufang Luo, Dongsheng Li, Chin-Yew Lin, Yuging Yang, and Lili Qiu. Longllmlingua: Accelerating and enhancing Ilms long context scenarios via prompt compression. arXiv preprint arXiv:2310.06833 2023.
[28] Huiqiang Jiang, Yucheng Li. Chengruidong Zhang, Qianhui Wu, Xufang Luo, Surin Ahn, Zhenhua Han, Amir H Abdi, Dongsheng Li, Chin-Yer Lin, et al. Minference 1.0: Accelerating pre-filling for long-context Iims via dynamic sparse attention. arXiv preprint arXiv:2407.024990 2024.
[29] James M Joyce, Kullback-leibler divergence. In International encyclopedia of statistical science, pages 720-722. Springer, 2011.

---

[30] Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, and François Fleuret. Transformers are mas: Fast autoregressive transformers with linear attention. In International conference on machine learning, pages 5156-5165, PMLR, 2020.
[31] Nikita Kitaev, Lukasz Kaiser, and Anselm Levskaya. Reformer: The efficient transformer. arXiv preprint arXhv:2001.04451, 2020.
[32] Heejun Lee, Geon Park. Youngwan Lee, Jina Kim, Wonyoung Jeong, Myeongjae Jeon, and Sung Ju Hwang. Hip attention: Sparse sub-quadratic attention with hierarchical attention pruning. arXiv preprint 1.Xi::2406.008222) 2024.
[33] Yucheng Li. Huiqiang Jiang, Qianhui Wu, Xufang Luo, Surin Ahn, Chengruidong Zhang, Amir H Abdi, Dongsheng Li, Jianfeng Gao, Yuqing Yang, et al. Scbench: A kv cache-centric analysis of long-context methods. arXiv preprint arXiv:2412. 2024.
[34] Yuhong Li, Yingbing Huang, Bowen Yang, Bharat Venkitesh, Acyr Locatelli, Hanchen Ye, Tianle Cai, Patrick Lewis, and Deming Chen. Snapky: Llm knows what you are looking for before generation. arXiv preprint arXiv:2404. 2024.
[35] Liu Liu, Zheng Qu, Zhaodong Chen, Yufei Ding, and Yuan Xie. Transformer acceleration with dynamic sparse attention. arXiv preprint arXiv:2 1229, 2021.
[36] Zichang Liu, Jue Wang, Tri Dao, Tianyi Zhou, Binhang Yuan, Zhao Song, Anshumali Shrivas- tava, Ce Zhang, Yuandong Tian, Christopher Re, et al. Deja vu: Contextual sparsity for efficient Ilms at inference time. In International Conference on Machine Learning, pages 22137-22176. PMLR, 2023.
[37] Zirui Liu, Jiayi Yuan, Hongye Jin, Shaochen Zhong, Zhaozhuo Xu, Vladimir Braverman, Beidi Chen, and Xia Hu. Kivi: A uuiin--rree asymmetric 2bit quantization for kv cache. arXiv preprint arXiv:2402.0222 02750, 2024.
[38] Enzhe Lu, Zhejun Jiang, Jingyuan Liu, Yulun Du, Tao Jiang, Chao Hong, Shaowei Liu, Weiran He, Enming Yuan, Yuzhi Wang, Zhiqi Huang, Huan Yuan, Suting Xu, Xinran Xu, Guokun Lai, Yanru Chen, Huabin Zheng, Junjie Yan, Jianlin Su, Yuxin Wu, Yutao Zhang. Zhilin Yang, Xinyu Zhou, Mingxing Zhang, and Jiezhong Qiu. Moba: Mixture block attention for long-context Ilms. arXiv preprint 23Xiv:2502.13/16 2025.
[39] Jesse Mu, Xiang Li, and Noah Goodman. Learning to compress prompts with gist tokens. Advances Neural Information Processing Systems, 36, 2024.
[40] Bo Peng. Eric Alcaide, Quentin Anthony, Alon Albalak, Samuel Arcadinho, Stella Biderman, Huangi Cao, Xin Cheng, Michael Chung, Matteo Grella, et al. Rwkv: Reinventing mns for the transformer era. arXiv preprint arXiv:2305.1300 13048, 2023.
[41] Bowen Peng, Jeffrey Quesnelle, Honglu Fan, and Enrico Shippole. YaRN: Efficient context window extension of large language models. In The Twelfth International Conference on Learning Representations, 2024. URL httpps///openrevier net/
[42] Jack Rae, Anna Potapenko, Siddhant M Jayakumar, and Timothy P Lillicrap. Compressive transformers for long-range sequence modelling. arXiv preprint arXiv: 191.::911.0050,,, 2019,
[43] Jeff Rasley, Samyam Rajbhandari, Olatunji Ruwase, and Yuxiong He. Deepspeed: System optimizations enable training deep learning models with over 100 billion parameters. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, KDD 220, page 3505-3506, New York, NY, USA, 2020. Association for Computing Machinery. ISBN 9781 4503399944 doi: 01145/339446.3446677 URL https //doi //doi.org/100 1141 3394553394466... 3406703
[44] David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel Bowman. Gpqa: A graduate-level google-proof q&a benchmark. In First Conference Language Modeling, 2024.
12

---

[45] Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoe Le, Geoffrey Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gate mixture-of-experts layer. arXiv preprint arXiv: /701.065.8, 2017.
[46]
Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding. Neurocoputting 568:127063. 2024.
[47] Yutao Sun, Li Dong, Shaohan Huang, Shuming Ma, Yuqing Xia, Jilong Xue, Jianyong Wang. and Furu Wei. Retentive network: A successor to transformer for large language models. arXiv preprint arXiv:2307.08621, 2023.
[48] Jiaming Tang, Yilong Zhao, Kan Zhu, Guangxuan Xiao, Baris Kasikei, and Song Han. Quest: Query-aware sparsity for efficient long-context IIm inference. arXiv preprint arXiv:2406. 10774, 2024.
[49] Philippe Tillet, Hsiang-Tsung Kung, and David Cox. Triton: an intermediate language and compiler for tiled neural network computations. In Proceedings of the 3rd ACM SIGPLAN International Workshop on Machine Learning and Programming Languages, pages 10-19, 2019.
[50] Lewis Tunstall, Edward Beeching, Nathan Lambert, Nazneen Rajani, Kashif Rasul, Youess Belkada, Shengyi Huang, Leandro von Werra, Clémentine Fourrier, Nathan Habib, et al. Zephyr: Direct distillation of Im alignment. arXiv preprint arXiv:2310.16944 2023.
[51] A Vaswani.
Attention is all you need. Advances in Neural Information Processing Systems,
2017.
[52] Hanrui Wang, Zhekai Zhang, and Song Han. Spatten: Efficient sparse attention architecture with cascade token and head pruning. In 202/ IEEE International Symposium High-Performance Computer Architecture (HPCA), pages 97-110. IEEE, 2021.
[53] Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. Efficient streaming language models with attention sinks. arXiv preprint arXiv:2309. /7453, 2023.
[54] Guangxuan Xiao, Jiaming Tang, Jingwei Zuo, Junxian Guo, Shang Yang, Haotian Tang, Yao Fu, and Song Han. Duoattention: Efficient long-context Ilm inference with retrieval and streaming heads. arXiv preprint arXiv:2410. 10819, 2024.
[55] An Yang, Baosong Yang. Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang. Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin Zhu, Rui Men, Runji Lin, Tianhao Li, Tingyu Xia, Xingzhang Ren, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yu Wan, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, and Zihan Qiu. Qwen2.5 technical report. arXiv preprint arXiv:2412.15115. 2024.
[56] Lijie Yang, Zhihao Zhang, Zhuofu Chen, Zikun Li, and Zhihao Jia. Tidaldecode: Fast and accurate Ilm decoding with position persistent sparse attention. arXiv preprint arXiv:2410.05076 2024.
[57] Songlin Yang. Bailin Wang, Yikang Shen, Rameswar Panda, and Yoon Kim. Gated linear attention transformers with hardware-efficiie training. arXiv preprint arXiv:2312.06635, 2023.
[58] Jingyang Yuan, Huazuo Gao, Damai Dai, Junyu Luo, Liang Zhao, Zhengyan Zhang, Zhenda Xie, YX Wei, Lean Wang. Zhiping Xiao, et al. Native sparse attention: Hardware-aligned and natively trainable sparse attention. arXiv preprint arXiv:2502. 1089, 2025.
[59] Manzil Zaheer, Guru Guruganesh, Kumar Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago Ontanon, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang. et al. Big bird: Transformers for longer sequences. Advances in neural information processing systems, 33: 17283-17297, 2020.
[60] Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can machine really finish your sentence? arXiv preprint
2019.
13

---

[61] Tianyi Zhang, Jonah Yi, Zhaozhuo Xu, and Anshumali Shrivastava. Kv cache is 1 bit per channel: Efficient large language model inference with coupled quantization. arXiv preprint arXiv:2405.03917, 2024.
[62] Xuechen Zhang, Xiangyu Chang, Mingchen Li, Amit Roy-Chowdhury, Jiasi Chen, and Samet Oymak. Selective attention: Enhancing transformer through principled context control. arXiv preprint 2rXfv:2411.128940 2024.
[63] Zhenyu Zhang. Ying Sheng, Tianyi Zhou, Tianlong Chen, Lianmin Zheng, Ruisi Cai, Zhao Song, Yuandong Tian, Christopher Ré, Clark Barrett, et al. H2o: Heavy-hitter oracle for efficient generative inference of large language models. Advances in Neural Information Processing Systems, 36:34661-34710, 2023.
[64] Qianchao Zhu, Jiangfei Duan, Chang Chen, Siran Liu, Xiuhong Li, Guanyu Feng, Xin Lv, Huanqi Cao, Xiao Chuanfu, Xingcheng Zhang, et al. Near-lossless acceleration of long context Ilm inference with adaptive structured sparse attention. arXiv preprint arXiv:2406. 15486, 2024.
14

---

A Appendix
A.1 Training SeerAttention with Customized GPU Kernel
In this appendix, we provide detailed design and implementation of our efficient kernel, highlighting key modifications FlashAttention and optimizations for long-context scenarios. We then evaluate the peak memory usage and additional latency overhead of our training kernel during the AttnGate training stage, showing that it incurs only minimal overhead in both memory and latency compared to training with FlashAttention-2.
Flash Attenion with --D-MaxPooling: A Customized training kernel. Section 3.2, we discussed the method for obtaining the ground truth attention map used to distill AttnGate. Specifically, we leverage the 2D-MMxxPoled attention map from full attention as the ground truth, aligning with the block-sparse attention definition. However, directly computing this attention map is challenging due to the quadratic memory complexity and the fused operation nature of FlashAttention. To overcome this, we developed a customized kernel based on Triton [9] that efficiently extracts the 2D-MaxPooled attention map by modifying the FlashAttention kernel while largely preserving its computation flow. Figure 8 shows the pseudo code and diagram of this customized kernel.
Inner Loop
-Pseuda Code of Customized Flash-Attn with MaxPooled AttnMap Input: Q, K, V; Output: O, A
KT
for i fran 1 to T,
Load 0,
for fron to
Load K,,
Computé S; dot(Q.,)), Fip (5,)
Q
Fgy Update H., max(n fay), by and O,,
Compute 1,, n, for fron T,
Load and Rescale Compute and Store 2 colnax(a;))
exp(ry, n//1,
Block Size B
Return O,
KTT
Store ri
Load
a, = exp( Fy m, )/1
8
Q
QKT
row max
entire row with final max and sum of exp
iterating
I
rescale
col max
Ag
Figure 8: Efficient FlashAttention kernel with pooling of attention map.
Normally, the softmax function ensures numerical stability by subtracting the maximum value before applying the exponential operation. FlashAttention computes the local row max of each block, and gradually updates the global maximum through iteration:
S Q.K7:
Tij rowmax (Stz);
my = max(m TQy).
where riy typically treated a temporary result. However, store it in HBM and rescale it later with the final global max m, and sum of exp I after the iteration:
as = exp(ri - m.)/4
(8)
This a.j represents the correct row max of the original attention block. With that, 2D-MaxPooling is achieved by applying a column max over a:y. This introduces only minor overhead (storing and rescaling riy) but significantly improves the efficiency of obtaining the ground truth. The overhead of memory and latnecy analysis is in Figure
Performance of the Training Kernel. We evaluate our customized FlashAttention kernel with 2D- MaxPooled attention map for scalable training of SeerAttention by comparing against with Py PyTorch naïve manual attention implementation and FlashAttention-2. As shown Figure the Py PyTorch kernel runs out of memory (OOM) when the sequence length exceeds 4k, while our customized kernel costs similar peak memory usage compared to FlashAttention-2... Regarding latency, since Py Torch encounters OOM for sequences longer than 8K, the attention operations per head into loop to assess
15

---

kernel-level latency. Figure 9b shows that the latency overhead introduced by the additional pooling operation is minimal compared PlashAttention-2. while the Py PyTocch implementation suffers from significant slowdown.
10
Hash-Attr-422
200
FlashAAlt-22
GPU Memory (GB)
Dustemized Hash-Alta with Pooling
Gustomized Flash-Aitn with Pooking
Naive marnul atin with tarch
Naive manndl with wih tarch
Latency
100
1.0
A100 Maximum
50
1k
2k
4k
8K
16k
32k
64k
2k
4K
8k
16k
32k
64k
Sequence Length
Sequence Longth
(a) Memory
(b) Latency
Figure Memory and latency of customized FlashAttention with max-pooling training kernel.
A.2 Preliminary Experiments of Fine-tuning with SeerAttention
YARN w/ SeerAttneion Sparsity 90% YoRN w/ SeerALineion Sparsity 50%
3.5
YARN Baseline
3.0
100
200
300
400
Steps
Figure 10: Fine-tuning Loss.
Figure 11: By incorporating SeerAttention with YaRN [41 to extend -lama-3-8B model from 8k
to 32k context length, the loss curves for 50% 90% sparsity are nearly identical to the dense YaRN baseline.
Table 4: Perplexity of YaRN baseline, SeerAttention after YaRN and YaRN fine-tuning with SeerAt-
tention.
YaRN
Post-training SeerAltention after YaRN
YaRN with SeerAttention
Sparsity
0.0
0.5
0.6
0.7
0.8
0.9
0.5
0.6
0.7
0.8
0.9
PG19
8.79
9.16
9.30
9.48
9.73
10.18
8.81
8.82
8.85
8.93
9.16
Proof-pile
2.46
2.53
2.57
2.61
2.68
2.85
2.47
2.47
2.51
2.60
In this preliminary experiment, we demonstrate that SeerAttention can be seamlessly integrated in Long-context extension fine-tuning stages. We follow YaRN [41] to extend the context size of a Llama-3-8B model from 8k 32k. The loss function is the summation of original LM loss and AttnGate loss. To ensure stable training, the AttCGaate are first initialized using the post-training self-distillation before tnnettuning the entire model. We integrate SecrAttention into YaRN and compare the performance against the YaRN dense baseline and the post-training time self-distillation of SeerAttention applied after YaRN. TTgure[Oppreeete the loss curves of the YaRN dense baseline and SeerAttention at 50% and 90% sparsity. The curve at 50% sparsity nearly overlaps with the baseline, while the curve 90% sparsity shows slightly higher loss. Table 4 displays the test perplexity on the PG19 and ProofPile datasets evaluated at 32k context length. The YaRN dense baseline achieves perplexity scores of 8.79 and 2.46, respectively. Post-training SeerAttention results in increased perplexity. When applying Seer during the YaRN extension fine-tuning, it maintains near-lossless performance 50% sparsity (with scores of 8.81 and 2.47), and even at 90% sparsity, the loss remains minimal.
16

---

A.3 Preliminary Results on Introducing Sparse Attention at Decoding Stage
Adjusting AttnGate for decoding The current AttnGate design mainly works for accelerating long-context prefill. However, applying the attention gate distillation idea is also feasible direction. This is important to improve the efficiency of reasoning models that generate longer sequences during inference before producing an answer, aka test-time scaling. Adjusting current design to compatible for decoding cases requires removing the sequence level pooling of Query to adhere to the token-by-toker generation fashion. Here is a modification of AttnGate design in (
Q. RoPE W gate reshape Quopee [..., d)).
(9a)
K. RoPE Wk gate concat mnax (Knope) P Pmin Teain(Knope (Koope); ), Pava (Knope)).
(9b)
S softmax Q K / /dgate).
(9c)
where, P mmxx, mmin. and Pawg stand for Max, Min and Average Pooling in sequence dimension, and I is the group size of GQA setting. d and dgate the hidden dimension of the original model and AttnGate for each head, respectively. S is the output score of each block from AttnGate. We aggregate Query heads within each group to share sparsity decisions improve the decoding efficiency. Specifically, a linear layer in the Q branch reduces each subgroup of queries (e.g.. 32 heads - heads for g=4) to single Qe head while keeping K heads unchanged, enabling shared sparsity among grouped queries. To compress K along the sequence dimension, we apply non- overlapping block-level pooling that concatenates Max, Min, and Average pooling outputs before a linear projection. Additionally, AttnGate reapplies RoPE on pre-RoPE Q and K, assigning each block the position of its first token, which we find improves the accuracy.
Evaluation on Reasoning Tasks We evaluate this design on three reasoning benchmarks: AIME24, MATH-500 (23] and EPQA-Diamon 44 using DeepSeck R1 Disttill-Qwen. 14B (2]. We compare against full attention and Quest [48) which is training-free sparse decoding method using query- aware KV cache selection. Both methods use a block size of 64 and apply sparsity to all layers for fair comparison. The maximum output length is fixed at 32, 26768 tokens across all settings. Accuracy is reported as average pass@1 over 64 (AIME24), (MATH-500), and 16 (GPQA) samples. For AttnGate distillation, we use OpenRI-MATH-222 [16] dataset with 800 steps and global batch size 16 on AMD MI300x GPUs, employing DeepSpeed ZeRO-2, AdamW W optimizer, and a le-3 learning rate with cosine decay.
Table 5: Performance comparison across different token budgets for ScerAttention-decodii and Quest using DeepSeek-R1 --it------ilii-ee------ model.
Method
Dataset
Token Budgets
Full
2k
4k
6k
8k
AIME24
55.78
66.35
67.50
66.82
67.50
SeerAttention-deecliing
MATH500
87.65
92.10
93.05
93.12
93.30
GPQA-Diamone
51.26
56.79
56.41
57.48
57.80
AIME24
25.83
46.67
53.75
60.00
67.50
Quest
MATH500
57.40
79.60
88.60
92.20
93.30
GPQA- 35.35
50.25
52.53
54.55
57.80
Table Table 5 shows the superior performance of our design compared to Quest across multiple reasoning benchmarks and token budgets. On AIME24, we consistently outperforms Quest, achieving 55.78 vs. 25.83 accuracy at 2k tokens and maintaining strong lead using more token budgets. Similarly, on MATH500 and GPQA-Diamond, SeerAttentiin-decodiin shows better budget-accuracy tradeoff compared with Quest, reflecting better information retention under sparse decodin owing to its learned gate that shares sparsity decisions, enabling more coherent block selection.
17

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: In abstract and introduction, provide clear description of motivation, methods and contribution of the paper.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We provide discussion of limitations the last section.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
18

---

Justification:
All the results are real experiment results without theoretical assumption.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: Training and testing details are listed in detail. The training data and testing benchmark are all public-available resources.
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
19

---

Answer:
[No]
Justification:
We did not provide code in the supplemental material.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.cm
cessiinnPoccyy for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/ cpipps.ccppullc//uuuii wwwiiccgguie///eeeeeee CoodeSubeiiieeee ssi wswonnolli onnolicy)))) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We have listed detailed hyperparameters of the training setting.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification:
The experiments results are mostly deterministic without an error bar.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, at least for the experiments that support main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
20

---

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of enrors is not verified.
For asymmetric distributions, the authors should be careful figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
show in tables or
error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We provide training GPU settings and typical GPU hours information.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics httpge///haurrgg.ccccc/aalclllaa
Answer: [Yes]
Justification: Our research does not involve human subjects, and all the training/testing dataset and models are commonly used open-source resources.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: This work is mainly about model efficiency. We believe the societal impact is mostly related to the model functionality itself instead of how fast it can run.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no socictal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
21

---

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [No]
Justification:
We use the common open- ppen-seureed models in our experiments.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: This paper properly credits and cites the creators or original owners of the assets used including code, data, and models.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the Datasets.papersstt paperswi thcode. cww/ datasets package should be provided. For popular datasets,
has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
22

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
We did provide code in submissions.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmoel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question:
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
23

---

Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Justification: This research does not involve LLMs any important, original, non- standard components
Guidelines:
The answer NA means that the method development in this research does involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy for what should or should be described.
24


Reviews:
[{"review_id": "hanYHV0Lf3", "reviewer": "Reviewer_DFJm", "summary": "The paper introduces SeerAttention—a post-training sparse attention method that augments the attention block with a learnable gating layer, which can identify block-sparsity patterns in attention maps. The gating layer introduced in the paper adds minimal overhead. It does not require from-scratch training, and the authors train it using a self-distillation technique, employing a KL-divergence loss with the ground-truth attention sparsity. They also augment the flash-attention kernel with their pooling scheme to ensure efficient attention during training of the gating layers. Following a block-sparsity pattern, the authors ensure compatibility with the FlashAttention kernel, making SeerAttention practically useful and demonstrating speedup over FA over large sequence lengths. The authors show extensive evaluation over several models and benchmarks and compare against various baselines. Their results show promising results with SeerAttention outperforming SOTA sparsity methods while being more efficient.", "questions": "- Can the authors show the results for LongBench and RULER with different sparsity levels for SeerAttention?\n- In the Short-Context evaluation, MMLU and ARC-c have very low average sparsities - 3.4 and 26. This is not the case for other short-context benchmarks. Have the authors analysed why that is the case? Is it possible that, depending on the dataset used to train the gate layers, it may not detect sparsity in certain downstream tasks?\n- What is the memory overhead of gate parameters added to the model?\n- SeerAttention applies block PE to the non-RoPE Q and K. As far as I know, the keys in KV-Cache are stored after RoPE application. How does SeerAttention handle this?", "limitations": "Limited discussion. While the authors do mention future work in their conclusion with potential for improvements to SeerAttention, having a dedicated section on Limitations would be helpful. There are several limitations which can be discussed: applicability to larger models due to post-training requirements, applicability to the decode stage in inference, and memory overhead due to added gating parameters.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 4, "strengths_and_weaknesses": "## Strengths\n- The information in the paper is very well presented. The authors describe their evaluation setup in detail and explain the choices made in their design and training of SeerAttention fairly well.\n- SeerAttention is compatible with FlashAttention, and shows practical speedup over the SOTA attention method for long context inference. This makes it deployable in practical settings.\n- The evaluation is extensive, with SeerAttention showing minimal drop in accuracy over multiple models and benchmarks. The attention maps identified by the gate layer also look promising.\n\n## Weaknesses\n- The choice of threshold in the evaluations is such that the sparsity levels for SeerAttention for both RULER and LongBench hover around 0.5-0.6. These are fairly low sparsity regimes, and attention maps have much higher sparsity. It would be great if the authors could show benchmark accuracy results for a wider range of sparsity with SeerAttention\n- The authors mention show ~7x speedup using SeerAttention, but that occurs at 90% sparsity, and there is no information in the paper to understand the accuracy trade-off with that kind of sparsity. At 50-60% sparsity, the speedups are more in the range of 1.5-1.6, which can be considered good.\n- While the training scheme for the gate layers developed by the authors is efficient and does not require full model retraining, it is still expensive. By the author’s own experiments, training the gate layers of a 7B model can take up to 40 A100 hours. For larger models like Llama 70B or 405B, this might be even more expensive. In this regime, this might be a big trade-off when compared to techniques that require no post-training.\n- SeerAttention only optimises the pre-fill stage and thus will not provide much benefit in decode-heavy workloads", "was_revised": true, "final_justification": "I maintain that SeerAttention is a novel, sound and practically relevant sparse attention technique that outperforms SOTA techniques like DuoAttention. The method is extensively evaluated, well presented and practically efficient. My major concerns were as follows:\n\n- Evaluation on wider sparsity values: The authors provide data in the rebuttal to show that SeerAttention performs well even in higher sparsity regimes. They also offer accuracy vs speed tradeoff evaluations for different sparsity levels.\n- Cost of training the gating layers: While this remains a limitation of the method, the authors have shown that in comparison to other SOTA methods like DuoAttention, this cost is the same or less for SeerAttention.\n- Applicability to Decode Settings: While the paper focuses on prefill-only acceleration, which is a completely valid and important setting, the authors also provide results on how their method can be modified to work with decode. This is impressive and enhances the paper's contribution.\n\nIn summary, all of my major concerns have been addressed and my questions answered. Therefore, I maintain my score of accept.", "rebuttal": "We thank the reviewer for their thoughtful feedback and for recognizing the novelty.\n\n### **1. Weaknesses 1 & 2: Sparsity Ratio and Speedup**\nWe evaluated speedup–accuracy trade‑offs under varying sparsity. The table below shows kernel‑level speedups for different sparsity and sequence lengths:\n\n| Sparsity / SeqLen | 8k   | 16k  | 32k  | 64k  | 128k |\n|-------------------|------|------|------|------|-------|\n| 50%               | 1.73x| 1.78x| 1.81x| 1.84x| 1.84x |\n| 70%               | 2.58x| 2.80x| 2.94x| 3.01x| 3.01x |\n| 90%               | 5.91x| 6.93x| 7.87x| 8.28x| 8.63x |\n\nIn the end-to-end, by adjusting threshold T, SeerAttention flexibly balances sparsity and accuracy. RULER‑128k accuracy results:\n\n| Method                           | Sparsity | Accuracy (%) |\n| -------------------------------- | -------- | ------------ |\n| DuoAttention 0.5| 0.50     | 75.32 |\n| DuoAttention 0.7| 0.70     | 70.35        |\n| DuoAttention 0.75| 0.75     | 55.48        |\n| SeerAttention $T=5\\times10^{-5}$ | 0.57  | 75.71  |\n| SeerAttention $T=1\\times10^{-4}$ | 0.62     | 75.15 |\n| SeerAttention $T=2\\times10^{-4}$ | 0.72     | 74.94|\n| SeerAttention $T=4\\times10^{-4}$ | 0.80     | 73.81  |\n| SeerAttention $T=5\\times10^{-4}$ | 0.83   | 73.37  |\n\n\n### **Tradeoffs between Speedup and Accuracy**\n\nTo evaluate the tradeoffs, we count the average sparsity and accuracy to present.In LongBench, SeerAttention employs a threshold of 2e-3 for all AttnGates. With the same threshold, different attention gates can exhibit varying sparsity ratios, and longer context data tends to be sparser.\n\n\n#### **LongBench performance & efficiency results on Llama-3.1-8B-Instruct**\n\n| Methods| 0-4k  | 4-8k  | 8k+   | Avg. Accuracy | Avg. Sparsity | Avg. Speedup |\n|----------|-------|-------|-------|-----------|---------------|---------|\n|  Baseline (Full Attention)  | 55.32 | 53.98 | 52.90 | 54.07| 0.00  | 1.00x    |\n| MInference| 55.23 | 53.78 | 52.18 | 53.73     | 0.31     | 0.72x|\n| MoA| 50.74 | 49.84 | 51.89 | 50.82     | 0.35| 1.11x|\n| DuoAttention    | 53.77 | 52.17 | 51.27 | 52.40     | 0.50*   | 1.07x|\n| SeerAttention | 55.43 | 54.49 | 52.69 | 54.20 | 0.50 | 1.32x |\n\n\\* 50% streaming heads, the real sparsity < 50%\n\nIn RULER, we employ a threshold of 5e-4 under 4k-128k test benches. \n\n#### **RULER performance & efficiency results on Llama-3.1-8B-Instruct**\n\n| Methods       | 4k    | 8k    | 16k   | 32k   | 64k   | 128k  | Avg. Accuracy | Avg. Sparsity | Avg. Speedup |\n|---------|-------|-------|-------|-------|-------|-------|--------|---------|-------------|\n|  Baseline (Full Attention)| 95.53 | 92.37 | 92.01 | 87.63 | 84.39 | 76.26 | 88.01    | 0.00             | 1.00x            |\n| MInference    | 95.53 | 92.64 | 91.37 | 85.71 | 83.24 | 67.02 | 85.92    | 0.36 | 0.83x    |\n| DuoAttention  | 95.64 | 92.08 | 90.71 | 84.75 | 83.24 | 75.32 | 86.96| 0.52    | 1.09x|\n| SeerAttention | 95.53 | 92.71 | 92.02 | 88.49 | 83.48 | 73.37 | 87.60| 0.55| 1.41x|\n\nThese results demonstrate that SeerAttention achieves the best overall accuracy–efficiency trade‑off, offering higher sparsity and greater speedup without degrading accuracy compared to prior methods.\n\n### **2. Weakness 3 & Question 3: Training Cost and Parameter Overhead**\nWe appreciate your suggestion to clarify the training cost and parameter overhead. In Llama-3-8B-Instruct model,\n\n- SeerAttention: ~40 A100 hours in total (**5 hours on 8×A100**)\n\n- DuoAttention: **“several hours on 8×A100”**\n- MInference: ~2 hours on 1×A100 for sparsity profiling/calibration but is less accurate and slower in inference\n\nThe training cost of SeerAttention is comparable to or lower than DuoAttention. Although MInference calibrates faster, it sacrifices inference accuracy and speed. SeerAttention introduces only a small number of trainable parameters. As shown in Fig. 9 (Appendix A.1), we design customized training kernels to distill the gate, and the memory/latency overhead during distillation is < 5% compared to native FlashAttention. For LLaMA‑3.1‑8B, the gate adds only 101 M parameters (~1.3% of the model), making the method lightweight and scalable. \n| Base Model                   | AttnGate Size |\n| ---------------------------- | ------------- |\n| LLaMA‑3.1‑8B‑Instruct| 101 MB  |\n| LLaMA‑3.1‑70B‑Instruct| 503 MB        |\n| Qwen2.5‑7B‑Instruct| 77 MB         |\n| Qwen2.5‑14B‑Instruct| 189 MB|\n| Qwen2.5‑32B‑Instruct| 252 MB        |\n| DeepSeek‑R1‑Distill‑Qwen‑7B  | 101 MB|\n| DeepSeek‑R1‑Distill‑Qwen‑14B | 189 MB        |\n| DeepSeek‑R1‑Distill‑Qwen‑32B | 252 MB        |\n\n\n### **3. Weakness 4: Decoding-Stage Inference:**\n\nWe appreciate the suggestion to discuss decode‑stage evaluation. In this work, we primarily focus on accelerating the prefill stage and leave decode‑stage optimization for future exploration. Prefill and decode are two distinct phases, and several PD‑disaggregation serving systems, such as DistServe [1] and Mooncake [2], have explored separating prefill and decode to improve TTFT/TPOT. Many prior works also focus on only one phase: for instance, MInference mainly targets prefill, while Quest [3] and H2O [4] focus solely on decode.\n\nOur approach can naturally extend to decode sparsity with a minor modification of the AttnGate design. Specifically, we remove the Q‑compression branch (pooling + linear) and instead distill using a 1D max‑pooling ground truth of the attention map (along the sequence dimension) from the base model. In other words, during distillation for decode, we directly use the original Q without sequence‑level compression, enabling each token’s Q to select important KV blocks at inference.\n\nWe evaluated this design on the DeepSeek‑R1‑Distill‑Qwen‑14B reasoning model using two challenging math tasks and compared it against Quest. Below we provide a brief summary of the results (block size = 64). A more complete version can be included in the appendix upon acceptance. The results show that SeerAttention achieves higher accuracy than Quest, even with larger KV blocks and sparser attention. \n\n#### **Model: DeepSeek-R1-Distill-Qwen14B**\n\n| Method (**AIME**)                                       | Acc   | Avg Sparsity | Avg Decode Length |\n|----------------------------|-------|--------------|-------------------|\n| Full Attention                                   | 70    | 0            | 10k               |\n| Quest (Block Size 16, Token Budget 8192)         | 53.33 | 36%          | 17k               |\n| Quest (Block Size 64, Token Budget 8192)         | 16.67 | 65%          | 28k               |\n| SeerAttention (Block Size 64, Threshold=5e-3)    | 66.7  | 86%          | 11k               |\n\n\n| Method (**MATH500**)                                          | Acc   | Avg Sparsity | Avg Decode Length |\n|--------------------------------------------------|-------|--------------|-------------------|\n| Full Attention                                   | 94    | 0            | 4k                |\n| Quest (Block Size 16, Token Budget 2048)         | 83.4  | 73%          | 6k|\n| SeerAttention (Block Size 64, Threshold=7e-3)    | 94    | 85%          | 4k|\n| SeerAttention (Block Size 64, Threshold=1e-2)    | 92.8  | 89%          | 4k|\n\n\n### **4. Question 2: Short Context:**\nWe appreciate the suggestion to clarify why certain short‑context tasks (e.g., MMLU, ARC‑c) exhibit lower sparsity.\nFirst, as the input sequence length decreases, the attention map — Softmax(QKᵀ) — naturally becomes denser: shorter sequences lead to more concentrated information, and the softmax distribution assigns higher average attention scores to each token. This behavior is inherent to attention computation itself and not specific to our gate predictor design.\nIn SeerAttention, we can freely control sparsity by selecting different thresholds or Top‑K ratios. For a fixed threshold, the resulting sparsity naturally decreases for short contexts and increases for long contexts because the gating mechanism’s softmax rows sum to 1, so tokens in shorter sequences inherently receive larger attention scores.\nMoreover, attention computation is rarely the bottleneck for short contexts, whereas sparse attention provides significant efficiency gains for long‑context inference. Attention only becomes a major bottleneck when the sequence is long, where sparsity leads to meaningful speedups.\nWe thank the reviewer for pointing this out — we have reorganized Table 3 accordingly. As sequence length decreases, the observed sparsity drops (e.g., 3.4% and 26%), which directly reflects this inherent behavior.\n\n \n|       | MMLU | ARC-c | HellaS. | GSM-8K |\n|-------|------|-------|---------|-----|\n| Full Attention| 68.1 | 60.7  | 80.1| 75.7   |\n| SeerAttention| 67.9 | 60.2  | 79.8| 75.6   |\n| Avg Sparsity| 3.4  | 26| 50.4    | 52.1   |\n| Avg SeqLens| 118  | 395   | 840| 872    |\n\n\n### **5. Question 4: Rope in KV cache**\nThe rope design in decoding version is similar to prefill case. In decoding, SeerAttention maintains a small compressed K cache (after RoPE) specifically for gating. This avoids loading the full K cache at every step and allows selective V-loading, maximizing decode-stage speedup. Without this, sparsity-based selection would be limited to a **~2× upper-bound** speedup due to full K-cache loading.\n\nOnce again, we sincerely thank the reviewer for their positive assessment of SeerAttention’s **novelty, technical soundness, and practical impact**. We will incorporate your suggestions to further strengthen the final version of the paper.\n\n[1] DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving, https://www.usenix.org/system/files/osdi24-zhong-yinmin.pdf\n\n[2] Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving, https://arxiv.org/abs/2407.00079\n\n[3] Quest: Query-aware sparsity for efficient long-context llm inference, https://arxiv.org/abs/2406.10774\n\n[4] H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models, https://arxiv.org/abs/2306.14048"}, {"review_id": "GWoGW7kPM3", "reviewer": "Reviewer_UEJ9", "summary": "This paper introduces SeerAttention, a post-training attention mechanism aimed at improving the efficiency of long-context inference in LLMs by learning sparse attention patterns through a self-distilled gating mechanism named AttnGate. Inspired by the gating mechanism in Mixture of Experts (MoE), SeerAttention adds a learnable gate to each attention layer that identifies salient Q-K blocks via pooled Q/K representations. The gating module is trained via KL-divergence to mimic block-level 2D max-pooled full attention outputs, using a lightweight self-distillation setup that freezes the base model. At inference time, a threshold or TopK is applied to the gating outputs to control sparsity. \n\nEmpirically, it demonstrates favorable accuracy-speed trade-offs on long-context tasks like LongBench and RULER, with significant speedups (up to 7.3× at 128k sequence length) and minimal accuracy degradation even at high sparsity than existing sparsity based prefilling techniques, such as MInference and DuoAttention.", "questions": "1. Comparison to DuoAttention/MInference in Training Cost: How does the training cost of SeerAttention (in GPU hours) compare to  DuoAttention/MInference (lightweight heuristics)? If significantly higher, can the authors provide a cost-benefit analysis?\n\n2. Can ground-truth sparsity (gt) be used directly? In Eq. (2), AttnGate is trained to approximate gt. Why not directly use gt to select top-k blocks (as an accuracy upper bound)? How much accuracy is lost compared to using gt?\n\n3. Is pooling method model-dependent? In section 3.1 and Fig. 2, different pooling strategies yield different performance. Are these consistent across model architectures (e.g., GQA vs MQA heads)? Does the best pooling generalize?\n\n4. Can the authors list and compare tunable parameters required by SeerAttention vs DuoAttention?\n\n5. In Table 1, it is better to show the individual sparsity under different length. For difference context lengths, the sparsity would be difference.", "limitations": "1. Training Resource Requirement: Although only the AttnGate is trained, the cost would be higher than heuristic methods. This might limit use in low-resource or deployment-only settings.\n\n2. The paper focuses exclusively on the pre-filling stage and does not discuss how SeerAttention could be extended to the decoding phase. If the proposed gating mechanism or block-sparse attention kernel is not easily compatible with decoding, this could significantly limit the method’s practical applicability, especially in real-time LLM deployments where decoding dominates inference latency.\n\n3. Lack of memory consumption analysis, especially for long context inference.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n\t1. The proposed method is well-justified and thoughtfully engineered, with a solid motivation tied to dynamic sparsity in attention. The use of self-distillation from full attention as a ground truth is conceptually sound and computationally efficient due to their custom max-pooling kernel.\n\t2. The work targets a critical bottleneck—long-context inference latency—and achieves consistent improvements across multiple model sizes and benchmarks.\n\t3. The paper is mostly well-written and thorough.\n\nWeaknesses:\n      1. Comparative Cost: The training cost s non-trivial compared to heuristic methods like DuoAttention or MInference. A more detailed comparison of training overhead vs performance gain is missing.\n      2. Lack of training data efficiency analysis, the paper relies on the RedPajama dataset for self-distillation but provides no justification for this choice or analysis of how the choice of training data impacts the learned AttnGate. Especially comparing to MInference and DuoAttention. This is important in practical applications.\n      3. Dynamic sparsity introduces challenging in batch inference.", "was_revised": true, "final_justification": "The rebuttal addresses most of my concerns—particularly regarding training cost and training data comparisons with DuoAttention and MInference. These are critical factors for real-world applicability. I also appreciate the additional experimental results on decoding, though I note that the benchmarks used do not focus on long-context input tasks. The proposed method appears to achieve higher sparsity compared to existing sparse techniques, which is promising.", "rebuttal": "We thank the reviewer for recognizing that our method is well-justified and thoughtfully designed, especially the motivation around dynamic sparsity in attention, the use of self-distillation with our efficient max-pooling kernel from **both system and algorithm prospective**. This aligns well with our goal to address the long-context prefilling challenge.\n\n### **1. Training cost vs accuracy gain**\n\nWe appreciate your suggestion to clarify the training cost and parameter overhead. In Llama-3-8B-Instruct model,\n\n- SeerAttention: ~40 A100 hours in total (**5 hours on 8×A100**)\n\n- DuoAttention: **“several hours on 8×A100”**\n- MInference: ~2 hours on 1×A100 for sparsity profiling/calibration but is less accurate and slower in inference\n\nSeerAttention requires a training budget comparable to or lower than DuoAttention. Both methods train only the gating module without updating the base model weights, so the cost mainly comes from forward passes. Compared to post-training approaches such as SFT or RL finetuning, this cost is negligible.\n\nFor accuracy, the gain should be considered together with sparsity. DuoAttention has a fixed sparsity ratio (~50%), whereas SeerAttention achieves similar accuracy at **much higher sparsity (80–90% on 128k RULER)**. MInference is faster to profile since it only calibrates a sparsity head on a given dataset, but it is less accurate. Moreover, MInference often **suffers from runtime overhead** caused by head-by-head index search and execution flow, as reported in GitHub issues (e.g., https://github.com/microsoft/MInference/issues/43), where users observed that MInference can even run slower than the dense baseline.\n\nAs shown in Fig. 9 (Appendix A.1), we design customized training kernels to distill the gate, and the memory/latency overhead during distillation is < 5% compared to native FlashAttention. For LLaMA‑3.1‑8B, the gate adds only 101 M parameters (~1.3% of the model), making the method lightweight and scalable. \n\n| Base Model| AttnGate Size |\n| ---------- | ------ |\n| LLaMA‑3.1‑8B‑Instruct| 101 MB  |\n| LLaMA‑3.1‑70B‑Instruct| 503 MB|\n| Qwen2.5‑7B‑Instruct| 77 MB|\n| Qwen2.5‑14B‑Instruct| 189 MB|\n| Qwen2.5‑32B‑Instruct| 252 MB|\n| DeepSeek‑R1‑Distill‑Qwen‑7B  | 101 MB|\n| DeepSeek‑R1‑Distill‑Qwen‑14B | 189 MB|\n| DeepSeek‑R1‑Distill‑Qwen‑32B | 252 MB|\n\n#### **About training data:** \nThank you for this question. Selecting training data is less critical in our case as the gate only learns the **distribution instead of the real end-to-end knowledge** from the data. Regarding training data, our gate learns to approximate **distribution patterns rather than full knowledge**, so dataset choice is less critical. We use RedPajama [1] for its popularity in long-context training. MInference and DuoAttention rely on downstream or BookSum [2] datasets, and none of these works provide ablation on dataset choice. This remains an open question but does not undermine our method’s core merit.\n\n### **2. Use GT directly in inference**\nWe appreciate the reviewer’s interest in this question, which was a key consideration in our system–algorithm co-design. Our experiments show that directly using GT to select sparse blocks results in virtually no accuracy loss, even under high sparsity. However, computing GT requires evaluating the full quadratic softmax term, $\\text{softmax}(QK^\\top)$. If this computation is already performed, there is little performance benefit from only applying sparsity to the $ \\text{AttentionScore} \\cdot V $ part.\n\nIn summary, GT is accurate but expensive, which motivates the need for a fast, low-cost AttnGate.\n\nRegarding the question “How much accuracy is lost compared to using GT?”, intuitively:\n\n- Full attention provides the upper bound for GT-based sparsity.\n\n- GT-based sparsity provides the upper bound for our AttnGate-based sparsity.\n\nAs noted above, GT computation is even more expensive than full attention because we must offload all attention scores to CPU to avoid out-of-memory issues before using them as GT. Therefore, we use full attention as the baseline when comparing accuracy across different sparsity settings (see Tables 1–3 in the paper and additional results in the rebuttal to another reviewer: https://openreview.net/forum?id=Nf8yfPDFTl&noteId=yPnYUXTMey).\n\n### **3. Pooling selection**\nOur pooling experiments are consistent across all tested GQA-based models. The best pooling strategy generalizes well within this setting. However, pooling selection remains an open question, especially for other attention architectures like MQA/MLA. Our main contribution is introducing the **gated attention concept**, predating NSA [3] and MoBA [4] (another reviewer asked). Future work may explore alternative gate structures such as strided MLPs or convolutions, inspired by MoE router literature.\n\n### **4. Tunable parameters for each layer**\nWe would like to clarify to the reviewer that by “tunable parameters” we consider two categories: (1) trainable parameters and (2) hyperparameters. Below, we list and compare these parameters for SeerAttention and DuoAttention accordingly.\n\n#### **For trainable parameters**\n\n- DuoAttention: a scale vector multiplied on streaming head and dense head.\n- SeerAttention: a linear layer from compressed QK to gate hidden vector.\n\nIf we compare the number of trainable parameters, DuoAttention indeed has fewer parameters. However, even for SeerAttention, the additional parameters are minimal—less than 1% of the total. For instance, in LLaMA-3.1-8B, the gating module introduces roughly 101M parameters, which is about 101M / 8B ≈ 1.3% of the model size. This overhead is very small and keeps the method efficient to train. As shown in Figure 9 of Appendix A.1, we design customized training kernels to distill the gate, and the memory and latency overhead during this distillation stage are minimal compared to native FlashAttention.\n\n| Base Model| AttnGate Size |\n| ------| ------ |\n| LLaMA‑3.1‑8B‑Instruct| 101 MB|\n| LLaMA‑3.1‑70B‑Instruct| 503 MB|\n| Qwen2.5‑7B‑Instruct| 77 MB|\n| Qwen2.5‑14B‑Instruct| 189 MB|\n| Qwen2.5‑32B‑Instruct| 252 MB|\n| DeepSeek‑R1‑Distill‑Qwen‑7B  | 101 MB|\n| DeepSeek‑R1‑Distill‑Qwen‑14B | 189 MB|\n| DeepSeek‑R1‑Distill‑Qwen‑32B | 252 MB|\n\n#### **For hyperparameters**\n\nSeerAttention introduces an additional hyperparameter, block size, as well as the number of training steps, in addition to the usual hyperparameters required for model fine-tuning. DuoAttention, on the other hand, requires selecting the percentage of streaming heads and full-attention heads, along with the number of training steps and other standard fine-tuning hyperparameters.\n\n\n### **5. Modify results organization**\nThank you for the suggestion since we just present the overall sparsity for different sequence length inside the sentences. We will revise the presentation if accepted.\n\n### **6. About decoding**\nWe appreciate the suggestion to discuss decode‑stage evaluation. In this work, we primarily focus on accelerating the prefill stage and leave decode‑stage optimization for future exploration. Prefill and decode are two distinct phases, and several PD‑disaggregation serving systems, such as DistServe [5] and Mooncake [6], have explored separating prefill and decode to improve TTFT/TPOT. Many prior works also focus on only one phase: for instance, MInference mainly targets prefill, while Quest [7] and H2O [8] focus solely on decode.\n\nOur approach can naturally extend to decode sparsity with a minor modification of the AttnGate design. Specifically, we remove the Q‑compression branch (pooling + linear) and instead distill using a 1D max‑pooling ground truth of the attention map (along the sequence dimension) from the base model. In other words, during distillation for decode, we directly use the original Q without sequence‑level compression, enabling each token’s Q to select important KV blocks at inference.\n\nWe evaluated this design on the DeepSeek‑R1‑Distill‑Qwen‑14B reasoning model and compared it against Quest. Below we provide a brief summary of the results (we choose block size = 64 for K block). A more complete version can be included in the appendix upon acceptance. \n\n| | Dataset | 2k| 4k | 6k | 8k | Full |\n| --- | --- | --- | --- | --- | --- | --- |\n| SeerAttention-R  | AIME24 | 55.78 | 66.35 | 67.50 | 66.82 | 67.50 |\n|  | MATH500 | 87.65 | 92.10 | 93.05 | 93.12 | 93.30 |\n|  | GPQA-Diamond | 51.26 | 56.79 | 56.41 | 57.48 | 57.80 |\n| Quest  | AIME24 | 25.83 | 46.67 | 53.75 | 60.00 | 67.50 |\n|  | MATH500 | 57.4 | 79.6 | 88.6 | 92.2 | 93.30 |\n|  | GPQA-Diamond | 35.35 | 50.25 | 52.53 | 54.55 | 57.80 |\n\n### **7.About batch inference**\n\nWe appreciate the reviewer’s interest in this question about batching inference. In fact, unlike MoE, batch inference is less challenging for attention. During prefill, **variable-length** inputs can be packed along the sequence dimension, hiding batch size within sequence length. For decoding, modern infrastructures (like vllm or sglang) support **paged attention**, allowing our block-sparse method to selectively activate pages naturally.\n\nReference:\n\n[1] RedPajama, https://github.com/togethercomputer/RedPajama-Data\n\n[2] BookSum: A Collection of Datasets for Long-form Narrative Summarization, https://arxiv.org/abs/2105.08209\n\n[3] Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attention, https://arxiv.org/abs/2502.11089\n\n[4] MoBA: Mixture of Block Attention for Long-Context LLMs, https://arxiv.org/abs/2502.13189\n\n[5] DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving, https://www.usenix.org/system/files/osdi24-zhong-yinmin.pdf\n\n[6] Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving, https://arxiv.org/abs/2407.00079\n\n[7] Quest: Query-aware sparsity for efficient long-context llm inference, https://arxiv.org/abs/2406.10774\n\n[8] H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models, https://arxiv.org/abs/2306.14048"}, {"review_id": "wNUnobKgnS", "reviewer": "Reviewer_ebQ8", "summary": "The paper proposes a dynamic block sparse attention approach to accelerate the prefilling stage of large language models (LLMs). Specifically, it pools query (Q) and key (K) matrices into blocks along the sequence dimension, projecting them via learnable linear layers, and computes inner products to predict blocks with high attention values. Only the blocks estimated to have high attention values are computed explicitly. The authors also propose a corresponding training scheme using KL-divergence to align dense and sparse attention masks.", "questions": "1. Could you provide additional results comparing sparse prefill with dense decoding across MoA, DuoAttention, and SeerAttention to ensure fairness and isolate the effect of sparse prefill?\n2. Could you include a comprehensive figure or table illustrating the performance-efficiency trade-offs (e.g., benchmark scores versus runtime) of various sparse attention methods **across different density levels**? This would effectively demonstrate SeerAttention’s relative strengths across conditions.", "limitations": "Yes. The authors transparently acknowledge the limitation of exclusively optimizing prefilling rather than decoding. This focus is acceptable, given future work could explore efficient decoding strategies.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 3, "originality": 2, "strengths_and_weaknesses": "**Strengths**\n\n1. **Effective design:** The method is straightforward and effective. Experimental results demonstrate that SeerAttention outperforms state-of-the-art sparse attention methods across the LongBench and RULER benchmarks.\n2. **Practical speedup:** The proposed custom kernels successfully translate theoretical improvements into practical wall-clock speedups. At 80% sparsity, SeerAttention achieves a 1.41x prefilling speedup across sequence lengths of 4K-128K in the RULER benchmark, showing superior scalability with increasing sparsity.\n\n**Weakness**\n\n1. **Fairness of comparisons:** SeerAttention exclusively uses sparse attention in the prefilling phase, whereas baseline methods (MoA and DuoAttention) apply sparse attention during both prefill and decoding. To ensure a fair comparison, the authors should either:\n   - Evaluate all methods with sparse prefill and dense decoding, isolating the impact of sparse decoding.\n   - Alternatively, discuss the efficiency-performance trade-offs more comprehensively across varying input-output lengths to highlight differences due to sparse decoding.\n\n2. **Clarity of illustrations and explanations:** The presentation can be improved for clarity:\n   - It is unclear whether the parameters $W_q$ and $W_k$ in Equation 1 are newly introduced learnable parameters distinct from the original query/key projection layers. Also it probably is, the author is advised to explicitly clarify that in the paper.\n   - Lines 80-81 conflate the concepts of \"dynamic\" (varying by input) and \"heterogeneous\" (varying across heads and layers).\n   - The specific method used for generating attention masks during evaluation (top-k or threshold-based) is not explicitly stated.\n3. **Novelty relative to prior work:** The relationship to previous works such as MInference, which also uses block-sparse attention with pooling-based block estimation, should be clarified in greater detail. Clearly articulating the differences from existing methods would strengthen the paper’s novelty claims.", "was_revised": true, "final_justification": "SeerAttention delivers speed and accuracy gains under prefill-only and decoding-only sparse attention setups. Although a fully unified comparison remains lacking, I see positive signs towards this end during the rebuttal phase, thus raising my score to borderline accept. If the paper gets accepted, I strongly encourage the authors to complete the promised fair-comparison study and detail the novelty discussion in the final version.", "rebuttal": "We sincerely thank the reviewer for your thoughtful and constructive feedback. We are glad that the reviewer recognized the **effectiveness, practical speedup, and scalability** of SeerAttention. The comment have helped us identify ways to improve the clarity and fairness of our presentation, and we have carefully addressed each point below.\n\n### **1. Comparisons setting**\nThank you for pointing out this issue. In our experiments, we directly used the original open-source implementations of MoA and DuoAttention. We agree that a more controlled comparison (using dense decoding for all methods) would be fairer. We are re-implement MoA and DuoAttention frameworks and doing the following the experiments settings: variable input length (prefilling) + variable output length (decoding) to fairly compare. Due to the rebuttal time limit, we could not re-implement and rerun all experiments with modified decoding settings.\n\n\n### **2. SeerAttention Decoding**\n\nWe appreciate the suggestion to discuss decode‑stage evaluation. In this work, we primarily focus on accelerating the prefill stage and leave decode‑stage optimization for future exploration. Prefill and decode are two distinct phases, and several PD‑disaggregation serving systems, such as DistServe [1] and Mooncake [2], have explored separating prefill and decode to improve TTFT/TPOT. Many prior works also focus on only one phase: for instance, MInference mainly targets prefill, while Quest [3] and H2O [4] focus solely on decode.\n\nOur approach can naturally extend to decode sparsity with a minor modification of the AttnGate design. Specifically, we remove the Q‑compression branch (pooling + linear) and instead distill using a 1D max‑pooling ground truth of the attention map (along the sequence dimension) from the base model. In other words, during distillation for decode, we directly use the original Q without sequence‑level compression, enabling each token’s Q to select important KV blocks at inference.\n\nWe evaluated this design on the DeepSeek‑R1‑Distill‑Qwen‑14B reasoning model using two challenging math tasks and compared it against Quest. Below we provide a brief summary of the results (block size = 64). A more complete version can be included in the appendix upon acceptance. The results show that SeerAttention achieves higher accuracy than Quest, even with larger KV blocks and sparser attention. \n\n#### **Model: DeepSeek-R1-Distill-Qwen14B**\n\n| Method (**AIME**)                                       | Acc   | Avg Sparsity | Avg Decode Length |\n|----------------------------|-------|--------------|-------------------|\n| Full Attention                                   | 70    | 0            | 10k               |\n| Quest (Block Size 16, Token Budget 8192)         | 53.33 | 36%          | 17k               |\n| Quest (Block Size 64, Token Budget 8192)         | 16.67 | 65%          | 28k               |\n| SeerAttention (Block Size 64, Threshold=5e-3)    | 66.7  | 86%          | 11k               |\n\n\n| Method (**MATH500**)                                          | Acc   | Avg Sparsity | Avg Decode Length |\n|--------------------------------------------------|-------|--------------|-------------------|\n| Full Attention                                   | 94    | 0            | 4k                |\n| Quest (Block Size 16, Token Budget 2048)         | 83.4  | 73%          | 6k|\n| SeerAttention (Block Size 64, Threshold=7e-3)    | 94    | 85%          | 4k|\n| SeerAttention (Block Size 64, Threshold=1e-2)    | 92.8  | 89%          | 4k|\n\n\n### **3. Clarity of illustrations and explanations**\n#### **3.1** Thank you for the suggestion. We will update the notation to use different superscripts to avoid confusion with the original model weights. The gate weights, $W_q^{\\text{gate}}$ and $W_k^{\\text{gate}}$, are newly introduced learnable parameters, and only the gate is trained during self-distillation. For LLaMA‑3.1‑8B, the gate contains 101M parameters (~1.3% of the total model parameters).\n\n\n#### **3.2** We appreciate the comment on the wording (dynamic vs. heterogeneous) and will revise it accordingly.\n\n\n#### **3.3** For mask generation, we illustrate the process using an example within a single head of one layer. Suppose the sequence length is $S$ and the block size is $B$. The output of AttnGate in this head is $o = [S/B, S/B]$.\n\n\n- For the top‑$k$ method, we apply a torch.topk operation on each row, obtaining indices with shape $[S/B, k]$.\n\n\n- For the threshold‑based method, we compute a binary mask as $mask = o > \\text{threshold}$. Blocks marked as True in this mask are directly activated in our kernel.\n\n\n### **4. Novelty relative to prior work**\n\nBlock-sparse attention indeed dates back to much earlier works such as BigBird [5] and BlockSparse-BERT [6], and MInference is also a more recent approach that adopts block-sparse attention with pooling-based block estimation. However, our method is fundamentally different: pooling-based selection itself is not the core contribution here. The key novelty lies in introducing **a learnable gating mechanism for sparse attention**, similar to **how a trainable router in MoE outperforms static pruning or heuristic compression methods**. Our approach enables dynamic and data-dependent sparsity decisions guided by full-attention supervision, which sets it apart from heuristic or static approaches like MInference. Also, there are some following works like NSA [7] and MoBA [8] adopt this approach to pre-training stage.\n\n\n### **5. Question 2: performance-efficiency trade-offs**\nThank you for this helpful suggestions. We agree that our presentation of the performance–efficiency trade-off could be improved. While the paper already contains the necessary data (e.g., accuracy in **Table 2** and speedup in **Figure 6**), we did not present them together in a unified figure or table, which may have reduced clarity.\n\nTo better illustrate this trade-off, we provide additional results on the 128k RULER benchmark below, showing how accuracy varies with sparsity under different thresholds (T = threshold):\n\n| Method| Sparsity | Accuracy (%) |\n| ---------------- | -------- | ------------ |\n| DuoAttention 0.5| 0.50     | 75.32 |\n| DuoAttention 0.7| 0.70     | 70.35|\n| DuoAttention 0.75| 0.75     | 55.48|\n| SeerAttention $T=5\\times10^{-5}$ | 0.57  | 75.71  |\n| SeerAttention $T=1\\times10^{-4}$ | 0.62| 75.15 |\n| SeerAttention $T=2\\times10^{-4}$ | 0.72| 74.94|\n| SeerAttention $T=4\\times10^{-4}$ | 0.80| 73.81  |\n| SeerAttention $T=5\\times10^{-4}$ | 0.83| 73.37  |\n\nFor the kernel-level efficiency, we measured different sequence length latency under different sparsity setting:\n\n| Sparsity / SeqLen | 8k   | 16k  | 32k  | 64k  | 128k |\n|-------------------|------|------|------|------|-------|\n| 50%               | 1.73x| 1.78x| 1.81x| 1.84x| 1.84x |\n| 70%               | 2.58x| 2.80x| 2.94x| 3.01x| 3.01x |\n| 90%               | 5.91x| 6.93x| 7.87x| 8.28x| 8.63x |\n\nFor tradeoffs between speedup and accuracy, we employ a threshold of 5e-4 for all AttnGates under 4k-128k test benches and count the average sparsity and accuracy to present RULER performance & efficiency results on Llama-3.1-8B-Instruct.\n\n| Methods       | 4k    | 8k    | 16k   | 32k   | 64k   | 128k  | Avg. Accuracy | Avg. Sparsity | Avg. Speedup |\n|---------|-------|-------|-------|-------|-------|-------|--------|---------|-------------|\n|  Baseline (Full Attention)| 95.53 | 92.37 | 92.01 | 87.63 | 84.39 | 76.26 | 88.01    | 0.00             | 1.00x            |\n| MInference    | 95.53 | 92.64 | 91.37 | 85.71 | 83.24 | 67.02 | 85.92    | 0.36 | 0.83x    |\n| DuoAttention  | 95.64 | 92.08 | 90.71 | 84.75 | 83.24 | 75.32 | 86.96| 0.52    | 1.09x|\n| SeerAttention | 95.53 | 92.71 | 92.02 | 88.49 | 83.48 | 73.37 | 87.60| 0.55| 1.41x|\n\nThese results demonstrate that SeerAttention achieves the best overall accuracy–efficiency trade‑off, offering higher sparsity and greater speedup without degrading accuracy compared to prior methods.\n\nReference:\n\n[1] DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving, https://www.usenix.org/system/files/osdi24-zhong-yinmin.pdf\n\n[2] Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving, https://arxiv.org/abs/2407.00079\n\n[3] Quest: Query-aware sparsity for efficient long-context llm inference, https://arxiv.org/abs/2406.10774\n\n[4] H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models, https://arxiv.org/abs/2306.14048\n\n[5] Big Bird: Transformers for Longer Sequences, https://arxiv.org/abs/2007.14062\n\n[6] Blockwise Self-Attention for Long Document Understanding, https://arxiv.org/abs/1911.02972\n\n[7] Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attention, https://arxiv.org/abs/2502.11089\n\n[8] MoBA: Mixture of Block Attention for Long-Context LLMs, https://arxiv.org/abs/2502.13189"}, {"review_id": "GfVUdGlqnt", "reviewer": "Reviewer_HW7W", "summary": "This paper introduces SeerAttention, which employs a learnable method to achieve sparsity of attention computations. The paper also introduces self-distillation and kernels to achieve efficient training and inference. Experiments show that the method can achieve better efficiency while preserving the performance.", "questions": "Seeing Weaknesses", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths: \n* This paper introduces a new sparse attention, which can be obtained through distillation from the full attention. \n* The authors design kernels to achieve efficiency. \n* Experiments show the performance and efficiency of this method. \n\nWeaknesses:\n* **Lack of differences from other methods**: from the perspective of novelty, the core idea is similar to NSA and MoBA. In addition, I believe these attention variants can also be integrated into existing models via self-distillation rather than training them from scratch, as the authors proposed.\n* **Unfair comparison**: Compared with baselines, the trainable parameters are larger, and knowledge distillation is also employed in SeerAttention. Thus, I think the comparisons are not fair. I suggest that the authors can train the models with baselines using similar distillation methods.\n* **Lack of ablation studies**:\n    * w/o self-distillation\n    * Different settings of sparsity and block size.", "was_revised": true, "final_justification": "The authors respond to most of my questions and weaknesses.", "rebuttal": "We sincerely thank the reviewer for the constructive feedback and valuable suggestions. We address the main concerns below:\n\n### **1. Lack of differences from other method**\n\nWe respectfully disagree with the assessment that our method lacks novelty. SeerAttention introduces learnable sparse attention specifically for post-training, requiring only a small number of additional gate parameters that are distilled during this stage. In contrast, NSA and MoBA adopt a similar idea but focus on the pre-training phase. As shown in the comparison of trainable parameters and training tokens, NSA and MoBA require training all model weights with a large number of tokens and extensive GPU hours. Our approach, by contrast, is a lightweight and cost-effective self-distillation method.\n\n| Base Model                   | SeerAttention (trainable parameters)  |Training Tokens|\n| ---------------------------- | ------------- |------------- |\n| LLaMA‑3.1‑8B‑Instruct| 101 MB  | 0.5B|\n| LLaMA‑3.1‑70B‑Instruct| 503 MB        |0.5B|\n| Qwen2.5‑7B‑Instruct| 77 MB         |0.5B|\n| Qwen2.5‑14B‑Instruct| 189 MB|0.5B|\n| Qwen2.5‑32B‑Instruct| 252 MB        |0.5B|\n| DeepSeek‑R1‑Distill‑Qwen‑7B  | 101 MB|0.5B|\n| DeepSeek‑R1‑Distill‑Qwen‑14B | 189 MB        |0.5B|\n| DeepSeek‑R1‑Distill‑Qwen‑32B | 252 MB        |0.5B|\n\n| Base Model                   | MoBA (trainable parameters)  |Training Tokens|\n| ---------------------------- | ------------- |------------- |\n|  568M | 568M (whole)  | 10.8B |\n|  822M | 822M (whole)  |  15.3B |\n|  1.5B | 1.5B (whole)  |  27.4B |\n|  Llama-8B-1M| 16 GB (whole)  | 36.9B|\n\n| Base Model                   | NSA (trainable parameters)  |Training Tokens|\n| ---------------------------- | ------------- |------------- |\n|  7B-parameter transformer  | 7B (whole)  | 260B |\n\nWe have discussed this point with another reviewer, as seen in this comment (https://openreview.net/forum?id=Nf8yfPDFTl&noteId=LV1xhwZYfg). In fact, both of the two works have recognized acknowledged the novelty of SeerAttention. (ACs could help verify this if needed.) Therefore, we believe our work is indeed novel and offers a new direction for developing sparse attention methods.\n\n### **2. Fairness of comparisons (trainable parameters and self-distillation)**\n\nWe would like to clarify that the “self‑distillation” in our method is different from conventional knowledge distillation as understood in the review. Our self‑distillation **only trains the newly introduced gate parameters, while keeping the original model weights entirely frozen**. The additional parameters are **minimal**—for example, in LLaMA‑3.1‑8B, the gate introduces approximately **101M** parameters (about **101M/8B = 1.3%** of the original model size); see also our response to Reviewer DFJm (https://openreview.net/forum?id=Nf8yfPDFTl&noteId=yPnYUXTMey).\n\nAs illustrated in Figure 1, this distillation process simply guides the gate to mimic the attention pattern of the frozen full‑attention model, here we call it \"self‑distillation\". Hence, no additional knowledge distillation is applied to the backbone model itself. For this reason, it is not meaningful to ***“train the baselines with similar distillation methods,”*** since our approach does not perform distillation on the original model’s weights.\n\n### **3. Different settings of sparsity and block size.**\n\nWe appreciate the suggestion to include experiments with different sparsity and block size settings. \n\nFor **sparsity**, SeerAttention allows runtime adjustment via the threshold T, providing flexibility to trade off efficiency and accuracy. The table below shows that as T increases, sparsity rises while accuracy decreases slightly, and SeerAttention consistently outperforms DuoAttention at comparable sparsity.\n\n| Method                           | Sparsity | Accuracy (%) |\n| -------------------------------- | -------- | ------------ |\n| DuoAttention 0.5                 | 0.50     | 75.32        |\n| DuoAttention 0.7                 | 0.70     | 70.35        |\n| DuoAttention 0.75                | 0.75     | 55.48        |\n| SeerAttention $T=5\\times10^{-5}$ | 0.57     | 75.71        |\n| SeerAttention $T=1\\times10^{-4}$ | 0.62     | 75.15        |\n| SeerAttention $T=2\\times10^{-4}$ | 0.72     | 74.94        |\n| SeerAttention $T=4\\times10^{-4}$ | 0.80     | 73.81        |\n| SeerAttention $T=5\\times10^{-4}$ | 0.83     | 73.37        |\n\n#### **LongBench performance & efficiency results on Llama-3.1-8B-Instruct**\n\n| Methods| 0-4k  | 4-8k  | 8k+   | Avg. Accuracy | Avg. Sparsity | Avg. Speedup |\n|----------|-------|-------|-------|-----------|---------------|---------|\n|  Baseline (Full Attention)  | 55.32 | 53.98 | 52.90 | 54.07| 0.00  | 1.00x    |\n| MInference| 55.23 | 53.78 | 52.18 | 53.73     | 0.31     | 0.72x|\n| MoA| 50.74 | 49.84 | 51.89 | 50.82     | 0.35| 1.11x|\n| DuoAttention    | 53.77 | 52.17 | 51.27 | 52.40     | 0.50*   | 1.07x|\n| SeerAttention | 55.43 | 54.49 | 52.69 | 54.20 | 0.50 | 1.32x |\n\n\\* 50% streaming heads, the real sparsity < 50%\n\nIn RULER, we employ a threshold of 5e-4 under 4k-128k test benches. \n\n#### **RULER performance & efficiency results on Llama-3.1-8B-Instruct**\n\n| Methods       | 4k    | 8k    | 16k   | 32k   | 64k   | 128k  | Avg. Accuracy | Avg. Sparsity | Avg. Speedup |\n|---------|-------|-------|-------|-------|-------|-------|--------|---------|-------------|\n|  Baseline (Full Attention)| 95.53 | 92.37 | 92.01 | 87.63 | 84.39 | 76.26 | 88.01    | 0.00             | 1.00x            |\n| MInference    | 95.53 | 92.64 | 91.37 | 85.71 | 83.24 | 67.02 | 85.92    | 0.36 | 0.83x    |\n| DuoAttention  | 95.64 | 92.08 | 90.71 | 84.75 | 83.24 | 75.32 | 86.96| 0.52    | 1.09x|\n| SeerAttention | 95.53 | 92.71 | 92.02 | 88.49 | 83.48 | 73.37 | 87.60| 0.55| 1.41x|\n\nFor **block size**, we appreciate the reviewer’s interest in this factor, which was one of the key considerations in our system and algorithm design. Intuitively, using a larger block size **reduces memory allocation overhead** because the block‑wise attention map has shape $[bs, nh, seq/block, seq/block]$. However, at the algorithmic level, a larger block size leads to more **coarse‑grained distillation**. We have already conducted ablation studies in this direction. As shown below, larger block sizes generally improve perplexity at longer sequences, with **block size = 64** providing the best overall trade‑off. Nevertheless, the optimal choice may depend on the downstream task, and we will explore additional experiments in the revision if this factor is deemed critical.\n\n| Block Size | 8k    | 16k   | 32k   | 64k   | 128k  |\n| ---------- | ----- | ----- | ----- | ----- | ----- |\n| 32*32         | 10.49 | 10.36 | 10.46 | 11.09 | 11.91 |\n| 64*64         | 10.18 | 10.01 | 10.10 | 10.29 | 10.71 |\n| 128*128        | 10.25 | 10.04 | 10.12 | 10.31 | 10.61 |\n\nWe hope our responses have addressed your concerns and clarified the contributions of our work. Please let us know if there are further questions or suggestions, as we are happy to provide additional information."}]

Author rebuttal:

Meta-review: This paper presents SeerAttention, a dynamic block-sparse attention mechanism that learns block-level sparsity for the prefilling stage of large language models. Rather than relying on fixed sparsity patterns, SeerAttention uses a lightweight gating module (inspired by MoE) that pools and projects Q/K blocks to produce gating scores and selectively activate important attention blocks. Integrated with a block-sparse FlashAttention kernel and trained via self-distillation (KL alignment to dense masks), the method substantially reduces latency while maintaining or improving model accuracy.

Reviewers agree that SeerAttention is novel, technically sound, and practically relevant. The learnable block-wise gating distinguishes the approach from prior heuristics, yielding meaningful efficiency and memory improvements with modest overhead. The paper provides thorough empirical evaluation—kernel-level and end-to-end benchmarks, ablations (including block-size and training/decoding tradeoffs), and decoding-stage results—that convincingly demonstrate favorable accuracy–efficiency tradeoffs. The presentation is clear and well-structured. Overall, SeerAttention makes a noteworthy contribution to efficient long-context LLM inference and is judged by reviewers to be both methodologically innovative and empirically strong. I recommend acceptance, subject to minor revision to ensure reproducibility (e.g., releasing kernels/integration details and clarifying training costs and overheads).
Decision: Accept (poster)