Sparse Attention Adaptation for Long Reasoning
Authors: Yizhao Gao
Shuming Guo
Shijie Cao
Yuqing Xia
Yu Cheng
Lei Wang
Lingxiao Ma
Yutao Sun
Tianzhu Ye
Li Dong
Hayden Kwok-Hay So
Yu Hua
Ting Cao
Fan Yang
Mao Yang
Venue: ICLR 2026 Poster (2026)
Keywords: LLM
Sparse Attention
Reasoning
Primary area: foundation or frontier models, including LLMs

Abstract: We introduce SeerAttention-R, a sparse attention framework specifically tailored for the long decoding of reasoning models.
Extended from SeerAttention, SeerAttention-R retains the design of learning attention sparsity through a self-distilled gating mechanism, while removing query pooling to accommodate auto-regressive decoding. 
With a lightweight plug-in gating, SeerAttention-R is flexible and can be easily integrated into existing pretrained model without modifying the original parameters. 
We demonstrate that SeerAttention-R, trained on just 0.4B tokens, maintains near-lossless reasoning accuracy with 4K token budget in AIME benchmark under large sparse attention block sizes (64/128).
Using TileLang, we develop a highly optimized sparse decoding kernel that achieves near-theoretical speedups of up to 9x over FlashAttention-3 on H100 GPU at 90\% sparsity.

SPARSE ATTENTION ADAPTATION FOR LONG REASON- ING
Yizhao Gao 1224 Shuming Guo
Shijie Cao
Yuqing Xia²
Yu Cheng
Lei Wang'
Lingxiao Ma Yutao Sun
Tianzhu Ye² Dong Hayden Kwok-Hay So²
Yu Hua Ting Cao
Fan Yang"
Mao Yang²
'The University of Hong Kong
Microsoft Research
3 Huazhong University of Science and Technology
Peekngg University
"Tsnnghua University
ABSTRACT
We introduce SeerAttention-R, a sparse attention framework specifically tai- lored for the long decoding of reasoning models. Extended from SeerAttention, SeerAttention-R retains the design of learning attention sparsity through self- distilled gating mechanism, while removing query pooling to accommodate auto- regressive decoding. With lightweight plug-in gating, SeerAttention-P R is flexible and can be easily integrated into existing pretrained model without modifying the original parameters. We demonstrate that Stenntttn,,,,,,, trained just 0.4B tokens, maintains near-lossless reasoning accuracy with 4K token budget in AIME benchmark under large sparse attention block sizes (64/128). Using TileLang, we develop a highly optimized sparse decoding kernel that achieves near-theoretical speedups of up 9x over 3 on H100 GPU at 90% sparsity.
INTRODUCTION
Recent reasoning-focused models such OpenAl ol Jaech et al. (2024), DeepSeek-RI RI Guo et al. (2025), and Qwen3 Yang et al. (2025a) demonstrate that models' capabilities improve significantly through test-time scaling. By generating longer sequences during inference, these models are able to think and reason more effectively before producing an answer. Empirically, longer generations correlate with stronger reasoning performance. For instance, Qwen3-14B Yang al. (2025a) outperforms DeepSeek-RI --D.Diill--wee-leee 14B Guo et al. (2025) while producing longer responses average. Similarly, harder benchmarks such as AIME24 require more tokens per generation than easier ones like MATH-500 HATH--500|endryykks et al. (2020).
However, deeper reasoning introduces increasing efficiency challenges. Due the auto-regressive nature of decoding, later tokens must attend to a longer context, increasing compute and memory demands for the KV cache. As result, the per-token generation cost grows linearly, while the overall generation cost increases quadratically.
Sparse attention offers a promising approach addressing the long-sequence efficiency challenges. While it has been studied in general language modeling, its application to reasoning models, which require prolonged decoding, remains underexplored. Our experiment using oracle sparsity (Sec- tion 4.2p shows that attention in reasoning models is also inherently sparse, activating only a subset of important tokens is sufficient to maintain the model's 's reasoning capability. The key challenge lies in effectively identifying and leveraging this intrinsic sparsity.
'Equal Contribution.
1 Work done during internship at Microsoft Research Asia. Corresponning author.

---

In this work, we extend SecrAttention Gao et al. (2024) to SeerAttention- sparse attention framework aimed to improve the long decoding efficiency of reasoning models. ScerAttention was originally designed to improve prefill efficiency by selectively activating important attention blocks through lightweight, self-distilled attention gating mechanism at post-training time. SterAttention-E retains the core design of self-distilled attention sparsity and introduces modifications support efficient decoding. Specifically, it removes sequence-level pooling of query to accommodate auto- regressive decoding and adopts shared sparsity design aligned with Grouped Query Attention (GQA) to enhance hardware efficiency. SeerAttentions can be integrated into any standard transformer- based pretrained model by adding the learnable gate the attention layer, without fine-tuning original model parameters.
We apply SeerAttention-R to multiple reasoning-focuser open-source models, including Qwen3-4B, 14B Guo et al. (2025), and evaluate
8B, 14B Yang et al.] (2025a) and -eepsSeee---iissil-------- them on several reasoning benchmarks: AIME24, AIME25, MATH-500 (00H-550 Hendrrekk et al. (2020), and GPQA-Diamond Rein et al. (2024). Since SeerAttention-R only requires training the gating module, the distillation is lightweight with just 0.4B tokens from OpenR -MAHH-220K Face (2025) being sufficient. Across all models and tasks, SeerAttention- consistently outperforms the Quest Tang ct al. (2024) baseline and maintains near-lossless accuracy under 4k token budget. Notably, the accuracy gap further diminishes model size increases. More importantly. this learnable approach enables more coarse-grained sparse attention (e.g., block size 64 or 128), which further reduces the overhead from sparse attention scheme and improve hardware efficiency.
We implement the block sparse flash decoding kernel using both TileLang till and Triton Tillet et al. (2019), and benchmark on an H100 GPU with FlashAtteniion-3 (FA3) Shah et al. (2024) as the baseline. Across a range of combination of sequence lengths, batch sizes, and sparsity levels, our FileLang-based kernel consistently outperforms both Triton and FA3. The gains are especially pronounced at large sequence lengths and batch sizes. For example, at batch size 16 and sequence length 32k, our TileL.ang kernel achieves nearttheoretical speedups of up to 8.6x at 90% sparsity over the FA3 baseline, and delivers a 1.7x speedup compared to the Triton counterpart.
SEERATTENTION-P
2.1
RECAP OF SEERATTENTION
Popling
Linear Induuae to Ivheeds)
Pooling
Linear
Linear
RoPE
Catte U1by
Linear
RoPE
ROPE
Block Sparse Flash (Prefill)
RoPE
Block Sparse
Mattivull Softmax
MatiMul
Flash Decoding
Sparisfy
Sparidl
Black Indices
Attention Output
Black Indices
Attention Output
SeerAttention
(b) SecrAttention-R
Figure 1: SeerAttention (Sparse Prefill) and SeerAttention-R (Sparse Decode). In SeerAttention-P. no sequence dimension compressionppolling operation is applied in Query (Q). Given that modern architectures predominantly GQA, linear layer projects the Q from its original number of heads down to the number of KV heads, enabling shared sparsity selection in a GQA group.
SetrAttention Gao al. (024)) introduces self-distilled Attention Gate (AttnGate) that dynamically activates sparse blocks in attention computation for efficient long-context prefilling. Figure [TP shows the AutnGate architecture of SeerAttention, where Q. K tensors are both compressed (pooled) in the sequence dimension per block number of tokens. The compressed Q, K tensors are then passed through two newly added linear layers, which serve as learnable parameters in the AttnGate. With the following positional embedding, mutrix-multiplicateee attention, the AttnGate then generates the 2D block-level attention score estimation. Based on the output, can selectively activate blocks with higher scores while skipping the rest.
and softmax operation similar standard
2

---

In the distillation process, the AttnGate are trained to mimic the 2D block sparse distribution using the ground truth generated by the original pretrained model. This self-distillation training is efficient as the original model weights frozen. In this way, it brings accurate sparse attention to pretrained full-attention models without costly fine-tuning or pre-training. Powered by customized block-sparse flash attention kernels, SeerAttention achieves supreme accuracy-effiiency tradeoff in downstream long-context benchmarks.
2.2 SEERATTENTION-E R: ATTNGATE FOR SPARSE DECODING
This work introduces SeerAttention-R. an extension of SeerAttention tailored for the long-decoding phase of reasoning models. The foremost difference of AttnGate design in SeerAttention-R is that does not apply compression/pooling in the sequence dimension of Q to accommodate the token-by-token auto-regressive decoding process (shown in Figure[b)
Qgate RoPE Wa gate
Dataape(Quope.s (... g.l)),
(la)
K gatte RoPE W gate concat [P max (Knopa), Puin (Knopa); Pavg (Knope)))))
(1b)
softmax Qgato K /dggtt))
(Ic)
where, Pmax. Pmins and Payg stand for Max, Min and Average Pooling in sequence dimension, and is the group size of GQA setting. d and dgate are the hidden dimension of the original model and
AttnGate for each head, respectively. S is the output score of each block from AttnGate. The detailed design are discussed as follows.
Aggregation of Query Heads for Shared Sparsity in GQA Group Query Attention (GQA) Ainslie al. (2023) is widely used in LLMs to reduce KV cache size. In GQA, the query heads organized into groups, and each group shares a key-value head. Recent sparse attention works SAAP Mazare al. (2025b) and NSA Yuan et al. (2025) show that using identical attention sparsity choices for all queries in a group can improve the efficiency while achieving similar or better performance. In ScerAttention-R, we follow this practice and use linear layer in the Q branch of AttnGate to reduce each subgroup of queries to one single head. For example, with 32 query heads and 8 key-value heads (group size g = 4), there will be sets of linear weights in shape [dgate. each group of queries heads, resulting only 8 heads of Qgate- Since we keep the number of heads untouched K branch of AttnGate, the final output AutnGate will be key-value heads, achieving a shared decision of sparsity in group.
d] applying on
Pooling-basee Compression of Key We follow the practice of SeerAttention that uses pooling operations to compress the sequence dimension of K. The kernel and stride size of pooling are both equal to block size, which can also be understood as non-overlapping chunk-level pooling. To mitigate the potential information loss associated with pooling operations, employ composition of Max, Min, and Average pooling operations. The outputs from these pooling operations are concatenated prior being fed into the subsequent linear layer, similar SterAttention. The intuition behind this approach is that Max and Min Pooling can effectively capture outlier values, while Average Pooling helps keep the overall distribution intact.
Positional Embedding in AttnGate In line with ScerAttention, the decode AttnGate utilizes the pre-rope Q and tensors as inputs and reapplies RoPE Su et al. (2024) within AtmGate. Given that the branch is compressed along the sequence dimension, the position index is assigned to the initial token of each block. In our experiment, we found that the use of positional embedding AttnGate can consistently achieve better accuracy compared to the design without positional embedding.
2.3 DISTILLATION/TRAINII
Previous SeerAttention introduces AttnGate distillation method using the ground truth generate by LLM itself in the prefilling phase. The training process is efficient only the AttnGate trained. In SeerAttention-R. we extend this method to the decoding scenario by slightly changing the form of the ground truth. Figure shows the overall diagram of the training process.
3

---

Block
Pooling
Pseudo Code of Customized Atth Kemell to Generate Ground Truth in FND
Input: g, V; Dutput: D, (Haxpool Attn Hap)
Linear Layer
Load
For From 1
Load K,, V:
1D colln
Compute s. d ny rownax(S..)
Store Fgy
Compute final nax(augss) rub Ly og for fron
Seq
KLdiv Loss
Load Fig
Rescale gt., exp(ri v.1/1.
Store Bfa5
AltnGate Ouput
Ground Truth
Figure 2: Training Diagram and Training Kernel of SeerAttention R. (a) Self-distillation training of Attn in SeerAttention-R, It uses ID maxpooled attention scores from original model as ground truth train AttnGate. Query head reduction is not plotted in the diagram for simplicity. (2) Pseudo code of attention forward kernel for training that directly generates ground truth and attention output.
Ground Truth To train AttnGate for the auto-regressive decoding process, need adapt the ground truth generation method. Instead of performing 2D maxpooling of attention map in the prefill case, only do column-wise 1D) maxpooling shown in Figure 2p. This corresponds to the decoding AttnGate that does not compress in sequence dimension. Moreover, to accommodate the shared sparsity in GQA, the column- oolumn-pooled attention map is further maxpooled within each query heads subgroup, resulting in ground truth with key-value heads. Finally, the ground truth is normalized to sum to We then use the Kullback-Leibie divergence loss Joyce (01TT to train AttnGate in the distillation process.
Efficiently Obtaining Ground Truth during Training Explicitly calculating the full attention map softmax QKT /v and then perform the block-level pooling can cost huge GPU memory due the quadratic complexity. In SeerAttention-R. we also provide an efficient modification of FlashAttention- 2 Dao (2023) kernel that directly generates the ground truth along with the attention output. This kernel largely reuses the intermediate results (e.g. block-level rowmax) in Flash-Attentio and thus increases the efficiency the distillation process. The pseudo code is shown Figure 2p.
INFERENCE OF SHERATTENTION-R
Compression Cache
AttnGate
Pooling
Lineor
Update Once Block
Tokans Generated
Recent K Queue
i B i
K Cache
Block Indices
V Cache
Sparsely Load KV
Block Sparse Flash Decoding
Figure 3: Inference Diagram ScerAttention-R, During inference, a K Compression Cache used to cache the compressed key representation in AttnGate speedup sparse block prediction. This K Compression Cache only updates once per block number of tokens is generated (block=4 the plots for illustration). As result, the last block of sequence is always selected to compensate when the compression cache has not been updated yet. g is the group size of GQA.
4

---

3.1 SPARSIFY METHODS: TOKEN BUDGET VS THRESHOLD
During training, the AttnGate output S is distilled to mimic the distribution of the block-wise attention maps from the original model in real-valued (floating-poinl form. During inference, important key- value blocks can be selectively activated based on the predictions of AttnGate. In SeerAttention-R R, we apply two sparsity methods to convert the soft AttnGate outputs into binary block masks (or block indices). The first method is the token budger approach, which is widely adopted in sparse attention methods. Given a fixed token budget, it is first translated into a block budget by dividing the token budget by the block size. The AttnGate outputs are then sorted using Top-k kernel, where k corresponds to the block budget. While this method introduces an additional Top-k operation. it eliminates the need for softmax operation in AttnGate. The second method is the threshold approach, which simply selects blocks whose scores exceed a given threshold. The threshold method more self-adaptive as different heads may automatically infer different sparsity ratios. While these two methods involve different trade-offs between efficiency and accuracy, the token budget approach better suited for direct comparisons with other methods.
3.2 K COMPRESSION CACHE
Similar to KV cache, in SeerAttention-R, we use a K Compression Cache to store the compressed representation of K (after pooling plus linear) to speedup AttnGate prediction. Thus, AttnGate does not need to recompute K branch for past seen tokens. The update of K Compression Cache consists of two phases. First, when the sequence length not multiple of block size b. the new entry of K Compression Cache may not be accurate. During this time, the last block is always activated to eliminate unnecessary accuracy loss. Second, as long as number of new tokens are generated, the most recent b tokens will pass through the pooling and linear layer and update the K Compression Cache, In this way, the overhead of AttnGate can be minimized.
In practice, SeerAttention-R utilizes relatively large block size b, such 64, which significantly reduces the overhead of the K Compression Cache. Specifically when b = 64, the additional memory required for the K Compression Cache amounts only 1/128 (<1%) of the original KV cache size. This minimal overhead makes it highly efficient. Moreover, it introduces the possibility of offloading the larger KV cache to CPU or other storage. During inference, only the activated blocks need to be retrieved and transferred back to GPU memory on demand. Alternatively, sparse attention computations can even be performed on heterogeneous resources, such as the CPU, further optimizing memory usage and enabling efficient handling of long-context decoding tasks.
3.3 BLOCK SPARSF FLASH DECODING KERNEI.
To accelerate decoding under block-sparse attention, design a specialized kernel that extends the FtashAttention decoding pattern to support dynamic block spansity in the key/value memory. Our kernel adopts the grid scheduling strategy of flash decoding for GQA, using a three-dimensional launch space over (batch, heads kv, num num_split). This design supports concurrent computation across multiple query groups and key/value shards, maximizing block-level parallelism.
Our block sparse version of the decoding kernel takes the activated block indices from AttnGate (shape [batch, heads_kv max_ seeesseeeeeed__eeeeeeeeeeeleeeee blocks)), group of query heads. During execution, the kernel only traverses the selected indices and thus skips invalid entries. avoiding unnecessary computation and memory access. To improve load/compute balancing across Streaming Multiprocessors (SMs), we partition the key/value blocks along the num_spil sspli dimension using max_ souusselettedd_e blocks rather than the total number of blocks. This strategy ensures more uniform work distribution in the presence of sparsity indducdd irregularity.
selocted_blockss), which encodes the selected key/value blocks for each
On H100 GPUs, our kernel leverages the wgmma instructions for better Tensor Core usage by padding the number of query head groups to 64. We implement the kernel using TileLang til which automatically applies computation optimizations like tiling Zhu et al. (2022). warp specialization and pipelining Cheng al. (2025), and memory layout optimizations such tensorization, rasterization and swizzling Wang et al.] (2024) based on the target architecture. Additionally, we provide a Triton-based implementation with the same scheduling strategy, allowing for comparative evaluation.
5

---

4 EXPERIMENTS
4.1 EXPERIMENTS SETUP
Benchmarks, Models, and Baselines We evaluate SeerAttention-R on three math reasoning benchmarks: the American Invitational Mathematics Examination: AIME24, AIME25, and MATH- 500 500 Hendrycks et al. (2020), as well as GPQA-Diamont Rein et al. (002)). For model evaluation, we select four open-source pre-trained language models with strong reasoning capabilities: Qwen3-4B, 8B, 14B Yang et al. (2025a), and DeepSeck-R1-D RI-Distilll--war 14B Guo et al. (002)). All models are based on the standard Transformer architecture with Grouped Query Attention (GQA). We compare SeerAttention- R against standard full attention and Quest Tang et al. (2024). Quest is a training- free sparse attention algorithm applied during decoding, employing a query-aware key-value (KV) cache selection strategy. Specifically, Quest estimates the upper bound of attention scores within each K block (or "page") to select the most relevant blocks. By default, Quest uses a block size of 16, and keeps the first two layers fully dense to minimize error. In Section 4.3
we set the block size to 64 for both Quest and SeerAttention-E R, and apply sparse attention to all layers to enable a direct comparison. We also conduct ablation studies to analyze the impact of varying block sizes (Appendix [A.I) ) and incorporating hybrid dense layers (Appendix A.2). Note that SeerAttention-R enables shared sparsity selection within each GQA group, whereas Quest does not. Across all experiments, we the max output length to 32,768 tokens. While Qwen3 series of models extends this length to 38,912 for AIME24 and AIME25 in their official report, we fix this output length to ensure consistency and fair comparison across all settings. For SeerAttention- and the full attention baseline, we report average pass @ accuracy over 64 samples for AIME24 and AIME25, samples for MATH-500, and 16 samples for
Training Setup for Atenttttenion-R To distill AttnGate, we use the OpenR OppnRI-MAAIE -AATI------22000 Face (2025) dataset for training. Importantly, only the AttnGate trained, and the original model weights remain unchanged. Inputs are packed into sequences of up to 32k tokens with our variable-length Flash-Attention training kernel that also generates ground truth (Section 2.3 Training is performed with global batch size of 16 for 800 steps on AMD MI300x GPUs, utilizing DeepSpeed ZeRO-2 optimization. We use Adam W optimizer and a learning rate of 1c-3 with cosine decay schedule; a higher learning rate is feasible here because only the AttnGate parameters are trained.
4.2
ORACLE SPARSE ACCURACY: How SPARSE IS ATTENTION IN REASONING MODEL.S?
AIME24
AIME25
MATH-500
GiamoDi
Block Sire 32
100
Accuracy
Gradi: Sporset
Sparsz) Demse Banel
Grodi Sparsc)
Crack Sparse)
Deroe Baseline
Demuee Baxel
Demue Baseiie
100
Accuracy (%)
Block Size
Oradl Sporse!
Gracle Sparse) Dense Baael me
Crocie Sparse) Denoe Baxel
+
Grack Sparse)
Denoe Baseline
Denie Daseline
Block Size 128
100
Accuracy
50-
Grnele Spartel Bazding
Sparse)
Orncle Searse) Baselnng
Orccle Sparse)
Bussl
1k
2k
4k.
8k
1k
Zk
4kc
8k
1k
2k
4k
8k
1k
2k
4ic
8k
budget
budget
budget
budget
Figure 4: Oracle Sparse Results of Qwen3-14h 14B with block size 32, 64, 128.
In the first experiment, we aim to answer the question: How sparse is attention in reasoning models?
To investigate this, employ oracle block sparse selection, which uses the same block-level attention score ground truth described in Section 2.3 (column- -iiu ID max- max-pooling of the full attention map) to directly select the top-k KV blocks at each decoding step. This approach serves accuracy

---

upper bound: it reveals how well SeerAttention-R could perform if the gating mechanism perfectly predicted which blocks to attend. While this approach requires computing full attention to obtain the ground truth and does not provide any speedup, it allows to evaluate the accuracy upper bound achievable by SeerAttention-R under ideal sparse selection.
We evaluate Qwen3-144 -eeBB-14BBBa with three different sparse block sizes: 32, 64, and 128. The token budgets range from Ik to 8k. As shown in Figure[4] using oracle sparsity achieves lossless performance on all tasks when the token budgets reach 2k. For the more challenging AIME24 and AIME25 tasks, some accuracy degradation is observed with Ik token budget, particularly with the largest block size (128). However, this degradation negligible when using block sizes of 32 64. These results indicate that attention sparsity exists in the reasoning process. Based on this, we select a block size of 64 as the default for SeenAtttn:.
4.3
RESULTS OF SEERAITENTION-R AND QUEST
AIMEZ4
1000
Owen3 41 AcoJncy
6k
Accursey (8g
Owen3
2k
6k
8k
2k
42
6k
8k.
1k
2k
42
11.
1k
2%
4k.
64
100
Qwven'- 140
Acaincy (8g
140
DeepSeek Oheilllllooo
Accunscy
2k
6x
6k
8c
2k
4x
6k
8k
1k
2k
4%
6k
1k
2k
4k
6x
Token Budget
Taken Budget
Token Budget
Taken Budget
SeerAttention-R
Quest
Dense Baseline
Figure 5: Accuracy Results of Full Attention, SeerAttention-R. and Quest. The Quest sparse configuration is set to be the same as SeerAttention-R for fair comparison, which uses a block size of 64 and sparse attention in all layers.
Figure 51shows the results of all the models and benchmarks of Full Attention baseline, SeerAttention- R and Quest. As mentioned above, we modify the configuration of Quest to be the same as SeerAttention-R (block size 64 and using sparse attention from 2k, 4k, 6k, and 8k for AIME24 and AIME25, and 1k, 2k, 4k, and 6k for MATH-500 and GPQA- Diamond. This mainly because the typical averaged reasoning length from different benchmark not the same. For the more challenging AIME24 and AIME25, the averaged generated lengths of these models are around 11k-18k. While for the easier MATH-500 and GPQA, the averaged lengths reduced to 4k-9k. However, it is critical to note that across all combinations, the maximum generation lengths all reached the 32k token cap, underscoring the consistent demand for efficient long-context processing.
layers). We use token budgets
The results show that SeerAttention-P - achieves consistently better performance compared to Quest. This trend holds true across every benchmark and computational budget, underscoring the robustness
7

---

and effectiveness of SeerAttention-R, For the AIME24 benchmark, ScerAttention-F typically achieves lossless performance with 4k token budget while the previous oracle sparse only requires 2k. This is within expectation SecrAttention- is only approximation of ground truth with much less computation required. However, Quest fails to achieve lossless accuracy even using 8k token budgets under identical setting. For MATH-500 and GPQA-Diamone, the lossless token budgets reduce to 2k for SeerAttention-R while Quest requires around 8k to approach the full attention baseline.
key trend observed across the results is the relationship between model scale and tolerance for sparse attention. Larger models, such as the 14B variants, exhibit greater robustness to the information loss inherent in sparsity compared to their 4B and 8B counterparts. This phenomenon is particularly pronounced for Quest, where the accuracy gap at lower budgets shrinks significantly as the model size increases. For SeenAtten,,,n-,, the effect is also present. The 14B models close the final gap to the dense baseline more easily than smaller models on challenging benchmarks like AIME25. This indicates that as reasoning models continue to scale, the viability of sparse attention methods increases.
conclusion, the results demonstrate the superiority SeerAttento self-distilled approach over the training-free heuristics of Quest, especially in the challenging large block size configuration. Previous work LServe Yang et al. 2025b) also mentions the accuracy degradation of Quest over larger block sizes. They resolve this challenge by introducing Hierarchical Paging, system approach that uses an additional level of block(page abstraction called virtual logical page, which decouples the sparsity selection page size and physical page size. With SeerAttention- the sparse attention system design by using larger block size.
possibly simplify
4.4
KERNEL SPEEDUP
seqlen seqlen=8192
seqlen-1634
seqien seqlen=32768 32768
seqien 65536
seqlen 131072
Batch 5peedup
Batch Speedup
Speedup
Speedup
0.5
0.6 Sparsity
0.7
0.8
0.9
0.5
0.6
0.7
0.8
0.9
0.5
0.6
0.7
0.8
0.9
0.5
0.6
0.7 Sparsity
0.8
0.9
0.5
0.6
0.7 Sparsity
0.8
Sparsity
Sparsity
TileLang Sparse
Triton Sparse
FA3 Baseline
Figure 6: Kernel Speedup of our Block Sparse Wlssh-Decdiing Kernel on H100 GPU. Our Tilelanng implementation of the kernel achieves higher speedup ratio compared Triton implementation. For longer sequence length or larger batch size cases, the speedups approach the theoretical upper bound compared FA3 basline.
This section evaluates customized block sparse flash decoding kernel described in Section 3.3 We implement the kernel with both TileLang til and Triton and we use FlashAttention--3 (FA3) Shah al. (2024) baseline. The experiments
Nvidia H100 GPU with different input sequence
8

---

lengths (8k to 128k), batch sizes(1 to 16) and sparsity ratios (0.5 0.9). For the GQA configuration, we use 64 attention heads with key-value heads, and head dimension 128.
Figure 6 presents the detailed results. Each subplot corresponds to specific combination of input sequence length (seglen) and batch size (bs), with the x-axis showing different sparsity ratios and the y-axis indicating speedup. The TileLang implementation consistently outperforms the FA3 baseline and achieves greater speedup than the Triton implementation. In general, the sparse kernel delivers higher speedup when the input sequence length is longer or the batch size is larger. This is expected, as the decoding kernel is primarily I/O-bound. When the KV cache size is sufficient to saturate the bandwidth, such as when 1s=16 and seqlen 2 32k, our sparse kernel achieves near tharretceoll speedup (up to 9x at 0.9 sparsity). Even for moderate KV cache sizes, e.g. bs=4 and seqlen=32k, the kernel demonstrates significant speedup (up to 6x at 0.9 sparsity).
5 RELATED WORKS
5.1 TRAINING-FREE Vs. TRAINING-BASED SPARSE ATTENTION
Sparse attention research generally follows two directions: training- free (pre-defined or heuristic) and training- -ased methods. Training-free approaches adopt static patterns Xiao et al. 20 Fu et al. (2024); Xiao et al. (2024b) or heuristic-based algorithms Zhang et al. (2023); Tang et al. (2024)) ); ) Jiang et al.
et al. ) Yang et al. (2025b); Lai et al. 22025); Chen et al. (2024c); Liu et al. (022cc)
(2024a) Yang et al. (2024b); Hu et al. (2025) ; Zhang et al. (2025 ; Xu et al. (2025): Chen et al. (2025) ) relying on prior knowledge such as fixed patterns or head characteristics. In contrast, training-baser methods integrate sparse attention into models to reduce complexity with minimal accuracy loss. Early work explored ocal/global/block patterns Child et al. (2019) BBlltay al. (2020); Zaheer et al. (020)), while recent methods like NSA Yuan et al. (2025), ), MoBA et al. (2025), ACPLin et al. (2025b), MiniCPM4 Team (2025) train dynamic sparse modules during pre-training. SeerAttention offers middle ground, learning sparsity post-training without modifying model weights.
5.2 KV CACHE COMPRESSION
KV cache optimization is key for efficient inference, as smaller caches reduce bandwidth and memory
costs. Eviction- based methods Ge al. (2023); Li et al. (2024b): Zeng al. (2024); Zhang al.)
(2023); Liu et al. (2023); Adnan et al. (2024): Chen et al. (2024a): Behnam et al. (2025) permanently remove tokens, risking accuracy loss. A Alternativey, dynamic selection methods Tang et al. (2024);
Zhang et al. )0244); Hooper et al. (2024); Chen et al. (2025); Liu et al. (2024c); Hu et al. (2025) : Cai
ct al.
2025 Hao et al. (2025): ): Mazaré al. (2025a) retain all tokens but select subsets each step.
5.3
OTHER EFFICIENT ATTENTION ALGORITTMS
Beyond sparsity, efficient variants of multi-head attention include GQA Ainslie et al. (2023),
MQA |Shzzeer (2019), latent-based designs Liu et al. (2024a); Zadouri et al. (2025), ) and cross-
layer sharing approaches like YOCO Sun et al. (2024) and CLA Brandon et al. (2024). Linear attention Katharopoulos
al. 2020); Sun et al. (2023) Beck et (2024)) Yang et al. (024c);
Gu and Dao (2023); Dao and Gu (2024)
; Peng et al. (2023); Yang et al. (2023) enables parallel training and constant inference memory, though it struggles in long-context reasoning. Hybrid models combining linear and full attention show stronger performance Dong et al. (0024); Li at. 2025
6 CONCLUSION AND DISCUSSION
This paper introduces SeerAttention--R R, lightweight sparse attention framework that accelerates long decoding in reasoning models. As a plug-in gating module, it integrates into pretrained models without altering original parameters and requires only lightweight training. Despite coarse-grained block sizes, SeerAttention--R preserves near-lossless reasoning accuracy, while its TileLang kernel achieves near- therretcrett speedup at high sparsity ratios.
Several challenges remain. Achieving full end-to-end speedup will require integration with inference frameworks (e.g., vL.LM Kwon al. (2023), SGL.ang Zheng et al. (2024)) and compatibility with

---

PagedAttention, possibly combined with KV cache offloading Xiao al. (2024a); Liu et al. (2024c) Chen et al. (2024c); Hao et al. (2025). Another open problem is adaptive sparsity, as the trade- off between accuracy and efficincyy varies by task and sequence length. Top-p based sparsity selection Lin et al. (2025a); Chen et al. (202cc offers one promising direction. Finally, unifying sparse prefill and decoding remains challenging: prefill benefits from parallelism while decoding does not. Approaches such as multi-token prediction Gloeckle et al. (2024); Liu et al. (2024b) or
speculative decoding Leviathan et (2023) may help align them under single gating mechanism.
In summary. SeerAttention-R shows that post-training sparse attention can deliver efficiency with minimal accuracy loss, while future work lies in adaptive sparsity, unified prefill/decoding, and system-level integration.
Limitations. The current evaluation of SeerAttention-R focuses on mathematical reasoning bench- marks (AIME24, AIME25, MATH-500, GPQA-Diamond) and does not cover other task domains such as long-context retrieval, code generation, or open-domain question answering. While we believe the learned sparsity patterns are general, transferability to non-mathematical tasks remains to be validated in future work. Additionally, our comparison is primarily against Quest; we did not compare with other post-training sparse attention methods for decode that are not evaluated on reasoning tasks such TidalDecode (Yang et al.) (2024)) and MagicPIG (Chen et al. 2024b). We also note a conceptual similarity to the later work DeepSeek Sparse Attention (DSA) (Liu al.) 2025a): both learn attention sparsity via celf-distillation. The key difference is the attention variant: DSA targets MLA and achieves token-level selection.
ACKNOWLEDGEMENTS
This work supported in part by the National Natural Science Foundation of China (NSFC) under Grant No. U22B2022.
REFERENCES
TileLang. URL Uttpptpp.//gtthho..om//////aaaaa
Muhammad Adnan, Akhil Arunkumar, Gaurav Jain, Prashant Nair, Ilya Soloveychik, and Pu- rushotham Kamath. Keyformer: Kv cache reduction through key tokens selection for efficient generative inference. Proceedings of Machine Learning and Systems, 6:114-121 127, 2024.
Joshua Ainslie, James Lee-Thorp. Michiel De Jong, Yury Zemlyanskiy, Federico Lebrón, and Sumit Sanghai. Gqa: Training generalized multi-query transformer models from multi-head checkpoints. arXiv preprint arXiv:2305. 13245, 2023.
Maximilian Beck, Korbinian Pöppel, Markus Spanring, Andreas Auer, Oleksandra Prudnikova, Michael Kopp, Günter Klambauer, Johannes Brandstetter, and Sepp Hochreiter. xlstm: Extended long short-term memory. arXiv preprint wrXiv:2405.045177 2024.
Payman Behnam, Yaosheng Fu, Ritchie Zhao, Po-An Tsai, Zhiding Yu, and Alexey Tumanov. Rocketkv: Accelerating long-context I1m inference via two-stage kv cache compression. arXiv preprint arXiv:2502.1400 2025.
Iz Beltagy, Matthew Peters, and Arman Cohan. Longformer: The long-document transformer. arXiv preprint arXiv:2004.0511 055500, 2020.
William Brandon, Mayank Mishra, Aniruddha Nrusimha, Rameswar Panda, and Jonathan Ragan- Kelley, Reducing transformer key-value cache size with cross-layer attention. In The Thirty-eighth Annual Conference Neural Information Processing Systems, 2024.
Zefan Cai, Wen Xiao, Hanshi Sun, Cheng Luo, Yikai Zhang, Ke Wan, Yucheng Li, Yeyang Zhou, Li-Wen Chang, Jiuxiang Gu. Zhen Dong, Anima Anandkumar, Abedelkadir Asi, and Junjie Hu. R-kv: Redundancy-aware kv cache compression for training-free free reasoning models acceleration. arXiv preprint 2505::2505.24333 2025.
10

---

Guoxuan Chen, Han Shi, Jiawei Li, Yihang Gao, Xiaozhe Ren, Yimeng Chen, Xin Jiang, Zhenguo Li, Weiyang Liu, and Chao Huang. Seplim: Accelerate large language models by compressing one segment into separator. arXiv preprint arXiv: 2024a.
Yaoqi Chen, Jinkai Zhang, Baotong Lu, Qianxi Zhang, Chengruidong Zhang, Jingjia Luo, Di Liu, Huiqiang Jiang, Qi Chen, Jing Liu, Bailu Ding. Xiao Yan, Jiawei Jiang, Chen Chen, Mingxing Zhang. Yuqing Yang, Fan Yang, and Mao Yang. Retroinfer: A vector-storage approach for scalable long-context Ilm inference, 2025. URL
org/ absaabss25550 02922
Zhuoming Chen, Ranajoy Sadhukhan, Zihao Ye, Yang Zhou, Jianyu Zhang, Niklas Nolte, Yuandong Tian, Matthijs Douze, Leon Bottou, Zhihao Jía, et al. Magicpig: Lsh sampling for efficient Ilm generation. arXiv preprint (rXiv:2410.1611 2024b.
Zhuoming Chen, Ranajoy Sadhukhan, Zihao Ye, Yang Zhou, Jianyu Zhang, Niklas Nolte, Yuandong Tian, Matthijs Douze, Leon Bottou, Zhihao Jia, et al. Magicpig: Lsh sampling for efficient Ilm generation. arXiv preprint arXiv:2410..611 16179, 2024c.
Yu Cheng, Lei Wang, Yining Shi, Yuqing Xia, Lingxiao Ma. Jilong Xue, Yang Wang, Zhiwen Mo, Feiyang Chen, Fan Yang. Mao Yang. and Zhi Yang. PipeThreader: Software-detined pipelining for efficient dnn execution. In 19th USENIX Symposium on Operating Systems Design and Imple- mentation (OSDI 25), 2025. URL httph:t/////mm. http:://wwwww..ew...e usenix org. /tt.///enne//esaiiiiiic osdi25 presentation/chent
Rewon Child, Scott Gray, Alec Radford, and Ila Sutskever. Generating long sequences with sparse transformers. arXiv preprint arXiv:1904. 10509, 2019.
Tri Dao. Flashattention-22: Faster attention with better parallelism and work partitioning. 2023. URL
https://arxiv org/ /bagg/ab////ee / 08691
Tri Dao and Albert Gu. Transformers are ssms: Generalized models and efficient algorithms through structured state space duality, arXiv preprint 4rXiv:2405.21000 2024.
Xin Dong, Yonggan Fu, Shizhe Diao, Wonmin Byeon, Zijia Chen, Ameya Sunil Mahabaleshwarkar, Shih- Shih-Yang Liu, Matthijs Van Keirsbilck, Min-Hung Chen, Yoshi Suhara, et al. Hymba: hybrid- head architecture for small language models. arXiv preprint arXiv:2411.13677 2024.
Hugging Face. Open r1: A fully open reproduction of deepseek-rl, January 2025. URL https
//github.com/ wwt..///oo/ggggggg.gc.e.e.eoeeee wtwpa//g..ccceeeeeorrer
Tianyu Fu, Haofeng Huang, Xuefei Ning, Genghan Zhang, Boju Chen, Tianqi Wu, Hongyi Wang. Zixiao Huang, Shiyao Li, Shengen Yan, et al. Moa: Mixture of sparse attention for automatic large language model compression. arXiv preprint arXiv:2406. /4909, 2024.
Yizhao Gao, Zhichen Zeng, Dayou Du, Shijie Cao, Peiyuan Zhou, Jiaxing Qi, Junjie Lai, Hayden Kwok-Hay So, Ting Cao, Fan Yang, al. Seerattention: Learning intrinsic sparse attention in your llms. arXiv preprint 1rXiv:2410. 13276, 2024.
Suyu Ge, Yunan Zhang, Liyuan Liu, Minjia Zhang, Jiawei Han, and Jianfeng Gao. Model tells you what to discard: Adaptive kv cache compression for llms. arXiv preprint arXiv2231000800,, 2023.
Fabian Gloeckle, Badr Youbi Idrissi, Baptiste Rozière, David Lopez-Paz, and Gabriel Synnaeve. Better & faster large language models via multi-token prediction. arXiv preprint 19X1v:2404.197777, 2024.
Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces, arXiv preprint arXiv:2312.00752. 2023.
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Pciyi Wang, Xiao Bi, et al. Deepseck-rl: Incentivizing reasoning capability in 1lms via reinforcement learning, arXiv preprint arXiv:2501.12948, 2025.
Jitai Hao, Yuke Zhu, Tian Wang, Jun Yu, Xin Xin, Bo Zheng, Zhaochun Ren, and Sheng Guo. Omnikv: Dynamic context selection for efficient long-context llms. In The Thirteenth International Conference Learning Representations, 2025.
11

---

Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. arXiv preprint 00Xiv:2000.03300, 2020.
Coleman Hooper, Sehoon Kim, Hiva Mohammadzadeh, Monishwaran Maheswaran, June Paik, Michael W Mahoney, Kurt Keutzer, and Amir Gholami. Squeezed attention: Accelerating long context length Ilm inference. arXiv preprint arXiv:24/
1 1.09688. 2024.
Junhao Hu, Wenrui Huang, Weidong Wang, Zhenwen Li, Tiancheng Hu, Zhixia Liu, Xusheng Chen, Tao Xie, and Yizhou Shan. Efficient long- long-decoding inference with reasoning-awan attention sparsity. arXiv preprint arXiv:2502. / 147, 2025.
Aaron Jacch, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Careey, et al. Openai ol system card. arXiv preprint arXiv:2412.16720, 2024.
Huiqiang Jiang, Yucheng Li, Chengruidong Zhang, Qianhui Wu, Xufang Luo, Surin Ahn, Zhenhua Han, Amir Abdi, Dongsheng Li, Chin Yew Lin, et al. Minference 1.0: Accelerating pre-filling for long- cong-contet Ilms via dynamic sparse attention. arXiv preprint arXiv:2407.02490 2024.
James M Joyce. Kullback-leibler divergence. In International encyclopedia of statistical science,
pages 720-722. Springer, 201
Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, and François Fleuret. Transformers are rnns: Fast autoregressive transformers with linear attention. International conference on machine learning, pages 5156-5165. PMLR, 2020.
Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles, 2023.
Xunhao Lai, Jianqiao Lu, Yao Luo, Yiyuan Ma, and Xun Zhou. Flexprefill: A context-aware sparse attention mechanism for efficient ong-sequence inference. arXiv preprint 2rX6v.250..20766, 2025.
Yaniv Leviathan, Matan Kalman, and Yossi Matias. Fast inference from transformers via speculative decoding. In International Conference on Machine Learning, pages 19274-19286. PMLR, 2023.
Aonian Li, Bangwei Gong, Bo Yang, Boji Shan, Chang Liu, Cheng Zhu, Chunhao Zhang, Congchao Guo, Da Chen, Dong Li, et al. Minimax-01: Scaling foundation models with lightning attention. arXiv preprint ar 2rXhv:2501.083133 2025.
Yucheng Li, Huiqiang Jiang, Qianhui Wu, Xufang Luo, Surin Ahn, Chengruidong Zhang, Amir H Abdi, Dongsheng Li, Jianfeng Gao, Yuqing Yang, al. Scbench: A kv cache-centric analysis of long-context methods. arXiv preprint arXiv:2412.10319, 2024a.
Yuhong Li, Yinghing Huang, Bowen Yang, Bharat Venkitesh, Acyr Locatelli, Hanchen Ye, Tianle Cai, Patrick Lewis, and Deming Chen. Snapkv: Llm knows what you are looking for before generation. Advances in Neural Information Processing Systems, 37:22947-22970, 2024b.
Chaofan Lin, Jiaming Tang, Shuo Yang, Hanshuo Wang, Tian Tang, Boyu Tian, Ion Stoica, Song Han, and Mingyu Gao. Twilight: Adaptive attention sparsity with hierarchical top-p pruning. arXiv preprint arXiv:2502.02777 2025a.
Zhixuan Lin, Johan Obando-Ceron, Xu Owen He, and Aaron Courville. Adaptive computation pruning for the forgetting transformer. arXiv preprint arXiv:2504.064) 06949, 2025b.
Aixin Liu, Bei Feng, Bin Wang, Bingxuan Wang, Bo Liu, Chenggang Zhao, Chenggi Dengr, Chong Ruan, Damai Dai, Daya Guo, et al. Deepseek-v2: strong, economical, and efficient mixture-of- experts language model. arXiv preprint 24055:2405.04444 2024a.
12

---

Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437. 2024b.
Aixin Liu, Aoxue Mei, Bangcai Lin, Bing Xue, Bingxuan Wang, Bingzheng Xu, Bochao Wu, Bowei Zhang, Chaofan Lin, Chen Dong, et al. Deepseek-v3. 2: Pushing the frontier of open large language models. arXiv preprint arXiv:251.002556, 2025a.
Di Liu, Meng Chen, Baotong Lu, Huiqiang Jiang, Zhenhua Han, Qianxi Zhang, Qi Chen, Chen- gruidong Zhang. Bailu Ding, Kai Zhang, al. Retrievalattention: Accelerating long-context Ilm inference via vector retrieval. arXiv preprint arXiv:2409. 10516, 16, 2024c.
Ruikang Liu, Yuxuan Sun, Manyi Zhang, Haoli Bai, Xianzhi Yu, Tiezheng Yu, Chun Yuan, and
Lu Hou. Quantization hurts reasoning? an empirical study on quantized reasoning models. arXiv preprint arXiv:2504. 04823, 2025b.
Zichang Liu, Aditya Desai, Fangshuo Liao, Weitao Wang, Victor Xie, Zhaozhuo Xu, Anastasios Kyrillidis, and Anshumali Shrivastava. Scissorhands: Exploiting the persistence of importance hypothesis for I1m kv cache compression test time. Advances in Neural Information Processing Systems, 36:52342-52364, 2023.
Enzhe Lu, Zhejun Jiang. Jingyuan Liu, Yulun Du, Tao Jiang, Chao Hong, Shaowei Liu, Weiran He, Enming Yuan, Yuzhi Wang. Zhiqi Huang. Huan Yuan, Suting Xu, Xinran Xu, Guokun Lai, Yanru Chen, Huabin Zheng, Junjie Yan, Jianlin Su, Yuxin Wu, Yutao Zhang, Zhilin Yang, Xinyu Zhou, Mingxing Zhang, and Jiezhong Qiu. Moba: Mixture of block attention for long-context llms. arXiv preprint arXiv:2502. 13189, 2025.
Fieran-Emmanuel Mazaré, Gergely Szilvasy, Maria Lomeli, Francisco Massa, Naila Murray, Hervé Jégou, and Matthijs Douze. Inference-time sparse attention with asymmetric indexing. arXiv preprint 2r0i 050.00846. 2025a.
Therre-Emmanuee Mazaré, Gergely Szilvasy, Maria Lomeli, Francisco Massa, Naila Murray, Hervé Jégou, and Matthijs Douze. Inference-time sparse attention with asymmetric indexing. arXiv preprint ar arXiv22502.08246 2025b.
Bo Peng, Eric Alcaide, Quentin Anthony, Alon Albalak, Samuel Arcadinho, Stella Biderman, Huanqi Cao, Xin Cheng, Michael Chung, Matteo Grella, et al. Rwkv: Reinventing mns for the transformer era. arXiv preprint ar arXiv:2305. 13048, 2023.
David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R Bowman. Gpqa: graduate-level google-proot q&a benchmark. In First Conference on Language Modeling, 2024.
Jay Shah, Ganesh Bikshandi, Ying Zhang, Vijay Thakkar, Pradeep Ramani, and Tri Dao. Flashattention-3: Fast and accurate attention with asynchrony and low-precision. Advances in Neural Information Processing Systems, 37:68658-6688 -66655, 2024.
Noam Shazeer. Fast transformer decoding: One write-head is all you need. arXiv preprint
arXiv:1911.021500 2019.
Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding. Neurocomputingg 568: 127063, 2024.
Yutao Sun, Li Dong, Shaohan Huang, Shuming Ma. Yuqing Xia, Jilong Xue, Jianyong Wang, and Furu Wei. Retentive network: A successor to transformer for large language models. arXiv preprint arXin:2307.0862), 2023.
Yutao Sun, Li Dong, Yi Zhu, Shaohan Huang, Wenhui Wang. Shuming Ma, Quanlu Zhang, Jianyong Wang, and Furu Wei. You only cache once: Decoder-decoder architectures for language models. Advances in Neural Information Processing Systems, 37:7339-7361, 2024.
Yutao Sun, Tianzhu Ye, Dong Li, Yuqing Xia, Jian Chen, Yizhao Gao, Shijie Cao, Jianyong Wang, and Furu Wei. Rectified sparse attention. arXiv preprint 108. 2025.
13

---

Jiaming Tang. Yilong Zhao, Kan Zhu, Guangxuan Xiao, Baris Kasikci, and Song Han. Quest: Query-aware sparsity for efficient long-context Ilm inference. arXiv preprint arXiv:2406.10774, 2024.
MiniCPM Team. Minicpm4: Ultra-efficient IIms on end devices. 2025.
Philippe Tillet, Hssian-Tsumg Kung, and David Cox. Triton: an intermediate language and compiler for tiled neural network computations. In Proceedings of the 3rd ACM SIGPLAN International Workshop on Machine Learning and Programming Languages, pages 10-19, 2019.
Lei Wang, Lingxiao Ma, Shijie Cao, Quanlu Zhang, Jilong Xue, Yining Shi, Ningxin Zheng, Ziming Miao, Fan Yang, Ting Cao, Yuqing Yang, and Mao Yang. Ladder: Enabling efficient low-precision deep learning computing through hardware-aware tensor transformation. In 18th Operating Systems Design and Implementation (OSDI 24), pages 307-323, Santa Clara, CA, July 2024. USENIX Association, ISBN 978-1-939133-400 URL https
USENIX Symposium
/www. , www.unnnnn.... wwt.:///ooooooooooooooooo wtw.://wwn..c..ooooo.o.oooeeeeea
lei
Chaojun Xiao, Pengle Zhang, Xu Han, Guangxuan Xiao, Yankai Lin, Zhengyan Zhang, Zhiyuan Liu, and Maosong Sun. Infllm: Training-free long-context extrapolation for Ilms with an efficient context memory. arXiv preprint arXiv:2422.04617 2024a.
Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. Efficient streaming language models with attention sinks. arXiv preprint arXiv:2309. / /7453, 2023.
Guangxuan Xiao, Jiaming Tang, Jingwei Zuo, Junxian Guo, Shang Yang, Haotian Tang, Yao Fu, and Song Han. Duoattention: Efficient long-context Ilm inference with retrieval and streaming heads. arXiv preprint arXiv:2410.1005 2024b.
Ruyi Xu, Guangxuan Xiao, Haofeng Huang, Junxian Guo, and Song Han. Xattention: Block sparse attention with antidiagonal scoring. arXiv preprini arXiv:2503. 16428, 2025.
An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report. arXiv preprint arXiv:2505.08 099388, 2025a.
Lijie Yang, Zhihao Zhang, Zhuofu Chen, Zikun Li, and Zhihao Jia. Tidaldecode: Fast and accurate Ilm decoding with position persistent sparse attention. arXiv preprint arXiv:2410.05076, 2024a.
Shang Yang, Junxian Guo, Haotian Tang, Qinghao Hu, Guangxuan Xiao, Jiaming Tang, Yujun Lin, Zhijian Liu, Yao Lu, and Song Han. Lserve: Efficient long-sequence Ilm serving with unified sparse attention. arXiv preprint arXiv:2502. 2025b.
Shuo Yang, Ying Sheng, Joseph E Gonzalez, Ion Stoica, and Lianmin Zheng. Post-training sparse attention with double sparsity. arXiv preprint arXiv:2408. 07092, 2024b.
Songlin Yang, Bailin Wang, Yikang Shen, Rameswar Panda, and Yoon Kim. Gated linear attention transformers with hardware-efficient training. arXiv preprint arXiv:2312.06633 2023.
Songlin Yang. Bailin Wang, Yu Zhang, Yikang Shen, and Yoon Kim. Parallelizing linear transformers with the delta rule over sequence length. arXiv preprint arXiv:2406.06433 2024c.
Jingyang Yuan, Huazuo Gao, Damai Dai, Junyu Luo, Liang Zhao, Zhengyan Zhang, Zhenda Xie, YX Wei, Lean Wang, Zhiping Xiao, al. Native sparse attention: Hardware-aligned and natively trainable sparse attention. arXiv preprint arXiv:2502.11088 2025.
Ted Zadouri, Hubert Strauss, and Tri Dao. Hardware-efficient attention for fast decoding. arXiv preprint arXiv:2505.2 /448, 2025.
Manzil Zaheer, Guru Guruganesh, Kumar Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago Ontanon, Philip Pham, Anirudh Ravula, Qifan Wang. Li Yang, et al. Big bird: Transformers for longer sequences. Advances in neural information processing systems, 33:17283- 2020.
Zihao Zeng. Bokai Lin, Tianqi Hou, Hao Zhang, and Zhijie Deng. In-context kv-cache eviction for llms via attention-gate. arXiv preprint arXiv:2410. / /2876, 2024.
14

---

Hailin Zhang, Xiaodong Ji, Yilin Chen, Fangcheng Fu, Xupeng Miao, Xiaonan Nie, Weipeng Chen, and Bin Cui. Pqcache: Product quantization- based kvcache for long context Ilm inference. arXiv preprint arXiv:2407.12820, 2024.
Jintao Zhang, Chendong Xiang, Haofeng Huang, Jia Wei, Haocheng Xi, Jun Zhu, and Jianfei Chen. Spargeattn: Accurate sparse attention accelerating any model inference. In International Conference on Machine Learning (ICML), 2025.
Zhenyu Zhang, Ying Sheng, Tianyi Zhou, Tianlong Chen, Lianmin Zheng, Ruisi Cai, Zhao Song. Yuandong Tian, Christopher Re, Clark Barrett, Zhangyang Wang, and Beidi Chen. H2o: Heavy- hitter oracle for efficient generative inference large language models. Thirty-seventh Confer- Neural Information Processing Systems, 2023. URL https://// http::::///ppneeeeeeeeeeeese Eorum?id=RkRrz 7GKO.
Lianmin Zheng, Liangsheng Yin, Zhiqiang Xie, Chuyue Livia Sun, Jeff Huang, Cody Hao Yu, Shiyi Cao, Christos Kozyrakis, Ion Stoica, Joseph E Gonzalez, al. Sglang: Efficient execution of structured language model programs. Advances in Neural Information Processing Systems, 37: 02587-62583, 2024.
Hongyu Zhu, Ruofan Wu, Yijia Diao, Shanbin Ke, Haoyu Li, Chen Zhang, Jilong Xue, Lingxiao Ma, Yuqing Xia, Wei Cui, Fan Yang, Mao Yang. Lidong Zhou, Asaf Cidon, and Gennady Pekhimenko. 16th USENIX Symposium on Operating Systems Design and Implementation (OSDI 22), pages 233 248, Carlsbad, CA, July 2022. USENIX Association. ISBN 978-1-939133-28-11 URL https::pp/..... / www. useni usenix.org/ 2oo.ferrence/ooal222/reeeettttt
ROLLER:
Fast and efficient tensor compilation for deep learning.
15

---

A APPENDIX
A.1
BLOCK SIZE FOR SPARSE ATTENTION
Qwen3-4B
Q-eB3-8B
100
Quest
Quest SeerAttention-R
AIME24 (%)
80
SeerAttention-R
60
40
20
16
128
16
32
64
128
Block Size
Block Size
Figure 7: AIME24 results using different block sizes with 4k token budget. SeerAttention-R achieves almost consistent performances on different block sizes. However, Quest gets lower accuracy when block size gets larger. Note that in this experiment, SeerAttention-R enables shared sparsity selection within each GQA group, whereas Quest does not.
The token block size for sparse attention is critical factor that affects overall system performance. If the block size is too small, it incurs significant overhead in sparse block prediction, including increased computational cost and larger metadata requirements such as compression caches and block indices. While larger block size also potentially improve the utilization of GPUs.
Figure 7 presents AIME24 results on the Qwen3-4B and Qwen3-8B models across block sizes ranging from 16 to 128. By default, Quest uses a block size of 16. The results indicate that Quest's performance decreases as the block size increases. However, SeerAttention-P achieves consistent accurate sparse block selection at different block sizes. Remarkably, this robustness lies under the assumption of the additional mask sharing in the GQA group dimension. We excluded a block size of 16 from experiments due to its inefficiency during both training and inference. It often leads to out-of-memory errors because of the large intermediate attention maps generated during training.
A.2
HYBRID DENSE ATTENTION IN THE FIRST TWo LAYERS
Quest
SeerAttention-R
100
Sparse Layers
Sparse Layers
80
Hybrid Dense First Layers
Hybrid Dense First Layers
Accuracy (%)
60
40
20
6k
Token Budget
Token Budget
Figure 8: AIME24 results of whether using dense attention in first two layers (Qwen3-4B).
Some post-training sparse attention algorithms employ hybrid dense attention in certain layers to mitigate accuracy loss. By default, Quest applies dense attention in its first two layers. However, for evaluate both Quest and SeerAttention-R using purely sparse attention across
fair comparison, all layers in previous evaluation. This approach allows us to isolate and analyze the effects of sparse attention without the confounding influence of hybrid attention.
16

---

To further investigate the impact of hybrid dense attention, conduct ablation study using the Qwen3-4B model on the AIME24 benchmark with block size of 64. As shown Figure g incorporating hybrid dense attention in Quest yields significant improvement in accuracy, whereas SeerAttention- only sees marginal benefits. This difference may be due to the already accurate sparse prediction by SeerAttention-R in the first two layers, reducing the potential gains from hybridization.
A.3 THRESHOLD VS TOKEN BUDGETS
0.8
Activate Tokens (k)
Accuracy
0.7
0.6
Threshold
4k Token Budgets Threshold 4e-3
Token Budget
Dense Baseline
0.5
0.65
0.70
0.75
0.80
Generate Length (k)
32k Sparsity
(a)
(b)
Figure 9: Threshold vs. Token Budget. Results obtained using Qwen3-4B models on AIME24 benchmark. (a) Difference of activated tokens distribution of two methods. (b) Sparsity vs Accuracy tradeoff of two methods. Thresholds: 2e-3, 3e-3, 4e-3, 5e-3, 6e-3. Token Budget: 8k, 6k, 4k. 2k.
In SeerAutention-R, we employ two AutnGate sparsification strategies, threshold and token budget, to convert real- valued gate scores into discrete block selections. The token budget method offers straightforward way to align sparsity and compare with different methods. However, the threshold method is extremely simple to implement and avoids the need of sorting. Figure 9a illustrates the distribution of activated tokens across varying sequence lengths using a threshold of 4e-3 and a token budget of 4K the AIME24 benchmark with Qwen3-4B model. The token budget approach results in strict piecewise linear activation pattern, whereas the threshold method yields a smoother, curved distribution. Figure 96 compares the sparsity-accuracy trade-off's of the two methods. The threshold method shows slightly better accuracy in high sparsity region.
A.4 IMPACT OF SPARSE ATTENTION ON GENERATE LENGTII
Table 1: Qwen3-8B 8B AIME24 Accuracy Reasoning Length.
Token Budgets
2k
4k
6k
8k
Accuracy
13.3
44.2
52.5
59.6
Quest
Gen. Length(k)
30.0
22.9
19.6
17.2
Accuracy
56.6
72.3
74.2
75.1
SeerAttention-
Gen. Length(k)
19.8
16.3
15.3
15.1
We observed that using inaccurate sparse attention (too small budget low recall) can increase output token lengths in reasoning tasks. Table Tablessshows using Qwen3-8B model. The baseline accuracy of full attention and the generated length are 74.5 and 15.1 k, respectively. We can see that Quest, and ScerAttention-R with 2k budget cases, all incur much longer reasoning paths compared full attention. A similar phenomenon has been reported in quantization Liu et al. (2025b). where inaccurate quantization algorithms lead to longer reasoning paths. We believe this effect is universal across different post-training efficiency optimizations of reasoning model, as such methods can introduce enrors that accumulate over the long reasoning chains. These additional reasoning steps potentially undermine the original goal of improving efficiency. Therefore, accurate sparse attention selection algorithm is crucial to mitigate this effect. Another promising approach to eliminate the accumulated errors is to use Rectified Sparse Attention Sun et al.) (2025), which periodically performs dense rectification of the KV cache.
shows the AIME accuracy and reasoning length
17

---

A.5 TRAINING BUDGET
Table Training Budgets
Training Tokens
GPU Hours
Qwen3-4B 10.9
Qwen3-8B 12.2
Qwen3-14B
0.4B
18.6
As lightweight distillation process where only the AttnGate parameters trained, ScerAttention-R is also highly efficient in terms of training, In our experiments, we set the global batch size to 16 and trained for just 800 steps, utilizing DeepSpeed Stage 2 optimization on MI300x GPUs. Each data batch is packed to a sequence length of 32k with our custom variable-length FlashAttention forward kernel, as described in Section 2.3] Table summarizes the GPU hours required for training models of various sizes. Notably, distilling an 8B model requires only 12 GPU hours, demonstrating the efficiency of our approach. Increasing the quantity, quality, and diversity of training data may lead to further improvements.
18


Reviews:
[{"review_id": "FxNwd012kh", "reviewer": "Reviewer_Maso", "summary": "This paper proposes SeerAttention-R extending SeerAttention which was originally designed to improve pre-fill efficiency by sparsely activating important attention blocks with a self distilled gating mechanism at post-training time. The proposed SeerAttention-R keeps the original design choices of SeerAttention but removes sequence-level query pooling to support autoregressive decoding and adopts a shared sparsity design for Grouped Query Attention (GQA). They experimented with different reasoning models and benchmarks and showed good speedups while maintaining the accuracy, by training the gating function on 0.4B tokens.", "questions": "Check the weaknesses.", "limitations": "", "rating": 2, "confidence": 2, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1)  This paper is based on an already proposed sparse attention mechanism. Their contribution is to accommodate the original idea to make it work for autoregressive decoding by removing sequence level query pooling and adapted for recently proposed Grouped Query Attention.\n\n2) They also developed a sparse decoding kernel  using TileLang that achieves up to 9x speedups over FlashAttention-3 on H100 GPU at 90% sparsity.", "weaknesses": "1) The results and design is based on GQA shared sparsity. How well does SeerAttention-R transfer to non-GQA like MHA models without the group-sharing trick? Can you provide comparative results on that?\n\n2)  It’s unclear how the reported gains translate in practical inference stacks—e.g., vLLM with FlashAttention—and in settings that use speculative decoding or KV-cache quantitation. Can you quantify the benefits under these configurations?\n\n3)  The proposed method requires additional training, unlike training-free approaches such as TidalDecode [1] and Quest. The paper also didn't a compare with TidalDecode. Can you report results on TidalDecode?\n\n[1] Yang, Lijie, et al. \"Tidaldecode: Fast and accurate llm decoding with position persistent sparse attention.\" arXiv preprint arXiv:2410.05076 (2024).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "K3V2txxM0w", "reviewer": "Reviewer_UjW9", "summary": "The paper introduces SeerAttention-R, a sparse attention framework designed to enhance the efficiency of long-context autoregressive decoding in reasoning models. Building upon the original SeerAttention, SeerAttention-R employs a self-distilled gating mechanism to learn attention sparsity, eliminates query pooling to support autoregressive decoding, and offers a lightweight plug-in gating system for seamless integration into existing pretrained models without altering their original parameters. The authors demonstrate that SeerAttention-R, trained on 0.4 billion tokens, maintains near-lossless reasoning accuracy with a 4,000-token budget in the AIME benchmark under large sparse attention block sizes (64/128). Additionally, they develop a highly optimized sparse decoding kernel using TileLang, achieving near-theoretical speedups of up to 9x over FlashAttention-3 on an H100 GPU at 90% sparsity.", "questions": "1. How does the self-distilled gating mechanism compare to other attention sparsity learning methods in terms of efficiency and accuracy beyond Quest?\n2. What are the limitations of the sparse decoding kernel on hardware other than the TileLang with H100 GPU tensor core, and how does it perform on different infra configurations? \n3. How does it performance on other long-context benchmarks with larger token budget?\n4. Are there any trade-offs between the level of sparsity and the model's reasoning accuracy, and how can these be balanced?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The self-distilled gating mechanism effectively learns attention sparsity, enhancing computational efficiency without significant accuracy loss.\n2. The approach of learning attention sparsity through self-distillation is innovative, addressing the challenge of long-context autoregressive decoding in reasoning models. \n3. Empirical results show up to 9x speedup over existing methods like FlashAttention-3 on H100 GPUs at 90% sparsity, validating the proposed optimizations.", "weaknesses": "1. While the method shows promising results on the three math reasoning benchmark, its performance on other large-scale datasets or tasks remains unaddressed, raising questions about generalizability. \n2. The reported speedups are based on H100 GPUs; performance on other hardware configurations is not discussed, which may limit the applicability of the optimization. \n3. The paper lacks a direct comparison with other state-of-the-art sparse attention mechanisms, making it difficult to contextualize the proposed method's advantages. \n4. There is a lack of detailed ablation studies to isolate the contributions of individual components, such as the self-distilled gating mechanism and the sparse decoding kernel.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "28ubn36WWl", "reviewer": "Reviewer_rta5", "summary": "SeerAttention-R is a sparse attention framework for long decoding in reasoning models. It is a modification of SeerAttention: removes query pooling, keeps key pooling and the rest of its gating mechanism. It is lightweight mechanism that integrates into existing models (plug-in gating), needs a fraction of a billion tokens to train (learnable gate). SeerAttention-R maintains reasoning accuracy for large sparse attention block sizes (64/128) and modest reasoning token budgets (e.g. 4K for AIME benchmark). Further, to accelerate decoding under block-sparse attention a special kernel is implemented in TileLang (resulting in close to theoretical speedup). Empirical results (i) on sparse attention demonstrate that SeerAttention-R performs favorably to the approach in Quest (only 4K token to reach the full-attention accuracy) and (ii) on decoding, implementation is faster than one based on Triton.", "questions": "Quest is a training-free approach, so it does not incur the (small) gate learning (training) overhead, however suffering an accuracy loss for shorter token budgets.\n- What about the accuracy/training overhead tradeoff for approaches like those mentioned in Lines 444-446 in Section 5.1?\n- How would these compare to SelfAttention-R?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "- Important and timely enhancement proposals of SeerAttention for reasoning that are implemented alongside a \"matching\" decoding kernel using state-of-the-art libraries/languages/tools.\n\n- Empirical results (largely \"summarized\" in  Figure 6 for (i) and Figure 7 for (ii)) demonstrate clean improvements across all dimensions (token budgets, block sizes, baseline LLMs and benchmark datasets).\n\n- Pragmatic plans for integration with popular inference frameworks.", "weaknesses": "- Comparison to only one another sparse attention framework (Quest) could be considered a weak point (on the other hand it keeps the presentation take-aways crisper and easier to consume).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "BgedH4eJsr", "reviewer": "Reviewer_gHf7", "summary": "This paper proposes SeerAttention-R, a sparse attention framework designed for long-sequence decoding in reasoning models. As an extension of SeerAttention, SeerAttention-R retains the core design of learning attention sparsity through a self-distilled gating mechanism while removing query pooling to accommodate auto-regressive decoding. Experiments show that SeerAttention-R, trained on 0.4B tokens, maintains comparable reasoning accuracy on AIME benchmarks, even with large sparse attention block sizes. The optimized sparse decoding kernel implemented using Triton achieves 9× speedup over FlashAttention.", "questions": "See the Weaknesses described above.\n\nOne more question regarding Section \"Training Setup for SeerAttention-R\": The paper uses the AdamW optimizer with a learning rate of 1e-3 and cosine decay schedule. Could you clarify what weight decay value was used? Additionally, why was 1e-3 chosen as the learning rate, which seems relatively large compared to typical fine-tuning settings?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper is well-structured and clearly demonstrates the differences between SeerAttention and SeerAttention-R. Figure 1 contrasts the two methods, allowing to better understand the key insights (removing query pooling, adding shared sparsity design for GQA, etc).\n\n2. The paper provides comprehensive evaluation across multiple models (Qwen3-4B 8B and 14B, DeepSeek-R1-Distill-Qwen-14B) and benchmarks (AIME24/25, MATH-500, GPQA-Diamond).\n\n3. The high training efficiency is interesting. SeerAttention-R requires only 0.4B tokens for lightweight distillation, demonstrating its practical applicability.\n\n4. The paper provides Triton kernel implementation, which is valuable for practical deployment.", "weaknesses": "1. The entire paper suffers from citation formatting. Academic papers should properly distinguish between \\citep and \\citet. The author requires to carefully review every citation in the paper and apply the correct format consistently. \n\n2. The evaluation is restricted to mathematical reasoning tasks (AIME, MATH, GPQA) with training on 0.4B tokens. Generalization to other important domains remains unvalidated, particularly long-context probe tasks and scientific reasoning benchmarks, which are crucial for assessing whether the learned sparsity patterns transfer beyond mathematical reasoning.\n\n3. The paper proposes using a composition of Max, Min, and Average pooling for K compression (Equation 1b) but provides no ablation study comparing this design against alternatives. It would be valuable to see experimental results comparing: (a) using only Max pooling, (b) using only Average pooling, (c) Max+Avg combination, and (d) the full Max+Min+Avg composition. Understanding which pooling components contribute most to performance would strengthen the technical contribution.\n\n4. The experimental comparison is primarily limited to Quest, lacking comparisons with other recent sparse attention methods including NSA, MoBA, and alternative efficient attention mechanisms like linear attention and DeltaNet. Including comparisons with at least a subset of these methods would better position SeerAttention-R's contributions.\n\nNevertheless, I really like that the authors provide a Triton kernel implementation in addition to TileLang, which significantly enhances the reproducibility and practical utility of this work - this is a major reason for my Weak Accept rating.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "xiOLC2oDpj", "reviewer": "Reviewer_BPwo", "summary": "This paper proposes a sparse attention framework designed to accelerate auto-regressive decoding in large language models. The method builds upon SeerAttention, which learns block-level attention sparsity through a lightweight gating network trained via self-distillation. Unlike the original SeerAttention, the proposed approach removes query pooling to make the mechanism compatible with incremental token-by-token decoding. The model further introduces group-wise sparsity sharing aligned with GQA, enabling more efficient decoding with reduced computational overhead. Experiments are conducted to evaluate the proposed method, demonstrating its effectiveness in maintaining model accuracy while improving inference efficiency.", "questions": "See the weakness above.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The paper aims to improve the efficiency of auto-regressive decoding in large language models through sparse attention mechanisms. Given the growing importance of long-context inference and low-latency generation, this topic is both relevant and valuable for the ICLR community.\n\n- The main idea is clean and easy to follow. The motivation for each design choice is well explained. The figures and descriptions effectively convey the framework’s structure and its differences from previous work.", "weaknesses": "- The proposed approach makes minor modifications to SeerAttention, primarily by removing query pooling to enable auto-regressive decoding. This adaptation is technically reasonable and practically useful, but it does not introduce novel algorithmic ideas or theoretical insights. The contribution feels more like an engineering extension of prior work rather than a fundamentally new method.\n\n- The experimental section could be better organized. Sections 4.3 and 4.4 are clear and easy to follow, but I found Section 4.2 somewhat hard to get. Maybe I missed something, but it is not entirely clear what this part aims to show, and I am also confused about how the “oracle block sparse selection” is defined and implemented. Could the authors provide more explanation or additional details here? My feeling is that this section aims to show that attention in reasoning models is inherently sparse during decoding. Is that correct? If so, this experiment might be better suited as an appendix study or as an initial motivation to support the use of sparse attention in reasoning.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)