SeerAttention-R: Sparse Attention Adaptation for Long Reasoning
Authors: name: Yizhao Gao; name: Shuming Guo; name: Shijie Cao; name: Yuqing Xia; name: Yu Cheng; name: Lei Wang; name: Lingxiao Ma; name: Yutao Sun; name: Tianzhu Ye; name: Li Dong; name: Hayden Kwok-Hay So; name: Yu Hua; name: Ting Cao; name: Fan Yang; name: Mao Yang
Published: 2025-06-10 15:17:26

Abstract: We introduce SeerAttention-R, a sparse attention framework specifically
tailored for the long decoding of reasoning models. Extended from
SeerAttention, SeerAttention-R retains the design of learning attention
sparsity through a self-distilled gating mechanism, while removing query
pooling to accommodate auto-regressive decoding. With a lightweight plug-in
gating, SeerAttention-R is flexible and can be easily integrated into existing
pretrained model without modifying the original parameters. We demonstrate that
SeerAttention-R, trained on just 0.4B tokens, maintains near-lossless reasoning
accuracy with 4K token budget in AIME benchmark under large sparse attention
block sizes (64/128). Using TileLang, we develop a highly optimized sparse
decoding kernel that achieves near-theoretical speedups of up to 9x over
FlashAttention-3 on H100 GPU at 90% sparsity. Code is available at:
https://github.com/microsoft/SeerAttention.