Less Is More: Training-Free Sparse Attention with Global Locality for
  Efficient Reasoning
Authors: name: Lijie Yang; name: Zhihao Zhang; name: Arti Jain; name: Shijie Cao; name: Baihong Yuan; name: Yiwei Chen; name: Zhihao Jia; name: Ravi Netravali
Published: 2025-08-09 21:10:33

Abstract: Large reasoning models achieve strong performance through test-time scaling
but incur substantial computational overhead, particularly from excessive token
generation when processing short input prompts. While sparse attention
mechanisms can reduce latency and memory usage, existing approaches suffer from
significant accuracy degradation due to accumulated errors during
long-generation reasoning. These methods generally require either high token
retention rates or expensive retraining. We introduce LessIsMore, a
training-free sparse attention mechanism for reasoning tasks, which leverages
global attention patterns rather than relying on traditional head-specific
local optimizations. LessIsMore aggregates token selections from local
attention heads with recent contextual information, enabling unified cross-head
token ranking for future decoding layers. This unified selection improves
generalization and efficiency by avoiding the need to maintain separate token
subsets per head. Evaluation across diverse reasoning tasks and benchmarks
shows that LessIsMore preserves -- and in some cases improves -- accuracy while
achieving a 1.1times average decoding speed-up compared to full attention.
Moreover, LessIsMore attends to 2times fewer tokens without accuracy loss,
achieving a 1.13times end-to-end speed-up compared to existing sparse
attention methods.