Title: One Stone Three Birds: Training-free Core-context-aware Attention for Efficient LLM Prefilling, Decoding, and KV Caching
Primary area: foundation or frontier models, including LLMs
Keywords: Large Language Models, Efficient Attention

Abstract: The quadratic computational complexity of self-attention poses a critical bottleneck for large language models (LLMs) processing ultra-long contexts. Existing efficient attention methods either accelerate only the prefill stage through static or dynamic sparse patterns, or compress the KV cache solely during decoding, failing to accelerate both stages via a unified solution. Moreover, these approaches often rely on predefined patterns or uniform compression strategies, overlooking the head-specific and context-dependent redundancies. In this paper, we propose Training-free Core-context-aware Attention (TFCA-Attention), a dynamic sparse attention that achieves “one stone three birds”: it unifies acceleration for prefilling, decoding, and KV cache reduction through a consistent sparsity mechanism. TFCA-Attention features an offline calibration phase that determines head-specific sparsity budgets and an online token selection phase that adaptively retains core context tokens using a lightweight redundancy metric. Theoretically, we provide a bounded approximation error guarantee, ensuring long context modeling accuracy. Extensive experiments demonstrate that TFCA-Attention achieves a 2.8× speedup and reduces KV cache by 61% at 128K context length while maintaining performance comparable to full attention across various benchmarks, offering a practical plug-and-play solution for efficient long-context inference.

TL;DR: A dynamic sparse attention method for unified acceleration of both prefilling, decoding and KV cache reduction in LLM inference.