Title: SparCas: A Dimension-First Cascade for Efficient Long-Context LLM Inference
Primary area: foundation or frontier models, including LLMs
Keywords: LLM Inference, KV Cache, Sparsity

Abstract: Large language models (LLMs) have demonstrated strong capability in handling long-context sequences, but inference efficiency is bottlenecked by the continuously growing KV cache. KV cache selection methods mitigate this by retaining only “important” tokens for attention, yet existing solutions either rely on coarse page-level selection or expensive token-wise hashing, failing to balance efficiency and accuracy. We introduce Sparsity Cascade (SparCas), a novel dimension-first cascade that resolves this trade-off. SparCas is grounded in a key observation that token importance ranking exhibits robust stability when low-magnitude cache dimensions are discarded. We instantiate this approach as a prune-in-prune design: (i) intra-token sparsity prunes non-critical dimensions to form a compact key slice, and (ii) cross-token sparsity ranks tokens within this slice, retaining only the most salient ones for full attention. This cascaded sparsity effectively preserves the quality of token-level cache selection while significantly reducing the computational overhead relative to hashing-based methods. Across extensive evaluations on PG-19, LongBench, and RULER, SparCas consistently matches or outperforms dense attention and prior baselines, achieving oracle-level accuracy with budgets as small as ≤ 1% of tokens at a 32K-token context. Integrated into FlashInfer, SparCas delivers up to 3.01× faster self-attention and 1.64× end-to-end speedups at the same context length. Our project is anonymously available at https://anonymous.4open.science/r/sparcas/.

TL;DR: SparCas cascades sparsity from dimensions to tokens for efficient KV cache selection, reducing memory traffic while preserving dense-attention accuracy in long-context LLMs.