NeurIPS 2025

AttentionPredictor: Temporal Patterns Matter
for KV Cache Compression
Qingyue Yang1*, Jie Wang1†, Xing Li2‡, Zhihai Wang1, Chen Chen2, Lei Chen2,
Xianzhi Yu2, Wulong Liu2, Jianye Hao2, 3, Mingxuan Yuan2, Bin Li1
1MoE Key Laboratory of Brain-inspired Intelligent Perception and Cognition,
University of Science and Technology of China
2Huawei Noah’s Ark Lab
3College of Intelligence and Computing, Tianjin University
{yangqingyue, zhwangx}@mail.ustc.edu.cn,
jiewangx@ustc.edu.cn, li.xing2@huawei.com
Abstract
With the development of large language models (LLMs), efficient inference through
Key-Value (KV) cache compression has attracted considerable attention, especially
for long-context generation. To compress the KV cache, recent methods identify
critical KV tokens through static modeling of attention scores. However, these
methods often struggle to accurately determine critical tokens as they neglect the
temporal patterns in attention scores, resulting in a noticeable degradation in LLM
performance. To address this challenge, we propose AttentionPredictor, which
is the first learning-based method to directly predict attention patterns for
KV cache compression and critical token identification. Specifically, Attention-
Predictor learns a lightweight, unified convolution model to dynamically capture
spatiotemporal patterns and predict the next-token attention scores. An appealing
feature of AttentionPredictor is that it accurately predicts the attention score and
shares the unified prediction model, which consumes negligible memory, among
all transformer layers. Moreover, we propose a cross-token critical cache prefetch-
ing framework that hides the token estimation time overhead to accelerate the
decoding stage. By retaining most of the attention information, AttentionPredictor
achieves 13× KV cache compression and 5.6× speedup in a cache offloading
scenario with comparable LLM performance, significantly outperforming the state-
of-the-arts. The code is available at https...