AttentionPredictor: Temporal Patterns Matter for KV Cache Compression
Authors: Qingyue Yang
Jie Wang
Xing Li
Zhihai Wang
Chen Chen
Lei Chen
Xianzhi Yu
Wulong Liu
Jianye HAO
Mingxuan Yuan
Bin Li
Venue: NeurIPS 2025 poster ()
Keywords: Large Language Model
KV Cache Compression
Long Context
Primary area: deep_learning

Abstract: With the development of large language models (LLMs), efficient inference through Key-Value (KV) cache compression has attracted considerable attention, especially for long-context generation.
To compress the KV cache, recent methods identify critical KV tokens through static modeling of attention scores. However, these methods often struggle to accurately determine critical tokens as they neglect the *temporal patterns* in attention scores, resulting in a noticeable degradation in LLM performance. 
To address this challenge, we propose **AttentionPredictor**, which is the **first learning-based method to directly predict attention patterns for KV cache compression and critical token identification**. 
Specifically, AttentionPredictor learns a lightweight, unified convolution model to dynamically capture spatiotemporal patterns and predict the next-token attention scores. An appealing feature of AttentionPredictor is that it accurately predicts the attention score and shares the unified prediction model, which consumes negligible memory, among all transformer layers. Moreover, we propose a cross-token critical cache prefetching framework that hides the token estimation time overhead to accelerate the decoding stage. By retaining most of the attention information, AttentionPredictor achieves **13$\times$** KV cache compression and **5.6$\times$** speedup in a cache offloading scenario with comparable LLM performance, significantly outperforming the state-of-the-arts. The code is available at https://github.com/MIRALab-USTC/LLM-AttentionPredictor.

AttentionPredictor: Temporal Patterns Matter for KV Cache Compression
Qingyue Yang", Jie Wang"t, Xing Li Zhihai Wang', Chen Chen', Lei Chen', Xianzhi Yu', Wulong Liu?, Jianye Hao Mingxuan Yuan?, Bin Li'
MoE Key Laboratory of Brain- inspirid Intelligent Perception and Cognition, University of Science and Technology of China Huuawi Noah's Ark Lab
College of Intelligence and Computing, Tianjin University (yangqingyue, zhwangx) @mail uste. edu. en. jewangxusstc. edu. cn, 1i www.iiinnnwwweic.ommmm
Abstract
With the development large language models (LLMs), efficient inference through Key-Value (KV) cache compression has attracted considerable attention, especially for long-context generation. To compress the KV cache, recent methods identify critical KV tokens through static modeling of attention scores. However, these methods often struggle to accurately determine critical tokens as they neglect the temporal patterns in attention scores, resulting in noticeable degradation in LLM performance. To address this challenge, we propose AttentionPredicctrr, which is the first learning-bas based method to directly predict attention patterns for KV cache compression and critical token identification. Specifically, Attention- Predictor learns a lightweight, unified convolution model to dynamically capture spatiotemporal patterns and predict the next-token attention scores. An appealing feature of Predictor is that it accurately predicts the attention score and shares the unified prediction model, which consumes negligible memory, among all transformer layers. Moreover, we propose cross-token critical cache prefetch- ing framework that hides the token estimation time overhead to accelerate the decoding stage. By retaining most of the attention information, AttentionPrediccto achieves 13x KV cache compression and 5.6x speedup in a cache offloading scenario with comparable LLM performance, significantly outperforming the state- of-the-arts. The code is available at AttentionPredictor
Introduction
Large language models (LLM) like OpenAl ol [ have shown impressive scalability and effectiveness tackling complex tasks through long chain-of-thouge (CoT) reasoning and multi-turn conversa- tions 225 However, these long-context tasks require LLMs to handle extremely lengthy contexts, presenting computational and memory challenges for LLMs (6) Specifically, the key-value cache (KV cache), which holds the attention keys and values during generation to prevent re-computations, consumes huge GPU memory [7]. For example, a model with 7 billion parameters, the parameters consume only 14 GB of memory whereas the KV cache requires around 72 GB with the 128K prompt length [8]. As the decoding latency and memory footprint scale with the KV cache, it is important to compress the KV cache as the prompt expands.
This work was done when Qingyue Yang was intern at Huawei. Project lead.
Corresponndig author. Email: jiewangx @ustc.edu.cn.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Cache Eviction
Cache Retrieval
Training Based e.g. SoeeA ttention
A AtlentionPredictor uOurs)
R.g. H20
e.g. Quest
Spotial seriees
(0
Quay
Daggooo satles
Sunol axmu
Bindtend Lawe
1+]
Layer
Statically model attention pattems
Ignore direct
Indirect spatial only
Spetiotemporal prodictor captures dynamic pattems
attention petterns
predictor
Figure 1: A comparison of H2O, Quest, SeerAttention, and AttentionPredictor for identifying critical tokens in the next step with history attention score. Our learning-ban spatiotemporal predictor captures the dynamic attention patterns and accurately predicts next-step attention scores.
Many attention sparsity- -baee methods have been proposed to compress the KV cache in the sequence dimension. Previous works have shown that a small portion of tokens dominate the attention distribu- tion, substantially determining token generation precision [1] Therefore, we can dramatically reduce the cache size by computing attention sparsely with only the critical tokens, while maintaining LLM performance. Cache eviction methods [1133 use heuristic ranking with attention scores to identify critical keys and evict the less relevant ones. However, heuristic scoring methods can only model fixed patterns and struggle to identify critical tokens accurately, causing these methods to suffer from LLM performance degradation. Recently, SeerAttention [14] and Attention Gate (I5) use learnable modules to model dynamic patterns and retrieve critical tokens. However, they only represent keys or hidden states, rather than directly modeling the attention score distribution. As result, they often exhibit limited accuracy after compressing KV cache (Table 8 for details). This motivates the need for a more effective solution that directly targets the prediction of attention scores for enhanced KV cache compression.
In our paper, we thus explore the importance of temporal patterns in attention scores for identifying critical KV cache. Previous research indicates that attention scores exhibit repetitive patterns intrinsic to LLMs 10 18). We further observe that attention patterns have temporal characteristics, such as re-access, sequential, and seasonal, as illustrated in Figure 2 Our theoretical analysis in Section 333 shows these patterns originate from intrinsic model properties such as query similarity and position encoding. These patterns are both stable and predictable across time, forming a two-dimensional (2D) temporal map with local continuity and translation invariint characteristics.
KV cache compression with critical token identification can be formulated to attention score prediction which is a 2D time series prediction problem. Inspired by the theoretical understanding of attention mechanisms, we introduce scores for critical token identification, as shown in Figure T| Since temporal attention patterns are dynamic and difficult to capture with existing heuristic methods, we learn lightweight but accurate prediction models to address this challenge. Specifically, we design convolutional module to capture 2D patterns in the input attention scores, allowing a single predictor to be shared across all transformer layers. These choices shrink our unified prediction model to roughly one millionth the size of the LLM (LLaMA-3.. 1-8), resulting negligible memory consumption. In contrast, SeerAttention's per-layer prediction model amounts to 101MB in total, whereas ours is only 21KB just 0.02% of its size. Additionally, we employ distribution error calibration by periodically computing dense attention, and apply block-wise attention compression to further improve the efficiency of prediction. By accurately identifying critical tokens, iitentionPredictor retains most of the attention information after KV cache compression. To further accelerate I.I. decoding, we apply Attentionnrediccor to our proposed KV cache management framework, a cross-token KV cache prefetching system. In contrast to the existing cross-layer approach [19], our cross-token method can capitalize on longer transfer times, and adapt to more sophisticated and accurate methods for identifying critical tokens.
AitentionPredictor. time-series prediction approach to predict attention
1) Based on our observation of the temporal patterns in the attention score, we propose AttentionPreedictrr. To the best of our knowledge, is the first learning- method to directly predict attention patterns for KV cache compression and atfention sparsity. 2) We propose the first cross-token prefetching framework, which effectively mitigates prediction delays and KV cache transfer latency in the L.M decoding stage. 3) Experiments demonstrate that our approach achieves 13x KV cache compression with comparable L.LM performance and 5.6 speedup to full cache offloading.
Our contributions:

---

2 Related works
2.1 Efficient LLM Inference
Several efforts have optimized pre-trained model inference efficiency from different perspectives. Speculative decoding methods 240 use smaller draft models to predict multiple future tokens for parallel validation by target LLM. In contrast, our approach instead predicts the next token's s attention scores to estimate KV cache compression. Inference systems [2] accelerate the prefilling stage with techniques like prefix cache. Aieentoonrrddchae compatible with them, for applied to the decoding stage. At the same time, block- based sparsification can also be applied 25 and other computational accelerations.
to paged attention
2.2 KV Cache Compression
Many methods compress the KV cache by leveraging the finding that attention scores are sparse, which allows for estimated sparse computation on high-score positions.
Cache eviction. These methods use heuristic approaches to identify key-value pairs of high im- portance and evict the less relevant ones. StreamingLLM [[2] observes that the earliest tokens have high attention scores during inference, so it only retains the initial and the recent few tokens. H2O [11] accumulates all historical attention scores as an estimation, but suffers from the accu- mulation error of scores from the first few tokens being too frequent. SnapKV [13 accumulates attention scores within recent window as estimate and performs one-time filtering during the prefill stage. MInference [16] and FlexPrefill [30] inductively define several attention patterns and determine the compression strategy for each head to accelcrate the prefilling stage. All these methods are Meuristic-basee and statistically model attention patterns. They struggle to capture the dynamic temporal patterns within attention scores accurately.
Cache retrieval. These methods, such as Quest [i. InfLLM 32], and PQCache [33, retrieve critical tokens by estimating attention weights over compressed key representations. However, such compression can degrade fidelity and introduce retrieval errors. In particular, Quest is sensitive to the page size, and the accuracy significantly drops with large page sizes and small budgets, shown in Figure 8] Instead, our method employs a lightweight temporal predictor to estimate attention weights directly, yielding higher retrieval accuracy. Other methods involving KV cache quantization [18 34
36) and KV cache budget allocation [37 8 (38 39] are orthogonal to our token scoring approach and can be combined with our AttentionPredictor..
Training- based Approaches. Recently, a number of training- cache compression approaches have been proposed. MoBA [40] extends the cache retrieval method by integrating sparse attention during training. Similarly, SeerAttention [4] and NSA [41 both train linear models to encode and retrieve key blocks more accurately. However, these approaches typically require training separate model for each layer, which limits their scalability and generalization. In contrast, our framework employs single plug-and-play module that can be applied uniformly to all layers and heads within the same LLM. Our model is only 0.02% the size of SeerAttention. Moreover, whereas NSA relies on the fine-tuning of LLMs achieve optimal results, our method achieves comparable improvements without any additional fine-tuning, making it both more efficient and broadly applicable.
KV cache cross-layer prefetching. These methods hide part of the cache transfer time based on offloading the cache to the CPU. However, as the sequence length increases, the transfer time is too long to be hidden. InfiniGen [9 combines cache retrieval and prefetching by approximating the attention score for the next layer to load the critical cache. However, the estimation time increases significantly the sequence grows, and the inference time for single layer is insufficient cover this. In contrast, our cross-token prefetching framework can hide longer estimation and transfer time within the per-token inference time.
3 Method
In this section, we introduce AttentionPredicctr, the first learning-based method to directly predict attention patterns for KV cache compression and attention sparsity., along with the cross-token prefetch framework for improved cache management. We begin with the problem formulation for attention prediction in Section 3.2 followed by the observation and theoretical analysis of temporal
3

---

attention patterns in Section 3.3 Then we introduce our novel AttentionPredictor in Section 3.4 Finally, Section 3.5 tross-token prefetch framework that efficiently hides both evaluation and cache loading latencies.
3.1 Preliminary
In the language model decoding stage, denote Q E Rlxd, K Rtdd the query tensor and key tensor used for generate token i, respectively, Specifically, we denote K; E Ixd where i E (1,2.....4 t), as the key tensor for token i, and K K the complete key tensor. The
attention score at step t is calculated as: A, Softmax J Q
[ 2....
................................ 0t,r gixt.
The sparsity- KV cache compression seeks find a subset keys with budget B that preserves the most important attention values. Specifically, the subset of key indices is S {81, 82, 8B), where each sj an index within the total & tokens. We define the attention recovery rate as:
ΣS=1 01,33
Rrec
||A:|1
which reflects the amount of information preserved after compression. higher recovery rate Ryec indicates less information loss caused by KV cache compression. Therefore, the goal of KV cache compression can be formulated as finding the index set S that maximizes Ryee-
3.2 Problem Formulation
Accurate next-step attention score modeling is critical to maintain high attention recovery rate with limited KV tokens and improve final LLM generation performance with KV cache compression. Therefore, KV Cache compression with token selection can be formulated as attention score pre- diction, which a two-dimensional (2D) time series prediction problem. Specifically, we predict the attention in step +++ as A.41 and select the critical token positions with S = ('opB(A(41)) Since stacking attention vectors the temporal axis corresponds to decoding steps and the spatial axis corresponds to token positions in context as Figure I] We thus frame attention score prediction as a time series prediction problem. The input to our predictor a 2D spatiotemporal sequence Ai = (Acomp)) ( representing the H most recent block-wise compressed attention scores, where Acomp is generated via max-pooling on the original attention score A; (detailed in Sec. (344). We then train lightweight model predict the attention scores for step + A F(An), where F(-) denotes the model function.
time yields 2D spatiotemporal attention map A., (A,) 1- where
3.3 Attention Temporal Patterns
Observation. We observe that attention exhibits consistent and structured temporal behaviors, which we term attention patterns. At a local level, these patterns are remarkably stable. We have identified three fundamental patterns that reflect strong invariances under spatiotemporal shifts. However, these patterns reveal a complex dynamic globally, exhibiting long-term evolu- tion. To fully understand this dynamic, we will first provide analysis of the stable patterns.
Revacress
Sequertial
Senvonal
Figure 2: Visualization of three predictable tempo- ral attention patterns. Re-access shows repeated attention to specific tokens. Sequential shows at- tention progresses toward the next tokens. Sea- sonal exhibits periodic recurrence as alternating bands of high and uniform attention scores.
Three A Attention Patterns. Our work moti- vated by the observation that attention exhibits three distinct temporal patterns- re-access, se-
quential, and seasonal, shown in Figure 2 These patterns collectively suggest that attention evolves with both temporal continuity and structural regularity across tokens. We unify the three local spatiotemporal attention patterns the following approximate invariance under joint temporal and spatial shifts:
Ot RC 06+56.
(2)

---

where and ō, and 6; represent temporal and spatial offsets, respectively. (1) The re-access pattern corresponds to (6, = 6, = 0). where the model repeatedly attends the same tokens. (2) The se- quential pattern corresponds to (6: = 1, 6; = 1); reflecting progressive loken-wise attention. (3) The seasonal pattern corresponds to (6, > 1,6; = 0), indicating periodie recurrence of attention to fixed positions. While our re-access and sequential patterns resemble shapes defined in MInference [T6], our time-series analysis also reveals a novel seasonal pattern, and our dynamic predictor improves prediction accuracy by 5% over fixed-template approaches (see Appendix C.10).
To better understand why attention follows spatiotemporal patterns, we investigate the underlying theoretical mechanisms. We find out that the high continuity between queries and position embedding plays a central role in shaping attention behavior.
High Query Self-similarity We observe that the query embedding sequence {a: exhibits strong temporal self-similarity, which is consistent with prior findings in [19]. Specifically, the one-step cosine autocorrelation is 0.87 as detailed in Appendix [A.]] indicating a high degree of alignment between queries. As only the most recent KV cache is updated, high query self-similarity indicates high stability of the attention score over consecutive time steps, which is highly predictable.
For theoretical clarity, we define the raw attention scores as Ā, = =q omitting both the 1 1/va factor and the softmax. This allows us foeus the underlying structure of attention logits, as softmax a monotonic function and does not alter the relative ranking of attention weights. For step ++ 1, supposing the query vector evolves as qt+1 = q: + Aq. the attention step + is:
Äμ1 q0+1k1a41 = (qc + [ka [441]" ] =ukk + Aqkia + (a: 1q)
(3)
Focusing on the first t entries of Ā4t1, we obtain:
Aet1(1 [1 : = q + Agh = Ao + AA,
where A = Agk Noting that || is small due to the high autocorrelation, and the key matrix k remains relatively stable, the term A is of small magnitude in norm:
| S AAll |K,||
Therefore, A, A441 holds pointwise to a good approximation when the query exhibits high self-similarity. This implies that the relative ranking and distribution of attention weights over past tokens change slowly across decoding steps. other words, attention exhibits temporal smoothness and inter-step consistency, which form the basis of the observed re-access and sequential patterns.
Position Embedding. The attention patterns are closely related position embedding, particularly Rotary Positional Embedding (RoPE), which applies position-dependeee rotations to queries and keys. In RoPE, the query and key vectors are partitioned into d/2 groups along dimensions and apply rotational matrices with different frequencies to each group. The attention weights are calculated
with RoPE encoding as 17 42
d/2
RoPB(g:) RoPE(k))
Σ
(a
(m)
R((j -i)0m) (10m))mmm))))
(5)
d/2
l1 (m))
[1111/mmmm) cos (olm) + ( :)0m) (m)m))
where ("") (m))) E 2R2 denote the components of q and kj in the m-th group, Om is the frequency parameter, and (f(m) is the angle between ((m) and k.fm) For the query and key are stable across steps, the formulation reveals that the inner product depends primarily on the relative position j -. When j i A is fixed, the cosine terms remain stable across decoding steps, producing consistent attention along diagonals, showing as the sequential pattern. Furthermore, the periodicity of the cosine function explains the periodic slashes of sequential patterns observed in Figure 2
Dynamic Patterns. These foundational patterns are not static and exhibit dynamic evolution even within a single generation sequence. As the decoding process progresses, patterns can shift, appear, or fade as illustrated in |Figure 2] This dynamism driven by the decay of query-key similarity from inherent query shifting, and is especially evident in long-output tasks like reasoning.
5

---

Atteutlonnreedictor
Crosstoken
Spelid
Prefetch
Decoding
Tokzn
Updas
Prodict
Top-K
CLM
Prodetee
Ataction Ag
Bietion A
Compresend
Stomeoyy
Attrntios
Stared
Commolutiona
Unitled Lighbreicht
Next Shep Bion Aea
Critcal Blook Indices 8
Black Todoss
Logar2
Tobes trd
Model
Full Cadlee
Layer2
Figure 3: Overview of AttentionPrediccoor and cross-token prefetching framework. (a) Attention- Predictor formulates the history attention scores as a spatiotemporal sequence, and predicts the attention at the next step with pre-trained model. To enhance efficiency, the attention history is updated in compressed form at each decoding step. (b) The cross-token prefetching framework asynchronously evaluates critical tokens and fetches KV for the next token during the LLM inference. thereby accelerating the decoding stage.
Predict Attention with Time Series Methods. Since the query similarity and position embedding are inherent features of LLMs and independent of input, the resulting attention pattern is also inherent to LLMs. This allows single shared predictor that generalize across heads and datasets. For the pattern also changes as the sequence grows, this dynamic requires the predictor not only to recover the stable structures but also to adapt shifts. Building on theoretical insights into attention, propose that the pattern can be treated as two dimensional spatiotemporal process suited to time series methods. In the next section, we introduce architecture that this view to capture and forecast the evolution of attention over time.
3.4 AttentionPredictor: A Spatiotemporal Predictor
Overall Process. As shown in Figure 3
and Algorithm 1 AttentionPredictor prepares attention history sequence An in the prefilling stage, and predicts attention during the decoding stage. First, the At from the LLM compressed to A Ar using block-wise attention compression. Next, Air is updated with 4 Acomp, The next step attention A then predicted with the pretrained model F. From Å.- the top-K positions are selected with budget of B/b, since Ac+1 in compressed form. Finally, the indices are expanded with b to obtain the final critical token indices S.
Model Design. To capture spatiotemporal fea- tures, introduce convolutional neural net- work (CNN) composed of two 2D convolution layers followed by a ID convolution layer. The 2D convolutions capture spatiotemporal features multiple scales, while the ID convolution fo- cuses on the time dimension, extracting tem- poral patterns across time steps. By replacing
Algorithm 1 Identify Critical Tokens
Input: Attention scores Ar, Attention history se- ries Aμ, Block size b, KV budget B
Output: Critical KV token indices set S
1: Pad A, to the nearest multiple of b with zero
2: Aromp <- MaxPooling (Ae, b)
3: Ah + Concat(AAtt
+ :). A.omp)
the fully connected layer with a 1D convolu-
4: A
tion kernel, the model adapts to the increasing
spatial dimension, without data segmentation or training multiple models. Compared to an auto-regressive LSTM 43. the CNN is more lightweight and offers faster predictions, main- taining prediction time shorter than the single-token inference latency. Additionally, when compared to a MLP 44] on time-series dimension, the CNN is more effective at capturing spatial features, which improves prediction accuracy.
5: Scomp Top-K(AAt1,, B/b)
6: S St -UESS.mmm bi i.b+1, i-b (b-1))
Return KV cache indices set S
Training Data and Strategy. Our model both data-efficient and generalizable. We train the model only on a small subset of attention data, specifically approximately 3% extracted from the dataset. The model performance on the entire dataset shows our model effectively captures the patterns (see Section 4.2 Additionally, the temporal patterns of attention are inherent in LLMs, single model can generalize well across various datasets. For example, our model trained on LongBench also performs well on the GSM8K dataset, highlighting the generalization capability of AttentionPredicctr..

---

Block-wise Attention Compression. To speed up prediction, we apply attention compression before computation. By taking advantage of the attention's locality, AttentionPredictor predict attention and identify critical tokens in blocks. Inspired by Quest 31. we use the maximum attention value in each block as its representative. Specifically, max- ppooling applied on A, with a kernel size equal to the block size b, as A, A = Maxpooling( A, b), reducing prediction computation to roughly 1/b.
Distribution Error Calibration. Due to the sparsity of attention computation, the distribution of attention history Ah used for prediction may deviate from the distribution of dense attention. This deviation tends to accumulate over decoding, particularly as the output length increases. To mitigate this issue and enhance prediction accuracy. we introduce distribution error calibration technique to correct these deviations. Specifically, we store the full attention score every M steps, effectively balancing accuracy with computational efficiency. Note that the full attention is only used to update the history. During L.LM decoding. we still use the sparse KV cache, strictly adhere to the budget. Additionally, we predict and update the position S every several steps for efficiency.
3.5 KV Cache Cross-token Prefetching
To address the increased memory cost of longer contexts, current LLM systems offload the KV cache to the CPU, but I/O transfer latency becomes significant bottleneck in inference. KV cache prefetching offers a solution by asynchronously loading the sparse cache in advance. We introduce the cross-token KV cache prefetching framework, which differs from the cross-layer method [19] by leveraging longer transfer time budgets and enhancing data integration. They are helpful for improving prediction accuracy with larger models and improving transfering efficiency. Specifically, our implementation involves a prefetching process for each layer. As illustrated in Figure 3] during the decoding phase, AttentionPredictor forecasts the critical token indices for the next step. The framework then prefetches the sparse KV cache with S for the next step from the full cache to the GPU. At the same time, LLM inference of other layers is also ongoing. Subsequently, the prefetched sparse KV cache is used to calculate the sparse attention of the next token. The timeline of cross-token prefetching can be seen in Figure 10 in Appendix B.T
4 Experimental Results
4.1 Settings
Tasks. 1) We use the LongBench 45 InfiniteBench (46) and RULER QA dataset for long context evaluation. LongBench is widely used benchmark for long-context LLM inference. It consists of 16 datasets covering 6 tasks, including single- and multi-document QA (SQA and MQA), summarization, few-shot learning, synthetic tasks, and code completion. The metries for each task are in Appendix B.3. InfiniteBench has up to extreme 124K average context length. 2) We employ the AIME 48] dataset, tough mathematics dataset, for evaluating the performance of our method with the long-reasnn reasorigg output scenario. 3) We use GSMSK math dataset to evaluate the long n-shot CoT task. 4) We use MMLU [49 and GPQA [50] dataset to evaluate the multi-choice task. 5) We use the Needle In A Haystack [51] experiment to evaluate the in-context retrieval capabilities.
Baselines and LLMs. We select four sparse attention approaches our baselines. These include four accumulative
cache eviction methods, StreamingI_LM [12 and its advanced method Cascading [22
approaches H2O [1I and SnapKV 13) and two cache retrieval method Quest 31 and learning- based SecrAttention [4]. We use the official implementations of all bascline experiments. A detailed description of each baseline is provided in Appendix B.2 We choose two widely used 1. [53 with 32K context length and
long-context models for our evaluation:
LLaMA-3.1-8D 1.-MA--31-8B-Insttrue [54] with 128K context length.
Implementation details. We set the history step H to 64, the block size b 16, and the calibration step M to 5. Performance analysis of these hyperparameters is discussed in Section 4.6] Follow Quest 31.. we did not apply our method or any other algorithms to the first two layers the LLM. Following the settings of H20 and Streamingl.LMM, We allocated the budget equally to the prefix and local tokens, assigning 64 tokens each. The remaining KV budget is allocated to intermediate tokens, determined by the prediction model. We conducted experiments on NVIDIA A800 (80GB) GPUs.
7

---

Table 1: The evaluation results on the LongBench dataset across IK, 2K, and 4K KV cache budgets.
Budget
Method
SQA MQA Summurr Few-shol Synthetic Code Average?
SQQ
MQA Summury Few-shot Synthetic Cole Averaget
Longchat-v1.5-77 7bb-3k
L.aMA
Full
31.07
23.95
24.10
63:80
15.25
54.86
433.59
46,00
68.78
52.29
StreamingL.MM
22.89
19,84
21.91
60:65
3.75
53.23
30.84
31.73
37.51
21,95
61.69
51.25
51.26
42.35
Cascadu
22.66
21.10
19.91
59.84
6.00
9.76
24.30
35.61
38.85
23.93
62.98
51.25
42.5%
42. 42.16
1024
H20++
26.98
22.38
22.92
61.10
14.00
54.67
13.58
37.00
42.28
24.32
60.81
54,28
52.21
44.25
SnapKy
28.16
23.85
22.98
62.75
14.00
55.84
34.63
10.62
43.50
21.78
67.14
53.81
52.57
46.37
35.29
AttentonPredicttrr
24.20
63.81
14.00
54.38
35.62
StreamingL.IMM
24.79
20.44
23.62
61.52
5.75
52.62
31.99
34.39
36.61
24.63
64,80
45.13
44,99
41.64
Cascade
24.97
22.84
20.79
62.22
7.00
10:33
26 26.46
18.41
41.35
74.46
65.30
$2.25
43.0K
41 9K
2048
H20+
27.51
22.91
21.51
62.58
14.50
54.66
34,
40.55
45.78
24.94
65.51
53,36
53.58
46, 34
SnapKV
29.27
24.27
23.21
63.53
14.25
56.69
35.
42.77
44.79
53.32
54.35
47.44
24.51
36,
54.25
AftentooPredicctr
30.91
24.37
64.60
15.25
54.52
36.15
42.78
45.25
68.26
55.38
51.81
Streamingg..IMM
20.63
20.01
23.64
56.18
7.75
55.36
32. 36
41.63
98.82
24.44
53.95
41.15
52.27
42.63
Cascade
27.94
22.83
21.40
63.68
8.25
10.29
27.57
11.80
44.33
25.34
66.08
$1.59
43.558
45.57
4096
H20+
30.07
23.72
21.74
63.16
14.75
54.66
35.06
43.14
45.21
25.52
66. 66.14
53.29
53.61
47.45
SnapKV
30.52
24.35
24.54
63.80
14.50
56.70
35.
12.87
45.34
25.79
67.63
54,09
53,39
47.84
21.61
53.62
25.77
53.21
47.59
AtlentionPredicto
30.78
24.50
24.65
64.19
14.75
54.85
36. 10
43.87
44.57
25.69
69.07
33.38
53.44
48.17
4.2 Main Results
Results on LongBench. We evaluate our method on various long-context tasks in the LongBench benchmark, with the KV cache budgets ranging from 1024 to 4096. We restrict H2O's attention to the past 64 steps to ensure faimeess and denote this variant as H20+. As shown in Table AttentionPredictoor surpasses the performance of all SOTA KV cache eviction and retrieval methods across various KV budgets and LLMs. Notably, the average performance loss compared to the full cache is less than 0.5% across all cache budgets. This demonstrates the ability of our method to effectively model attention patterns and precisely predict the locations of critical tokens. Furthermore, in an extremely sparse budget setting of 8 % (1K/13K),
approach achieves full cache performance, while all other methods suffer at least a 1 1.25% reduction. Additional results of SeerAttention, long-context benchmark InfiniteBench and RULER QA are in Appendix C.2
C.3
and C.4
00
Quest
Atbettanirodlictor
767
Decode Labeetyy (ree
250
H20-1/4 Bodget
Rodgat
60
AL
150
Ouz- 1/B Bulget
5.6x
40
33.8
Ouzz 1/16 Budget
1667
50
2058
Z1K.
XV Cache Budget
Costest Length
Figure 4: Evaluation results on the long- output reasoning task AIME2024 with QwQ-32B.
Figure 5: Per-token decode latency of LLaMA-3. 1-8BB under varying tokens, comparing ours with PlashAtten- tion2 (full cache) and H20.
Results CoT reasoning. Reasoning significantly enhances the problem-solving capabilities of ILMss. For instance, QwQ-32B achieves an accuracy of 76.7% % on the challenging AIME benchmark with the aid of detailed reasoning. However, generating complex answers leads output length of up to 31.5K tokens. This sharply contrasts with typical long-contex: scenarios, which involve long inputs but short outputs. Such shift introduces distinct challenges for KV cache compression during decoding. Under this demanding setting, we evaluate AttentionPredicaa acceleration baselines, H2O and Quest. As shown in Figure 4) AttentionPredietor consistently outperforms both baselines. It achieves comparable accuracy of 76.7 % with a KV budget ratio of 0.13 (2K/15K average), whereas Quest drops sharply to 43.3%. Moreover, our approach attains this accuracy with an average output of only 15K tokens, offering significantly higher efficiency compared to H2O (21K) and Quest (20K). Additional results of GSM8K, multi-choice benchmarks MMLU and GPQA are in Appendix C.I
against two decoding-phaat
and C.6
8

---

4.3 Needle In A HayStack
We conduct the "Fact Retrieval Across Context Lengths" (Needle A Haystack) experiment. Our approach achieved perfect 100% % retrieval accuracy with a 64K-token context and 1/64 cache compression ratio, surpassing all existing methods. This outcome underscores the effectiveness of our method in preserving critical information necessary for accurate retrieval.
Acc-
SnapKV Ace- 84.5%
02000
Quest Acce
AllectionPrddicloo Ace=
Figure 6: Results the Fact Retrieval Across Context Lengths ("Needle In A HayStack") test in LLaMA-3.1-SD Instructt with 64k context size in 1024 KV cache size.
4.4 Efficiency
Decode latency. We evaluate decoding latency a cache offloading setup 55 essential for long contexts where the KV cache exceeds GPU memory. In this scenario, the standard approach (FlashAttention2) creates a bottleneck by prefetching the entire KV cache from CPU to GPU, which method optimizes. Further details the setup are in Appendix B.4 As shown in Figure 5 our method consistently outperforms both FlashAttention2 and H2O across all context lengths. Notably, at 32K tokens, we achieve 5.6x acceleration compared to FlashAttention2, demonstrating the dramatic acceleration enabled by loading only a small fraction of the cache. Moreover, we attain 1.6x speedup over the cecuulationefffciiee H2O, showing the effectiveness of our cross-token prefetch system.
Overload breakdown. Table 2 details the per-layer over- head of our prefetching system. key result that this total overhead remains remarkably stable, increasing only from 47.3ms to 50.0ms as context length grows from 4K to 32K. This stability is core advantage of our asynchronous design. The growing latency from Predict Attention and Transfer Cache is effectively hidden by being absorbed into the Wait for Main LLM period, during which the main model is busy processing the current token's inference. Due to parallel execution across layers, this per-layer over- head is equivalent the final per-token latency.
Table Per-layer overhead breakdown (in ms) of our cache prefetching system, averaged across all layers.
Context Length
4K
8K
16K
32K
Predict Attention
0.7
1.2
2.3
4.5
Transfer Cache
3.9
13.2
Walt for Main LLM
44.4
43.4
39.7
32.3
Total Per-Layer Overhead
47.3
48.6
49.6
50.0
4.5 Prediction Accuracy
Compare with Baselines. We evaluate the prediction accuracy to evaluate our predictor, and the metric is Rgred/ /Raarget On the three representative tasks-QA. summary, and mathematical reasoning-wei wwith different KV cache budgets, AttentionPredictoor consistently achieves a higher average recovery rate compared to H2O and Quest. shown in Table This demonstrates that method accurately identifies the positions critical tokens, thereby minimizing information loss. Notably, with the KV budget of 512. AttentionPredictoo shows significant advantage over Quest, achieving 7% higher average recovery rate, emphasizing our robustness under the extremely high (20x) compression ratio.
Generalization across datasets. evaluate our model's generalization by training exclusively on single dataset and then evaluating prediction accuracy on all remaining, unseen tasks within LongBench. As shown in Figure 7] the model consistently achieves over 95% accuracy across every task, demonstrating strong cross-task transferability and generalization.

---

Table 3: The attention prediction accuracy
(%) across different cache sizes.
904.
Llama-3.1-0
O Summery
97.4
97
91-2
16
M01
95.6
Bodget
Method
Summory
56.9
97.2
96.7
96.
96
85.1
H20
55 M0
57
93.39
19:25
312
Quesi
$6.84
60.15
63.15
63.36
91.17
88.10
95.40
Kode
97.0
49.72
M00
Syatheo
H2O
95
12.61
97.09
94.51
2048
Quest
55.00
02.35
94.21
94.10
Difiext
94.63
95.65
Figure 7: The prediction accuracy (%) of dif- ferent dataset from LongBench.
4056
98.96
97.87
90.14
98.33
4.6
Ablation Study
The effects of hyperparameter block size AttentionPrediccor performance studied. Other
hyperparameters, predictor structure and training efficiency are shown in App. C.8 C.7 and D.1
Block size b. Figure 8 shows that Preditoo maintains superior average performance with block sizes 8-64. While larger block sizes degrade performance from coarser token po- sitioning. AttentionPredictor declines more mildly than Quest's block-wise KV cache retrieval. Quest's s performance drop may stem from losing total attention score information when its min- max based per-block attention score upper bound estimation used for retrieval, causing inaccurate block identification. In contrast, AttentionPredicttor accurately captures attention patterns, improving critical block identification and mitigating block-wise retrieval drawbacks.
Accunay (80)
10
Fill Cadle
Qust
12
Figure 8: Evaluation of Block Size
Longchat with Lengh
Conclusion
We present AttentionPredictor the first learning-based method directly predict attention patterns for KV compression and attention sparsity method. With lightweight convolution model, AttentionPre- dictor captures spatiotemporal patterns attention score and predicts the next-token attention score accurately. Additionally, we propose the first tross-token prefetching framework, which effectively mitigates prediction and transfer delays in the L.LM decoding stage. Experiments on the long-context datasets demonstrate that ittentionPredictor achieves 13x KV cache compression and 5.6x speedup with comparable accuracy of LLM.
Acknowledgments
This work was supported in part by National Key R&D Program of China under contract 2022ZD01 19801, National Nature Science Foundations of China grants U23A20388, 62021001, U19B2026, and U19B2044. This work supported in part by Huawei well. We would like to thank all the anonymous reviewers for their insightful comments. This research was also supported by the advanced computing resources provided by the Supercomputing Center of the USTC.
10

---

References
[1] OpenAl. Learning to reason with LLMs.
https://openai.com conpinex.co///nx////// index/ 2024. [Accessed 30-01-2025].
reason -earning-to-resson---- with 11ms/
[2] Runquan Gui, Zhihai Wang. Jie Wang, Chi Ma, Huiling Zhen, Mingxuan Yuan, Jianye HAO,
Defu Lian, Enhong Chen, and Feng Wu. Hypertree planning: Enhancing LLM reasoning via hierarchical thinking. In Forty-second International Conference on Machine Learning, 2025.
URL httpps:://opeererieer wtt..n/ooooommmddd5heeiete.
[3] Shervin Minaee, Tomas Mikolov, Narjes Nikzad, Meysam Chenaghlu, Richard Socher, Xavier Amatriain, and Jianfeng Gao. Large language models: A survey. arXiv preprint arXiv:2402.06196, 2024.
[4] Xu Huang. Weiwen Liu, Xiaolong Chen, Xingmei Wang, Hao Wang, Defu Lian, Yasheng Wang, Ruiming Tang, and Enhong Chen. Understanding the planning Ilm agents: A survey, arXiv preprint
2024.
[5] Haoyang Liu, Jie Wang, Yuyang Cai, Xiongwei Han, Yufei Kuang, and Jianye HAO. Optitree: Hierarchical thoughts generation with tree search for LLM optimization modeling. In The Thirty-ninth Annual Conference on Neural Information Processing Systems, 2025. URL https //openreview.o ntw/ferum Ejoyyyddddjjjyoooooe
[6] Zixuan Zhou, Xuefei Ning, Ke Hong, Tianyu Fu, Jiaming Xu, Shiyao Li, Yuming Lou, Luning Wang, Zhihang Yuan, Xiuhong Li, et al. survey on efficient inference for large language models. arXiv preprint arXiv:2404.14290 2024.
[7] Haoyang Li, Yiming Li, Anxin Tian, Tianhao Tang, Zhanchao Xu, Xuejia Chen, Nicole Hu, Wei Dong. Qing Li, and Lei Chen. A survey on large language model acceleration based on kv cache management. arXiv preprint arXiv:2412. 19442, 2024.
[8] Dongjie Yang. Xiaodong Han, Yan Gao, Yao Hu, Shilin Zhang, and Hai Zhao, Pyramidinfer: Pyramid KV cache compression for high-thre high-throughhut LLM inference. In Findings of the Association for Computational Linguistics, ACL 2024, Bangkok, Thailand and virtual meeting, 2024.
August 11-16, 2024, pages
[9] Zichang Liu, Jue Wang, Tri Dao, Tianyi Zhou, Binhang Yuan, Zhao Song, Anshumali Shrivas- tava, Ce Zhang, Yuandong Tian, Christopher Re, et al. Deja vu: Contextual sparsity for efficient Ilms at inference time. In International Conference on Machine Learning, pages 22137-22176. PMLR, 2023.
[10] Suyu Ge, Yunan Zhang, Liyuan Liu, Minjia Zhang, Jiawei Han, and Jianfeng Gao. Model tells you what to discard: Adaptive KV cache compression for 1lms. In The Twelfth International Conference Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024, 2024.
[11] Zhenyu Zhang, Ying Sheng, Tianyi Zhou, Tianlong Chen, Liammin Zheng, Ruisi Cai, Zhao Song. Yuandong Tian, Christopher Ré, Clark W. Barrett, Zhangyang Wang, and Beidi Chen. H2O: heavy-hitter oracle for efficient generative inference of large language models. In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine, editors, Advances in Neural Information P Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 16, 2023, 2023.
[12] Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. Efficient streaming language models with attention sinks. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024. URL https://opeereevixx wew.:eooommddd-G//.......
[13] Yuhong Li, Yingbing Huang, Bowen Yang. Bharat Venkitesh, Acyr Locatelli, Hanchen Ye, Tianle Cai, Patrick Lewis, and Deming Chen. Snapkv: Llm knows what you are looking for before generation. arXiv preprint 244.422404.14469, 2024.

---

[14] Yizhao Gao, Zhichen Zeng. Dayou Du, Shijie Cao, Peiyuan Zhou, Jiaxing Qi, Junjie Lai, Hayden Kwok Hay So, Ting Cao. Fan Yang. et al. Secrattention: Learning intrinsic sparse 2024.
attention in your llms. arXiv preprint arXiv:2410.
[15] Zihao Zeng, Bokai Lin, Tianqi Hou, Hao Zhang, and Zhijie Deng. In-context kv-cache eviction for Iims via attention-gate arXiv preprint arXiv:2410.12876, 2024.
[16] Huiqiang Jiang, Yucheng Li, Chengruidong Zhang, Qianhui Wu, Xufang Luo, Surin Ahn, Zhenhua Han, Amir H Abdi, Dongsheng Li, Chin-Yew Lin, et al. Minference 1.0: Accelerating pre-filling for long-context Iims via dynamic sparse attention. arXiv preprint arXiv:2407.022490 2024.
[17] Federico Barbero, Alex Vitvitskyi, Christos Perivolaropoulos, Razvan Pascanu, and Petar Veličković. Round and round we go! what makes rotary positional encodings useful? In The Thirteenth International Conference on Learning Representations, 2025.
[18] Xing Li, Zeyu Xing, Yiming Li, Linping Qu, Hui-Ling Zhen, Wulong Liu, Yiwu Yao, Sinno Jialin Pan, and Mingxuan Yuan. Kvtuner: Sensitivit--wware layer-wise mixed pre- cision kv cache quantization for efficient and nearly lossless Ilm inference, 2025. URL https://arxiv. org/ abs.//02.22...... 04420
[19] Wonbeom Lee, Jungi Lee, Junghwan Seo, and Jaewoong Sim. [InfiniGen]: Efficient generative inference of large language models with dynamic {KV) cache management. In 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24), pages 155-172, 2024.
[20] Zhihai Wang, Jie Wang, Jilai Pan, Xilin Xia, Huiling Zhen, Mingxuan Yuan, Jianye HAO, and Feng Wu. Accelerating large language model reasoning via speculative search. In Forty-second International Conference on Machine Learning, 2025. URL attps::/openreet net/ forumfiid=og0t5exii1
[21] Yuhui Li, Fangyun Wei, Chao Zhang, and Hongyang Zhang. EAGLE: speculative sampling requires rethinking feature uncertainty. In Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024, 2024.
[22] Hanshi Sun, Zhuoming Chen, Xinyu Yang, Yuandong Tian, and Beidi Chen. Triforce: Lossless acceleration of long sequence generation with hierarchical speculative decoding. arXiv preprint arXiv:2404.1 1912, 2024.
[23] Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 technical report. arXiv preprint arXiv:2412. /9437, 2024.
[24] Tianle Cai, Yuhong Li, Zhengyang Geng, Hongwu Peng, Jason D. Lee, Deming Chen, and Tri Dao. Medusa: Simple LLM inference acceleration framework with multiple decoding heads. In Forty-first International Conference Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024, 2024.
[25] Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng. Liammin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the 29th Symposium on Operating Systems Principles, pages 61 1266,666600 2023.
[26] Mingcong Song, Xinru Tang, Fengfan Hou, Jing Li, Wei Wei, Yipeng Ma, Runqiu Xiao, Hongjee Si, Dingcheng Jiang, Shouyi Yin, al. Tackling the dynamicity in production Ilm serving system with sota optimizations via hybrid prefilldeecode/verffty scheduling on efficient meta-kernels. arXiv preprint arXiv:2412.188006 2024.
[27] LMDeploy Contributors. Lmdeploy: A toolkit for compressing, deploying, and serving 11m. 2023.
[28] Lianmin Zheng, Liangsheng Yin, Zhiqiang Xie, Chuyue Sun, Jeff Huang, Cody Hao Yu, Shiyi Cao, Christos Kozyrakis, Ion Stoica, Joseph Gonzalez, et al. Sglang: Efficient execution of structured language model programs. arXiv preprint arXiv:2312.07100 2024.
12

---

[29] Zihao Ye, Lequn Chen, Ruihang Lai, Wuwei Lin. Yineng Zhang, Stephanie Wang. Tianqi Chen, Baris Kasikci, Vinod Grover, Arvind Krishnamurthy, et al. Flashinfer: Efficient and customizable attention engine for Ilm inference serving. arXiv preprint arXiv:2501.01000 2025.
[30] Xunhao Lai, Jianqiao Lu, Yao Luo, Yiyuan Ma, and Xun Zhou. Flexprefill: A context- aware sparse attention mechanism for efficient long-sequence inference. In The Thirteenth International Conference on Learning Representations, 2025.
[31] Jiaming Tang. Yilong Zhao, Kan Zhu, Guangxuan Xiao, Baris Kasikei, and Song Han. QUEST: query-aware sparsity for efficient long-context LLM inference. In Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024, 2024. URL http://ppweeeeeeeeeeee//rorrmddo
[32] Chaojun Xiao, Pengle Zhang, Xu Han, Guangxuan Xiao, Yankai Lin, Zhengyan Zhang, Zhiyuan Liu, and Maosong Sun. Infllm: Training-free free long-context extrapolation for Ilms with an efficient context memory. In The Thirtt-eighth Annual Conference on Neural Information Processing Systems, 2024.
[33] Hailin Zhang, Xiaodong Ji, Yilin Chen, Fangcheng Fu, Xupeng Miao, Xiaonan Nie, Weipeng Chen, and Bin Cui. Pqcache: Product quantization- based kvcache for long context Ilm inference. arXiv preprint arXiv:2407. 12820, 2024.
[34] Zirui Liu, Jiayi Yuan, Hongye Jin, Shaochen Zhong, Zhaozhuo Xu, Vladimir Braverman, Beidi Chen, and Xia Hu. KIVI: tuning-free asymmetric 2bit quantization for KV cache. In Forty- first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024, 2024.
[35] Coleman Hooper, Sehoon Kim, Hiva Mohammadzadeh, Michael W Mahoney, Yakun Sophia Shao, Kurt Keutzer, and Amir Gholami. Kvquant: Towards 10 million context length 1lm inference with ky cache quantization. arXiv preprint PrXXi:2401.18073 2024.
[36] Zhenyu Zhang, Shiwei Liu, Runjin Chen, Bhavya Kailkhura, Beidi Chen, and Atlas Wang. Q-hitter: A better token oracle for efficient Ilm inference via sparse-quantized kv cache. Pro- ceedings of Machine Learning and Systems, 6:381-394, 2024.
[37] Zijie Geng, Jie Wang, Ziqi Liu, Feng Ju, Yiming Li, Xing Li, Mingxuan Yuan, Jianye Hao, Defu Lian, Enhong Chen, and Feng Wu. Accurate KV cache eviction via anchor direction projection for efficient LLM inference. In The Thirty-ninth Annual Conference on Neural Information Processing Systems, 2025.
[38] Zefan Cai, Yichi Zhang, Bofei Gao, Yuliang Liu, Tianyu Liu, Keming Lu, Wayne Xiong, Yue Dong, Baobao Chang, Junjie Hu, al. Pyramidkv: Dynamic kv cache compression based pyramidal information funneling. arXiv preprint arXiv:2406.020m6 2024.
[39] Yuan Feng, Junlin Lv, Yukun Cao, Xike Xie, and S Kevin Zhou Ada-kv: Optimizing
kv cache evietion by adaptive budget allocation for efficient Ilm inference. arXiv preprint arXiv:2407. / 2024.
[40] Enzhe Lu, Zhejun Jiang, Jingyuan Liu, Yulun Du, Tao Jiang, Chao Hong, Shaowei Liu, Weiran He, Enming Yuan, Yuzhi Wang, al. Moba: Mixture of block attention for long-context Ilms. arXiv preprint arXiv:2502.1 1389 2025.
[41] Jingyang Yuan, Huazuo Gao, Damai Dai, Junyu Luo, Liang Zhao, Zhengyan Zhang, Zhenda Xie, YX Wei, Lean Wang, Zhiping Xiao, et al. Native sparse attention: Hardware-aligne and natively trainable sparse attention. arXiv preprint arXiv:2502. 1089, 2025.
[42] André Jonasson. Rotary outliers and rotary offset features in large language models, arXiv preprint 2503.0182.. 2025.
[43] Alex Graves and Alex Graves. Long short-term memory. Supervised sequence labelling with recurrent neural networks, pages 37-45, 2012.
13

---

[44] David E Rumelhart, Geoffrey E Hinton, and Ronald J Williams. Learning representations by propagating errors. nature. 323(6088) 1986.
[45] Yushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu, Jiankai Tang, Zhidian Huang, Zhengxiao Du, Xiao Liu, Aohan Zeng, Lei Hou, Yuxiao Dong, Jie Tang, and Juanzi Li. LongBench: A bilingual, multitask benchmark for long context understanding. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pupers), pages 11397, Bangkok, Thailand, August 2024. Association for Computational Linguisties. doi: 172. URL BtLLhttts////ccandtiie
10.18653/11/2224.eom
org/ 2024. acl socl-1ope long. 172
[46] Xinrong Zhang, Yingfa Chen, Shengding Hu, Zihang Xu, Junhao Chen, Moo Khai Hao, Xu Han, Zhen Leng Thai, Shuo Wang, Zhiyuan Liu, et al. bench: Extending long context evaluation beyond 100k tokens. arXiv preprint
2en.22422.11711 2024.
[47] Cheng-Ping Hsich, Simeng Sun, Samuel Kriman, Shantanu Acharya, Dima Rekesh, Fei Jia, and Boris Ginsburg. Ruler: What's the real context size of your long-context language models? In First Conference on Language Modeling, 2024.
[48] MAA. American invitational mathematics examination-AIME https://maa. org/ math- cowpetitions/ amer ataaammriianna can- feb 2024. Accessed February 2024 from American Invitational Mathematics Examination- AIME 2024.
matanallmmtemmtiiiii cs examinationnanl aime
[49] Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. In International Conference on Learning Representations, 2021.
[50] David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang. Julien Dirani, Julian Michael, and Samuel R Bowman. Gpqa: A graduate-level google-proof q&a benchmark. In First Conference on Language Modeling, 2024.
[51] G. Kamradt. Limtest needleinahaystack. GitHub repository. 2023. https://github. com/ gkamradt
Last accessed 2025-5-15.
[52] Jeffrey Willette, Heejun Lee, Youngwan Lee, Myeongjae Jeon, and Sung Ju Hwang. Training- free exponential context extension via cascading kv cache. In The Thirteenth International Conference Leurning Representations, 2025.
[53] Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Jason Flinn, Margo I. Seltzer, Peter Druschel, Antoine Kaufmann, and Jonathan Mace, editors, Proceedings of the 29th Symposium on Operating Systems Principles, SOSP 2023, Koblenz, Germany, October 23-26, 2023, pages 611-626. ACM, 2023. doi: 10. 0114445600006.66111111 URLhhtpss://da http://oi...o/10..11111110111 3613165
[54] Meta AI. Introducing Llama 3.1. http:///A....aaaccooooo//ooaaaaa 2024 [Accessed 09-01-2025).
2024.
[55] Thomas Wolf. Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. Transformers: State-of-the- art natural language processing, In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38-45, Online, October 2020. Association for Computational Linguistics. URL httph: //ww: www. aclweb. www.wwwbb.g..................... anthology/ 2020. emn emnlp-demos.
[56] Tri Dao. Flashattention-22: Faster attention with better parallelism and work partitioning. The Twelfth International Conference Learning Representations, 2024.
[57] Zhongchao Yi, Zhengyang Zhou, Qihe Huang. Yanjiang Chen, Liheng Yu, Xu Wang, and Yang Wang. Get rid of isolation: continuous multi-task spatio-temporal learning framework. Advances in Neural Information Processing Systems, 37: 17:110011133672 2024.
14

---

[58] Zhengyang Zhou, Qihe Huang, Binwu Wang, Jianpeng Hou, Kuo Yang, Yuxuan Liang, Yu Zheng, and Yang Wang. Coms2t: complementary spatiotemporal learning system for data- adaptive model evolution. IEFF Transactions on Pattern Analysis and Machine Intelligence, 2025.
[59] Zhuoming Chen, Ranajoy Sadhukhan, Zihao Ye, Yang Zhou, Jianyu Zhang. Niklas Nolte, Yuandong Tian, Matthijs Douze, Leon Bottou, Zhihao Jia, and Beidi Chen. MagicPIG: LSH sampling for efficient LLM generation. In Adaptive Foundation Models: Evolving Al for Personalized and Efficient Learning. 2024. URL SFavqiNUsh
[60] Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.144 2021.
15

---

A High Similarity of Queries and Keys
A.1 Qeury Similarity
Our model predicts the attention at step + from a recent history. Since the temporal dimension corresponds to the query sequence (one query per step), the similarity between adjacent queries (qt and (t+1) is the most critical factor for demonstrating step-by-step temporal continuity. Therefore, evaluate the similarity of consecutive queries. Specifically, extract query data from LLM using LongBench and compute the cosine autocorrelation of adjacent queries with a lag of I:
(1c l6+1)
p ビ Σ
-
181l 1941111''
As shown in Figure 9 the heatmap illustrates the query similarity across each layer and head of the LLM. The average similarity is 86% for Longchat and 87% for LLaMA-3.1, indicating strong continuity. Furthermore, the similarity in LLaMA is more consistent across layers, while Longchat shows higher similarity in the shallower layers.
Quey Similarty Oageil) of Longchat
Quary Sinilatte (ag=1) of LL
mas Man
a cown hin
ds gl i
O to
G ES in e al C
ic tt
N in e
e
E
K
L
D
D
ST D
D
R H D
D
D
TE
US
IT
Heal
Had Iule
Figure 9: Query similarity of Longchat and LLaMA-3.1.
Table 4: Self similarity of query and key over different distances.
Distance
10
20
50
Query Similarity 0.87 0.83 0.80
0.78
0.77
0.76
Key Similarity
0.82 0.77 0.74 0.72
0.71
0.67
To provide more detailed analysis of the similarity of queries at a far distance, we measure the average query cosine similarity across various distances (lags). The results are presented in Table 4] The query similarity gradually decreases as the distance increases, which is expected. However, the similarity remains relatively high even at longer distances. This finding suggests that while the influence of the most recent tokens is strongest, the attention patterns exhibit strong local stability that decays slowly over time. This supports our model's 's effectiveness in capturing these evolving yet persistent patterns using a recent history window.
A.2 Key Similarity
In addition to queries, we also investigate the similarity of keys over time. When analyzing the atten- tion scores between pairs such as A and A441,, both attend to the same position i. Consequently. the keys involved in the two computations are exactly the same (k;), and therefore no issue of key self-similarity arises in this case.
In contrast, when analyzing patterns like the "Periodic Slashes" pattern, the comparison involves pairs such as Ar, and A041,44, where the attention scores are influenced by different keys (k; and
16

---

kj-1). In this case, the self-similarrty of keys becomes relevant. To quantify this, we compute the cosine similarity between keys at different positions with various lags:
1
(kis ki+Tr | II"
p(T)
2
|all
where denotes the distance between keys. The results are also presented Table 4 and we can find that they indeed exhibit strong self-similarity. These consistently high similarity values indicate that keys vary slowly over time. Our findings also consistent with prior works such as KIVI M and KVQuant [35), which reported that keys often contain fixed, massive channels, suggesting limited variation across positions.
B Implement Details
B.1 Implement Details of AttentionPredictor
Model Architecture Details. The architecture of the CNN-based AttentionPredictor model is summarized in Table This design ensures efficiency and adaptability varying sequence lengths, while maintaining sufficient capacity to capture temporal patterns.
Table 5: CNN-based AttentionPredictor Architecture.
Layer No.
Layer Type
Details
2D Convolution
Kernel: (3,3), Padding:
Channels:
2
ReLU Activation
3
2D Convolution
Kernel: (3,3), Padding: Channels: 16 > 32
4
ReLU Activation
5 Adaptive Avg. Pooling Collapses temporal history dimension size 1
6 ID Convolution
Kernel: I, Channels: 32
Training Details. We capture the attention scores during full-cache inference
data. We package the attention scores H steps, together with the (H + 1)-th step's score, as input-output pairs. The training data is block-compressed to align with the usage during LLM model inference. Specifically, the input history attention is Ay R "xx padding with zero for short attentions and the output attention score is C E RIx Attention scores with insufficient lengths
padded with zeros at the end to ensure that all data within a package has the same length. For the attention at step t + 1, we only predict the first positions because the last position is derived from the newly generated key, which does not need to be compressed or transferred. Notably, only the attention scores from the decoding phase used output data. As result, the training dataset includes the final H steps of the pre-filling attention and all attention scores from the decoding stage. To train the model, we calculate the discrepancy between the predicted and real attention scores using mean squared error (MSE) loss. The training epoch is set to 30, and we select the model with the best attention prediction accuracy as shown Equation I
Cross-token Prefetching. We leverage GPU parallel streams and CPU multi-threading to parallelize the token retrieval and cache transfer across different layers. Consistent with the main experiment, we maintain uncompressed caches for the first two layers to ensure higher inference accuracy. The timeline of prefetching framework in Figure 10
B.2 Implement Details of Baselines
Our experiments are based on LLMs using 16-bit floating points, and we utilize FlashAttention2 [56] during the prefilling stage to reduce GPU memory consumption. Since FlashAttention2 does not output intermediate attention scores, for methods that rely on historical attention, we additionally compute partial prefill attention to obtain the scores. Our batch size is fixed at 1. For all baselines, we use the official code implementations to ensure the effectiveness of the methods. Additionally, for all methods, we skip compress the first two layers of the LLM.
17

---

Takmn
Tolmm
Low Petth KV
Fetch KV
Scomp Query
Comp Top-1
Atts FFN
ChComp Query
CCmmp Top-k
Feech KV
S Alt FFN
Comp Queary
Cump Top-
Fatch KV
TEN
Tokamn
Toloes/
Timaline
Cross- token prefetch KV
Laper Cump Queny
Ligen22
Altn PEN
Cozzp Query
Altn
Cortp Queny
Attn FFN
Comp Query
Atte FTN
FPN
Pottectir Prndicerr
Predench KV
Asyynchooous besks
Euntin
hide
Timeline of our proposed cross-token prefetching. By asynchronously loading the critical KV cache for the next token, our framework hides the token evaluation and transfer latency, accelerating the decoding stage of LLM inference.
Figure 10:
StreamingLLM [2] finds that the sink token is very important, so retains the initial and the most recent few tokens. For StreamingLLMM, evenly distribute the budget between sink tokens and local tokens.
H2O [1] accumulates historical attention scores as an estimation. We evenly distribute the budget between the heavy budget and the recent budget. Since computing the full historical data can cause OOM with longer inputs, we use the 64 steps of historical information to calculate the heavy hitters, and denote this as H20+. Note that accumulating all prefill attention scores causes H2O to focus on initial tokens, so this an improvement to the method's effectiveness.
SnapKV 13 accumulates attention scores within recent window as an estimate and performs one-time filtering during the prefill stage. We the time window size of 64 to align with our method. Since Snapk only performs cache compression once during the prefill stage, the number of KV tokens it uses continues to grow during decoding, resulting in a relatively lower compression ratio compared to other methods.
Quest [31] uses the current query and paged key to approximate the attention score. We use a chunk size of 16, corresponding the block size in method, which is also the parameter value used in the original paper.
B.3 Details of LongBench Dataset
LongBench is carefully crafted benchmark suite designed to evaluate the capabilities of language models in processing extended documents and complex information sequences. was developed for multi-task assessment of long-context inputs. The details of the metrics, number of words, language, and data used in LongBench are presented in Table
45
B.4 Details of Efficiency Experiment Setup
Our efficiency evaluation is conducted under the cache offloading scenario, which is essential for long-contexi generation when the KV cache size exceeds the available GPU memory. Unlike GPU- based eviction/compression, offloading keeps the full cache on CPU, preserving model accuracy. This setting is also adopted in recent works [2 33
We conduct experiments under this offloading setup on FlashAttention2, H20, and our proposed AttentionPredietor:
FlashAttention2 [6
We utilize the standard KV cache offloading implementation provided by the Hugging Face library. In this setup. at each decoding layer, the entire KV cache for the upcoming layers is prefetched from CPU GPU to compute the full attention.
H2O 11 Also uses cross-layer prefetching but transfers sparse KV cache, making the
comparison more direct evaluation of the prefetching framework's 's efficiency.
AttentionPredictor Runs on an offloading setup but prefetches across tokens, improving efficiency.
18

---

Table 6: An overview of the dataset statistics in LongBench. 'Source' denotes the origin of the context. Avg len' refers to the average number of words, which is shorter than the token length after tokenization. Accuracy (CL.)² refers to classification accuracy, while "Acuuracy (EM) refers to exact match accuracy.
Dataset
Source
Avg len
Metric
Language
#data
SingleDDcuumce QA
NarrativQQ
Literature, Film
18,409
English
200
Qusper
Science
3,619
English
200
MulhiicldQAoou
Multi-ficld
4,559
English
150
Mald--ocummn Document Q4
HotparQA
Wikipedia
English English
2WikMultihoPOA
Wikipedia
4,887
200
MuSiQue
Wikipedia
11,214
Faglish
200
Summarisation
GovRepont
Government report Moeting
8,734
Rouge-L Rouge-L Rouge L
English English English
200
Q
MultiNews
News
2,113
200
chot Learning
TREC
Web question
5.177
Accuracy (CLS)
English
200
TriviaQA
Wikipodia, Web
8.209
English English
200
SAMSum
Dialogue
6,258
Rouge-L
200
Synthetic Task
PassageCount
Wikipedia Wikipedia
11.141
Accuracy (EM)
English English
200
Pnssageeetieralenn
9,289
Accuracy (EM)
200
Code Completion
LCC
Github
1,235
Edit Sim
Pythoe/Chilaaa Python/lava
500
RepoBench-P
Github repository
4,206
Edit
C Additional Experimental Results on Accuracy
C.1 Results on Long N-shot CoT GSM8K.
scenario, which is different with long output We evaluate the reasoning task under a evaluate method on the mathematical reasoning scenario in Section 4.2) As shown in Table 7 dataset GSM8K with LLaMA-3.1.
To simulate CoT tasks within long-context, we increased the number of few-shot examples. Specifi- cally, we randomly selected fixed number of questions and standard CoT answer pairs as prompts, along with the questions to be tested. We chose 25, 47, and 97 few-shot examples, resulting in input lengths of approximately 4K, 8K, and 16K tokens respectively. The few-shot data were sourced from the GSM8K training set. Since the test set does not overlap with the training set, the answers remain undisclosed to the LLM.
Unlike long-co contex tasks LongBench, CoT mathematical reasoning tasks present distinct challenges. For example, the retrieval-based SOTA method, Quest, excels on long-context benchmarks but performs poorly on CoT tasks, showing 16.91% % accuracy drop at a sequence length of 16K. In contrast, ArrentionPredietor achieves significantly smaller accuracy loss of just 0.91 %. Moreover, across all sequence lengths, our method outperforms baselines in most cases.
Table Evaluation results on the long n-shot CoT GSM8K task with different input context lengths with Llama-3.1 188-Innruett
GSMBK+LLaMA-311
Prompt length
Method
Budget
4K
8K
16K
Average
Full cache
Full
56.79
55.27
54.13
55.40
StreamingL
IK
54.74
49.96
50.94
51.88
H20+
IK
57.16
52.01
52.16
53.78
Quest
IK
48.52
45.26 37.22
43.67
SnapkV
IK
$3.45
48.67
49.20
50.44
AttentionPredictor
IK
56.48
53.30
53.22
54.33
19

---

Table 8: The detailed evaluation results from LongBench of all tasks. Since the official pre-trained model of SeerAttention for LongChat is unavailable, we only evaluate its performance on LLaMA-
Single- Document(A
Multi DecumnentAA
Summary
Few-shot Learning
Syuthetic
Code
Budget
Method
NovO
Mosige
SAMSum
Averaget
Longelat-<1.5-7-3333
Fulli cache
20.70
20.11
43.00
33.05
14
14.56
22.70
26.61
65.50
50
83 83.09
40.90 00
0.00
10.50
$3.94
56
06
Streamingt.IMM
15.03
17.31
27. 14
27.00
21.52
9:09
21.51
19.39
21.97
55.00
81.07
37.0
0.50
5.00
53.05
28.65
Cascade
15.56
19.02
24.16
23,57
22.1 13
7.88
11.76
19.50
15.77
51.50
82.44
36.73
1.00
7.50
6.07
12:45
22.50
1120+
19.01
20.00
33.08
30.60
22.47
12.23
21.01
21.59
22.07
50.50
83.42
39.62
0.00
27.50
51.35
$4.63
52.07
512
SnapKV
18.78
25:21
38.18
33.26
23.79
13.22
21 14
21.15
22.58
59.00
84 84.20
38.41
0.00
25.50
54.33
57.43
33.56
Quest
16,80
31.43
38.01
27.95
23.75
10.003
27.91
22.5
25.68
64.50
X1.27
39.05
0.00
41.55
51.95
50.22
54.34
AtrestonPredicter
17:69
78:49
41.63
34.13
23.67
14.18
28.40
21.82
25.52
64.00
84.66
40.36
1.00
25.50
54.42
55.89
35.10
SLreaminng.L.M
16.30
20.06
32.50
28.
21.37
9.56
25.
20.36
23.51
60.50
82.15
39:20
1.50
6:00
50.
53.71
30.84
Csocade
19:30
21:31
27:47
30.24
22.29
10.78
14 60
20.63
19.19
57.00
83.43
39 10
2.50
9.50
6.69
12.83
24.80
H120-
15.92
24.65
37.38
32,90
22.28
12.46
22.72
21.62
24.22
59.00
84.39
39.90
0.00
28.00
53.74
55.60
53.55
1024
S
19.46
26.97
38.96
34,01
23.22
14.33
23.07
21.46
24.50
64.00
84.33
39.93
0.00
28.00
$3.96
$7.72
34.6.
Quest
18.51
31.09
41.32
29.91
22.95
11.35
30.36
22.67
26.52
66,00
X3.35
40.86
0.00
35.00
50.77
54. 16
35.29
AfteationPPediittr
19.36
29.50
41.67
33.64
24.30
14.91
29.92
22.30
26.19
66,00
N4.85
40.58
0.00
28.00
53.22
56.54
35.62
StreamingL.L.MM
15.52
22.66
36.30
29:113
22.15
10.13
27.59
21,04
26.20
64.00
X1.93
38.62
2.00
9.50
19.75
55
31.99
Cascade
19.49
24.30
31.63
33.48
22.68
12.37
16.82
20,82
20.76
63.00
X3.51
40.32
3
11:00
7.28
13:38
26.46
H20=
19:10
26.62
16.82
33.23
22:65
12.83
25.
21.95
21.09
62.50
84:65
40
0.00
20
$3.33
55.09
34.00
2048
SnapKV
20.02
78
40.61
54.13
24.02
14.67
26.05
21.82
25.59
65.00
84.71
40.89
0.00
38.50
54.93
58,44
35.50
Quest
19:12
32,06
43.72
31,91
24.97
14.83
31.56
23.65
26.36
65.30
85.09
41.69
0.00
31,50
50.53
55.75
36.1 1d
AttemtunPredicttor
20.07
30.06
42.80
34.26
24:20
14.64
30.53
22.07
26.61
67.00
84.94
41.86
0.00
30.50
53.45
56 36.59
36.15
StreamingL.LM
19:45
27.90
41.53
28.54
20.50
10,99
26.55
20.73
26.54
58.00
75.71
34.82
2.50
13.00
54.31
56.41
32.36
Caocade
27.99
35.38
21.33
84.99
7.48
13.09
H20
28.24
41.41
23.87
14.52
27.91
21.76
04.00
40.90
52.71
56.61
35.06
4196
SnapKY
42.56
24.46
26.54
84
54.70
58.69
35.09
43.90
23.95
84
18
51.62
55.61
AtrentinPrrdicttrr
20.33
42.88
24:27
15.10
30:74
41.42
30.00
36.10
LLLMMA-311-0-- Instructt
29.26
45.36
56.12
48.90
32.43
33:77
25.20
27,05
91.48
40.85
99.50
56,85
48.15
48.17
StrvamingI.
24:91
18.64
20.34
92.32
45.05
41.17
Cascade
30.09
64.75
22,93
24.16
21.09
97.49
40.61
45.58
40.00
29.34
23.28
99.50
56,05
43.02
SnapKV
39:81
52.05
54.60
26.92
23.22
24.16
65.50
90.23
-01-1
99:00
55. 66
46
02
StcrAtication
20.46
43:07
52.82
48.86
43.54
26.32
29
22.88
26.37
61.50
42.51
4.50
54.00
40,09
39.79
Quest
44.01
$3.99
58.08
96
38.84
32.90
34. 50
26.80
70.00
88
39:42
11.16
99.50
51.65
42.03
33
AtrettinPredicttr
27.53
43.92
36.88
51.78
45.30
27.00
30:21
24.93
26.54
67.30
92.28
00:45
3.50
99.50
59.11
46.81
47.06
StreamingI.I.M
26
21.55
64:00
89.42
40.65
04.50
56.
46
43.35
Cuscade
27.14
37.17
42.51
53.05
37.75
25.73
26. 24
22.25
25.61
58.50
89 89.70
40.73
6.50
96.00
46.85
38.27
42.16
I120+
26:24
36.
48:47
$3.97
45.76
27.08
26. 11
23.59
25.05
53.00
88.54
40.88
9:05
99.50
58.2
89:22
44.25
1024
Snapky
27.63
41.52
52.37
55.83
46.39
27.09
26. 26
24.15
25,89
70.00
90166
40.76
5.12
99.50
53.55
49.55
46.37
SterAtieation
23.64
46.13
54.41
54.61
38.78
28.13
31.47
24.25
26.69
65.50
29:21
41.79
6.00
82,00
43.97
35:49
43.38
Quest
27.46
43.58
54.95
57.22
44.5%
20.86
33.64
25.37
26.68
72.00
85.61
40.05
9.55
99.50
$3.92
41.82
46.92
Attestionrredittrr
28.30
46.66
55.81
56.87
50067
30.10
32.34
25.05
26.66
74.00
92.11
42.15
7.17
100.00
59.12
S0.26
48.58
StreamingI.I.M
25.79
32:35
45.03
40.84
36.77 71
23.20
20
23.45
26.83
69:00
86-87
38.51
7.25
83.00
43.32
46 66
41.64
Cascade
25.09
40.74
46.05
51.86
45.54
26.55
25. 98
22.42
26. 89
63.50
90.55
41.35
10.50
94.00
47.03
39.11 1.3
43.96
1120+
25.07
40.35
53:22
54.81
44.75
31.77
28.3
23.50
26.37
64.50
90.71
41.32
6.71
100.00
$1.97
89 19
46.34
2048
Smapk
25.00
45.34
54.98
57.97
45.89
20.50
28.92
24.29
26.32
71.50
W9.36
40.71
0.64
300.00
58.79
49.90
47.44
ScerAtteation
27.56
46.05
55.70
15.83
47.07
29.09
32.02
24.25
26.82
68.50
88.24
40.14
6.00
93.50
46.94
37.74
45.34
Quest
20.25
46.85
$5.90
57.5 91
45.51
30.83
34,05
24,99
26.93
74,00
92.41
41.77
8.50
100.00
$4.29 45.72
45 00
ArteufiomPrediettrr
29.52
45.72
$4.30
$6,50
26.95
33.45
28
24.68
26.85
72.00
91.79
42.20
10:05
99.50
57.47
49.09
483.33
StreamingI.LMM
25.88
43.45
55.55
40.90
37.78
28.79
25.
22.15
26
62.00
75.53
36.32
8.29
74.00
58,63
45.91
42.63
Cuscade
28.45
46.21
50.77
55.02
47.60 68
30.29
31.47
23.75
26.93
65.50
90.03
41.70
S.67
94.50
47.14
40.02
45.57
H20+
28:37
45.33
55.72
56.27
74
31,63
30.91
24.15
26.89
65.50
90.96
41.97
7.06
99.50
57.52
49.70
47.45
4996
44.03
55.40
30.86
31
24.73
72.00
90.31
40.58
99.50
57.75
49.05
47.84
SecrAtieation
30.43
46.06
56.00
45.75
31.08
32
24.89
68.50
89.44
42.94
99.50
46.13
39.46
46.39
54.66
85.88
45.06
AtteationPredicttr
47.07
24.76
C.2 Additional Results on Longbench Benchmark
Due to space constraints, we presented the test results after aggregating task types in Section 4.2 Here, we present the individual results for all tasks. The (Table 8 shows that our method surpasses the majority of the SOTAS on most tasks, and the average performance under all budgets and L.MMs exceeds all compared methods, demonstrating the effectiveness of our approach.
SecrAttention utilizes a learnable module to aid cache compression. We use the officially released ScerAttention model for our tests. Since the official pre-trained model of SecrAttention for LongChat is unavailable, we evaluate its performance on LLaMA-3. 1--.A-...-----Im- In--truut
Table 8 indicates that our method outperforms SeerAttentin,, which also employs a learnable module. In particular, with a cache budget of 512, our method achieves an average accuracy 7.27% higher than SeerAttention. This could be attributed to SterAttention encoding keys and queries without directly modeling attention scores, thus failing to accurately retrieve important token blocks under extreme resource constraints. Moreover, by employing a single unified model for all LLM layers, our
20

---

approach results in a total model size that is merely 0.02% of SeerAttention's 's further attesting to the effectiveness of method.
C.3 Results on InfiniteBench
To evaluate the effectiveness of our proposed method, we conduct experiments on selected tasks from the InfiniBench [46] benchmark, an extremely long-context benchmark with an average context length 214K. We use LLaMA-3-8B as the base model and focus on the Math Find and Choice tasks, which require numerical reasoning and multi-option selection, respectively. We compare our approach against several state-of-the-art KV compression baselines under an 8K token KV cache budget.
As shown in Table 9, AttentionPredictor achieves the highest average performance (44.9), outper- forming Quest (43.4), SnapKV (42.5), and InLLLM (33.7). Notably, on the Math Find task, which is particularly sensitive to the preservation of precise attention information, our method attains 34.3 significantly higher than all baselines. These results suggest that AttentionPredictor is more effective at identifying and preserving the most relevant attention information under constrained memory, enabling better retention of long-range dependencies.
Table 9: Evaluation on InfiniteBench.
InfiniteBench with Llama-38B
KV Budget
Method
Math Find Choice Average
Full
Full cache
21.7
44.1
32.9
8K
InfLLM 32
23.7
43.7
33.7
8K
SnapKV 13
27.4
57.6
42.5
8K
Quest 31
32.3
54.5
43.4
8K
AttentionPredictor
34.3
55.5
44.9
C.4 Results on RULER QA Benchmark
We further evaluate our method on RULER QA benchmark 47, which designed to assess long-context understanding under varying context lengths and constrained KV cache budgets. The results are summarized in Table 10 The results demonstrate that with a highly constrained budget of only IK tokens, our AttentionPrediccoor consistently and significantly outperforms the SnapK baseline across all tested context lengths.
Table 10: Evaluation results the RULER QA benchmark with different input context lengths with Llama-3. --B-a-..-------tterre
RULER QA Benchmark with Llama-3.1-88 --8--.111--t--rreee Instructt
Context I Length
Method
KV Budget
4K
8K
16K
Full cache
Full
75.0
73.0
69.0
Snap KV
IK
63.7
71.7
68.0
PrtentoonPreddcctrr
IK
73.4
71.9
68.8
C.5 Results on MMLU Benchmark
To further assess the effectiveness of our approach, we conduct experiments on the MMLU benchmark Since the default evaluation mode in the official Im-eval library is single-step prediction task, does not fully reflect the benefits our method, which is designed to optimize multi-step decoding. Therefore, adopt the cot fewshot configuration, which applies n-shot Sfain-of-Thoughh (CoT) prompting strategy and involves a multi-step reasoning process. This setup provides a more suitable testbed for evaluating our decoding-stage optimizations.
449.
As shown in Table IT our proposed AttentionPredictor achieves performance very close that the full-cache model while consistently outperforming the SnapKV baseline under different KV cache
21

---

budgets. These results demonstrate that our method remains effective even on relatively short input tasks, provided that the decoding process requires multi-step reasoning.
Table 11: Evaluation results the MMLU benchmark with few-shot CoT.
MMLU Benchmark with Llama-3.1 Hama-3.1-8B-Insttee Instrruttttt
Budget
Method
512
1024
Full cache
67.17
67.17
Snap KV
65.60
66.17
AttentionPredictor
66.27
66.64
C.6 Results on GPQA Benchmark
To further validate our approach on expert-level tasks, we also tested on the GPQA benchmark 500 GPQA is challenging dataset of graduate-level multiple-choice questions curated by domain experts across biology, physics, chemistry, and philosophy. is similar in format to MMLU but significantly more difficult, requiring deep expert-level reasoning.
As shown in Table 12 our method again proves effective, maintaining high accuracy in a scenario requiring deep reasoning.
Table 12: Evaluation results on the GPQA benchmark with n-shot COT.
Method
Budget
5-shot
10-shot 15-shot
Average
Llama-3.1 1---e...--8--Inttrue
Full
31.31
37.88
31.31
33.50
SnapKV
1K
22.73
29.80
23.23
25.25
AttentionPredictor
IK
31.31
33.33
31.82
32.15
C.7
Prediction Model
Our prediction task can be framed as modeling spatio-temporal information inherent in the token sequences, problem for
Table 13: Prediction accuracy and parameter numbers of different model implementations.
which various architectures have been explored 43
57 58
We evaluate the performance of various prediction model plementations. MLP is applied to the sequence dimension to handle variations in token length. For LSTM, attention is di- vided into 16 width blocks and predictions are independent for each block, referred to as LSTM-bloce in Table 13] Similarly, CNN-block utilizes block-wise prediction approach, employ- ing a fully connected layer for fixed block lengths. Finally, the CNN model, the primary setting in experiments, predicts all attention scores simultaneously. As reported in TTalle 13]
BongBench+Longchhe
Model type
Total params Recovery rufe
LSTM-block CNN-block
3.2M
91.75
21K
91.94
CNN
4.9K
CNN outperforms other models, achieving the highest attention prediction accuracy by effectively capturing attention patterns. ML. failed to account for neighboring token interactions, while LSTT-- block and CNN-block were restricted to block-level information without global context. Notably, CNN also required the fewest parameters and was the most memory-efficient during inference.
C.8 Ablaioon Study Results of Hyperparameters
We evaluate the impact of hyperparameters on our method. Specifically, we train the predictor using the default parameters of the main experiment, i.e., b- 16, 64. We employ full attention without sparse computation during training, which is equivalent to performing distribution eтror calibration each step, i.e., M 1. Subsequently, we test the performance under different hyperparameters using the trained model. Other parameters during testing remain consistent with the main experiment, with KV budget of 1 1.. We evaluate six representative tasks Longbench. The results are shown in Table 14
22

---

Calibration step M. Table 14 shows the average performance of AttentionPrediitor with calibration steps ranging from to 20. Performance improves with shorter calibration intervals, indicating that the calibration scheme reduces cumulative errors caused by differences between sparse and original attention distributions. However, higher calibration frequencies increase computational costs, creating a trade-off between accuracy and efficiency.
History step Il. As depicted in Table 14, we evaluate the average accuracy of Attentionnredicttor over a range of history steps. Overall, the performance gap compared to the full cache remains consistently small. Notably, the middle value of H maximizes performance by providing the necessary information for pattern recognition, while mitigating redundancy caused by the decaying self-correlation of attention scores.
Table 14: Results of AttentionPredictorr with different hyperparameters.
LongBench+Longchatt
Hyperprameter
SQA
MQA
Summary
Few-shot Synthetic
Code
Average
MF-en
HotpotQA
QMSum
TriviaQA
Pre
Lec
Full
33.05
83.99
30.50
$2.94
44.39
41.83
34.00
22.22
84.45
28.00
51.87
43.73
History
32
41.60
33.90
22.32
84.81
30.00
$2.53
44.19
Step
64
41.67
33.64
22.30
84.85
28.00
52.22
43.78
128
41.51
34.39
22.45
84.35
26.00
52.96
43.61
42.89
33.27
22.35
84.70
28.50
52.53
44.04
Calibration
42.47
33.61
22.43
84,90
28.00
52.13
43.92
Step
5
41.67
33.64
22.30
84.85
28.00
$2.22
43.78
10
41.10
33.87
22.37
84.85
28.00
52.36
43.76
20
41.54
33.99
22.12
84,90
28.00
$2.50
43.84
43.09
34.61
22.38
85.29
28.00
52.76
44.36
Block
16 41.67
33.64
22.30
84.85
28.00
52.22
43.78
Size
32
40.85
34.47
22.46
84.87
25.00
$3.02
43.45
64
42.60
34.04
22.01
84.85
19.00
52.86
42.56
C.9 Necessity of the Learnable Attention Predictor
pertinent question arises regarding the necessity of a learnable attention predictor: given the potential for high similarity between attention distributions of adjacent tokens layers, could one simply utilize the attention map from a preceding token or layer as a direct estimate? To address this, we conducted comparative experiment evaluating our proposed AttentionPPediictrr against two such heuristic baselines: (1) Previous-Layer Attention, which uses the attention map from the corresponding position in the immediately preceding layer, and (2) Previous- Token Attention, which employs the attention map of the immediately preceding token within the same layer.
Table 15: Performance comparison LongBench with Llama3. 1. 11-8B-IInstunn
Single-Decumen D
Multi- A
Summary
Few-shot Learning
Santhelic
Code
Budget
Method
TRIE
Averugef
Full eache
29.25
56.
56.67
49.00
32.43
33.77
27.08
74,00
1024
Previous layer
26.04
46. 17
46
26
36.27
21.94
25.81
64.00
83.
10.75
60
58
42.51
1124
Presious token
28.90
34.34
55.37
55.63
48.66
30.30
27.36
23.45
25.73
69.50
$3.95
43.01
3,50
99.56
56.96
47.40
46.82
1074
AttentionProddeeeer
28.50
47.35
54.99
57.53
45.91
31.63
32.52
24.51
26.78
73,00
80.54
42.12
6.49
99.56
56.96
47.43
47.80
The results of this evaluation, performed on the LongBench benchmark using Llama3.1 1-B- -IIntruut presented in Table 15 Employing previous- attention yields an average score of 42.51 1,,
which is a substantial 5.29% lower than our AttentionPredictor' average of 47.80% This performance decrement underscores the limitation of direct cross-layer attention transference, primarily because attention patterns often exhibit considerable variation across different layers, making the previous layer's 's attention less reliable proxy for identifying critical tokens in the current layer.
The previous-token attention strategy achieves a higher average score of 46.82% This is 4.31% % better than the previous-layer approach and aligns with the observation that attention mechanisms frequently display sequential patterns along the token dimension within the same layer. However, despite this improvement, relying solely on the single immediately preceding token is still suboptimal.
23

---

Our learnable AttentionPredictor,, with an average score of 47.80%, outperforms the previous-token attention baseline by 0.98%. This superiority stems from our model's capacity to learn and leverage more complex, dynamic attention patterns from several previous tokens, rather than being confined to the heuristic of the single last token, which may not adequately capture evolving contextual dependencies.
Therefore, these comparative results affirm that while attention similarities exist, simplistic heuris- tic estimations are insufficient. The empirical evidence highlights the necessity and distinct advantage of our proposed learnalle AttentionPredictor for achieving more accurate anticipation of attention distributions.
C.10 Comparison between MInference and AttentionPredictor
To further evaluate the effectiveness of our proposed AttentionPredictor, we compare its performance against MInference [16]. MInference operates by categorizing attention heads into three predefined sparse patterns (A-shape, Vertical-Slash and Block-Sparse) and uses kemel-aware search to assign the optimal pattern each head offline. During inference, it dynamically builds sparse indices based on the assigned pattern and the specific input.
MInference (prefill): The standard version where the pattern-fitting algorithm is run only during the pre-filling stage. (2) MInference (decode): A stronger, more dynamic variant where the computationally expensive pattern-fitting algorithm is re-run at every decoding step to adapt to new contexts.
We consider two variants of MInference for a comprehensive comparison: (1)
AttentionPredictorr consistently outperforms both MInference variants across all tasks, achieving an average accuracy of 94.25%. This demonstrates that by learning from recent history instead of relying fixed pattern templates, AttentionPredictor more accurately captures the dynamic nature of attention during decoding.
As shown in Table 16
Table 16: Attention prediction accuracy (%) on Llama-3 3.1 11-8B-Innsrune with 1K KV budget.
Method
QA
Summary Math Average
MInference (prefill)
90.51
84.45
93.47
89.48
MInference (decode) 92.18
88.74
94.67
91.86
AttentionPredictor 94.95
91.72
96.10
94.25
C.11 Top-K Selection Strategy Analysis
Why Top-K Outperforms Sampling? While prior works such as MagicPIG [59 suggest that Top-K K may introduce bias certain tasks, our experiments reveal critical insight: prediction accuracy matters more than the design of the selection policy. As shown in Table 17 we compare the performance of our AttentionPredictt and MagicPIG on the LongBench benchmark using Llama- 3.1-8B-Instruc (MagicPIG's results taken from the original paper). The results demonstrate that when combined with our high-precision AttentionPredictor,, even a simple Top-K strategy achieves superior performance compared to MagicPIG's 's more sophisticated sampling approach. This indicates that high-qualit attention predictions can enable basic selection policies to outperform more complex alternatives. Furthermore, it is important note that our AttentionPredicea is decoupled from the selection policy and can be flexibly integrated with various methods, including both Top-K K and importance sampling.
Table 17: Longbench task results on (.lma-..1-8B----
with 10% KV budgets.
Method
Qasper RB-P
Lee
Pre
TREC TriviaQA
Average
Full Cache
45.36
48.15
56.86
99.50
74.00
91.48
69.22
MagicPig
43.96
46.45
57.06
100.00
74.40
91.38
68.87
AttentionPredictor
45.50
50.26
59.41
99.67
74.00
88.61
69,58
24

---

D Additional Experimental Results on Efficiency
D.1 Training Efficiency
We evaluate the impact of the training data ratio prediction accuracy on the hotpotqa dataset of Longbench. The predictor's 's performance improved only slightly, from 96.0% 97.5%, as the proportion of training samples increased from 3% to 70%. This efficiency can be attributed to two factors. First, the predictor only needs to capture the relative patterns rather than precisely predict attention scores, which is casier. Second, since attention patterns are inherent characteristics of the LLM, the predictor exhibits strong generalization capabilities. These findings suggest that a relatively small amount of training data is adequate for the predictor to effectively learn the underlying attention dynamics.
Table 18: Accuracy of predictor with different training sample ratios. The test set is a 20% split from the datasets.
Training Samples Ratio 0.03 0.05 0.1 0.2 0.5 0.7
Accuracy (%) on Test Samples 96.0 96.2 95.9
96.6
97.2 97.5
D.2
Computational Overhead of AttentionPredictor.
We have quantified the inference overhead of our predictor model. As shown in Table 19 both the latency and memory footprint are minimal, ensuring that the prediction process does not become a bottleneck.
Table 19: The computational overhead of the prediction model.
Context Length Prediction Latency (ms) Peak Memory Usage (MB)
4K
0.28
97
8K
0.47
194
16K
0.91
388
32K
2.22
776
D.3 Memory Efficiency of Cross-Token Prefetching
We have evaluated the memory footprint of our proposed cross-token prefetching framework during decoding. As shown in Table 20 the framework consistently consumes less GPU memory compared the standard cross-layer approach. This advantage arises because the memory savings from loading only a small, sparse KV cache outweighs the potential overhead from the cross-token prefetching mechanism.
Table 20: Peak GPU memory consumption (GB) of L.L.A-3.1--3 during decoding.
Context Length Standard Cross-Layer Our Cross-Token Memory Reduction
4K
32.9
30.1
8%
8K
35.9
30.3
16%
16K
41.9
30.5
27%
32K
53.9
31.1
42%
E
Limitations
Our current investigation is primarily centered specific range of LLM architectures and sizes, and the generalizability of observed attention patterns to vastly different model types warrants further study. The necessary conditions for specific attention pattern can be explored in the future work.
25

---

Boarder Impacts
Our study of LLM attention score patterns and their predictability may offer insights into attention dynamics, potentially informing future research on these characteristics. By exploring attention score patterns, this work could also be relevant advancements in KV cache compression, possibly contributing to the development of more memory- model designs.
Academic Impact:
Through the potential implications for KV cache compression via attention pre- dictability, this research could contribute to more efficient LLM inference. This might lead to reduced latency in Al applications and lower energy consumption. Such improvements could potentially enhance the accessibility of LLMs and support more sustainable Al practices.
Societal Impact:
G License
We include the following licenses for the code, models, and benchmarks we used in this paper.
Benchmarks:
LongBench 45
License
AIME 2024 48
License GSM8K [60] License Needle
In A Haystack [51]: License InfiniteBench [46]
License
Models: -LLMA-3.1-8B-In-t-
[54]
License LongChat (53 License
Code: H2O 11
License
Minference [16 License, SeerAttention
14 License
26

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
This paper analyzes the temporal patterns in the attention score and proposes AttentionPredictorr to predict attention scores and dynamically identify critical tokens. Moreover, a cross-token prefeching system is proposed to accelerate the decoding stage. Experiments long-context benchmarks shows AttentionPredietor achieves 13x cache compression with comparable performance.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. It is fine to include aspirational goals as motivation long it is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
The limitations of our AttentionPredictor are detailed in Appendix
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged
the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
27

---

Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification:
The theoretical results are given in Section S
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
Justification:
The implementation of our approach are detailed in Section 3 and Appendix B
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open- open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
28

---

5. Open access to data and code
Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Question:
Answer: [Yes]
Justification:
We provide the code and dataset in the supplemental material.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://mips.co
for more details.
While encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https for more details.
7 //nips. htcp//blcc//uudeeeeeeeeeeeeeeeee
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: The details about experiment setting are in Section 4.1 and Appendix
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification:
The results are in Section C.8 and Section 4.6
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi-
dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
29

---

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out of range (c.g. negative emror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
The details about hardware specification are in Section 4.1
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification:
We have reviewed and conformed with the Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: The details of broader impacts are included in Section
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
30

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Question:
Answer: [NA]
Justification: Our model and data are focused on KV Cache compression, with no risk of misuse.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: Licenses in Appendix G
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the Datasets.papersstt
package should be provided. For popular datasets, paperswi thcode. cow/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
31

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
The details of the model detailed in this paper.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/cooe/mmoel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question:
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
32

---

Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, or originality the research, declaration is not required.
Answer: [NA]
Justification: The core method development in this research does not involve LLMs any important, original, or non-standard components.
Guidelines:
The answer NA
that the method development in this research does involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy for what should or should be described.
33


Reviews:
[{"review_id": "15OWG0QQxL", "reviewer": "Reviewer_spBc", "summary": "The paper presents AttentionPredictor, an innovative approach for KV cache compression in large language models (LLMs) by predicting next-token attention scores during the decoding phase. The method employs a lightweight convolutional model to identify and retain only critical tokens, significantly reducing memory usage while maintaining model performance. Additionally, the authors propose a cross-token prefetching framework to hide the computational overhead of attention score prediction. The approach achieves impressive results, including 13$\\times$ KV cache compression and 5.6$\\times$ speedup while maintaining comparable performance to full cache inference.", "questions": "See weaknesses.", "limitations": "See weaknesses.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 4, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1) The idea of learning-based attention predictor is novel. It also surprises me that the learned predictor generalizes well on unseen datasets.\n2) The unified convolutional model is remarkably lightweight (only 21KB) compared to alternatives like SeerAttention (101MB), making it practical for deployment.\n3) Comprehensive evaluations across multiple benchmarks demonstrate consistent performance improvements over state-of-the-art methods.\n4) The paper is well-motivated and clearly written.\n\nWeaknesses:\n1) It is a little confusing that since we get the indices of critical tokens from the attention predictor (see Algorithm 1), does it mean we need to keep all KV caches? If so, it is inappropriate to claim it is a method for KV cache compression.\n2) While the training process is described in Appendix B.1, more details about the CNN architecture (number of layers, kernel sizes, etc.) would be helpful.\n3) The computational overhead of the prediction model itself could be better quantified, especially since it runs at every decoding step.", "was_revised": true, "final_justification": "The authors have adequately addressed my concerns about the terminology, the architecture and the overhead of the proposed method. Therefore, I will keep my score.", "rebuttal": "We thank the reviewer for the insightful and valuable comments. We are encouraged by your positive feedback on the novelty, generalization, and lightweight nature of our approach. We respond to each comment as follows and sincerely hope that our rebuttal could properly address your concerns.\n\n### W1: Clarification on the term \"KV Cache Compression\".\n> It is a little confusing that since we get the indices of critical tokens from the attention predictor (see Algorithm 1), does it mean we need to keep all KV caches? If so, it is inappropriate to claim it is a method for KV cache compression. \n\nThank you for this insightful question. **You are correct** that **we maintain the full KV cache in CPU memory**. Our method achieves \"compression\" by **involving only a subset of the KV cache in the attention computation**, effectively reducing the active GPU memory footprint, which is the primary bottleneck during inference.\n\nTerms such as \"**KV cache retrieval**\" or **\"attention sparsity**\" may describe our method more precisely. We will revise the terminology in the final version of the paper to ensure greater clarity.\n\n### W2: Details of the CNN Architecture.\n> While the training process is described in Appendix B.1, more details about the CNN architecture (number of layers, kernel sizes, etc.) would be helpful. \n\nThank you for the suggestion. To provide a clearer and more quantifiable overview of our CNN architecture, we will add the following detailed breakdown to the paper's appendix.  **The architecture is in Table 1**. This design choice makes the model lightweight and adaptable to varying sequence lengths.\n\n***Table 1**: AttentionPredictor Structure.*\n| Layer No. | Layer Type | Details |\n| :--- | :--- | :--- |\n| 1 | 2D Convolution | **Kernel**: (3, 3), **Padding**: 1, **Channels**: 1 → 16 |\n| 2 | ReLU Activation | - |\n| 3 | 2D Convolution | **Kernel**: (3, 3), **Padding**: 1, **Channels**: 16 → 32 |\n| 4 | ReLU Activation | - |\n| 5 | Adaptive Avg. Pooling | Collapses the temporal history dimension to size 1 |\n| 6 | 1D Convolution | **Kernel**: 1, **Channels**: 32 → 1 |\n\n\n\n### W3: Computational Overhead of the AttentionPredictor.\n> The computational overhead of the prediction model itself could be better quantified, especially since it runs at every decoding step. \n\nThank you for raising this important point. To clarify, the predictor **does not run at every decoding step**. To balance performance and overhead, we invoke the predictor periodically. In our experiments, we **update the critical token set only every two steps**, which effectively amortizes the prediction cost.\n\nWe have quantified **the overhead of our predictor model**. As shown in **Table 2**, both the latency and memory footprint are minimal, ensuring that the prediction process does not become a bottleneck. We will include this analysis in the final paper.\n\n***Table 2**: The computational overhead of the prediction model.*\n| Sequence Length | Prediction Latency (ms) | Peak Memory Usage (MB) |\n|---|---|---|\n| 4K | 0.28 | 97 |\n| 8K | 0.47 | 194 |\n| 16K | 0.91 | 388 |\n| 32K | 2.22 | 776 |\n\nAdditionally, **Table 3** shows a detailed breakdown of our cache prefetch system. \nOur cross-token prefetching framework is illustrated in the timeline in **Figure 10 (Appendix B.1)**. \n\n***Table 3:** Computation breakdown at 1/16 budget (in ms/token).*\nContext Length|Attention Prediction|Cache Transfer|Sync/Wait for next token|Total Overhead\n-|-|-|-|-\n4K|1.4|2.3|73.7|77.3\n8K|2.3|3.9|72.3|78.6\n16K|4.6|7.6|67.3|79.6\n32K|9.1|13.2|57.7|80.0\n\nThe \"Attention Prediction\" latency in Table 3 is greater than the values in Table 2 because it encompasses not only the prediction model inference but also additional matrix operations essential for the prefetching process. \n\nNote that since we perform attention prediction once every two tokens, the average per-token inference latency is further reduced."}, {"review_id": "3dL9QkiyUN", "reviewer": "Reviewer_gSQC", "summary": "The paper addresses the high memory and latency costs of maintaining the full Key–Value (KV) cache during long-context decoding in large language models. It proposes AttentionPredictor, the first learning-based method for predicting future attention scores via a lightweight, shared convolutional network. The method captures patterns in past attention history to identify critical tokens for KV cache retention. When combined with cross-token prefetching, which overlaps prediction and I/O, the approach maintains accuracy close to that of the dense model under high compression, resulting in significant generation speed-ups.", "questions": "Clarifications:\n* Line 35: Could you clarify what is meant by static patterns? My interpretation is that these are patterns constant across inputs, but methods like H2O are dynamic, selecting different critical tokens per input. If so, I don't understand this point.\n* Lines 36–38: What exactly are dynamic patterns? What does it mean to say \"they only represent keys or hidden states\"? Based on this, I don’t fully understand why this motivates modelling the distribution of attention scores.\n\nRequests for Additional Discussion:\n* Could you more clearly describe the efficiency experiment setup, and include a comparison where offloading is not used?\n* Could you explicitly discuss how token prefetching affects memory requirements?", "limitations": "No, the authors didn't properly addressed the limitations of their work which I discuss in Strengths and Weaknesses.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n* The paper proposes an efficient, learning-based method to predict attention scores, which performs well on downstream tasks and in CoT settings.\n* The authors discover a periodic seasonal pattern in attention score distributions.\n* Cross-token prefetching is used effectively to hide the overhead introduced by the AttentionPredictor.\n\nWeaknesses:\n\nMotivation and Contributions:\n* In Section 3.1, you state that \"attention recovery rate reflects the amount of information preserved after compression.\" Could you please define information formally here? The goal of KV compression is arguably to minimize the error in model output caused by compression, which may or may not correlate with attention recovery rate. MagicPig (Chen et al., 2024), in Appendix F, discuss why Top-K is a biased estimator of attention output.\n* You claim to discover temporal patterns in attention scores, but MInference, which you cite, has already identified the sequential and re-access patterns. In Lines 91–93, you write: \"They struggle to capture the dynamic temporal patterns within attention scores accurately.\" Could you provide evidence to support this claim?\n* Some theoretical explanations are unclear. In Lines 190–196, could you please further elaborate further how the formulation of RoPE leads to consistent attention along diagonals and how the periodicity of the cosine function explains the periodic slashes of the sequential patterns?\n\nEfficiency Claims and Measurements:\n* The abstract claims a 5.6× speed-up and states that the method \"significantly outperforms state-of-the-arts,\" suggesting a general breakthrough. However, this speed-up is achieved in a specific cache-offloading setup, which is not clearly defined or explained in Section 4.3.\n* The memory impact of cross-token prefetching is not discussed. My understanding is that prefetching the KV cache for the next token prediction increases GPU VRAM usage. If so, this would be unfair compared to baselines that use cross-layer prefetching.\n* There is no compute breakdown provided to understand the overhead caused by the AttentionPredictor.\n* The provided code is incomplete and only includes the implementation of the proposed method.\n\nPerformance Claims:\n* In Line 289, the authors claim that their method surpasses the performance of other methods. However, the paper lacks statistical analysis, which is required to establish the significance of the results.\n* The same issue applies to the CoT reasoning results. This is especially important given that (unless I'm mistaken) the benchmark uses a small number of samples.\n\nConference Checklist:\n* Q4–Q5: The authors do not release the code required to reproduce baselines, which limits understanding and reproducibility.\n* Q7: Standard deviations and statistical significance is not discussed.\n* Q8: Experimental setup details in Section 4.1 are minimal. The only information provided is: \"We conducted experiments on NVIDIA A800 (80GB) GPUs.\"", "was_revised": true, "final_justification": "Authors clarified all my concerns in the rebuttal. Thanks for that. I'm raising my score from 2 to 4. I'm keeping it at 4 because the work targets the KV off-loading setup which is more niche, therefore less significant.", "rebuttal": "We thank the reviewer for the insightful and valuable comments. We respond to each comment as follows and sincerely hope that our rebuttal could properly address your concerns. \n## Weaknesses\n### Motivation and Contributions\n> W1: Define information formally.\n\nWe thank the reviewer for this insightful comment. \n1. **On \"Attention Recovery Rate\" as a Metric:**\nWe agree minimizing output error is the ultimate goal. We argue that **attention recovery rate serves as an effective proxy**, as attention output $o = A_t · V$ depends directly on the attention scores. A high recovery rate means the compressed distribution $A_t'$ closely matches $A_t$, leading to an accurate $o' \\approx o$. This justifies its use to assess information preservation.\n2. **The primary contribution is the $AttentionPredictor$**, a universal module that predicts the next-token attention distribution. It is distinct from the selection policy and **can be paired with various methods, including Top-K or importance sampling**.\n3. **Top-K vs. Sampling & Why Our Method Excels:** While Top-K can be biased in some tasks (as noted by MagicPIG), our results show that **a precise predictor matters more than an unbiased selection policy**.\nOur AttentionPredictor captures spatiotemporal patterns to better estimate attention, so even with simple Top-K, **it outperforms MagicPIG’s complex sampling (Table 1)**, highlighting the predictor’s key role.\n\n***Table 1**: Longbench task results with 10% KV budgets.*\nMethod|Qasper|RB-P|Lcc|Pre|TREC|TriviaQA|Avg.\n-|-|-|-|-|-|-|-\nLlama-3.1-8B-Instruct|45.36|48.15|56.86|99.50|74.00|91.48|69.22\nMagicPig|43.96|46.45|57.06|100.00|74.40|91.38|68.87\n**AttentionPredictor**|45.50|50.26|59.41|99.67|74.00|88.61|**69.58**\n\n\n> W2: Provide evidence for the claim that MInference struggles to capture dynamic temporal patterns.\n\nThank you for the thoughtful question.\n1. **Temporal patterns are locally stable but evolve globally** during long inferences—e.g., shifting diagonal positions. Furthermore, we identified a **novel seasonal pattern**, where attention periodically shifts between token groups, a dynamic behavior not defined by MInference's categories. \n2. **Why MInference struggles to capture** these dynamics accurately:\n   * **MInference** uses **fixed templates per head**, making it hard to adapt to such evolving or periodic structures.\n   * In contrast, our **AttentionPredictor is fully dynamic** and learns these transitions from attention history. \n3. **Table 2** shows a **4.77% higher accuracy**, supporting our claim that modeling these dynamics leads to better attention prediction and information retention.\n\n***Table 2**: Attention prediction accuracy (%) on Llama-3.1-8B-Instruct with 1K KV budget.*\nMethod|QA|Summary|Math|**Average**\n-|-|-|-|-\nMInference|90.51|84.45|93.47|89.48\n**AttentionPredictor**|**94.95**|**91.72**|**96.10**|**94.25**\n>W3: How does RoPE lead to diagonal consistency and periodic slashes in attention patterns?\n\nThank you for the great question.\n1. **Consistent Attention along Diagonals**\n\nConsider the attention scores for two diagonal tokens, $A_{t,i}$ and $A_{t+1, i+1}$. Using RoPE, the score is:\n$$A_{t+1, i+1} = \\sum_{m=1}^{d/2}||q_{t+1}^{(m)}||||k_{i+1}^{(m)}||\\cos(\\phi'^{(m)}+(i-t)\\theta_{m})$$\nNote that $(i+1)-(t+1) = i - t$, so **the relative position stays the same** along the diagonal. Since RoPE encodes positions via relative angles, the cosine term is unchanged. The other terms in the sum ($||q||$, $||k||$, and the initial phase $\\phi$) vary slowly across time due to self-similarity (Sec 3.3), so overall:\n$$A_{t,i} \\approx A_{t+1, i+1}$$ \nThis leads to **stable attention scores along diagonals**.\n\n2. **Periodic Slashes**\n\nRoPE attention scores are sums of cosine functions over relative positions: \n$$A_{t,i} = \\sum_{m=1}^{d/2} C_m \\cos\\big(\\phi^{(m)} + (i-t)\\theta_m\\big)$$\nEach term is periodic with period $T_m = 2\\pi / \\theta_m$. If the sum is dominated by a single frequency $\\theta_{m*}$, **the attention score approximates a single cosine function**:\n$$A_{t,i} \\approx C_{m*} \\cos(\\phi^{(m*)}+(i-t)\\theta_{m*})$$\nThis causes attention to **peak at regular intervals in key positions**, forming **periodic slash patterns** with spacing ≈ $T_{m^*}$.\n\n### Efficiency Claims and Measurements:\n>W4 & Q3: The speed-up is not under general settings. Describe the efficiency experiment setup.\n1. **Why Offloading Matters**: Our cache compression work **focuses on the cache-offloading scenario**, which is critical for long-context generation when KV cache exceeds GPU memory. Unlike GPU-based eviction/compression, **offloading keeps the full cache on CPU**, preserving model accuracy. This setting is also adopted in recent works [1–4].\n\n2. **Experiment Setup.** \n    * **FlashAttention2**: Standard offloading implementation with cross-layer prefetching. At each decoding layer, it load the entire KV cache for the upcoming layers from CPU to GPU to compute the full attention. \n    * **H2O**: Also uses cross-layer prefetching but transfers sparse KV cache, making the comparison a more direct evaluation of the prefetching framework's efficiency.\n    * **AttentionPredictor**: Runs on an offloading setup but prefetches across tokens, improving efficiency.\n3. **Applicability to General Scenarios.** While designed for offloading, AttentionPredictor is adaptable—it can act as a cache eviction policy in memory-constrained setups, predicting less important blocks to evict and speeding up decoding.\n\n[1] InfLLM: Training-Free Long-Context Extrapolation for LLMs with an Efficient Context Memory. NeurIPS 24.\n\n[2] ShadowKV: KV Cache in Shadows for High-Throughput Long-Context LLM Inference. ICML 25 (Spotlight).\n\n[3] InfiniGen: Efficient generative inference of large language models with dynamic KV cache management. OSDI 24.\n\n[4] PQCache: Product quantization-based kvcache for long context llm inference. PACMMOD 25.\n\n>W5 & Q4: The memory impact of cross-token prefetching is not discussed. \n\nOur **cross-token prefetching framework actually consumes less GPU memory** during decoding compared to a standard cross-layer approach. See **Table 3**.\nThe reason is that the memory savings from loading only a **small, sparse KV cache** outweighs the potential overhead from the cross-token prefetching mechanism. \n\n***Table 3**: Peak GPU Memory Consumption (GB) during Decoding.*\nContext Length|Standard Cross-Layer|Our Cross-Token|Memory Reduction\n-|-|-|-\n4K|32.9|30.1|8%\n8K|35.9|30.3|16%\n16K|41.9|30.5|27%\n32K|53.9|31.1|42%\n> **W6**: Lack Compute breakdown.\n\n**Table 4** shows a detailed breakdown of our cross-token prefetching process (the timeline in Figure 10, Appendix B.1). Note that since we perform attention prediction once every two tokens, the average per-token inference latency is further reduced.\n\n***Table 4:** Computation breakdown at 1/16 budget (in ms/token).*\nContext Length|Attention Prediction|Cache Transfer|Sync/Wait for next token|Total Overhead\n-|-|-|-|-\n4K|1.4|2.3|73.7|77.3\n8K|2.3|3.9|72.3|78.6\n16K|4.6|7.6|67.3|79.6\n32K|9.1|13.2|57.7|80.0\n\n>W7 & W10: Lack baseline codes. \n\nWe used the official implementations released by the original authors to reproduce the baseline results.\n\n>W8 & W9 & W11: Lack statistical analysis.\n\nWe ran our method 5 times and the results are reported in **Table 5** and **Table 6**. Our method demonstrates both strong performance and consistency, outperforming other baselines.\n\n***Table 5**: LongBench results with LLaMA-3.1-8B-Instruct (Budget=1K)*\nMethod|Qasper|HotpotQA|QMSum|TriviaQA|Pre|Lcc|Avg.\n-|-|-|-|-|-|-|-\nFull Cache|45.36|56.67|25.20|91.48|99.50|56.86|62.51\nSnapKV|41.86|55.83|24.06|90.66|99.50|55.55|61.24\nQuest|43.58|57.22|25.37|88.61|99.50|53.92|61.37\n**AttentionPredictor**|**45.50 ± 0.94**|**55.40 ± 1.29**|**25.24 ± 0.13**|**91.71 ± 0.34**|**99.67 ± 0.29**|**59.41 ± 0.21**|**62.82 ± 0.35**\n\n***Table 6**: AIME-2024 CoT reasoning task*\nBudget|Method|Acc. (%)\n-|-|-\nFull|Full Cache|80.0\n2048|AttentionPredictor|**76.7 ± 2.8**\n>W12: Q8: Lack experimental setup details.\n\nWe will provide comprehensive information on the hardware specifications (CPU: Intel(R) Xeon(R) Gold 6330 CPU @ 2.00GHz, RAM: 3.9Ti, OS: Ubuntu 22.04) and software environment (PyTorch: 2.4.0, CUDA versions: 11.8).\n## Questions\n>Q1 & Q2: What is meant by static patterns and dynamic patterns? \n\nThank you for your insightful questions. \n1. **Static Patterns & H2O**: \n   * *Static patterns* are attention behaviors based on fixed rules that don’t adapt **during a single, long-output inference**.\n   * H2O uses cumulative attention scores to select tokens, favoring early, frequent ones. This helps **capture fixed-position patterns** (e.g., attention sinks) but limits responsiveness to new context later in generation.[5]\n2. **Dynamic Patterns & SeerAttention**: \n   * *Dynamic patterns* evolve throughout generation.For instance, in a sequential pattern, the position, interval, and intensity of the diagonal lines can shift over time (as in Figure 2 in our paper).\n   * SeerAttention and similar methods use learnable modules to capture dynamic patterns. However, they **predict attention distribution indirectly**. SeerAttention, for example, learns from a compressed representation of Q&K to infer the attention distribution, not the distribution itself.\n3. **Motivation**:\n   * This indirect approach can misalign with true attention scores. Our AttentionPredictor directly models next-token attention as a 2D time series, achieving much higher accuracy than SeerAttention—especially under tight budgets (Table 5, Appendix C.2). \n\n[5] Dialogue Without Limits: Constant-Sized KV Caches for Extended Responses in LLMs. ICML 25."}, {"review_id": "0HajiYvfzA", "reviewer": "Reviewer_Y2CJ", "summary": "This work finds Temporal Patterns in the attention scores of RoPE transformers. Based on this Temporal Patterns, this work use CNNs to predict critical tokens of time t+1 based on Attention scores patterns before t. This work computes the full attention score every M steps to avoid error accumulation. The author uses LongBench and simple Needle-in-a-HayStack tasks to evaluate the proposed method.", "questions": "- What is the reason of seasonal pattern? \n- What is the performance of the proposed method on other tasks? e.g. RULER qa tasks, GSM8K, MMLU?", "limitations": "yes", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "## Strengths: \n- The discovered Temporal Patterns in attention scores of RoPE transformers is interesting. \n- This work also provides a reaonable explaination of the sequential patterns from RoPE viewpoint. \n- Related works are well discussed. \n\n## Weakness:\n- The writing remains a large improvement space. E.g. line 146 mentioned the notion $A^{comp}$, but doesn't tell what is $A^{comp}$. \n- Some explaination are still questionable. E.g. line 171 High query self-similarity.  We know that softmax attention is a normalized score. This author only show the high similarity (0.87) of consecutive queries. To support the explaination in line 171, one need to also show the similarity of queries at far distance. Moreover, ther author doesn't explain the reason of Seasonal Pattern. \n- the calibration step M is 5, a small number. AKA, every 5 steps the proposed method needs to 1) compute attention scores on the whole sequence; 2) fetch KV cache of the whole sequence. The former requires high computation cost, while the later requires high memory cost. \n- The evaluation are conducted on narrow tasks, a simple needle-in-a-haystack task and a longbench task.", "was_revised": true, "final_justification": "maintain my original score.", "rebuttal": "We thank the reviewer for the insightful and valuable comments. We respond to each comment as follows and sincerely hope that our rebuttal could properly address your concerns. If so, we would deeply appreciate it if you could raise your score. If not, please let us know your further concerns, and we will continue actively responding to your comments and improving our submission.\n### **Weaknesses and Questions**\n\n> W1: The writing remains a large improvement space. E.g. line 146 mentioned the notion $A^{comp}$, but doesn't tell what is $A^{comp}$. \n\n\nThank you for your valuable feedback on our manuscript and for highlighting areas for improvement in our writing. \nWe will revise the text at its first mention.\n\n**Revision for lines 146-147:**\n\"The input to our predictor is a 2D spatiotemporal sequence \n$A_H = \\langle A_i^{\\text{comp}} \\rangle_{i=t-H+1}^t$ ,\nrepresenting the H most recent block-wise compressed attention scores, where each $A^{comp}_i$ is generated via max-pooling on the original attention score $A_i$ (as detailed in Section 3.4).\"\n\n> W2-1: Some explaination are still questionable. E.g. line 171 High query self-similarity. We know that softmax attention is a normalized score. This author only show the high similarity (0.87) of consecutive queries. To support the explaination in line 171, one need to also show the similarity of queries at far distance. \n\nThank you for this insightful question.\n\n**1. Rationale for Consecutive Query Similarity**. \n\nOur model predicts the attention at step $t+1$ from a recent history. Since the **temporal dimension corresponds to the query sequence** (one query per step), the similarity between adjacent queries ($q_t$ and $q_{t+1}$) is the most critical factor for demonstrating step-by-step temporal continuity. This is why we initially focused on the high **consecutive query similarity (lag-1) of 0.87**.\n\n**2. On Attention Normalization**. \n\nIt is also worth noting that the Softmax function normalizes attention scores **across the key dimension** for a single query; it does not influence the relationship between different queries across the temporal dimension.\n\n**3. Analysis of Query Similarity Across Distances**\n\nTo provide a more detailed analysis of long-range stability, we measure the average query cosine similarity across various distances (lags). The results are presented in **Table 1**.\nThe query similarity **gradually decreases as the distance increases**, which is expected. However, the similarity remains **relatively high even at longer distances**. This finding suggests that while the influence of the most recent tokens is strongest, the attention patterns exhibit a strong **local stability** that decays slowly over time. This supports our model's effectiveness in capturing these evolving yet persistent patterns using a recent history window. \n\n***Table 1**: Query Similarity over Different Distances.*\n**Distance**|1|2|5|10|20|50 \n:-|:-:|:-:|:-:|:-:|:-:|:-:\n**Similarity**|0.87|0.83|0.77|0.74|0.71|0.67\n> W2-2 & Q1: The author doesn't explain the reason of Seasonal Pattern.\n\nThank you. The seasonal pattern arises from combining two factors:\n1.  **RoPE's Periodicity**: Rotational embeddings are inherently periodic.\n2.  **Contextual Periodicity**: In structured data, queries separated by a period $S$ can be similar, so $q_t \\approx q_{t+S}$.\n\nThe attention score is \n$A_{t,i} = \\sum_m \\lVert q_t^{(m)} \\rVert \\cdot \\lVert k_i^{(m)} \\rVert \\cdot \\cos\\left( \\phi_{t,i}^{(m)} + (i - t)\\theta_m \\right)$. \nEach frequency $m$ in RoPE has a period $T_m = 2\\pi/\\theta_m$. We assume the input's seasonal period $S$ aligns with a dominant frequency's period ($S\\theta_{m^*} \\approx 2\\pi$).\n\nConsequently, the rotational component for the relative distance $(i-(t+S))$ aligns with that for $(i-t)$. When this RoPE-induced alignment combines with periodic queries ($q_t \\approx q_{t+S}$) and the same key vector $k_i$, we find that **$A_{t,i} \\approx A_{t+S, i}$**, which explains the seasonal pattern.\n\n> W3: the calibration step M is 5, a small number. AKA, every 5 steps the proposed method needs to 1) compute attention scores on the whole sequence; 2) fetch KV cache of the whole sequence. The former requires high computation cost, while the later requires high memory cost.\n\nWe thank the reviewer for their insightful comment regarding the computational and memory overhead of the calibration step. \n\nTo analyze this trade-off, we conducted **an ablation study on the calibration step M**. The results on LLaMA-3.1-8B are presented in Table 2, comparing different calibration frequencies (M=5, M=20) and no calibration (w/o M).\n\nBased on these results, we highlight three points:\n\n1. **The calibration step is an optional enhancement.** Our core method without calibration already outperforms SOTA. This demonstrates the strength of our prediction mechanism, with calibration serving only to further refine accuracy.\n2. **Our method offers a flexible accuracy-efficiency trade-off.** Increasing the calibration interval from M=5 to M=20 reduces the calibration computational overhead by 75% with only a minor performance decrease, still surpassing baselines.\n3. **The calibration overhead is minimal and can be hidden.** \n   * **Low-Cost Approximate Calculation**: The calibration only requires block-level attention scores, so we can therefore use **efficient approximation methods**, similar to those in Quest, to compute with minimal computational cost. \n   * **Parallel Offloading:** In modern inference systems that separate prefill and decoding stages, this lightweight **attention calculation can be offloaded to the CPU**. This allows it to run in parallel, ensuring no impact on the main model's inference throughput.\n\n***Table 2**: Sensitivity Analysis of Calibration Step M on Llama-3.1-8B*\nMethod|Single-Doc QA|Multi-Doc QA|Summary|Few-shot|Synthetic|Code|Average\n-|-|-|-|-|-|-|-\nFull Cache|56.12|56.67|25.20|91.48|99.50|56.86|64.31\nSnapKV|52.05|54.60|23.22|90.93|99.00|55.66|62.58\nQuest|53.99|58.08|24.50|88.55|99.50|51.65|62.71\n**AttentionPredictor (w/o M)**|54.71|54.92|24.22|91.47|99.50|59.74|**64.09**\n**AttentionPredictor (M=20)**|54.71|54.92|24.62|91.47|99.50|59.75|**64.16**\n**AttentionPredictor (M=5)**|54.94|54.85|25.32|91.48|99.5|59.62|**64.29**\n\n> W4 & Q2: The evaluation are conducted on narrow tasks, a simple needle-in-a-haystack task and a longbench task. What is the performance of the proposed method on other tasks? e.g. RULER qa tasks, GSM8K, MMLU?\n\nWe thank the reviewer for their insightful feedback and constructive suggestions.\nIn our initial submission, beyond the Needle-in-a-Haystack and LongBench tasks, we also included evaluations on:\n* **The GSM8K benchmark with a long n-shot CoT setting**, to assess performance on long-input mathematical reasoning. This is presented in Appendix C.1.\n* **A long-output reasoning task (AIME)**, which presents distinct challenges compared to long-input tasks. The results are detailed in Section 4.2 and Figure 4.\n* **The InfiniBench benchmark**, which involves extremely long contexts with an average length of 214K tokens. These results can be found in Appendix C.3.\n\n\nTo further address your comments with a more appropriate benchmark, we are conducting additional experiments on the **RULER qa benchmark**. We are providing the GSM8K results from our paper here for your convenience.\n\n**GSM8K Results (n-shot CoT)**\nAs shown in table 3 below (Table 4 from our paper), AttentionPredictor maintains high accuracy on the benchmark across various long-context prompt lengths, significantly outperforming other methods.\n\n***Table 3**: Evaluation results on the long n-shot CoT GSM8K task with Llama-3.1-8B-Instruct.*\nMethod|Budget|4K|8K|16K|Average\n-|-|-|-|-|-\n**Full cache**|Full|56.79|55.27|54.13|55.40\nStreamingLLM|1K|54.74|49.96|50.94|51.88\nH2O+|1K|52.16|52.01|57.16|53.78\nQuest|1K|48.52|45.26|37.22|43.67\nSnapKV|1K|53.45|48.67|49.20|50.44\n**AttentionPredictor**|1K|**56.48**|**53.30**|**53.22**|**54.33**\n\n### **RULER qa Results**\nFollowing the reviewer's suggestion, we have conducted new experiments on the RULER QA benchmark. The results are presented in **Table 4**.\n\n***Table 4**: Evaluation results on the RULER qa benchmark under various context lengths and KV cache budgets.*\nMethod|Budget|4K|8K|16K|\n-|-|-|-|-\nLlama-3.1-8B-Instruct|full|75.0|73.0|69.0\nSnapKV|1K|63.7|71.7|68.0\n**AttentionPredictor**|1K|**73.4**|**71.9**|**68.8**\n\nThe results demonstrate that with a highly constrained budget of only 1K tokens, our **AttentionPredictor consistently and significantly outperforms** the SnapKV baseline across all tested context lengths.\n\n**MMLU**: We appreciate the suggestion to evaluate on MMLU. However, we believe it is **not the most suitable benchmark** to demonstrate the effectiveness of our method for two main reasons:\n1.  MMLU tasks are resolved in a single generation step, which **only involves the prefill stage**. Since our AttentionPredictor method operates during the decoding stage, it is **not applied in these scenarios**.\n2.  Even with a standard 5-shot setting, the average context length for MMLU tasks is only around 500 tokens. This does not align with the **long-context** scenarios our work targets, where KV cache compression is a critical challenge.\n\nWe hope these clarifications and additional results adequately address your concerns. Thank you again for your valuable feedback, which has helped us strengthen our paper."}, {"review_id": "kiu9KN1bwA", "reviewer": "Reviewer_8ZdG", "summary": "The paper presents a design named AttentionPredictor, a learning-based method for Key-Value (KV) cache compression in large language models (LLMs), enabling efficient long-context inference. It utilizes a lightweight, unified convolutional model to capture spatiotemporal patterns in attention scores, predicting next-step attention scores, and integrates a cross-token prefetching framework to mitigate prediction and cache loading latencies.", "questions": "See weaknesses above. Also other minors:\n\nSections C.5 and 4.6 provide standard deviations but do not specify the error computation method (e.g., closed-form, bootstrap) or distributional assumptions (e.g., normality). Please provide these details.\n\nCalibration step M and history step H (Table 9) are empirically determined (lines 631-647). Please provide a theoretical justification or sensitivity analysis for their impact on performance.\n\nThe cross-token prefetching framework claims to hide latency (Figure 5), but lacks a detailed breakdown of overhead components (e.g., prediction time, data transfer, synchronization). Can you quantify the contribution of each component to latency and explore potential optimizations?", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\nThe solution introduces a learning-based approach for predicting direct attention scores and a cross-token prefetching framework, which differs from heuristic-based methods and per-layer models. Its lightweight unified prediction model enhances scalability and efficiency.\n\nIt also reduces memory and computational demands for long-context tasks, with implications for multi-turn dialogues, complex reasoning, and resource-constrained settings.\n\nThe paper articulates the theoretical basis, model design, and implementation details.\n\nWeaknesses:\n\nWhile standard deviations are provided, the paper does not specify the method for error computation or the distributional assumptions. Experiments are limited to LLaMA-3.1-8B and QwQ-32B, lacking validation on other architectures.\n\nWhile societal impacts, such as reduced latency and energy consumption, are noted, no deployment data for edge devices is provided.\nHyperparameter choices are empirically determined without theoretical justification or sensitivity analysis. The communication overhead analysis for cross-token prefetching lacks specific latency data.", "was_revised": true, "final_justification": "Thank the authors for the submission and the feedback during the rebuttal. Also, thanks for the other reviewers' comments, which all help the reviewer to finalize the evaluation (keep positive for the study).", "rebuttal": "We thank the reviewer for the insightful and valuable comments. We respond to each comment as follows and sincerely hope that our rebuttal could properly address your concerns. If so, we would deeply appreciate it if you could raise your score. If not, please let us know your further concerns, and we will continue actively responding to your comments and improving our submission.\n### **Weaknesses and Questions**\n\n> W1-1 & Q1: While standard deviations are provided, the paper does not specify the method for error computation or the distributional assumptions. Sections C.5 and 4.6 provide standard deviations but do not specify the error computation method (e.g., closed-form, bootstrap) or distributional assumptions (e.g., normality). Please provide these details.\n\n\nThank you for this point. To ensure the statistical significance of our results, we conducted each experiment five times and reported the mean and standard deviation of the outcomes.\n\nThe standard deviation was calculated using the common formula for **sample standard deviation**, which provides a robust measure of performance variance without assuming a specific underlying data distribution. The formula used is:\n\n$s = \\sqrt{\\frac{\\sum_{i=1}^{n}(x_i - \\bar{x})^2}{n-1}}$,\n\nwhere $s$ is the sample standard deviation, $n$ is the number of experimental runs, $x_i$ is the result of each individual experiment, and $\\bar{x}$ is the arithmetic mean of the five experimental results.\n\nThis method accurately reflects the variability in our results. We will add this clarification to the revised manuscript.\n\n> W1-2: Experiments are limited to LLaMA-3.1-8B and QwQ-32B, lacking validation on other architectures.\n\nWe appreciate the suggestion. As detailed in **Section 4.1** of our paper, the evaluation was conducted on three distinct models: **LongChat-v1.5-7b-32k, LLaMA-3.1-8B-Instruct, and QwQ-32B**. The results presented in **Table 1, Table 4, and Figure 4** demonstrate that AttentionPredictor is effective across these different model families. We agree that further validation is valuable and, we plan to extend our evaluation to other popular architectures like Mistral in our future work.\n\n> W2-1: While societal impacts, such as reduced latency and energy consumption, are noted, no deployment data for edge devices is provided. \n\nThank you for this question. Our work primarily targets **large-scale, cloud-based LLM inference systems**, where the immense memory footprint of the KV cache for long contexts is a critical bottleneck. While adapting KV cache compression for resource-scarce edge devices is an important research direction, it presents a different set of challenges and is beyond the primary scope of our current paper. We have clarified this target application scope in the manuscript.\n\n\n> W2-2 & Q2: Hyperparameter choices are empirically determined without theoretical justification or sensitivity analysis. Calibration step M and history step H (Table 9) are empirically determined. Please provide a theoretical justification or sensitivity analysis for their impact on performance.\n\nWe thank the reviewer for this point. We provide a sensitivity analysis for these key hyperparameters in **Section 4.6 and Appendix C.5**, with detailed results in **Table 9**. Our paper provides the following justification for these choices:\n* **For Calibration Step (M):** As discussed in the appendix on lines 638-642, there is a clear trade-off. More frequent calibration improves accuracy by reducing cumulative errors from distribution drift, but at the cost of higher computational overhead.\n* **For History Step (H):** As explained on lines 643-647, a moderate value for H is optimal because it provides sufficient historical information for pattern recognition, while a larger value can introduce redundancy due to the decaying self-correlation of attention scores.\n\n> W2-3 & Q3: The communication overhead analysis for cross-token prefetching lacks specific latency data. The cross-token prefetching framework claims to hide latency (Figure 5), but lacks a detailed breakdown of overhead components (e.g., prediction time, data transfer, synchronization).\n\nThank you for your insightful comment. Our cross-token prefetching framework is designed to hide this latency by overlapping asynchronous tasks with the main LLM computation. The process is illustrated in the timeline in **Figure 10 (Appendix B.1)**. The key components are:\n* **Prediction Time:** The prediction overhead is negligible because our unified CNN model is extremely lightweight at only 21KB.\n* **Data Transfer & Synchronization:** The primary latency comes from fetching the KV cache blocks. This is hidden by executing the transfer in parallel with the computations of other layers of the *current* token.\n\nTable 4 shows a detailed breakdown. Note that since we perform attention prediction once every two tokens, the average per-token inference latency is further reduced.\n\n**Table 4:** Computation time breakdown with 1/16 budget. All values are reported in milliseconds per token (**ms/token**).\n\nContext Length|Attention Prediction|Cache Transfer|Sync/Wait|Total Overhead\n-|-|-|-|-\n4K|1.4|2.3|73.7|77.3\n8K|2.3|3.9|72.3|78.6\n16K|4.6|7.6|67.3|79.6\n32K|9.1|13.2|57.7|80.0"}]

Author rebuttal:

Meta-review: ***Summary***

A great deal of research is currently focused on optimizing attention by reducing the size of the KV-cache, often by pruning unnecessary tokens.  In this paper, the authors introduce AttentionPredictor, which uses a small, lightweight, convolutional model to predict what portions of the cache will be necessary for the next decoding step.  The entire KV-cache must still be held in main memory on CPU, but only the predicted portions are prefetched and transferred to GPU, and the model only attends over the predicted portions, thus speeding up inference.

The authors do a qualitive analysis of attention patterns, and identify several temporal patterns that can be seen in attention, which they label "re-access", "sequential", and "seasonal".  The focus on temporal patterns distinguishes this work from prior work.  The authors also train a small 2D convolutional NN to predict these patterns, and use it to prefetch tokens.  

The authors compare their technique against a number of other KV-cache compression baselines, on a variety of tasks, and show that their method generally outperforms competitors.  They also show large latency improvements against a FlashAttention baseline at long context lengths.

***Meta-review***

Reviewer opinions on this paper were somewhat mixed and in the borderline range, with ratings of (5, 4, 4, 3).   As AC, I have gone over both the paper and the discussion, and I believe these ratings are too low.  This is an excellent paper, and should be clearly accepted.  

I am basing my decision on two factors.  First, the author rebuttal during the discussion period was very strong; the authors provided detailed and clear responses to all of the reviewer questions, and I do not believe that the current ratings reflect the strength of the rebuttal.  Second, I honestly think the ratings were too low to start with; I give my own opinions on the paper at the end of this review.  

The most detailed review by far was done by reviewer **gSQC (rating 4)**, who asked a number of good technical questions (thank you!). The authors gave a very long and detailed response, and there was further discussion afterwards.  In their final justification, reviewer gSQC writes "The authors clarified all of my concerns in the rebuttal... I am raising my score to 4.  I'm keeping it at 4 because the work targets the KV off-loading setup which is more niche, therefore less significant."  As AC, I appreciate the high-quality review from gSQC, but I disagree with their assessment of significance.  If there are no major technical issues, then I believe this work is significant enough to get a rating of 5; KV off-loading is a commonly used technique. 

The remaining reviews were all rather cursory, but the authors nonetheless wrote detailed responses to each reviewer concern.  Some examples are:

**Reviewer spBc (rating 5)**:
* The paper was "well-motivated and clearly written."  (As AC, I wholeheartedly agree.)
* "more details about the CNN architecture (number of layers, kernel sizes, etc.) would be helpful."  (The authors provided additional details, as requested.)

**Reviewer 8ZdG (rating 4)**:
* "Sections C.5 and 4.6 provide standard deviations but do not specify the error computation method."  (The authors provided the appropriate equations and justification.)
* "The cross-token prefetching framework claims to hide latency (Figure 5), but lacks a detailed breakdown of overhead components (e.g., prediction time, data transfer, synchronization)."  (The authors provided a table listing overheads.)
*  "no deployment data for edge devices is provided"  (Author response: "Our work primarily targets large-scale, cloud-based LLM inference systems.")

Reviewer 8ZdG's response was: "The rebuttal... addresses most of my concerns raised in the initial review."

**Y2CJ (rating 3)**  was unconvincing to me, and the reviewer did not adequately respond to the author rebuttal.  
* "The writing remains a large improvement space."  (As AC, I strongly disagree with this statement.)
* "The evaluation are conducted on narrow tasks, a simple needle-in-a-haystack task and a longbench task."  (Factually wrong, and corrected by authors in rebutttal.)
* "The calibration step M is 5, a small number. AKA, every 5 steps the proposed method needs to 1) compute attention scores on the whole sequence; 2) fetch KV cache of the whole sequence."  (The authors responded with a detailed explanation, and an experiment ablation with various different values for M.)

***AC opinion***

This paper has a number of strengths.

* The paper is very clear and extremely well-written, among the best in my batch.  It provides qualitative assessments, which give further insight into attention in general, along with solid quantitative analysis and good diagrams.  
* The literature review is fantastic.
* The problem is important, and the solution seems novel to me.
* The authors discuss various subtleties that weaker papers often overlook, e.g. line 240, "distributional error calibration."
* The implementation of their technique is solid, as can be seen from the author responses during rebuttal.  
* The experiments are comprehensive and convincing.
Decision: Accept (poster)