ProtoKV: Long-context Knowledges Are Already Well-Organized Before Your Query
Authors: Zhiyuan Yu
Shijian Xiao
Zhangyue Yin
Xiaoran Liu
Lekai Xing
Wenzhong Li
Nguyen Cam-Tu
Sanglu Lu
Venue: ICLR 2026 Poster (2026)
Keywords: Large Language Model
KV Cache
Primary area: foundation or frontier models, including LLMs
TL;DR: We discovered a new paradigm for key distribution in LLMs and used it to guide the KV cache compression strategy.

Abstract: Modern Large Language Models (LLMs) face fundamental challenges in processing long text sequences due to the quadratic complexity of attention mechanisms. Key-Value (KV) cache retention strategies mitigate this issue by selectively preserving salient KV pairs for autoregressive generation. However, existing methods fail to adequately and efficiently preserve the semantic integrity of the compressed representations. In this paper, we discover a prevalent phenomenon in LLM: within the key embedding space, while most tokens exhibit similarity with their contextual neighbors (we term position-determined tokens), a small subset of special tokens serving as semantic anchors consistently show local deviation property and form one or several clusters (we term semantic-anchored tokens). Motivated by this observation, we propose ProtoKV that separately processes these two token categories for KV cache compression. Within this framework, we first construct semantic prototypes based on the inherent characteristics of the two token categories, and subsequently form clusters of semantically similar tokens as basic compression units. This approach preserves semantic integrity with high computational efficiency. Experiments on LongBench demonstrate that ProtoKV achieves 2.11% higher accuracy than state-of-the-art methods under matched memory constraints.

Published as a conference paper at ICLR 2026
PROTOKV: LONG-CONTEXT KNOWLEDGES ARE AL- READY WELL-ORGANIZED BEFORE YOUR QUERY
Zhiyuan SuijShijaan Zhangyue Yin?, Xiaoran Liu², Lekai Xing',
Wenzhong L Nguyen Cam-Tu', Sanglu Lu'
State Key Laboratory for Novel Software Technology, Nanjing University, School of Computer Science, Fudan University zh thiyyunnyye monan-yu@mmill iuu.edu. en, lwz@nju edu. cn
ABSTRACT
Modern Large Language Models (LLMs) face fundamental challenges in process- ing long text sequences due to the quadratic complexity of attention mechanisms. Key-Value (KV) cache retention strategies mitigate this issue by selectively pre- serving salient KV pairs for autoregressive generation. However, existing methods fail to adequately and efficiently preserve the semantic integrity of the compressed representations. In this paper, we discover prevalent phenomenon L.L.: within the key embedding space, while most tokens exhibit similarity with their contextual neighbors (we term cosition-determen serving semantic anchors consistently show local deviation property and form one or several clusters (we term semantic-anchore tokens). Motivated by this ob- servation, we propose ProtoKV that separately processes these two token categories for KV cache compression. Within this framework, we first construct semantic prototypes based on the inherent characteristics of the two token categories, and subsequently form clusters of semantically similar tokens as basic compression units. This approach preserves semantic integrity with high computational effi- ciency. Experiments on LongBench demonstrate that ProtoKV achieves 2.11% 11% higher accuracy than state-of-the-art methods under matched memory constraints. Our code can be available in
tokens), a small subset special tokens
INTRODUCTION
Large language models (LLMs) have become revolutionary in modern artificial intelligence (Brown et al. 2020; Chowdhery et al. 2023 Touvron et al. 2023 showcasing remarkable capabilities in dialogue (Li et al. 2024a), question answering (Ho et al. 2020) and reasoning (Wei al. 2022).
However, deploying LLMs under fixed-memory hardware presents major computational hurdles due to the Key- Value (KV) cache, which stores historical KV vectors to avoid recomputation but consumes memory scaling as O(b n), with respect to batch size b and sequence length n.
To address this challenge, strategies have been proposed to optimize the KV cache. Architecture-level optimizations like MQA (Shazeer 2019) and GQA (Ainslie et al., (023) reduce memory via parameter sharing, and quantization approaches (Hooper et al.) 2024b] Shao et al., (024) lowers numerical precision. Both methods demonstrate limited efficacy when scaled to extremely long contexts. To this end, eviction strategies emerge to optimize the KV cache by prioritizing and retaining salient KV pairs in each head for generation. Importance of KV pairs is typically determined by various configurable schemes, including prior knowledge (e.g.. "attention sink" in (Xiao et al.) 2024)) and quantitative indicators like accumulative attention scores (Zhang et al. (023). Considering that measuring token importance isolatedly may compromise semantic integrity, recent methods (Li et al. 2024b Razzhigaev al., 2025) retain both high-value tokens along with their surrounding context to better maintain semantic coherence, leading to a significant boost in KV cache retention efficiency.
Equal contribution Corresponding author

---

Published as a conference paper at ICLR 2026
(a) Key embedding distribution & How it helps KV cache
(b) An Example of semantic-anchorer tokens (on SQUAD)
Color reprosents
Clusters formed by
Super Boxl
National Football Langue
Layer
(NFiL) for the 2015 sesson Demver Bronces detasted the Natienal Football Contenence (MTC)... samn thair
Chuuks formed by Position Determined Tokons
Chunk samantica
this
Encoded within SATs
Bowl, each Super Bood game with have boon known "Super L") Qusetion: Which represerted
40
40
(e) What are semantic-anchored tokens? (on GovReport)
20
20
22 Head Technical vocsburbry
Laper /Hedd 4: Diecnurna Markers
0
Spéci Pirtrrllaty
Mither decereing Wh Siiin
20
20
Doegin
Tats
Of text
Apgendix
Specifically
Lo
-40
-40
Aergndragi According
50
-50
50
PLC
Moresver such
According
<] </begin__of Sea_ionoofoe___e_p_eee textl2
Retained For high-order semantic information
Query Vector
Retained For local fine-grained knowledge
(aper laifh
sucht+
thoyeh
Figure 1: (a) T-SNE visualization of key embedding. As illustrated. semantic-anchin clusters within the key embedding space. (b) We used a sequence from SQuAD and highlighted the tokens corresponding to semantic anchors in a specific attention head, which are marked in red. (c) Different attention heads are dedicated to distinet cemantic-anchored tokens.
tokens typically form
However in natural language, proximity in position does not imply semantic similarity. Existing work has observed that when LLMs process input sequences, certain sequential information will be compressed into anchor tokens, e.g., label words for in-context learning (Wang et al., (023) or some punctuation marks (Razzhigaev et al. 2025). These tokens share similar properties and play analogous roles during inference, suggesting that they should be treated as a semantically cohesive unit (either collectively discarded or retained). In practice, however, the diversity of anchor tokens poses a significant challenge for systematically identifying and categorizing them.
Fortunately, we find that LLMs have inherently identified and organized these anchor tokens during prefilling stage. Specifically, by analyzing the key embedding distributions, we identify that tokens in long-context prompts mainly fall into two distinct categories.
(i) Position-Deterriine To- kens (PDT): These tokens exhibit strong key similarity with their contextual neighbors, corresponding to the majority tokens where proximity implies semantic relatedness; and (ii) Semantic-Anchored Tokens (SAT): These tokens violate such local adherence and demonstrate local deviation property. Previous studies have primarily focused on PDTs, suggesting that rotary position encodings (Su et 2023) cause key embeddings to exhibit manifold characteristics (Zandieh et al., 2024 Liu et al. 2025b c). while overlooking the existence of SAT'. this paper, we discover that although SAT' are few in number, they generally possess good clustering properties and serve as "semantic anchors" for long-context inference. In Section 3 we conduct a detailed quantitative study to investigate and analyze such properties of SATs, while also explaining the reasons behind such distribution pattern of the key embeddings. In Figure [ we illustrate the characteristics of this distribution through T-SNE visualization, accompanied by examples for SATs.
Through further exploration, we find it imperative for KV cache compression to simultancously accommodate the aggregated information encapsulated by SAT' and the fine-grained information inherent in PDT's. To this end, we propose ProtoK a novel KV cache retention strategy that first constructs hybrid semantic prototypes based on the inherent characteristics of the two token categories, and then dynamically reassigns importance scores. Specifically, we employ outlier degree metric to obtain a high-purity set of SATs, and apply locality-sensitive hashing to bucket them form SAT semantic prototypes that encapsulate anchor-level semantic information. For PDTs, we partition them into contiguous chunks to form PDT semantic prototypes that are prioritized for fine-grained information retrieval. Finally, an observation window-based selection mechanism is adopted to retain the semantic clusters most relevant to subsequent queries. Through comprehensive comparisons with existing KV cache retention strategies, demonstrate that our approach not only preserves the semantic integrity in a more reasonable manner, but also achieves superior robustness with higher computational efficiency. This dual advantage remains unattainable by existing methods.
Extensive experiments demonstrate that ProtoKV achieves excellent performance on both real-world and synthetic tasks. Specifically, ProtoKV delivers state-of-the art accuracy (with average gain of 2.21 %) on LONGBENCH, and also outperforms other baselines on the RULER benchmark. In the Needle In A Haystack evaluation, ProtoK maintains 97.3% retrieval accuracy with only 1.6%
2

---

Published as a conference paper at ICLR 2026
KV cache retention, demonstrating its powerful information retrieval capabiliiyy Moreover, ProtoK shows complementary benefits when combined with budget allocation methods.
PRELIMINARY
2.1 PROBLEM FORMULATION
Consider an autoregressive LLM with L layers and H attention heads. Let x, E Rd denote the input token embedding at decoding step I. To reduce recomputation, key-value pairs from previous steps are stored in a KV-Cache for each of the H heads. The attention output of E IR at step t is computed via a softmax attention mechanism using the current query vector and the cached key and value matrices. For each head h, our objective is to find compressed representations {K1:) (hh))
Jhml under
1:4 (h) o, ot ||2 is
a budget size B, such that || llo
| llo = B, and the approximation error ||o,
minimized. Here (() denotes the approximate output using the compressed KV pairs.
2.2 KV CACHE RETENTION
We focus on compressing long text prompts during the prefilling stage, i.c., the compression of {K (h) v(k) 1H I:N. 1:N 1==1- Theoretically, the optimal strategy for KV cache retention should prioritize tokens that consistently contribute the model's attention distribution throughout the entire generation process, which can be measured through inference-stage cumulative attention defined as follows:
Definition (Inference-stage Cumulative Attention). Given attention head h, let T denotes the total number of generation steps and qi the queries in step t. (k(h) (
li=ı denote the key vectors of all N
context tokens. The inference-stage cumulative attention (h) for context token is defined as:
(N)T (M) q
A (b)
softmax
dd
Precomputing Ah is infeasible since (h) is unpredictable during autoregressive generation. As result, strategies using accumulative attention scores (Zhang et al.) 2023 Chen al. [2024] Liu et al. 2024a Zeng et al. 2024) offer more flexibility. However, most of their methods measure token importance isolatedly, which may compromise semantic integrity for compressed KV pairs.
2.3 SEMANTIC-LEVEL COMPRESSION
The semantic-level compression paradigm addresses the limitations token-level compression by treating coherent semantic units as atomic elements KV cache management. Let C = {Ci,.... C denote a partition of the KV sequence into k semantic clusters, where each cluster C₁ contains contiguous semantically related tokens. The compression objective is preserve or discard entire clusters to maintain contextual integrity. Typically, clusters are ranked by importance scores (h) (C,). The top-k' clusters (k' < k) are retained to satisfy budget size B:
Σ Tailhh (C,)
(1(
P |G|| S B.
2)
The compressed KV pairs (K(M) mask (A)) {0,1) 1)' indicating preservation (M 1 if belong one of the selected clusters):
I.N. 1 retain only tokens from selected clusters, with binary
[(M)] K
K
(k) (())) (i(h) ((h))
O (A).
L:N
I:N
For instance, SnapKV [ et al. 2024b) and ChunkKV (Liu et al. 2025a) group tokens into contiguous chunks, and SentenceKV (Zhu et al. 2025) uses sentence boundaries for C. However in natural language, proximity in position does not necessarily imply semantic similarity. Clustering- based KV compression approaches (Liu et al. 2024b Hooper et al. 2024a) employ K-means clustering key embeddings, but are plagued by ineficiincyy and poor robustness. By contrast, our proposed ProtoKV is capable of simultaneously preserving semantic integrity and maintaining high efficiency.
3

---

Published as a conference paper at ICLR 2026
Head
Head
Hy
Hg
Hiz
His
Hza
Neighbor Sim.
0.8
0.8
Clustering Degree
1.5
0.6
0.6
1.0
1.0
0.4
0.4
0.5
0.5
0.2
1000
2000
1000
2000
1.5
2.0
25
3.0
1.5
2.0
2.5
3.0
Token Position (a) Layer 8
Token Position
Value of threshold P
Value of threshold P
(b) Layer 20
(a) Layer 8
(b) Layer 20
x-Neighborhood Similarity tend attain low values for small subset of imput tokens. Detailed results and analysis are in Appendix B
Figure 3:
Figure 4: SATs form progressively compact clustering with the increasing the threshold 3. Detailed results and analysis are Appendix
3. LOCAL DEVIATION PROPERTY ANALYSIS
3.1 PROPERTY ANALYSIS FOR SEMANTIC-ANCHOR TOKENS
In this section, highlight prevalent phenomenon during the prefilling stage of LLMs: within the key embedding space, the vast majority of tokens exhibit high similarity to their local neighbors, while compact subset demonstrably violates this locality prior, which we term Local Deviation Property Crucially, we find these spatially anomalous tokens consistently form one or several clusters, and play a pivotal role in long-context generation as semantic anchors. To identify these tokens, we first define the neighborhood similarity and outlier degree as follows.
Definition (x-Neighborrhod Similarity Outlier Degree). For token i with its key embedding k(t) in attention head h, we define its k-neighborhood similarity follows:
i+
(
(1) 2K
Σ
cos(k, (
In subsequent experiments, unless otherwise specified, we set = 5 by default. To identify tokens with local heterogeneity a normalized manner, we define outlier degree for token as:
( (1) (S. (1) E(S (0)) (VS) (i),
(5)
with E and represent the mean and variance of the neighborhood similarity across all input tokens.
Figure 3 reveals that although most tokens maintain high neighborhood similarity, there always exists a small subset exhibiting markedly lower values. We term tokens violating the locality property as Semantic-Anchored Tokens (SATs) due its aggregation of semantic information from partial sequences, while tokens conforming to locality are termed DositionnDeterminee Tokens (PDTs). is important to note that the boundary between SATs and PDTs is often ambiguous in practice (as illustrated Figure 2 therefore, our analysis focuses primarily the evolutionary trends of token proper- ties along a continuum of locality adherence/devvaation. We find that the following findings remain valid under the framework of Multi culti-Query Attention, which is verified in Appendices B to
0.90
Gummu Cummulative Flnquenl
Pequenncy
Frequency
0.16
0.8
Threshold between POTs (lef) and SATs (right)
0.10
0.4
0.06
SAMSuu
0.00
0.0
Outtier Degres
Figure 2: Outlier Degree distribution.
Property 1: SATS are clustered. To quantitatively validate the clustering property of SATs, by
setting a threshold 3, we group tokens with @ > B into cluster C as SATs. Obviously, a larger / indicates higher purity of the selected SAT'. We then define the clustering degree of C as follows:
Definition (Clustering Degree), Given cluster of tokens with their key representations (k (h) }. we define the intra-cluster similarity Suara (C) and inter-cluster similarity Sinter (C) as:
Σ cos(k) (h) k(()) (1)), Smer Exec,iec cos (h)
kkk (k)
Suna (C)
(6)
|C12
[C - (C)
JEEC
4

---

Published as a conference paper at ICLR 2026
Layer
Layer
Layer
TrivaQA
Narrative
2WidMuutthopOA
Altentonn
0.4
F1 F1
10
0.00
0.2
F1
0.05
0.0
(00135100
25 900052132
Drcopeed KV
Drapped
Droppeed
Layer
Layer
QMSum
RepoBench-
Altention
Rougee
Rouged
0.5
0.5
2
Edit
5 B
10
25 de 75 100125100
25 56 120455
25 00 100835150
Localnesa Degroe
Localnoss Degros
Localness Degree
Drooped KV
Drapped
Pairs
Diopoed Para
TrMitAA
SAMSsam
No Attention Sine
Altention
Attiantion Sink
Figure 5: Attention heads within all layers focus Figure Dropping KV pairs with highest outlier de- tokens with higher outlier degree. Detailed results and grees can cause sharp performance drop. Other results analysis are in Appendix
and analysis are in Appendix E
where N denotes the total number of tokens. The clustering degree of that measures its compactness then defined as T(C) = Smmra SmterrC
Building upon these formulations, we systematically evaluate the clustering properties of SATs selected under varying B thresholds. Figure shows the results on TriviaQA for layers (8 and 20) in L. 2llama2-BB-che Notably, when 8 exceeds critical threshold, the clustering degree exhibits sharp increase. This suggests that in the key embedding space, tokens with high outlier degrees tend to cluster. Moreover, we observe that in rare cases (e.g., when long-text semantics exhibit significant discontinuity), these SATs may form multiple distinct clusters, with representative examples provided in Appendix D
where increasing 33 leads to a more compact cluster of C with larger T(C).
salient for generation. To validate SATs' critical role in long-context inference, we analyze two complementary aspects. First, we examine if SATs are retrieved more frequently during long-text generation. We stratify tokens into 10 groups by their outlier degree (i) (Eq 5) and compute summed inference-stage cumulative attention (Eq [ per group for token significance measurement. As sshoos, the top 10% outlier tokens account for over 60% of cumulative attention in deeper layers, indicating that SATs act persistent anchors of attention.
Property 2: SATs
On the other hand, we study the impact of removing these SATs. Figure that pruning key- value pairs corresponding to highest outlier degree (i)((i) per head significantly degrades LLM's performance for long-context inference, which is similar to pruning by inference-stage cumulative attention. Random removal has no measurable effect. This confirms SAT's functional equivalence to the model's i inherent attention prioritization.
Conclusion.
Previous studies have focused mainly on PDTs Zandich et al. 2024 Liu al. 2025b c).
In this paper, we first detected the presence of outliers within them, which spontaneously deviate from their surrounding neighbors and cluster together, serving as anchors of attention across different tasks. Considering their properties, name them "semantic anchors".
3.2 FURTHER EXPLORATION
How SATs Cluster. In real-world experiments, we observe that SATs predominantly form a single, tightly-lustered group in the key embedding space across most attention heads. However, the clustering behavior exhibits certain variations under specific conditions: in the initial layers of the model, SATs may not yet emerge distinctly, resulting in a near-continuous distribution of key embeddings without clear outliers. In some cases, SATs are clustered but exhibit loosely-organize structure; and in low-coherence texts such as multi-document data, SAT' occasionally form multiple well-separated semantic groups. Despite these variations, subsequent experiments in Section 5.3 demonstrate that assuming SATs form a single compact cluster remains highly effective for KV cache compression. Comprehensive analysis for each scenario is provided in Appendix D
Causal Mechanisms Analysis. We attribute the above phenomenon to the following reason. First, modern LLMs commonly employ Rotary Position Embedding (RoPE) (Su et al., 2023) to inject positional information into keys, which leads to local similarity in the key representations ad- jacent tokens (Wang et al. 2024b Liu al. 2025ch D02iing pre-training, different attention heads
5

---

Published as a conference paper at ICLR 2026
spontaneously specialize in capturing distinct linguistic patterns (Conmy et al. 2023 Syed et al. 2023). and thus attend to their corresponding salient tokens. To efficiently access relevant value embeddings, LLMs learn to cluster these salient keys in the key space. Such clustering mechanism enables the heads to simultaneously assign higher attention weights to semantically relevant tokens during generation. Meanwhile, tokens that are less relevant head's specific role remain locally constrained due to the inductive bias introduced by RoPE.
"Semantic Anchors" "Attention Sink" (Xiao et al. 2024 observed the "attention sink" phe- nomenon where the initial tokens disproportionately absorb global semantic information. In Ap- pendix B we note that attention sinks generally exhibit high outlier degrees. However, our concept of semantic anchors is broader in scope. As shown in Figures and even after excluding attention sinks, the remaining SAT' continue to play an indispensable role in long-context generation.
"Semantic Anchors vs "Label words in In-context Learning" (Wang et al. 2023) demonstrated that in in-context learning scenarios, label words act as anchors that absorb sample semantics for text classification tasks. Our study reveals that these label word tokens also exhibit significant outlier characteristics. Detailed experimental results are presented in the Appendix
METHODOLOGY
4.1 MOTIVATION
Table I: Performance for Llama3-8B (256 budget).
Dataset
Considering the importance of SATs for long-context tasks, an intuitive approach is to retain KV pairs based on their outlier degree in descending order (we term SATKV). Compared to prior approaches, SATKV offers the advantage of eliminating the re- liance on attention matrix, thereby reducing compu- tational overhead (see Appendix HH.. However, we find this approach underperforms baselines like
Baseline
NrtvQA
HoiposQAA
SAMSum
Lec
SLM
17.98
37.83
34.82
54.84
H2O
23.67
41.57
40.19
57.52
SnapKV
23.32
42.70
39.78
60.27
SATKV
21.12
40.84
38.06
58.46
SnapKV [Table D. We attribute this to two factors: First, it neglects PDT selection patterns. Figure 5 shows PDTs comprise around 40% of attention, which necessitates simultaneous optimization for PDTs retention especially when the KV budget exceeds SAT capacity. Second, similar to accumu- lative attention methods like H2O, it assesses tokens individually and fails to maintain semantic coherence. Thus, we require a retention strategy that preserves semantic coherence while integrating both high-order semantic information (SATs) and local gneiigaieed knowledge (PDTs).
To this end, we introduce ProtoKV, novel method that constructs semantic prototypes by leveraging the inherent characteristics of both token categories. These prototypes form clusters of semantically similar tokens, serving as fundamental compression units. We compare our method with existing chunk-based and cluster-based approaches, demonstrating that our solution simultaneously preserves semantic integrity while improving efficiency. Pseudo-code for ProtoKV V is provided in Appendix
4.2 HYBRID SEMANTIC PROTOTYPE CONSTRUCTION
Considering that drawing hard distinction between PDTs and SATs is often challenging, instead of categorizing individual tokens, we propose extracting semantic prototypes holistically for patterns capture. Given an input sequence n tokens with corresponding key vectors T [k.) C IRd we first calculate the outlier degree (ii) of each token i according to Eq. 5). Tokens with top-p (i) are identified as candidate SATs = [k To better handle loosely-cluster or multi clutteclusterr scenarios, for these identified outlier tokens, we allocate them into hash buckets. Specifically, the key vector for jth token is projected into low-dimensional space using Random Fourier Features (RFF) mapping @ R R with Gaussian kernel approximation:
p(k,) - V2 cos (Wk, + b), with W ~ ('(0 21) and b ~ Uniform(O, 2T)
The real-valued projections are then binarized {0, 1} codes and subsequently interpreted as an r-bit integer for bucket allocation:
h, = I((kk) > 0) E 10, H(ky) = Σ 22 mod u,
(8)

---

Published as a conference paper at ICLR 2026
where I[(-) denotes the element-wise indicator function, h the ith bit of hj, and u the total number of hash buckets. To ensure clustering effectiveness, we typically require that u = 2". This binary-to- decimal conversion preserves the Hamming distance between original codes while enabling efficient bucket indexing. For other tokens of T O we partition them into v consecutive chunks {Cm h m==1 of equal length [(n 10-pp))) pp/p Based on this, we construct hybrid semantic prototypes of PDTs and SATs as follows:
Eneco.. KG
ΣKjEB, kj
(DD)
(SATS))
11 EK,KEc... kell2"
12 EB, kjlla
The hybrid semantic prototypes M is then obtained via: M (ch ( I U [. pendix DJ we show that our approach for constructing semantic prototypes effectively accommodates distinct patterns of key embedding distributions.
1--11 In Ap-
4.3 KV CACHE RETENTION VIA PROTOTYPE- PRUDDDY GUIDED ATTENTION
MM constructs semantic prototypes for both higher-level semantic anchors (c(SAT)) and fine-grained textual details (C(DDT)). For each token k, the pattern to which it belongs (denoted as c(k,)) is determined by the semantic prototype with the highest cosine similarity to it, formally:
k
c(kt = arg max
(10)
kk |2/c////
This assignment machanism divides all tokens into n clusters {Cz] and we calculate the impor- tance score (b( (C,)) (C₃) for each through an observation window-based selection mechanism (Li et al. 2024b). Given an input sequence with its length Lprmmee = Lprefix + Lobs. where Lobs denotes the observation window at the sequence end, for each attention head h E [H) (b())())) is defined as:
Leponut
(C)
K,
(11)
iEC; m=Loeetttt
The compressed KV representations are then constructed according Section 2.3
4.4 COMPARISON WITH EXISTING KV CACHE COMPRESSION STRATEGIES
In this section, we compare our ProtoKV with existing chunk-based and cluster-based, showcasing that it ensures more rational semantic coherence while maintaining compression efficiency.
Chunk-based strategies Chunk-based strategies assume that semantically similar tokens usually appear in contigu- ous sequences. As result, they partition input text into chunks and perform uniform retention/eviction operations on tokens within each chunk. However, our discovery of SATs reveals limitations in this assumption. For instance, noted in (Razzhigaev al. 2025), , certain punctuation marks carry critical information transmission and mem- ory functions, so their semantically similar counterparts should be other functionally equivalent punctuation marks, However, chunk-based methods fail preserve the seman- tic integrity of such tokens.
StreaminggLMM
Snapiv PysamisVVV
Squazze Attertion
H20
14 17
6
11 14
Number of Iteraiions
Number of iteraions
(e) Liama2-78-chat
(b) -B-ttruttt
Figure 7: Clustering- strategy requires 20 iterations before surpassing existing KV cache compression methods on TriviaQA.
Cluster-based strategies Clustering- based strategy (Liu al. 2024b; Hooper et al. 2024a) global captures semantically similar tokens through clustering within the key embedding space. Although these methods can identify SAT's, their iterative clustering process requires excessive computation. As illustrated in F Squeeze Attention (Hooper et al. 2024a) takes over 20 iterations to outperform baselines (compress only during the prefilling stage). Moreover, prototype clustering typically demands high-quality initialization, as poor initialization can cause the iterative process to converge to local optimum or lead to highly imbalanced token number distribution among clusters (Arthur ProtoKV directly obtains high-quality cluster centers
& Vassilvitskii 2007). By comparison, by leveraging the properties of SATs and PDTs, thus reducing the computational complexity from ((ent) to ((nt) for iterations achieving clusters.
7

---

Published as a conference paper at ICLR 2026
5 EXPERIMENT
5.1 IMPLEMENTATION
Dataset We primarily use LongBench (Bai et al. 2024) dataset to assess the performance of ProtoKV on tasks involving long-context imputs. LongBench comprises 14 English tasks and 2 code-related tasks, with an average length ranging from 5k to 15k tokens. We also use Ruler Hsich et al. 2024)
as benchmark. A detailed description of datasets is provided in the Appendix J
Baseline We benchmark our method against StreamingLLM (Xiao et al. 2024). H2O (Zhang et al. 2024d), SnapKV (Li et al. 2024b), PyramidKV (Cai. et al. 2024) and ChunkK'v (Liu al. 2025a). We use open-sourced LLMs include the Llama family (Llama-2-7B-che Mistral-7B-IIstruce
LlaMA-3-8B-imee -intttittttt) and
which can handle up to 32k context length. Detailed description of these three LLMs is provided in the Appendix K In Appendix [S] we also evaluate the performance of ProtoKV two recent LLMs: Phi-3.5-mimi-nnstruct and Mistral-7B-insttuuc-00..33
Experiment Setup All experiments use two NVIDIA 3090 GPUs (48GB total) with consistent
prompts across datasets. The Operating system ubuntu version 20.04.2 with CPU AMD Ryzen Threadripper PRO 5945WX (12-Cores). KV cache budget for evaluation range from 64 to 512. Our configuration balances experimental uniformity with task-specific optimizations.
5.2
RESULT ANALYSIS
5.2.1
REAL-WORLD AND SYNTHETIC BENCHMARKS
demonstrate the experimental results on LongBench across diverse KV cache configurations. Generally, our method maintains the best performance between 64-512 budgets, with an average improvement of 2. % As illustrated, ProtoK outperforms Sota baselines by 0.35% to 4.27% across diverse budget sizes. Task-specific experimental results are reported in Appendix L Additionally, Table 9 presents the performance of our ProtoKV on the Ruler dataset (NAIH and QA subsets), demonstrating that ProtoKV consistently achieves either the best second-best outcomes. Due its more rational partitioning of semantic clusters, our method achieves superior compression perfor- mance compared both SnapKV and ChunkKV. In appendix PP we further report the performance comparison on RULER benchmark with a 16K/32K context length.
Figure g
StreamingLLLM
SnapKV
ChunkKV
PyramidKV
H20
ProtoKV
Dataset
Method
NIAH
QA
Average Score
30.0
40.0
FullKV
99.6
46.0
3.6/
7.6/10.2
25.0
H2O
30.4/57.2
38.0/42.0
22.5
32.5
SnapKV
97.2/99.6
42,0/43.8
ChunkKV
97.2/99.6
42.4/42.2
20.0
30.0
64
128
206
512
64
95128
265
512
96128
256
512
PyramidVV
97.6/99.4
41.2/43.4
KV Budget Sice
KV Budgel Size
KV Budget Size
ProtoKV
98.2/99.6
42.8/43.6
(a) Llama2-7B-chat
(b) Mistral-7B--nstrue
(c) Llama-3-88-Inetruct
Figure Experimental results on LongBench under different KV cache budget conditions. Average score is reported across 16 tasks.
Table 2: Performance (%) on Ruler for LJama3-8B with budget size 128/512.
We further compare ProtoKV with ClusterK V (Liu et al. 2024b), Squeezed-Attentioon (Hooper et al.) 2024a), and SentenceKV V (Zhu et al.) 2025). These methods involve dynamically loading or offloading KV pairs during the in- ference stage. To ensure a fair comparison, we design two experimental configuratioss: one constrains these baseline methods to perform KV pair retention operations exclu- sively during the prefilling phase, while the other modifies ProtoKV to dynamically recall tokens at the granularity of semantic clusters during the decoding phase. As shown in Table 3 ProtoKV consistently outperforms other methods under both configurations. This advan- tage stems from its more robust and higher-quality clustering results compared to K-means, which ultimately enhances its performance over existing cluster-based approaches.
Performance
Approach
Decoding
Prefilling
SentenceKV
40.82
40.21
ClusterKV
41.32
40.45
SqueezeAtt ProtoKV
41.47
40.42
42.59
40.76
Table 3: Longbench performance for Llama3-SB under 512 budget.
8

---

Published as a conference paper at ICLR 2026
5.2.2 NEEDLE IN A HAYSTACK
We conduct Needle In A Haystack experiment (Liu al. 2024c Fu et al. (024a) using LlaMA- 3-8B-Instruct with up to 8K context length. This task requires precise information retrieval from extensive contexts, simulating real-world scenarios where relevant data is buried among vast irrelevant information. We compare other KV Cache techniques at a consistent cache budget size of 128 (a retention ratio up to 1.6%). Results in Figure 9 indicate that StreamingLLMM and H2O almost collapses retrieval task. Our ProtoKV attains 97.3% accuracy. which outperforms SnapK (94.2%) Interestingly, we observe that in the retrieval head (Wu et al. 2024). the needle text exhibits consistently higher outlier degrees than the haystack text (detailed results in Appendix G). This indicates that LLMs may detect semantic inconsistencies during the prefilling phase.
(a) ProtoKV (0.973)
(b) Sn (0.942)
(c) H20 (0.548)
(d) SLM (0.311)
2000
6000
4000
6000
5050
2000
4000
5000
2000
4000
600.
8000
Contaet: Langth
Contectt Langet
Corract Length
Conmeot Length
Figure 9: Results of the Needle In A HayStack experiment, where LLMs are required to retrieve a target sentence ("needle") inserted in long documents. The x-axis represents the context length while y-axis the depth where the needle inserted. E.g.. context length of 4000 and depth of 11.0 implies that the needle is inserted location 4000 x 11% 440 in sequence. The color indicates retrieval accuracy, the greener, the better.
5.3
FURTHER DISCUSSION
Tasks
ProtaV
ProtokV LA.
PmntKV HA.
Clustering
LSH Clustering
SDQA
32.87
33.12 (3.112()))4%)
33.58
Average Score
40.0
40.0
MDQA
25.31
25. 75.72
26.14
$7.5
37.5
SUM
24.41
24.89
(0. 47%)
25.29
(1.1599)
Few shot
66.43
56 85 (1.2)
67.34 67.34
88.0
35.0
SYN
37.15
37 43 80%))
37.91 (3.7339))
32.5
325
Code
59.38
59.77 (4.777(4
60.161)3227))
30.0
30,0
Table 4: Compatibility Analysis Mistral-7B-
KV Budget Size
KV Budget Size
Instruct with KV budget 256. 1 (-) denoting the improvement compared with SnapKV+LA.HHA.
(a) Mistral-7B-Instruse
(b) LlaMa-
Figure 10: Ablation Study for different KV budgets.
Compatibility Analysis Methods for KV cache budget allocation (Wang et al. 2024a Qin al.
2025 Feng et al. 2024 2025) intelligently distribute memory by the importance of each layer or head. We study their compatibility with our method, focusing on two strategies: Layer-wise Allocation (LA.) (Narrot et al. 2024) and Head-wise Allocation (HA.) (Feng et al.) 2024). As
shown Table ProtoKV demonstrates strong compatibility, with ProtokV /HA. achieving consistent improvement compared to SnapKV + L.A/HA..
Ablation Study We compare ProtoKV with two variants for ablation study: chunk-clustering that only uses chunk-based aggregation to obtain semantic prototypes, and LSHf-clusternng that buckets and groups all tokens via LSH in Eq, S Two variants adopt the same number of semantic prototypes (i.e.. cluster count) the original ProtoKV, along with other basic settings. Figure 10 shows both variants reduce KV cache compression performance, especially LSH-clustering.
41
SAT Protosype Number
8
$7.96
4
39.66
43.46
60.21
39.12
4
09.01
43.25
29.65
38.42
36
33
38
38.42
40.62
30 08
39.04
38
43
38.35
LeMSs Misuaa
LIMO3
Motral
37
37
1500
4000
Prototype Number
Prototype Number
Neighborood Window
Figure 11: Average results LongBench for Mistral-7B (LEFT) Llama3-8B (RIGHT).
Figure 12: Average results on LongBench for Llama3- 8B with KV budget size 256.

---

Published as a conference paper at ICLR 2026
Hyperparameter Analysis. As shown in Figure 11 optimal performance is achieved with around 500 prototypes, whereas SAT prototypes number require only 2-4. Figure [2 shows that selecting SATs via outlier degree requires merely 20-40 to optimize ProtoKV performance, since additional SAT prototypes may introduce PDT noise and impair clustering effectiveness. Moreover, the choice of neighborhood window size in outlier degree computation shows negligible influence the overall performance.
Computation Cost Analysis Fig- ure [3] illlutraates the computational cost of our ProtoKV. can be observed that our time consumed in evaluating the importance of se- mantic clusters s after prefilling is comparable with SnapKV, whereas cluster-based methods require up to 3.9 x time overhead. Furthermore, the overall computational time of ProtoKV remains relatively stable across different prototype numbers, which is reported here as the average time over various tasks on LongBench.
SnapKV
Nomallzed Time
ProtaKV ClusterKV
LiaMaS
Mistral
Time (h)
0.0-
0.6
0.4
16k
50
Sequenoe Length (a) Computation Cost
Prototype Number
(b) Time on LongBench
Figure 13: Computation Cost Analysis
CONCLUSION
In LLMs, we discovered an interesting phenomenon: within the key embedding space, while the vast majority of tokens exhibit manifold distribution along input position, a small subset of special tokens which serve as semantic anchors consistently form one or several clusters. Based on this insight, propose ProtoKV, a simplified approach to KV cache compression that leverages semantic prototypes to achieve a more rational and efficient semantic-level compression strategy.
ACKNOWLEDGEMENTS
This work was supported in part by the National Natural Science Foundation of China (Grant Nos. 22572236,62002011 222442,,,,441225 the Basic Research Program of Jiangsu Province (Grant Nos. BK20222003, 2K20251198, BK20253011) I), the Collaborative Innovation Center of Novel Software Technology and Industrialization, and the Sino-German Institutes of Social Computing.
ETHICS STATEMENT
The primary objective of this paper is provide KV cache compression framework that designed to accelerate inference. This work is based on the publicly available LongBench and Ruler dataset, which predominantly contains English text. We comply with all dataset licenses, and confirm the content contains neither private nor offensive information.
REPRODUCIBILITY STATEMENT
Pseudocode of this paper is shown in Appendix ! In Section 5.3 we provide a hyperparameter analysis and employ the optimal set to conduct our specific experiments and obtain the final reported results. We give the link the source code in abstract.
REFERENCES
Joshua Ainslie, James Lee-Thorp, Michiel de Jong, Yury Zemlyanskiy, Federico Lebrón, and Sumit Sanghai. GQA: training generalized multi-query transformer models from multi-head checkpoints. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. EMNLP 2023, Singapore, December 6- 10, 2023, pp. 4895 4901. Association for Computational Linguistics, 2023. doi:
10

---

Published as a conference paper at ICLR 2026
http:5///112222..E..-M.M emn peanlp-main... 298
URL http::///0....2g/111116633333332
David Arthur and Sergei Vassilvitskii. k-meanst+: the advantages of careful seeding. ACM-SIAM
Symposium on Discrete Algorithms, 2007. URL httlhhht///aat www.ppacca.nacccc....cooooo.
CorpusID: 782131
Yushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu, Jiankai Tang, Zhidian Huang, Zhengxiao Du, Xiao Liu, Aohan Zeng, Lei Hou, Yuxiao Dong, Jie Tang, and Juanzi Li. Longbench: A bilingual, multitask benchmark for long context understanding. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.). Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume Long Papers), ACL 2024, Bangkok, Thailand, August 11-16, 2024, pp. 3119- 3137. Association for Computational Linguistics, 2024. doi: 12111663/V120244AALLLL0000 URL
.ae44..cll-3 acl-long. 172
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D) Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models few-shot learners. Advances in neural information processing systems, 33: 2020.
Zefan Cai., Yichi Zhang, Bofei Gao, Yuliang Liu, Tianyu Liu, Keming Lu, Wayne Xiong, Yue Dong, Baobao Chang, Junjie Hu, and Wen Xiao. PyramidKV: Dynamic KV Cache Compression based Pyramidal Information Funneling, June 2024. URL httt:://axiv..rtee 02069 2XXv:2406.02066 [cs].
Yilong Chen, Guoxia Wang, Junyuan Shang, Shiyao Cui, Zhenyu Zhang, Tingwen Liu, Shuohuan Wang, Yu Sun, Dianhai Yu, and Hua Wu. NACL: A general and effective KV cache eviction framework for LLM at inference time. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2024, Bangkok, Thailand, August 11-16, 2024, pp. 7913-7926. Association for Computational Linguistics, 2024, doi: 10. 120653///2224AALLLLLOGGG https:://doi.rrg/11 1ttpr//.1.11153//11111 wwt.:0044.ccll--.. acl ong. 428
URL
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways. Journal of Machine Learning Reseurch, 141240) 2023.
Arthur Conmy, Augustine N. MarkrrParker, Aengus Lynch, Stefan Heimersheim, and Adrià Garriga-Aloo Alonso. Towards automated circuit discovery for mechanistic interpretability. ArXiv, hts/2304414997,, 2023. /tttpsss:://api wew.atteieccollo..oooooooooeoppa
25841 1844
Yuan Feng, Junlin Lv, Yukun Cao, Xike Xie, and S. Kevin Zhou. Ada-KV: Optimizing KV Cache Eviction by Adaptive Budget Allocation for Efficient LLM Inference, aug 2024. URL http:
//arxiv. wtg.iiiorrrrare abs /2wpa////22... 11550 15Xiv:2407.11550 [cs].
Yuan Feng, Junlin Lv, Yukun Cao, Xike Xie, and S Kevin Zhou. Identify critical kv cache in Ilm inference from an output perturbation perspective. arXiv preprint arXhv:2502.03805, 2025.
Yao Fu, Rameswar Panda, Xinyao Niu, Xiang Yue, Hannaneh Hajishirzi, Yoon Kim, and Hao Peng. Data engineering for scaling language models to 128k context. In Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024. OpenReview.net, nwt. ftnt:/forumiiddTaaae
2024a. URLhttts:: /ttttp::///oenrevviiee
Yu Fu, Zefan Cai, Abedelkadir Asi, Wayne Xiong, Yue Dong, and Wen Xiao. Not All Heads Matter: A Head-Level KV Cache Compression Method with Integrated Retrieval and Reasoning, nov 2024b. URL httt:::/atxii..oto// httpg//abs/2410
arXiv:2410.1923 [es].
Suyu Ge, Yunan Zhang, Liyuan Liu, Minjia Zhang, Jiawei Han, and Jianfeng Gao. Model tells you what to discard: Adaptive KV cache compression for LLMs. In The Twelfth International Conference on Learning Representations, 2024. URL holllhtps::/// http::::///peeeeieeee wtt.o//wwweeetttettteee id=uNrFpDPMyo
11

---

Published as a conference paper at ICLR 2026
Chi Han, Qifan Wang, Hao Peng. Wenhan Xiong, Yu Chen, Heng Ji, and Sinong Wang. LM-infinite: Zero-shot extreme length generalization for large language models. Kevin Duh, Helena Gomez, and Steven Bethard (eds.), Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies ( (Volume I: Long Papers), pp. 3991 --90-, Mexico City, Mexico, June 2024. Association for Computational Linguistics. doi: 106./v663vvl maa2002..aacl-oongg2220 URL pettps:///claantooooggo org/ 2024. nwacl long.og.g... 2227
Xanh Ho, Anh-Khoa Duong Nguyen, Saku Sugawara, and Akiko Aizawa. Constructing a multi- hop QA dataset for comprehensive evaluation of reasoning steps. In Donia Scott, Nuria Bel, and Chengqing Zong (eds.), Proceedings of the 28th International Conference on Computa- tional Lingnistics, pp. 6609-6625, Barcelona, Spain (Online), December 2020. International Committee on Computational Linguistics. doi: 1/226665//12222.colngg maingg--aan..80.. URL 580/
Xiaobin Hong, Wenzhong Li, Chaoqun Wang. Mingkai Lin, and Sanglu Lu. Label attentive distillation for gan-based graph classification.
TAA////AAAAA/AAA12/21 AAAI Press, 2024. ISBN 978- 1-57735-887-9, doi: 10. 10.1609%aaii. w3tp://a.iiii.......... 28693. BtLhhtts::/// org 10. 1609/a68a 73318.28692
Xiaobin Hong, Mingkai Lin, Xiangkai Ma, Wenzhong Li, and Sanglu Lu. Aggregation mechanism based graph heterogeneous networks distillation. Proceedings of the Thirty- International Joint Conference on Artificial Intelligence, IJCI '25, 2025, ISBN 978- 978-1-956792-06-5. doi: 2015//....22223232.
24249966ijcai.22255/222
Coleman Hooper, Sehoon Kim, Hiva Mohammadzadeh, Monishwaran Maheswaran, June Paik, Michael W Mahoney, Kurt Keutzer, and Amir Gholami. Squeezed attention: Accelerating long context length IIm inference. arXiv preprint arXiv:2411.0966 2024a.
Coleman Hooper, Schoon Kim, Hiva Mohammadzadeh, Michael W. Mahoney, Yakun Sophia Shao, Kurt Keutzer, and Amir Gholami. Kvquant: Towards 10 million context length LLM inference with KV cache quantization. In Amir Globersons, Lester Mackey, Danielle Belgrave, Angela Fan, Uirich Paquet, Jakub M. Tomezak, and Cheng Zhang (eds.), Ad- vances in Neural Information Processing Systems 38: Annual Conference on Neural Infor- mation Processing Systems 2024, NeurIPS 2024, Vancouver, BC, Canada, December 10 15, 2024, 2024b. URL 12Z5200E885330444444466222222233
nips oc cc/paper files/paper/ 2024//22////// //aah/ Abstractt wwn.erooon.eeeeeeeeeeeeee oat.rsaseeeatttaacceeeeeeeeeeeee
Cheng-Ping Hsieh, Simeng Sun, Samuel Kriman, Shantanu Acharya, Dima Rekesh, Fei Jia, Yang Zhang, and Boris Ginsburg. Ruler: What's the real context size of your long-context language models?, 2024. URL/htts:://arxx ptattttp:///xiivv.o httptt/ss//aaxiooooo//////////// 240/://bb/224444.4 06654
Jiawei Huang, Meiting Xue, Chenpu Li, Huan Zhang, and Bei Zhao. Dynamickv: Data storage strategy based on partition merging of log- structured merge tree. 2nd International Conference Computer, Vision and Intelligent Technology ICCVIT 2024, Huaibei, China, November 24- 27, 2024, pp. 1-6. IEEE, 2024. doi: 10. 109.:CCCVTT62992222222111 7/doi wtt/iorrg1000 10. www.9//CCCVV6392008 2024. 1082442..2222
URL https
classfiicatiino with graph convolutional networks, Thomas N. Kipf and Max Welling. 0290 2017. URL
Wonbeom Lee, Jungi Lee, Junghwan Seo, and Jaewoong Sim. InfiniGen: Efficient generative inference of large language models with dynamic KV cache management. In 18th USENIX Symposium on Operating Systems Design and implementation (OSDI 24). pp. 155- 172 Santa Clara, CA. July 2024. USENIX Association. ISBN 933-199933393333 URL htthttt:: /ww
usen x wtt.:///oootteeeeeeee//s////eeaa
Jianan Li, Quan Tu, Cunli Mao, Zhengtao Yu, Ji-Rong Wen, and Rui Yan. Streamingdialogue: Pro- longed dialogue learning via long context compression with minimal losses. In Amir Globersons, Lester Mackey, Danielle Belgrave, Angela Fan, Ulrich Paquet, Jakub M. Tomezak, and Cheng Zhang (eds.), Advances in Neural Information Processing Systems 38: Annual Conference on Neural Information Processing Systems 2024, NewrIPS 2024, Vancouver, BC, Canada, December
12

---

Published as a conference paper at ICLR 2026
10 15, 2024, 2024a. URL utL_tttee//ppppess.....cc/peeppp http://aappprtillss/a////////// hate://600...5aaaaccccccaaaaaaa0
html
Yuhong Li, Yingbing Huang, Bowen Yang, Bharat Venkitesh, Acyr Locatelli, Hanchen Ye, Tianle Cai, Patrick Lewis, and Deming Chen. Snapkv: LIM knows what you are looking for before generation. In Amir Globersons, Lester Mackey, Danielle Belgrave, Angela Fan, UIrich Paquet, Jakub M. Tomczak, and Cheng Zhang (eds.), Advances in Neural Information Processing Systems 38: Annual Conference on Neural Information Processing Systems 2024, NeurIPS 2024, Vancouver, BC, Canada, December 10 15, 2024, 2024b. URL http:: (3200000003333511133 185d1 9bde Abst Conecettoooffeeeeceee html
nips wwwppppr.ppp...cceee http://pprfflessseeeer files/paper/ 202/4//eee222////////// hash/
Mingkai Lin, Wenzhong Li, Ding Li, Yizhou Chen, and Sanglu Lu. Resource-efficien training for large graph convolutional networks with label-centric cumulative sampling. In Proceedings New York, NY, USA, 2022. doi: 10. 10.114573855447.552220
of the ACM Web Conference 2022, WWW '22, pp. 1170-1180
Association for Computing Machinery. ISBN 9781 9781450390965.
URL htt.hhttss///doo. org. 11..0.114553444 A5 3512165
Mingkai Lin, Wenzhong Li, Ding Li, Yizhou Chen, Guohao Li, and Sanglu Lu. Multi-domain general- ized graph meta learning. In Proceedings of the Thirty-Seventh AAAI Conference on Artificial Intel- ligence and Thirty-FFftth Conference on Innovative Applications of Artificial Intelligence and Thir- teenth Symposium on Educational Advances in Artificial Intelligence,
AAAI Press, 2023. ISBN 978-1-57733--80044
doi:
URL https
/00 609 /t/.///aaa-aa vv3....33......
Mingkai Lin, Wenzhong Li, Xiaobin Hong, and Sanglu Lu. Scalable multi-source pre-training for graph neural networks. ACM Multimedia 2024, 2024. BtLphttss://opeeeeettt net/
Mingkai Lin, Xiaobin Hong, Wenzhong Li, and Sanglu Lu. Unified graph neural networks pre-training for multi-domain graphs. In Proceedings of the Thirty-Ninth AAAI Conference Artificial Intelli- gence and Thirty-Seventh Conference on Innovative Applications of Artificial Intelligence and Fif- teenth Symposium on Educational Advances in Artificial Intelligence,
AAAI Press, 2025. ISBN 978-1-577735-897--
doi: 10.1609/aaai. 339111.33191111133333333 URL https?
1/doi.org 10 0. 609 2a.00//aaa..o www.aa..vvv.111111 33325
Di Liu, Meng Chen, Baotong Lu, Huiqiang Jiang, Zhenhua Han. Qianxi Zhang, Qi Chen, Chen- gruidong Zhang, Bailu Ding, Kai Zhang, et al. Retrievalattention: Accelerating long-context I1m inference via vector retrieval. arXiv preprint arXiv:2409. 2024a.
Guangda Liu, Chengwei Li, Jieru Zhao, Chenqi Zhang, and Minyi Guo. ClusterK V: Manipulating LLM KV Cache in Semantic Space for Recallable Compression, December 2024b. (ttp
// //arxiv. www./ogg/b......... /2122.2241... 03213 arXiv:2412.03211 [cs].
Nelson F. Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. Lost in the middle: How language models use long contexts. Trans. Assoc. Comput. Linguistics, 12:157-173, 2024c. doi: 10. 10.1162/TACI, A) 006338. URI. pietps://doo org/
10. 11 62/tac 00638
Xiang Liu, Zhenheng Tang, Peijie Dong, Zeyu Li, Yue Liu, Bo Li, Xuming Hu, and Xiaowen Chu. Chunkkv: Semantic-preserving kv cache compression for efficient long-context 11m inference, 2025a.
00299 00299
Xiaoran Liu, Ruixiao Li, Qipeng Guo, Zhigeng Liu, Yuerong Song, Kai Lv, Hang Yan, Linlin Li, Qun Liu, and Xipeng Qiu. Reattention: Training-free infinite context with finite attention scope, 2025b. URL httptttt///// http://pp///ax..oo//////////////
15176
Xiaoran Liu, Zhigeng Liu, Zengfeng Huang. Qipeng Guo, Ziwei He, and Xipeng Qiu. Longllada: Unlocking long context capabilities in diffusion 1lms, 2025c. URL https: : /ttp/xax.vv..///// abs/ /206. 14429
13

---

Published as a conference paper at ICLR 2026
Piotr Nawrot, Adrian Lancucki, Marcin Chochowski, David Tarjan, and Edoardo M. Ponti. Dynamic memory compression: Retrofitting Ilms for accelerated inference. In Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024. OpenReview.net, net forrfo wttt rAAkO
2024.
Ziran Qin, Yuchen Cao, Mingbao Lin, Wen Hu, Shixuan Fan, Ke Cheng, Weiyao Lin, and Jianguo Li. Cake: Cascading and adaptive kv cache eviction with layer preferences. arXiv preprint arXiv:2503.12491, 2025.
Anton Razzhigaev, Matvey Mikhalchuk, Temurbek Rahmatullaev, Elizaveta Goncharova, Polina Druzhinina, Ivan Oseledets, and Andrey Kuznetsov. Lim-microscope: Uncovering the hidden role of punctuation in context memory of transformers, 2025. URL https://aaxiiy:// org org/oa/goooooo /abs/ /
2502. 15007
Wenqi Shao, Mengzhao Chen, Zhaoyang Zhang, Peng Xu, Lirui Zhao, Zhiqian Li, Kaipeng Zhang, Peng Gao, Yu Qiao, and Ping Luo. Omniquant: Ommidirectionally calibrated quantization for large language models. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024. URL httptttttt//ppeeae net bet/foorum ? Wwwhhhhhl
Noam Shazeer. Fast transformer decoding: One write-head is all you need. CoRR, abs/1911.02150, 2019. URL
abt1111.1111110 C2150
Jianlin Su, Yu Lu, Shengfeng Pan, Ahmed Murtadha, Bo Wen, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding, 2023. URL / 09864
Aaquib Syed, Can Rager, and Arthur Conmy, Attribution patching outperforms automated circuit discovery, 2023.
/ttp://wwwi...rg/a////111...000 10348
Jiaming Tang, Yilong Zhao, Kan Zhu, Guangxuan Xiao, Baris Kasikci, and Song Han. QUEST: query- aware sparsity for efficient long-context LLM inference. In Forty-first International Conference Machine Learning, JCML 2024, Vienna, Austria, July 21-27, 2024. OpenReview.net, 2024. URL forumoormmm//mm 1adammmmd-==AAAATMTT
https::/
net /wt/
Hugo Touvron, Thibaut Lavril, Gautier Izacard. Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302. 13911, 2023.
Ao Wang, Hui Chen, Jianchao Tan, Kefeng Zhang, Xunliang Cai, Zijia Lin, Jungong Han, and Guiguang Ding. Prefixkv: Adaptive prefix kv cache is what vision citructtoonfollowiin models need for efficient generation. arXiv preprint arXi::2412.034499 2024a.
Lean Wang, Lei Li, Damai Dai, Deli Chen, Hao Zhou, Fandong Meng. Jie Zhou, and Xu Sun. Label words are anchors: An information flow perspective for understanding in-context learning, 2023. URL httphhtpp///aattae
org/ ab5. 14160
Zheng Wang, Boxiao Jin, Zhongzhi Yu, and Minjia Zhang. Model tells you where to merge: Adaptive kv cache merging for llms on long-context tasks, 2024b. URL /tittps://arxiv...tt wwwpeiivvorggaaao abbs/
2407. 08454
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H. Chi, Quoc V. Le, and Denny Zhou, Chain-of-thought prompting elicits reasoning in large language models. In Proceedings of the 36th International Conference Neural Information Processing Systems, NIPS '22, Red Hook, NY, USA, 2022. Curran Associates Inc ISBN 7781713871088
Wenhao WU, Yizhong Wang, Guangxuan Xiao, Hao Peng, and Yao Fu. Retrieval head mechanistically 15574
explains long-context factuality, 2024. URL
Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. Efficient streaming language models with attention sinks. In The Twelfth International Conference Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024. URL https://openrevv wttp://wwwweeeeettteettee.cm forrmriduruecoo $55 1FF
net
14

---

Published as a conference paper at ICLR 2026
Dongjie Yang, Xiaodong Han, Yan Gao, Yao Hu, Shilin Zhang, and Hai Zhao. Pyramidinfer: Pyramid KV cache compression for high- tigh-throughput LLM inference. Lun- Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Findings of the Association for Computational Linguisties, ACL 2024, Bangkok, Thailand and virtual meeting, August 1 11-16, 2024, pp. 3258-3270. Association for Computational Linguistics, 2024. doi: 10. SBB55////224FFNIINGG--CCC
htttss
11t5//11201653//22222244 wtw.:0.......nmi findings 191.
Zhiyuan Yu, Wenzhong Li, Zhangyue Yin, Xiaobin Hong, Shijian Xiao, and Sanglu Lu. Contextual structure knowledge transfer for graph neural networks. Proceedings of the Thirty-Ninth AAAI Conference Artificial Intelligence and Thirty-Seventh Conference Innovative Applications of Artificial Intelligence and Fifieenth Symposium on Educational Advances in Artificial Intelligence, AAAI25/1AAA22/EAAN
doi: 10. 1609/aaii. 3391... URL https: /ttpss://dooi. /oi. org / wttp://..tt....../////////..../. www.:///aaaa.... v39121. 34312 34381
AAAI Press, 2025. ISBN 978-1-57735-8977-
Amir Zandieh, Insu Han, Vahab Mirrokni, and Amin Karbasi. SubGin: Token Generation in Sublinear Time and Memory, February 2024. /RLpptpp///rrriv.oor////// arXiv:2402.06082 [cs].
06082
Zihao Zeng, Bokai Lin, Tianqi Hou, Hao Zhang, and Zhijie Deng. In-context KV-Cache Eviction for LLMS via Attention Gate, October 2024. http://tpe//aaar...ro///////2210 arXiv:2410.12876 [es].
Hailin Zhang, Xiaodong Ji, Yilin Chen, Fangcheng Fu, Xupeng Miao, Xiaonan Nie. Weipeng Chen, and Bin Cui. Pqcache: Product quantization- kvcache for long context Ilm inference. CoRR, 0os/2407.11820) 2024a. URL.htttss://dol/
48550/ a rxiv. 2407. 12820
Xuan Zhang, Cunxiao Du, Chao Du, Tianyu Pang. Wei Gao, and Min Lin. Simlayerkv: simple framework for layer-level KV cache reduction. CoRR, abs/2410. 1380244400.186,,,. 2024b. doi: 10.48550/ ARXIV.2410.13846 URL BRLhhttss:/ //ttps:///oi...... oww.::::://iii.o//11111
Yanqi Zhang, Yuwei Hu, Runyuan Zhao, John Lui, and Haibo Chen. Unifying kv cache compression for large language models with leankv. arXiv preprint arXiv:2412.031331 2024c.
Zhenyu Zhang, Ying Sheng, Tianyi Zhou, Tianlong Chen, Lianmin Zheng, Ruisi Cai, Zhao Song, Yuandong Tian, Christopher Ré, Clark W. Barrett, Zhangyang Wang, and Beidi Chen. H20: heavy-hitter oracle for efficient generative inference of large language models. In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine (eds.), Advances in Neural Information Processing Systems 36: Annual Conference on Neural In- formation Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 2023. URL htt.a:/ppppppperrrs.pp.../eeppep
6ceefa
html
Zhenyu Zhang, Shiwei Liu, Runjin Chen, Bhavya Kailkhura, Beidi Chen, and At- las Wang. Q-hitter: A better token oracle for efficient LLM inference via sparse- quantized KV cache. In Phillip B. Gibbons, Gennady Pekhimenko, and Christo- pher De Sa (eds.), Proceedings of the Seventh Annual Conference on Machine Learn- ing and Systems, MLSys 2024, Santa Clara, CA, USA, May 13-16, 2024. misys.org. 2024d. URL ptttp:://proceeeings..tt http://sswssaa../sppppesa//////p
htm
Junqi Zhao, Zhijin Fang, Shu Li, Shaohui Yang, and Shichao He. BUZZ: Beehive-structured Sparse KV Cache with Segmented Heavy Hitters for Efficient LLM Inference, October 2024. URL
abs//4/s///10 23079 arXiv:2410.23079 [cs].
Yuxuan Zhu, Ali Falahati, David H. Yang, and Mohammad Mohammadi Amiri. Sentencekv: Efficient Ilm inference via sentence-level semantic ky caching, 2025. URL htttst:s:: : /ttpxaarvvv//////////eea//aaaav. www.xii.org/aaa. org/absg
2504. 00970
15

---

Published as a conference paper at ICLR 2026
A USE OF LLMS
This study employs GPT-4 for text polishing and writing refinement. Moreover, we utilized Claude- 3.7-Sonnet assist us with code generation.
B OUTLIER PROPERTY VISUALIZATION AND ANALYSIS
We conduct detailed investigation into the properties of neighborhood similarity of token key embeddings in long-text contexts. Figures [4 to present results from various LLMs across different attention heads on three datasets, from which we summarize the following patterns.
(1) the initial layers of large models, we observe a universally high neighborhood similarity among token key embeddings for most of the attention heads, with nearly outlier tokens detected. This suggests that initial representations remain relatively homogencous and exhibit manifold charac- teristics within the semantic space. We also notice that key embeddings in certain attention heads remain relatively dispersed.
(2) As processing proceeds to shallower layers, the phenomenon of Attention Sink begins mani- fest. This is characterized by the initial tokens of the text aggregating global semantic information, resulting in their key embeddings exhibiting significant outlier tendencies. These tokens, often positioned at the beginning, absorb broad contextual attributes and become distinct from the more localized representations other tokens.
(3) Subsequently in deeper layers, the model shifts its focus to specific parts of the document, leading to the emergence of local deviation. This means that certain non-initial tokens also start to exhibit outlier characteristics, particularly those relevant to specialized or salient content within the text. This progression highlights the model's evolving semantic focus across layers, from broad contextual integration content-specific representation.
Meanwhile, we observed very interesting phenomenon on the MK3 dataset, where the neighborhood similarity exhibits comb-shaped distribution, meaning that an outlier token appears every few tokens. We present a sample from the MK3 dataset as follows:
A special magic uuid is hidden within the following text. Make sure to memorize it. I will quiz you about the uuid afterwards.
One of the special magic uuids for 44686665-22--446551771111222222 is: 73c55500--5511-4646 8060-8de 1ob0-8dd0facbbbbb6
One of the special magic uuids for 1ha773355...P2a444000bb22222226. is: 75b568d7-35eb-4225 9775-00015256e27.
can be observed that each segment follows key+uuid format. We found that the outliers across different heads typically around the colons and periods, which indicates that each key-uuid pair likely to be encoded into punctuation marks and the surrounding tokens, resulting in the periodic appearance pattern of outliers.
Figure 14: Token Noighborhood Similarity Across Dif- Figure 15: Token Neighborhood Similarity Across Dif-
ferent Attention Heads (Llama2- 7B on Samsum)
ferent Attention Heads (Llama3 8B Samsum)
16

---

Published as a conference paper at ICLR 2026
Figure 16: Token Neighborhood Similarity Across Dif- Figure 17: Token Neighborhood Similarity Across Dif-
ferent Attention Heads (Llama2-7B BB on TriviaQA)
ferent Attention Heads (Llama3-8B TriviaQA)
Figure 18: Token Neighborhood Similarity Across Dif- Figure 19: Token Neighborhood Similarity Across Dif- ferent Attention Heads (Llama2-7B on MK3)
ferent Attention Heads (Llama3-8B MK3)
C CLUSTERING PROPERTY ANALYSIS
In this section, we primarily investigate whether SATs exhibit clustering properties across different attention heads. Unlike the outlier property analysis experiment, where we analyzed specific long-contex, sequence, here we conduct analysis on the entire dataset by calculating the clustering coefficient of key embeddings with outlier degree above the threshold for each input sequence, and then average them across the whole dataset.
As shown in Figures 20123 23, for almost attention heads, the clustering coefficient values show an increasing trend as 3 rises, and eventually stabilize high value. We find that Stabilization occurs because the selected tokens are predominantly attention sinks most cases. Additionally, we observe that the trends in the first layer generally less pronounced, which aligns with our earlier analysis: semantic information is not yet well aggregated at the first layer.
Figure 20: Clustering Property Analysis for Llama2- 7B on 2WikimQA.
Figure 21: Clustering Property Analysis for Llama2- on TriviaQA.
Figure 22: Clustering Property Analysis for Llama3. 8B on NarrativeQA.
Figure 23: Clustering Property Analysis for Llama3- 8B HotporQA.
17

---

Published as a conference paper at ICLR 2026
1000
1000
1000
40
1000
30
750
750
750
20
750
10
500
500
500
500
-30
-20
250
-20
250
20
250
-10
-20
250
-40
-30
-40
-40
20
20
40
-40
-30
-40
20
continuous distribution
(h) single-cluster distribution
40
1000
1000
20
-20
250
250
-20
250
-40
-60-40-22
60
-60-40-20
-4
20
-20
(c) loosely-clustered distribution
(d) multi-cluster distribution
Figure 24: Diverse Clustering Pattern of Key Embeddings.
D KEY EMBEDDING DISTRIBUTION PATTERN ANALYSIS
In this section, we examine various distribution patterns of key embeddings, including: 1) the continu- ous distribution case, 2) the single-cluster distribution case, 3) the loosely-clustered distribution case, and 4) the multi-cluster distribution case. For each case, we elucidate why our proposed ProtoKV method achieves effective clustering performance.
(1) The continuous distribution case typically occurs in the earlier layers of the model, where semantic information has not yet been effectively aggregated. As shown in Figure 24) the key embeddings exhibit manifold characteristics without distinct outlier points. Under these circumstances, the semantic anchors selected via outlier degree may correspond arbitrary tokens, and the SAT semantic prototypes may also reside in arbitrary regions. Nevertheless, the SAT semantic prototypes have negligible impact on the clustering performance, as the PDT semantic prototypes can still be effectively identified, ultimately achieving the desired chunk parnnk-pattiieeedd clustering outcome.
(2) The single-cluster distribution case illustrated in Figure attention heads beyond the initial layers. In this scenario, our ProtoK V method is able to identify set high-purity semantic anchors through outlier degree measurement. These anchor tokens are tightly clustered within specific region of the semantic space. Consequently, even when employing a bashing-based bucketing strategy, the resulting semantic prototypes remain almost consistent. Although this approach may introduce a limited number of redundant SAT semantic prototypes, it has negligible impact on the final clustering performance.
is observed in the vast majority of
(3) The scenario. As illustrated in Figure 24 c), although the SATs are indeed concentrated within a certain region of the semantic space, the clustering effectiveness remains suboptimal, exhibiting dispersed and loosely-organized structure. Under such conditions, only single semantic prototype (cluster center) is selected, tokens near the cluster boundaries are likely be misassigned clusters governed by PDT semantic prototypes. To address this, our hashing-based bucketing strategy performs a coarse partitioning of these SATs, thereby expanding the coverage of the regions controlled by SAT semantic prototypes. This approach enhances the capability to capture higher-order semantic anchor patterns.
distribution case represents specific variant of the single-cluster distribution
(4) The multi-cluster distribution case typically occurs when the semantics of long texts exhibit low coherence. For instance, Figure (4(d) presents visualization of this phenomenon on the multi- document 2WikiMQA dataset. Our observations are as follows: (1) Even in multi-document datasets, only small number of attention heads exhibit multi-cluster distributions for keys; the vast majority still adhere to either single-cluster or loosely-clustered distributions. (2) Tokens from different clusters generally originate from distinct text segments, whereas query tokens with light colors may be assigned to different clusters, reflecting that the answer may come from different text segments. In multi-cluster scenarios, it is highly reasonable for our ProtoKV employ hashing- based bucketing strategy to construct semantic prototypes for different SAT clusters. Moreover, since these clusters well-separated, even simple hashing mechanism effectively distinguish them.
18

---

Published as a conference paper at ICLR 2026
E SATS ARE SALIENT FOR LONG-CONTEXT INFERENCE
In this section, we provide a detailed anal- ysis of the properties of SATs as semantic anchors. We present complete experimen- tal results from the main text, as illustrated in Figure 26 to 29 For Llama2, it can be observed that the first two layers, out- lier tokens are not the primary focus of the attention heads. However, starting from the third layer, the model begins to attend to outlying SATs, The situation is slightly different for LlaMa3. In the LlaMa3-8B model, it can be observed that all atten- tion heads tend to focus on outlier tokens, including those in the initial layers. Addi- tionally, found that even after removing the first several tokens (i.e., the attention sink), the model still attends to the first group of tokens with the highest outlier degree, we give the results in Appendix [F] This demonstrates that SAT are distributed throughout the text and commonly serve as semantic anchors.
TriviaQA
Narrative
2WikiMultthop@A
F1 Score
8
F1 Score
20
30
88
10-
20-
10
25
100115160
10015116
Dropped Parss
Dropped
Dropped Parrs
aM
SAMSum
RepoBench-P
Rouge-L
20-
Rouge-L
40
Edit
10-
20
25 20 it 100 125
150
22 50 75 100 150
25 20 75 100 125 150 Drapped KV Parss
Drapped KV Pars
Dropped KV Pars
Random
Attontion Score
Outter Degree
Figure 25: Dropping KV pairs with highest outlier degrees can cause sharp performance drop for llama3.
Additionally, we also present the impact of removing the top-k KV pairs with the highest outlier degrees on Llama3's performance in processing long-text inputs (while the main text demonstrates results for Llama2), As shown in Figure [5] removing 25 KV pairs with the highest outlier degrees causes the model to almost completely malfunction. This indicates that tokens exhibiting outlier properties generally function as semantic anchors.
Figure 26: Group-wise semantic anchor property anal- ysis. (Llama2-7B on TriviaQA)
Figure 27: Group-wiss semantic anchor property anal- ysis. (LlaMa3 8B on TriviaQA)
Figure 28: Group-wise semantic anchor property anal- ysis. (Llama2-7B on Samsum)
Figure 29: Group- wise semantic anchor property anal- ysis. (LlaMa3-8B on Samsum)
F ALIGNMENT WITH PRIOR WORKS
We compare our findings with prior work. Specifically, (Xiao et al. 2024) observed the "attention sink" phenomenon during large model generation, where the initial tokens disproportionately absorb semantic information. In the visualization of Appendix | we can observe that the initial few tokens often exhibit extremely low neighborhood similarity.
Furthermore, to validate whether the SATs we propose as semantic anchors are equivalent to the attention sinks mentioned in prior work, we conducted additional experiments. Specifically, we removed the first ten tokens from the long-text prompt (prior studies indicate that the first four the most critical; here, we adopt a more aggressive approach by excluding the first ten). The
19

---

Published as a conference paper at ICLR 2026
remaining tokens were then divided into ten groups based their outlier degree, and measured their inference-stage cumulative attention.
As illustrated in Figure 30, we observed that even after removing the attention sinks, large models still tend to focus on tokens with higher outlier degrees. This demonstrates that our SATs not only encompass attention sinks but are also capable of identifying anchors located elsewhere in the text.
(Wang et al. 2023) demonstrated that in in-context learning, label words act as anchors that cluster sample semantics. Our study reveals that label word tokens also exhibit significant outlier char- acteristics. We conduct experiments for LlaMa3-8B on two text classification datasets. These two datasets are: the binary-class movie review dataset IMDB and the multi-class news dataset AGNews.
Layer
Ruviow (lat)
Sorkmort (lazel)
Figure 30: Dropping Attention Sink Tokens
Figure 31: Label words are outliers.
The constructed prompt templates for each dataset are as follows: 1) For IMDB, the template is "Review: [text]. Sentiment: [label]," where "text" represents the movie review content and "label" indicates positive/negative sentiment. 2) For AGNews, the template is "Article: [text]. Category: [label]," where "text" represents the news article content and "label" denotes the category label. We constructed multiple sets of examples and combined them into a final prompt, which was then fed into the large language model for prefilling. Subsequently, we measured the outlier degree of different tokens in the "text" portion and the "label" portion respectively. The results, as illustrated in Figure 3T, show that the outlier degree for "text" is almost zero, while the outlier degree for "labe" is generally higher and exhibits an increasing trend with deeper layers. This indicates that label words gradually become semantic anchors as layer depth increases, which aligns with findings in (Wang et al. 2023
G NEEDLE IN A HAYSTACK ANALYSIS
We investigate the trends in the distribution of key embeddings of tokens in the "NEEDLE IN HAYSTACK" experiment. First, we present the key conclusion: in retrieval heads, the neighborhood similarity of needle texts lower than that of haystack texts, indicating that large language models (LLMs) can perceive textual incoherence and assign higher outlier degree incoherent texts, making them more likely to be retrieved subsequently.
Specifically, retrieval heads (Wu al. 2024 refer to specialized attention heads in transformer-based mod- els that are mechanistically responsible for retrieving critical information from long-context inputs. These heads dynamically activate identify and prioritize salient contents (i.e., "needles") within extensive con- texts (i.e., "haystacks"), enabling the model main- tain factual consistency and coherence in long-range dependencies. Taking Llama2 as an example, we fo-
Layer @Head
Layer 21 Head 30
Neighbutoodd Smiltrity
Neaca Hayttck
Needan Haystace
0.50
0.
0.70
0.70
0.06
0.66
c.
02
04
0.00
600
0.4
6a
Poeklon
Postion
Figure 32: Needle contexts are outliers.
cus on two retrieval heads identified by prior work: Head 16 in Layer 9 and Head 30 in Layer 21. As shown in Figure 32 we observe that the neighborhood similarity of needle texts in these retrieval heads is consistently lower, reinforcing their role in detecting out-of-context salient information.
H SELECTING KV PAIRS VIA OUTLIER DEGREE
In the main text, we propose using the outlier degree as a criterion to select and retain important KV pairs, termed SATKV. For long-text input of length n, always retain the last & KV pairs to
20

---

Published as a conference paper at ICLR 2026
ensure query integrity, and select from the preceding - KV pairs based on their outlier degree. is noteworthy that SATKV eliminates the need for query-based attention computation; instead, achieves effeettie KV compression using keys alone. This implies that SATKV's compression results may exhibit universality across diverse queries. Moreover, since x is typically smaller than both the budget B and the observation window size, SATKV operates more efficiently.
Table 5: Time Complexity Comparison
LLM
H2O
SnapKV SATKV
((1)
(nBd)
((nCored) (
I PSEUDOCODE
We present the pscudo-code for our ProtoKV as follows, it is worth noting that all loop statements in this code can be executed in parallel. We analyze the time complexity for it, assuming a key dimension of d. Computing the k-Neighborhood Similarity to obtain the SAT requires O(ned) time. Constructing the hybrid semantic prototypes has complexity of ((d)), while assigning prototypes to each token incurs O (nkd) time. Finally, the observation window selection step takes (nL time. Since k is typically larger than both Loss and in practical experiments, the overall time complexity can be approximated as O(nkd).
Algorithm Algorithm ProtoKV
Require: Key vectors (kr [/= chunk size v, hash bits r, buckets SAT selected number observation window size Lobs, attention head number H, Key- Value budget size B Ensure: Compressed KV cache (K(M) (( h=1
1: for each head h E [H] do
2: Calculate (h)()) according to Eq.
3: O - Top-p of (t)
4: for m 1 to do
5:
Cm < [k 1mmmmm))///JHH km[nej
(PDTs) Σ 2m.
6:
C.n [0]
7:
end for
8:
for kj E O do
9: o(K,) 22/r cos (Wy, + b)
10:
H(k,) + Binarize (okk))
end for
12: for
1 to do B, (k) (k)) }}
13:
EB, kj [BJ
14:
(SATs)
15:
end for
16: M t (CPDTRi jm U (c (SATSI)
for each token k; do
18:
C(ki) + arg Max k. 19: end for
20:
1 to n do 21: ((h)
for
Lpongs
9 K;
m= Ineets
22:
(1)) f- [ Σo(k) (th)
23:
end for
24: ((h) <- Top-(B - Loos) indices of 5(N)
R(h) [k(")) JiEroN) U U(k(h)
25:
(h) + {v: (1)) ( hero (v, (h) Leprerno
26:
27: end for
21

---

Published as a conference paper at ICLR 2026
J LONGBENCH DATASET DETAILS
Dataset LongBench is a large-scale benchmark dataset designed for evaluating language models' capabilities in understanding and generating long texts. It covers various types of tasks including, but not limited to, SingleDooummen Question Answering (QA), Multi conlti-Doeument QA, Summarization, Few-shot Learning, and Synthetic tasks. The aim is to comprehensively assess models across different application scenarios.
Here are some specific tasks included in the LongBench dataset along with their characteristics:
NarrativeQA: Focuses on understanding narrative texts, requiring models read and answer questions about stories or narratives.
Qasper: Involves asking and answering questions based on academic articles, testing the model's ability to understand scholarly literature.
MultiFieldQA-en: Covers QA tasks across multiple fields, enhancing the model's capability to understand texts from diverse domains.
HotpotQA, 2 2WikiMultihopAA. MuSiQue: These tasks emphasize reasoning and information integration across multiple documents, challenging the model's ability to find answers in multi- document environment.
GovReport, QMSum, MultiNews: Concentrate on extracting key information and generating summaries from lengthy texts, assessing the model's summarization capability.
TREC, TriviaQA, SAMSum: Evaluate the model's learning ability and domain-specific knowledge acquisition through few-shot examples.
Synthetic tasks designed to test the model's performance under specific conditions, such as document counting or retrieval accuracy.
PassageCount, PassageRetrieval-en:
LCC, RepoBench-P: Involve code understanding and evaluation of editing similarity, catering the unique requirements of programming languages.
Each task comes with its own set of evaluation metrics (e.g., Fl Score, Rouge-L, Accuracy) to quantify model performance. Moreover, LongBench includes texts from different languages and domains, ensuring broad applicability and linguistic diversity of the models. This dataset plays crucial role in advancing the field natural language processing, especially in improving models' abilities to handle long texts. Detailed information is demonstrated in Table 6
Task
Task Type
Source
Eval metric
Avg len
Language
NarrativeQA
Single-Doc. QA
Literature, Film
F1
18,409
EN
MIT License
Qasper
Single-Doc. QA
Science
F1
3,619
EN
MIT License
MuliiFietQQA-en
Single-Doc. QA
Multi-field
F1
4,559
EN
MIT License
HotpoiQA
Multi-Doe. QA
Wikipedia
FI
9,151 EN
MIT License
2WikiMulthhuQAA
Multi-Doc. QA
Wikipedia
FI
4,887 EN
MIT License
MuSiQue
Multi-Doc. QA
Wikipedia
11,214 EN
MIT License
RovRep
Summarization Goverment report Meeting
Rouge-L.
8,734
EN
MIT License
QMSum
Summarization
Rouge-L
10,614
EN
MIT License
MultiNews
Summarization
News
Rouge-L
2,113 EN
MIT License
TREC
Few shot
Web question
Accuracy
5,177 EN
MIT License
TriviaQA
Few shot
Wikipedia, Web
Fl
8,209
EN
MIT License
SAMSum
Few shot
Dialogue
Rouge-L
6,258
EN
MIT License
PassageCoun Count
Synthetic
Wikipedia
Accuracy
11.141
EN
MIT License
PassugeRetrieval-enn
Synthetic
Wikipedia
Accuracy
9.289
EN
MIT License
LCC
Code
Github
Edit Sim
1,235
Python/CHilaaa
MIT License
RepoBench-P
Code
Github
Edit Sim
4,206
Python/Java
MIT License
Table An overview the dataset statistics in LongBench.
K LLM MODEL DETAILS
Meta-Llama-23 --B-aa-------n-nuut an 3B-parameter instruction-tun variant of LLaMA-3, optimized for dialogue tasks. Using transformer architecture with SFT and RLHF, it features 128K vocabulary and GQA for efficiency. The model supports 8o-ctnt (extendable to 128K) and demonstrates strong performance in text generation and reasoning tasks.
22

---

Published as a conference paper at ICLR 2026
Distral-7B--nstructt-0.. is a 7.3B- 1-BB-parameter instruction-tuned model by Mistral AI, featuring 32K context length via optimized RoPE embeddings. With grouped-query attention for efficiency, it excels in conversational and coding tasks while supporting GGUF quantization. Benchmarks show it outperforms comparable 7B models, particularly in code generation.
Llama2-7B-chat is Meta's 7 billion parameter chat-optimized language model, fine-tuned for dialogue applications using RLHF. The model features a 4K token context window and demonstrates improved safety and helpfulness compared its base version. It achieves strong performance in conversational tasks while maintaining efficient inference through optimized transformer architecture.
Configuration
LlaWA-33SB-IIstrucct
Mistrall7B--nnstrutt-...
Llama2-7B-chat
Hidden Size
4.09
4,09
4.096
Layers
32
32
32
Q Heads
32
32
32
KV Heads
8
8
32
Attention Heads
32
32
32
Max Position Embeddings
8.192
32,768
4,096
Intermediate Size
14,336
14,336
11,008
Vocabulary Size
128,256
32,000
32,000
Table 7: Configuration of Models.
L DETAILED RESULTS FOR LONGBENCH
Singl-Deeament Q4
Molti-Treemmnt QA
Summarization
Fow-shee Learnig
Symthetic
Code
Moda
Size
Methed
NArQA
Quiper
MP MF-08
HappiQA
3WIIMQA
Musique
GevRqport
QMSaa
MoltNNws
TREC
Trivia24
SAMSuM
PCiun
RB-P
18409
1619
4559
11214
8734
10514
2113
5177
11141
4206
FullKV
25.16
30.29
40.43
45.35
37.04
23.84
28.60
23.34
26.33
75.00
90.23
42.65
5.10
70.0
50
55 60
42:34
SLM
17.47
8.55
21.31
32.86
26.28
15.54
17.91
20.42
211.66
45.00
73.36
30.7%
5.75
68.50
4%
49.31
31.29
H20
21.58
12.54
28.57
99.85
78. 62
18.55
30.23
22.16
20.14
35.50
85.62
39.19
5.83
69.
50
54
46
50.81
34.66
LiMAA-1-88
SnapKV
19.69
21.36
20.07
51.84
35.92
Ioanec
ChunkKV
23.08
16.32
50.18
37.25
28.60
19.12
19.49
20.74
20.45
53.00
88.25
39:57
5,81
68.00
58.13
53.75
36.36
Pyramod
21.80
16.65
30.75
38.48
29.80
19.25
19.92
22.06
20.12
56.50
85.95
38.20
5.92
68.00
57.88
51.54
37.16
PunaKY
17.05
19.35
19.83
22.31
37.52
SLM
17.95
11.09
23.85
37.83
16.02
20.94
$2,00
79.68
69.50
H20
23.67
10.85
32.70
41.57
31.08
18.91
22.25
22.91
23.69
41.00
90.36
40.19
5.54
69.50.7.55
52.16
36.55
SnipKV
23.32
20,31
$7.35
42,70
51.08
30.47
22.63
23.04
23.03
71.00
90.39
30.78
5.50
69.
60
55.62
39.51
ChunkKV
20.12
20.68
22.94
23.02
69.
Pyramid
23.46
18.76
35.05
42.33
31.55
20.73
23.37
23.11
24.37
72.00
90.45
39.54
5.50
69.50
59.25 54,87
39.61
ProtoKV
23.58
19.92
36.38
43.72
32.29
20.89
23.25
22.96
23.42
70.00
90.81
40.07
5,50
69.50
62
55 0.69
39.95
PullKV
25.07
32.92
69.34
39.77
27.32
16,83
32.87
24.24
27.10
70,00
85.37
43.30
2.75
59,25
56.86
50.48
40:29
SLM
35.11
27.25
9:80
19.02
43.50
75.12
36.50
2.67
27.17
43.79)
28:34
H20
19.99
20.34
58.60
28.50
21.63
12.88
201.5
22.61
22.08
$3.00
81.29
39.75
2.20
75.58
49
44.27
33.53
Mostral-18
SnapKV
22.14
21.14
42.95
32.96
22.12
14.12
19.19
21.89
21.01
54.00
83.77
39.92
2.51
06.50.51.811
46.51
35.84
ChuckyV
22.04
14.29
22.95
65.25
51
36.22
Pyramid
22.30
22.52
43.65
33.07
22.45
15.73
20.56
22.52
21.36
64.00
83.84
43
2.74
67.95
51.655144440
46.47
36.29
ProsoKV
23.11
23.70
44.89
36.12
22.88
15.57
21.63
23.75
22.49
67.50
81.39
41.96
3.30
53.34.33..5441
47.95
37.54
SLM
19.26
17.78
36.82
27.74
22.78
10.53
74.47
19.84
25.48
$1.00
76.39
40.24
2.50
31.90
46
15 56
31 14
23.22
41.76
14.03
23.53
22.96
53.50
1.66
50.77
46.70
26.39
SnapkV
23.08
25:95
45.04
34.79
24.75
14.41
24.14
23.69
24.47
67.50
85.64
41.51
1.95
68.11
53.7449.31
38.19
ChuniiVV
23.60
26.22
43,53
35.16
25.45
14,53
24.41
23.52
24.13
67.50
84.31
41.42
2.60
70.25
53.82
50.64
38.54
Pyramil
13.65
24.79
48.23
38.37
ProsoKV
23.76
26/02
48.82
34.96
26.32
14.66
34.60
23.62
24.91
70.50
85.02
42.76
2.50
71,40
55.62
51.14
39:25
Full
14.82
9.5
23.76
7.35
10.71
9.23
25.63
23. 79
26.51
65.00
89.16
34.28
2.50
9.50
68.24
51 83
29.72
SLM
10.12
4.94
15.8
5.93
9.07
3.05
18.07
19:30
18:30
42.50
76.97
24.18
2.00
3.11
61.47
44.44 25
22:93
4.55
21
24.09
Liamat-71 cba
SnipKV
13.13
5.84
21.62
7.12
9.19
3.90
18.91
21.41
18:21
45.00
81.12
27.85
1.60
7.02
61.48
54.87
25
ChunkaVV
13.42
5.81
21,80
7.47
9.02
3.85
19.50
21.42
13.09
51.00
84.32
26.72
1.50
7.95
62.80
55.47
25.51
Pyramid
13.78
5.75
22.37
7.62
9:68
3.95
19.24
20.47
59,00
84.38
29.42
1.50
8.22
62.24
54.51
26.45
PussaCV
5.94
22.08
7.76
9.29
4.099
201.5
21.601
59.50
84.69
1.50
8.18
63.22
56.97
26.94
SLM
12.74
4.94
15.8
5.93
9.12
3.48
25.70
19:31
24.57
54,00
81.67
31.47
2.00
4.38
61.87
52.20
25.60
H20
14.55
5.95
18.67
6.42
8.67
4.17
23.45 60
22.07
22.72
56.00
82.66
30.48
2.50
8.89
58.83
56.83
26.45
75
SnspKV
4.12
22.39
1.52
64.94
56.88
28.01
ChunkkVV
17.55
7.03
21.29
7.26
10.22
4.47
24.15
22.75
23.15
65.00
85.52
37
1.68
8.54
65
12
28.39
Pyramod
17.84
7.28
20.37
7.14
10.47
4.29
23.59
22 30
22.41
54.00
85.17
32.72
2.67
8,23
65.75
57.50
28,30
ProsoYY
17.29
7.34
20.96
7.58
11.43
4.80
24.73
22.57
22.89
58.00
85.78
34.51
1.68
7.67
66.38
50.34
29.11
Figure 33: Performance comparison the LongBench dataset for full KV cache, extant KV baselines (including StreamingLLMM, H20. SnapKV, ChunkKV, PyramidKV) and our ProtoKV.
23

---

Published as a conference paper at ICLR 2026
M RELATED WORKS
KV Cache Compression focuses on retaining critical key-value pairs while permanently discarding
unimportant ones to optimize memory and inference. Two dominant strategies emerge: (1) static
methods with prefill-phase token selection (Ge et 2024 Li et al. 2024b Zeng et al. 2024), and
(2) dynamic approaches updating cached entries via attention-basee metrics or structural patterns
during decoding (Xiao et al. 2024: Han et al. 2024 Zhang al. 2023 Zhao et al.. 2024 Recent
advancements address persistent eviction challenges through multi-tie caching and asynchronous
retrieval (Lee al., 2024 Tang al., 2024 Zhang al., 2024a Hooper et al. 2024a Liu et al.
2024a). However, most of these solutions fail to efficiently preserve semantic coherence, leading to
suboptimal selection decisions.
Semantic-Level KV Cache Selection Optimizing KV cache at semantic level is crucial to maintain- 2025a) group tokens into semantic chunks, retaining the most informative segments and discarding redundant ones to enhance long-context inference efficiency. Recently, clustering- based KV compression approaches (Liu et al.) 2024b; Hooper et al.) 2024a) clusters tokens semantically and recalls them at the granularity of semantic clusters.
ing output coherence. (Li al. 2024b
Discussion: Why Clustering? In our view, clustering- -ased KV compression can be explained via graph neural networks (GNNs) Kipf & Welling (017): Hong et al. (2025); Lin et al. (2025); ) Yu et al. (2025); [Lin et al. (2023), as the two are closely related. In GNNs, a node's importance is influenced by its neighbors, similarly, KV pair's importance comes from aggregating its neighboring KVs. SnapKV defines neighbors as fixed-window tokens and uses max pooling, while clustering-based methods group tokens by key similarity and apply mean pooling, which is more reasonable. However, like GNNs, clustering- methods also face scalability challenges Lin et al. (2022) : Hong et al.
(022)) Lin et al. (2024).
KV Cache Budget Allocation LLMs' hierarchical layers exhibit distinct information extraction patterns, motivating adaptive memory allocation across layers/heads. Layer-wise strategies (Cail
et al.] 2024] Yang et al.] [224]] Huang et al.) [022]] Zhang et al., (2024b) prioritize resource distribution
by analyzing attention concentration gradients, where lower layers retain uniform contextual signals
while higher layers preserve semantic focal points. Head-wise approaches (Feng et al.) 2024 Zhang
al.] [024c] Fu al. 2024b) further enable finer-grained optimization through intra-layer importance
differentiation.
N FURTHER ANALYSIS FOR LOCAL DEVIATION PROPERTY
GPTO56-200 de AME7404
U1ted 10.00-7..00.00 AMM2200
Oweed-42l on ABMEHOM
Sovcsddr AM-
Ou Freeutityy
0.0
Our Curtse Freeushcy
0:55
E005-
Degne
Outter Degres
Outler Doges
Outler Dogno
Figure 34: Outlier degree distribution analysis for different models on AIME2024 datasct.
E:127
Ourrelthe Freco
cde
L1100
0.52
0.00
082
0:00
0.00
080
E:00
Outeer Ougres
Orterr Degrus
Outtr Dogns
Outler Dagin
Figure 35: Outlier degroe distribution analysis for different models on HMMT datasct.
GP7055--200 on VCSUM
Ladeo 73 VnVCSUM
Oween--0 VCSUM
Starcoder VCSLM
OuT Pa Freeuretty
dhe Frieuaecy
odi-
E1100
E005
Oegree
Outter Cogres
Outber Dogis
Out Dogno
Figure 36: Outlier degree distribution analysis for diffcrent models on VCSUM dataset.
24

---

Published as a conference paper at ICLR 2026
OFTI0S008 es YCSMM
VO3UM
26111
Figure 37: Layer-wise distribution analysis for SAT' across different models using VCSUM dataset.
In this section, we further validate the widespread presence of the Local Deviation Property across diverse large language model architectures and various types of datasets. To this end, we examine representative models from three major attention architecture families: the original multi-head attention (MHA), represented by L1ama2- grouped-query attention (GQA), represented by the recent Dwen3-4B, and multi-query attention (MQA), which is less commonly used today and for which we employ the earlier model St. St.rcoder. Additionally, test GPTOSS-20ES an siitteentioo-wwh-siiiinn implements.
open-source mixture-of-experts model with
In terms of datasets, alongside LongBench used in the main text, conduct key distribution anal- yses on the 1IME-202 and HMMT datasets, both of which are mathematical problem datasets. For instance, AIME comprises challenging problems from the Ameri can Invitat a Mathematisn ics Examinat ion, characterized by rigorous, logically sound solutions with minimal noise. Each problem is formatted as "Problem: [...], Solution: [...]". and then fed into to the large language model for prefilling. After that, we extract the key vectors head-wise, and analyze the outlier degree distribution of them.
Our experimental results demonstrate that:
The Local Deviation Property persists across all tested attention architectures, including the mixture-of-expprts model. This is reflected in the presence of tokens with high outlier degrees when processing various types of long-context data, as visualized in Figures 34/ee63336366
The phenomenon is observed consistently across both long-context benchmarks (LongBench and RULER) and mathematical reasoning datasets. Moreover, the density distribution of outlier degrees appears to be model-dependent rather than dataset-specific.
We also provide layer-wise distributions of SATs in Figure 37 Specifically, we quantify the number of outlier keys per KV head within each layer and visualize their number distributions using box plots, with SAT detection thresholds set at 1.5, 1.75, and 2. The results indicate that th's e majority of KV heads contain outlier key vectors (i.e., SATs), which aligns fully with the conclusions presented in the main text.
DYNAMIC DECODING DESIGN
Different from focusing on the prefilling stage, Dynamic Decoding Design focuses on the management of KV cache during the reasoning stage of LLM. Specifically, we allocate memory for two components: a cache of bud- get size Buulge to store retained KV tokens, and buffer of size Bruffer for newly generated text tokens. The total memory requirement is thus Busul Brudger Buufier- Af- ter the model generates each fixed-length text segment in the buffer, R-KV performs KV cache compression. At the end of each text segment, the last a tokens are al- ways retained in the cache as observation tokens. Next, we concatenate the existing Bhudge tokens in the cache with the first Bruffer - a tokens in the buffer, resulting in Brudget Braffer a candidate KV tokens. Each can- didate is assigned a selection score, and we select the top k Boudge tokens to fit in the rest of the cache budget, in addition to the a observation tokens. This process compresses the KV cache while preserving critical context, enabling efficient memory utilization during autoregressive decoding.
Table 8: Performance (%) for dynamic de- coding setting with Llama3-8B.
KV Budget
Benchmark
Method
1024
2048
FullKV
82.38
82.38
H2O
71.64
78.15
MATH
SnapKV
74.43
80.50
ChunkKV
76.69
80.42
ProtoKV
79.67
81.39
FullKV
49.79
49.79
H2O
19.51
29.87
AIME
SnapKV
17.73
32.76
ChunkK
18.32
34.98
ProtoKV
21.05
38.26
25

---

Published as a conference paper at ICLR 2026
We evaluate the performance of different KV Cache compression algorithm the MATH and AMIE dataset, both of which are mathematical problem datasets. shown, our method achieves consistent improvements The results are presented in Table 8 As existing compression techniques. For instance, under a KV budget of 1024, ProtoKV outperforms H2O by 8.03%, SnapKV by 5.24%, and V by 2.98%. When the KV budget increases 2048, ProtoKV still maintains clear advantage. On the more challenging AIME benchmark, ProtoKV demonstrates even more significant advantages. With 1024 KV budget, it surpasses H2O, SnapKV, and ChunkKV by 7.89%, 18.73%, and 14.90% respectively. When the KV budget expands to 2048, the performance gap further widens, with ProtoKV achieving improvements of 8.39% over H2O, 5.50% over SnapK V, and 3.28% over ChunkK V. These observations indicate that: ProtoKV remains effective not only during the prefilling stage but also under dynamic decoding scenarios.
P ADDITIONAL EXPERIMENTAL RESULTS ON RULER
In the main text, we evaluated various KV compression meth- ods using the RULER benchmark with 4K context length. To further assess the scalability of these methods under more demanding conditions, we extend the evaluation to longer con- texts of 16K and 32K. We report the performance of Llama3- 8B with budget size of 512 under different KV compression approaches. As detailed in Table 9 our proposed ProtoKV achieves superior performance, attaming 68.3% and 57.2% on the 16K and 32K contexts, respectively. This represents a sub- stantial margin over competing methods. Notably, ProtoKV outperforms H2O remarkably under the 16K and 32K settings. When compared to other strong baselines, ProtoKV maintains consistent and non-trivial improvements: 6.7-7.1% over SnapKV, 3.6-4.6% over ChunkKV, and 6. 6.146.4% over PyramidKV. While the performance gain ProtoKV over other advanced methods was marginal in the 4K context setting discussed in the main text, these new results demonstrate that its advantage becomes pronounced and increasingly significant as the context length extends, highlighting its superior scalability and effectiveness in long-context scenarios.
Context
Method
16K
32K
FulIKV
85.7 79.9
H20
27.4
24.6
SnapKV
64.0 53.4
ChunkKV
65.9
54.7
PyramidKV
53.9
ProtoKV
68.3 57.2
Table 9: Performance (%) on Ruler for Llama3-8B with budget size 512.
Q CONFIDENCE INTERVALS ANALYSIS WITH TEMPERATURE SAMPLING
To more rigorously validate that the performance improvements of our method are robust and not merely attributable to the inherent stochasticity of large language models, we for the first time conduct a significance analysis of different KV cache algorithms. Specifically, we configure the model to perform sampling during the decoding phase, with temperature of 0.6 and top-k value of 20 (i.e., sampling is constrained to the top 20 tokens in the vocabulary at each generation step). We perform five decoding runs each sub-task dataset in LongBench and report the mean and standard deviation for LlaMa3-SB.
[a] Single Decumest QA
Hutel Decument QA
(4) Sammartaation
Fycans
24
KMI Dudgeet Size
KV Budget She Synthetic
KV Dudgee Sizw In Cade Generation
Fle--Shee Shorthhttth Learning
Figure 38; Results for Temperature Sampling.
26

---

Published as a conference paper at ICLR 2026
We present the experimental results in Figure 38 The analysis reveals three key findings: First, our approach consistently achieves state-of-the- performance across various subtask benchmarks. Second, in specific domains such as Summarization, the lower confidence bound of our method exceeds the upper bounds of competing baselines, demonstrating that the observed improvements are statistically significant and not merely artifacts of L.LM stochasticity. Third, method exhibits the smallest variance in performance scores, indicating enhanced robustness and more reliable model outputs compared existing techniques.
R
COULD PROTOKV ACHIEVES FULLKV'S LEVEL OF PRECISION?
In our preliminary experiments, observed discemible performance gap between our pro- posed ProtoKV and the FullKV baseline. To
Table 10: Average score on Longbench for Llama-3B.
10%
20%
30%
investigate whether ProtoKV can attain the pre- cision level of FullKV under an increased bud- get allocation, we conducted additional experi- ments on LongBench using LlaMa3-8B. Specif- ically, we adhered to the experimental protocol of ChunkKV and evaluated performance using budget percentages relative to the total context size. As illustrated in Table 10 given Long average input length of approximately 10K tokens, a 20% kens, which suffices to nearly match FullKY's performance. Moreover, with 30% budget, ProtoK slightly surpasses FullK V, which we attribute to the selective eviction of certain KV cache entries that helps the model focus more effectively on task-relevant text segments while reducing attention to less relevant information.
Budget
Budget
Budget
FullKV ProtoKV
42.34
42.34
42.34
41.66
42.27
42.49
Improvement
-1.61%
-0.16%
+0.37%
to roughly 2048 to-
S EXPERIMENTS FOR MORE RECENT LLMS
In addition to the three large language models discussed in the main text, we have extended our evaluation to include two more recent and advanced models: thi-3.5-mini-insttuue and Mistral-7B. Instruct-vO.3.
Phi-3.5-mini is a lightweight, state-of-the open model constructed from the dataset used for Phi-3, which incorporates synthetic data and filtered publicly available web data, with strong emphasis on high-quality, ienoning-intensivv information. As member of the Phi-3 model family, supports an extensive context length of 128K tokens. The model has undergone rigorous enhancement process that combines supervised fine-tune tuninunnng, proximal policy optimization, and direct preference optimization to ensure precise instruction adherence and robust safety measures.
Mistral- 7aerral-7--nstuuct--oo is a 7-billion parameter instruction-tune tuned language model developed by Mistral AI. It utilizes advanced architectural features such as Grouped Srouped-Query Query Attention and Sliding- Window Attention, which enhance computational efficiency and performance across various tasks including question answering, text generation, summarization, and logical reasoning. The model fully open-source and designed for efficient deployment on consumer- grade hardware, making suitable for both research and practical applications.
In Table II we present the performance of these two models on the LongBench benchmark using different KV cache compression strategies. In terms of average results, the ProtoKV method achieves the most significant performance improvement on both models. For Mistral- Mistral-7B-Innttree In-trrtuutt-- Pro- toKV attains an average score of 43.56, which represents 1.6% improvement over the second-best compression method, ChunkKV (42.89). On the Phi-3.5-mini model, ProtoK V achieves an average score of 18.49. outperforming the second-best SnapK V method (17.80) by 3.9%. These results indicate that ProtoK V delivers consistent and significant performance improvements across different recent models.
27

---

Published as a conference paper at ICLR 2026
Single Document
Model
Medied
NimvA
Quaper
MF- cc
HoppoRAA
2WIUMMA
Musique
CovRapon
QMSum
MotuNows
TREE
TrimiQA
SAMSum
18409
3619
4559
9151
4857
11214
8734
10614
2113
5177
$209
6258
11141
9389
1235
4206
PullKV
22.17
22.29
37.63
25.19
27.59
15.21
32.41
21.96
23.39
67.50
86.34
15.32
3.65
79.79
34,89
41.53
37.45
Phi 5-5 mini
SLM
3.52
11.89
10.64
8.58
7.29
3.88
26.75
11.00
14.68
48.84
4.36
6.94
1.87
5.12
17.51
15:51
12:38
H20
5.63
11.77
13:41
13992
10.00
4.83
25.35
15.54
15:44
28.75
15.67
19.24
2.54
57
03.58
18.82
14.67
15.81
ME5
SnapKV
5.67
16.65
17:23
18.56
15.24
7.56
29.22
16.95
15.42
48.
20.95
20.11
2.52
16,00
17.88
17.00
17:80
ChunkXV
6.18
16.32
16.18
16.25
14,60
7.12
27.49
17.74
16.45
48.25
21.25
19.57
3.21
15.00
19.99
15.73
17.61
Pyramid
6.00
16.38
16.30
16.79
13.11
7.13
27.40
17.00
16.15
69.29
21.86
19.20
2.83
15,50
18.16
16.26
17.51
PonaKV
6.26
17.05
16.95
17:87
16.25
8.35
26.91
18.32
16,92
49.37
22.05
20.74
3.37
16,00
19.84
17.61
18:49
FallKY
25.55
40.03
$1.40
45.28
36.39
24.53
33.85
24.51
27.16
73.00
91.33
50.65
6.50
93.08
62.13
62.70
46.76
Mistal Insuutt-i..
SI.M
18.99
24.40
38.23
32:57
20.04
14.73
20.02
19.35
26.23
69.50
68.44
18.98
6.00
8.83
18.00
34.58
20.24
1120
23.11
32.99
48:33
42.66
34.23
22.19
28.09
22.8
25.57
48.00
$8.95
46.81
3.00
64.50
58.25
55.58
42.08
1634
SnapKV
23.89
36.25
49.38
43.65
35.28
22.95
28.59
23.5
25.20
72.30
$8.75
47.01
4.50
64.50.5
58.51
58,35688800 58.39
42.69
ChunkXV
23.51
36.11
49.49
44.99
35.49
25.45
29.32
22. 86
25.47
72.00
88.76
47.54
4.50
64.55
54
58.45
42.89
Pyrumid
23.43
36,04
49.67
48.81
35.67
25.12
29.36
22.90
26.13
72.00
$8.91
47.18
4,50
64.50
58.98
58.12
42:84
PrnoKV
23.55
36.50
50.63
45.13
36.18
24.34
30.31
23.82
26.90
73.00
89.75
48.32
5.00
64,30
59.89
59.31
43.56
Table 11: Performance comparison the LongBench dataset for recent L.MMs.
28


Reviews:
[{"review_id": "cq17GdgvB8", "reviewer": "Reviewer_mS4R", "summary": "The authors discover that while most tokens demonstrate high similarity (in key space) with their contextual neighbors (position-determined tokens, PDTs), a subset of tokens (dubbed \"semantic-anchored tokens\", SATs) deviate from this property while accumulating a significant amount of attention. The authors construct lsh-based prototypes for SATs and chunk-based protoypes for PDTs as compression units. Clusters are ranked according to an importance metric, tokens are assigned to these clusters, and tokens from the top-ranked clusters are retained until the budget is met. This approach (ProtoKV) outcompetes baselines by >2% on LongBench and outcompetes other baselines at a lower budget and ties others at higher budgets.", "questions": "- See weaknesses. \n - How does the method perform on RULER 16K or 32K?\n - Is H2O truly **that** bad on RULER (Table 2)? This doesn't seem to concur with other literature. \n - How does this approach fundamentally differ from the Reformer, which also chunks and groups tokens according to LSH buckets?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- This appears to be the first work to discover SATs. The experiment distinguishing them from sinks validates this unique token type. \n\n- The clustering approach is principled according to the token types. \n\n- ProtoKV defeats several popular strategies across varying model families on LongBench. \n\n- The success of the method at a very small budget (64 tokens) is attractive for severely resource-constrained systems.", "weaknesses": "- LongBench and RULER are known to stack lots of noisy context around sparsely distributed signals, thus possibly rendering the appearance of SATs as unique to these types of benchmarks. The appearance of this token type does not appear to be explored over a greater variety of long-context tasks.\n\n- Besides Llama-3-8B Instruct, only older models are tested. The authors should consider evaluating their approach on newer Qwen, Phi models, and/or the latest Mistral-7B. \n\n- The performance gain on RULER is quite minimal. While the average improvement on LongBench is +2%, the individual numbers on LongBench in Figure 34 are far less significant, where ProtoKV either incrementally wins or even loses against other baselines on a variety of tasks. This makes it difficult to determine whether ProtoKV is truly a worthwhile compression strategy.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "fZNCziGOWd", "reviewer": "Reviewer_Y76g", "summary": "The paper proposes ProtoKV, a semantic-aware KV cache compression framework for large language models (LLMs) that mitigates long-context inference costs. It introduces two token categories—Semantic-Anchored Tokens (SATs) and Position-Determined Tokens (PDTs)—and constructs hybrid semantic prototypes for each, guiding KV retention through cluster-based attention relevance. The method preserves semantic integrity while maintaining computational efficiency, outperforming baselines such as SnapKV, H2O, and ChunkKV by up to 2.11% on LongBench and achieving 97.3% retrieval accuracy in Needle-in-a-Haystack tests.", "questions": "1.\tCould the authors provide layer-wise or head-wise distributions of SATs to clarify whether semantic anchoring is model-general or architecture-specific?\n2.\tHow does ProtoKV behave under streaming or dynamic decoding, where prefilling-only assumptions may not hold?\n3.\tCan the authors include significance analysis or confidence intervals to verify the reported 2.11 % average improvement?", "limitations": "", "rating": 4, "confidence": 2, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "•\tThe identification of SATs as clustering outliers in the key embedding space (Fig. 4–6) provides a new lens for understanding token semantics in LLMs. This insight grounds ProtoKV’s prototype-based compression design and distinguishes it from previous attention- or position-driven methods.\n•\tThe framework (Sec. 4.2–4.3) integrates Random Fourier Feature hashing (Eq. 7–8) and prototype-guided selection (Eq. 10–11), avoiding costly iterative clustering (Fig. 8). Pseudocode and reproducibility details are given (Appendix J), enhancing transparency.\n•\tProtoKV is compared with multiple baselines across three architectures (LLaMA-2, LLaMA-3, Mistral) and two benchmarks (LongBench, Ruler), showing robustness under varying KV budgets (64–512) and across tasks (Fig. 9, 10, 12–14). The ablation studies further isolate the roles of prototype number and SAT count.", "weaknesses": "•\tWhile the “local deviation property” (Eq. 4–6) is empirically supported, the causal explanation (Sec. 3.2) remains qualitative. There is no analytical or statistical validation that SATs correspond to meaningful semantic units across layers or models.\n•\tKey hyperparameters such as neighborhood window $\\kappa$, prototype number, and threshold $\\beta$ are only briefly tuned (Fig. 12–13) without robustness metrics or cross-dataset variance, limiting confidence in generalizability.\n•\tAlthough computational cost is compared (Fig. 14), there is no wall-clock latency or memory breakdown versus model size (e.g., >8 B models), and no statistical significance tests for accuracy gains (Table 2–4).\n•\tThe contribution of each stage (SAT detection, LSH clustering, observation window) is only partially evaluated; removing or modifying these modules’ effects is not explicitly quantified.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Y5r1lM5lxN", "reviewer": "Reviewer_KnCx", "summary": "The paper introduces ProtoKV, a novel framework designed to enhance the efficiency of Key-Value (KV) cache retention in large language models (LLMs) when processing long text sequences. The authors identify two categories of tokens within the key embedding space: Position-Determined Tokens (PDTs), which maintain strong similarity with their contextual neighbors, and Semantic-Anchored Tokens (SATs), which exhibit local deviation and form clusters. By leveraging the unique properties of these two token types, ProtoKV constructs semantic prototypes that improve KV cache compression while preserving semantic integrity. Experimental results demonstrate that ProtoKV outperforms existing state-of-the-art methods by achieving an average accuracy improvement of 2.11% on the LongBench benchmark, showcasing its effectiveness in maintaining high retrieval accuracy with minimal KV cache retention.", "questions": "Please refer to the weakness part.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Good Presentation. The presentation of this paper is very clear, the structure is reasonable, and the presentation of the figures and tables is also very precise.\n- Reasonable Idea:  The paper presents a novel perspective on token categorization in LLMs, particularly the identification and utilization of SATs as semantic anchors, which is a significant contribution to the field.\n- Good Performance: The authors provide comprehensive experiments across various benchmarks, clearly demonstrating the advantages of ProtoKV over existing methods in terms of accuracy and efficiency.\n- Insightful Analyses.", "weaknesses": "- Complexity of Implementation: The proposed method may introduce additional complexity in the implementation of LLMs, which could be a barrier for adoption in certain applications or by practitioners with limited resources.\n- Limited Comparison with Other Methods: While the paper provides lots of experiments to evaluate ProtoKV, a more extensive analysis involving **more recent SOTA approaches** could strengthen the argument for its superiority.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "O2DGUi9WBP", "reviewer": "Reviewer_m3F1", "summary": "The paper introduces ProtoKV, a KV-cache compression method for LLMs. It first exploits a key structural insight: most tokens (“position-determined”) cluster with their neighbors, while a small set of “semantic-anchored” tokens (SAT) consistently deviate and form clusters. By splitting tokens into these two classes, ProtoKV builds separate semantic prototypes and compresses each class into semantically coherent clusters, preserving meaning while cutting memory. On LongBench it outperforms prior state-of-the-art techniques by 2.11\\% accuracy under the same memory budget.", "questions": "1. The paper claims that SATs are salient for generation, so why not select all SATs directly? The results as shown in Figure 12 illustrate that the SAT prototype number does not influence the accuracy, because ProtoKV selects all SATs, even though they are classified into different clusters. Additionally, what type of RFF-based hashing is it, locality-sensitive or random?\n\n2. Figure 14(b) shows that the average compression time exceeds half an hour. However, full attention computation usually takes several minutes for a long context. Do the users have to wait half an hour for KV cache compression?\n\n3. As shown in Figure 34, none of the methods match FullKV in terms of accuracy. Could ProtoKV achieves FullKV’s level of precision with a larger budget size, e.g., 1024 tokens?\n\n\n\nMinor comments:\n- Line 461, Eq. equation 8 -> Eq. 8\n- The caption of Figure 34 states that bold indicates the best performance and underline the second performance, but no text in the figure is bolded or underlined.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "1. The paper is well written and excellently presented. Figure 1 clearly illustrates the paper's novel findings.\n\n2. The paper finds that semantic-anchored tokens, which exhibit the local deviation property, are important for generation. It provides exhaustive experimental results to demonstrate this.\n\n3. ProtoKV builds separate semantic prototypes and compresses each class into semantically coherent clusters. This method improves accuracy compared with existing baselines.", "weaknesses": "1. The rationale behind locality-sensitive hashing applied to SAT is unclear. (Question 1)\n\n2. The time cost of ProtoKV may hinder its applicability. (Question 2)\n\n3. There are no error bounds for the key results. Quantitative robustness indicators such as standard deviations would help validate generalizable conclusions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)