FreeKV: Boosting KV Cache Retrieval for Efficient LLM Inference
Authors: Guangda Liu
Chengwei Li
Zhenyu Ning
Jing Lin
Yiwu Yao
Danning Ke
Minyi Guo
Jieru Zhao
Venue: ICLR 2026 Poster (2026)
Keywords: LLM inference
KV cache
Primary area: infrastructure, software libraries, hardware, systems, etc.
TL;DR: We propose FreeKV, an algorithm-system co-optimization framework for LLM inference to enhance KV retrieval efficiency while preserving accuracy.

Abstract: Large language models (LLMs) have been widely deployed with rapidly expanding context windows to support increasingly demanding applications.
However, long contexts pose significant deployment challenges, primarily due to the KV cache whose size grows proportionally with context length.
While KV cache compression methods are proposed to address this issue, KV dropping methods incur considerable accuracy loss, and KV retrieval methods suffer from significant efficiency bottlenecks.
We propose FreeKV, an algorithm-system co-optimization framework to enhance KV retrieval efficiency while preserving accuracy.
On the algorithm side, FreeKV introduces speculative retrieval to shift the KV selection and recall processes out of the critical path, combined with fine-grained correction to ensure accuracy.
On the system side, FreeKV employs hybrid KV layouts across CPU and GPU memory to eliminate fragmented data transfers, and leverages double-buffered streamed recall to further improve efficiency, enabling effective overlap with computation, full latency hiding, and practical speedups from speculative recall.
Experiments demonstrate that FreeKV achieves near-lossless accuracy across various scenarios and models, delivering up to 13$\times$ speedup compared to SOTA KV retrieval methods.

Published as a conference paper at ICLR 2026
FREEKV: BOOSTING KV CACHE RETRIEVAL FOR EF- FICIENT LLM INFERENCE
Guangda Liu'
Chengwei Li'
Zhenyu Ning'
Jing Lin"
Yiwu Yao?
Danning Ke²
Minyi Guo'
Jieru Zhao
School of Computer Science, Shanghai Jiao Tong University
2 Huawei Technologies Co., Ltd
ABSTRACT
Large language models (LLMs) widely deployed with rapidly expanding con- text windows to support increasingly demanding applications. However, long contexts pose significant deployment challenges, primarily due the KV cache whose size grows proportionally with context length. While KV cache compres- sion methods have been proposed address this issue, KV dropping methods in- cur considerable accuracy loss, and KV retrieval methods suffer from significant efficiency bottlenecks. We propose FreeKV, a training-free free algorith--systen co- optimization framework to enhance KV retrieval efficiency while preserving ac- curacy. On the algorithm side, VreeKV introduces speculative retrieval shift the KV selection and recall processes out of the critical path, combined with fine-grained correction to ensure accuracy. On the system side, FreeK em- plays hybrid KV layouts across CPU and GPU memory to eliminate fragmented data transfers, and leverages doublc-buffered streamed recall to further improve efficiency, enabling effective overlap with computation, full latency hiding, and practical speedups from speculative recall. Experiments demonstrate that FreeKV achieves near-lossless accuracy across various scenarios and models, delivering up to a 13x speedup compared to SOTA KV retrieval methods. Code is available at ht https:// //g//////// gig.ub.......... com/s sjju- zhao- lab/F / FreeKV.
INTRODUCTION
Large language models (LLMs) have gained remarkable prominence for their ability to excel across diverse tasks and have been widely deployed in a variety of applications, such document analy- sis, chatbots and coding assistants (Chew et al., 2023; Jiang et al., 2024b). To process increasingly complex tasks such as long-document QA, multi-turn dialogue, and repository-lews standing, the context window sizes of LLMs are rapidly expanding to accommodate longer inputs. Mainstream LLMs now support context windows of 128K tokens (Qwen et al., 2025; DeepSeek-Al AI et al., 2025), with frontier models reaching up to million tokens (DeepMind, 2025; xAI, 2025).
code under-
While larger context windows unlock new capabilities for applications, handling long context presents significant challenges for efficient deployment. These challenges arise from the KV cache in LLMs, which stores the key-value states of previous tokens to avoid recomputation during infer- ence, causing its size to grow proportionally with the context length. On the hand, the size of KV cache can exceed the capacity of GPU memory. For instance, the KV cache for a single request can reach 40GB for Llama-3-70B with context length of 128K (Meta, 2024). On the other hand, since the LLM decoding is memory-bound accessing a large KV cache significantly degrades the decoding speed (Fu, 2024).
To mitigate these issues, based on the sparsity of attention computation, previous works proposed compressing the KV cache, i.c., utilizing only a portion of the KV cache for inference. These com- pression methods can be broadly classified into two categories: KV dropping and KV retrieval (Li al., 2025). KV dropping methods only retain KV cache for important tokens and permanently evict unimportant ones. The identification of important tokens can be performed either statically (Xiao et al., 2024a) or dynamically (Li et al., 2024). contrast, KV retrieval methods maintain the entire KV cache but dynamically select subset for inference (Chen et al., 2024b; Liu et al., 2024).
"Correspondence Jieru Zhao (jt.eeuue.i..uu.cnn

---

Published as a conference paper at ICLR 2026
Fu
Static Drop
Dynamic Drop
Retheval
Normaized Accuracy
100
No Offload
Selection
InfiniGen
Recall
60
ShadowKV
Others
ArkVale
0
40
80
120
100
200
240
280
NIAH
Summarization
Reasoning
Time Per Output Token (ms)
Figure I: Left:
Accuracy comparison of KV dropping and retrieval methods across different tasks.
Right: Latency breakdown of KV retrieval methods with offloading.
While both KV dropping and retrieval methods can maintain acceptable model accuracy under spe- cific scenarios and tasks, recent studies reveal significant accuracy degradation with KV dropping methods, particularly on tasks like summarization and reasoning (Gao al., 2025a; Liu et al., 2025). This degradation stems from the dynamic nature of token importance, where tokens previously deemed unimportant and permanently dropped may become crucial in later steps (Chen et al., 2024a; Liu et al., 2024). For complex tasks involving long generation, the omission of a large number of such important tokens results in severe accuracy decline. This issue is further exacerbated with the advent of reasoning models, where extended thinking processes lead to generation lengths reach- ing 32K tokens or more (OpenAI, 2024; DeepSeek-Al et al., 2025; Qwen, 2025b), highlighting the limitations of KV dropping methods. In Fig. 1, we compare the accuracy of static KV dropping (Ra- zorAttention (Tang et al., 2024a)), dynamic KV dropping (RaaS (Hu et al., 2025)) and KV retrieval (Quest (Tang et al., 2024b)) under similar KV cache budgets, across tasks of Needle In A Haystack (NIAH), summarization and reasoning (Kamradt, 2023; Bai et al., 2023; Huggingface, 2025). Both static and dynamic drop methods exhibit significant accuracy degradation on summarization and rea- soning tasks. In contrast, KV retrieval methods maintain robust accuracy across all tasks. Therefore, KV retrieval methods are better suited for more general and practical scenarios.
Despite their superior accuracy performance, KV retrieval methods face significant efficiency challenges. First, since the complete KV cache must be retained, retrieval methods often offload the KV cache to CPU memory to circumvent GPU memory limitations. For methods without offloading like Quest, out-of-memory errors inevitable for long contexts and large sizes. However, for offloading methods, due to the low bandwidth of the CPU-GPU connection, recalling the selected KV tuples from CPU GPU memory incurs long latency. Second, KV retrieval methods select KV tuples from the entire context, leading to considerable selection overhead, even though most retrieval methods adopt page-wise selection to alleviate this issue. In Fig. 1, we present the latency break- down of SOTA offloading methods, using Llama-3.1 3.8--8-1-BBBB- --8--nnttruct with a batch size of 1 and context length of 32K. For Vrk (Chen al., 2024a), recall and selection contribute approximately 94% of the overall latency. Similarly, while ShadowKV (Sun et al., 2025) introduces key cache recon- struction to recall only the value cache, recall and selection still comprise about 73% of the total latency. InfiniGen (Lee et al., 2024) overlaps recall with computation by re-projecting the query and key vectors to prefetch the KV cache for the next layer. However, as shown in Fig. 1, InfiniGen's 's recall latency cannot be fully hidden due to its inefficient token-wise recall, with the remaining unoverlapped portion still accounting for nearly 53% of the overall latency. Moreover, InfiniGen incurs substantial overhead from the re-projection and selection. All these retrieval methods lead to significantly higher latency compared to inference with the full KV cache without offloading.
To overcome these challenges, optimization framework that significantly boosts the efficiency of KV retrieval, while maintain- ing near-lossless model accuracy across diverse scenarios. On the algorithm side, leveraging the high similarity of query vectors between adjacent decoding steps, FreeKV introduces speculative re- trieval, which shifts the selection and recall processes out of the critical path via step-wise KV reuse, thus avoiding inference blocking. As shown in Fig. 2a, this approach allows selection and recall to overlap with other operations, effectively hiding their overhead. To counter potential accuracy losses from pure KV reuse, FreeKV incorporates gra-igraied correction to preserve model accuracy with minimal impact on efficiency. On the system side, FreeKV V employs hybrid KY layouts across CPU and GPU memory to eliminate inefficient fragmented data transfers and avoid layout conversion overhead during inference. In addition, FreeKV implements a double-buffering mechanism to facil- itate streamed recall, further improving recall efficiency by overlapping CPU-GPU and GPU-GPU
introduce FreeKV, a training-free algorithm-system co-
2

---

Published as a conference paper at ICLR 2026
ArkVale: Vanilla KV retrieval with offloading
QKV Proj
Seleciin
KV Recal
Attention
FFV
ShadowKV: Reconstruct K using SVD & recall only V
Time
RazorAttention
Freek//
QKV Prot.
Selection
Reconstrust
Amendon
QKV Proi.
Efficiency
Raas
V Recul
Thme
InfiniGen: Prefetch for the next layer with re-projected Q and K
Dropping: Efficient
ShadowKV
QKV Pray.
Re-praj.
Sel.
Attention
FFV
QKV Proj. Ri proi & Set.
Atlention
but inaccurate
InfiniGen
KV Recal the next layar
Time
FreeKV: Fast KV retrieval with fully overlapped KV recall
Retrieval: Accurate
ArkVale
but Inefficient
QKV Pray
Attention
FFW
Pky
Attention
FFN
QKV Proy
Quest
Selection
Fast KV Recad
Selection
Fastt KV Recal
Time
Accuracy
(a)
(b)
Figure 2: (a) Comparison of timelines for KV retrieval methods, FreeKV shifts the selection and recall out of the critical path. (b) Accuracy-effiiiecc trade-off of K. compression methods.
data transfers. These system-side optimizations dramatically reduce recall latency, enabling effec- tive overlap with computation, full latency hiding, and practical speedups from speculative recall. As shown in Fig. 2b, FreeKV strikes balance between accuracy and efficiency, establishing new Pareto frontier. Extensive experiments show that FreeKV maintains near-lossless accuracy across diverse scenarios and models, delivering up x3x speedup over SOTA KV retrieval methods.
BACKGROUND AND RELATED WORK
2.1 PROBLEM FORMULATION
The decoding process of an attention head in LLMs be expressed as softmax(qk") /Vayv, where q.o o E 1xd are the query vector and attention output of the current token, respectively, and K, V E RLxd represent the K and V states of the preceding tokens. For modern LLMs with Grouped Query Attention (GQA) (Ainslie et al., 2023), the number of attention heads and KV heads are denoted as Ago and nkv. respectively. The output of attention head is given by Deg/ngg á is the corresponding KV head. And is the group size, representing the number of attention heads within group that share the same KV head.
O₁ = softmax(q,Kī,, Vd) where
G The
KV retrieval methods select a subset of KV tuples for attention computation based on attention weights derived from q and K, defined Zh = Sel(q", )*), where Ih represents the indices of selected KV tuples, and ! = B specifies a preset KV cache budget. In practice, retrieval methods consistently retain KV tuples for S sink tokens at the beginning and W tokens within the local window, leaving B S - W tuples available for selection.
For GQA models, the space required for retrieved KV tuples O x nie) the selection is group- consistent, meaning the indices of KV tuples selected by all attention heads within the same group are identical, i.e., (m=1)) selection is not group-consistent, the required space increases O(B ngo), resulting in G times higher costs in both space and memory accesses.
((m-1)xG+2
TmxG for m = 1, 2, ..,,, 7ak If the
2.2 RELATED WORK
KV dropping KV dropping methods can be further categorized into static and dynamic dropping. Static dropping methods evict KV states using fixed patterns determined before inference. For stance, StreamingLLLM (Xiao et al., 2024b) retains KV tuples only for the initial sink tokens and those within local window. Based on this, RazorAttention (Tang et al., 2024a) and DuoAtten- tion (Xiao et al., 2024a) retain full KV cache for designated retrieval heads, while limiting other heads to sink tokens and a local window. Static dropping methods are computationally efficient, incurring minimal overhead during inference, with GPU memory usage proportional sparsity s and the context length L. However, their fixed nature overlooks dynamic patterns during inference, leading to significant accuracy losses. Dynamic dropping methods, on the other hand,
preset
3

---

Published as a conference paper at ICLR 2026
Table I: Comparison of KV cache compression methods.
AtzorAttn
RaaS
Quest
Arklele
ShadowKV
InfiniGen
FreeKV
Category
Statie Drop
Dynamic Drop
Retrieval
Retrieval
Retrieval
Retrieval
Retrioval
Long Generation
V
X
x O O(2 +
GPU Mem. Usage
((sL)
((B) x
O(E)
(
O
L)
O(8)
Group-consiseent
Elficiency
High
High
No Offiedd
Slow Recall Overlap
Pool Overlap
SVD Cost
Re Re-proj. Cost Pool Overlup
Full Overlap
Fast Recall
evict KV tuples based on attention scores calculated online during inference (Zhang et al., 2023; Li et al., 2024). While most dropping methods do not support the onnggenerration scenarios, RaaS (Hu ct al., 2025) addresses these scenarios by evicting tokens that have not received sigmificant attention scores for a sustained period. Dynamic dropping methods retain and score only a fixed budget of the KV cache, achieving good efficiency despite the additional scoring overhead.
KV retrieval Both static and dynamic dropping methods incur permanent information losses, re- sulting in notable accuracy degradation, particularly in Long-generation scenarios. In contrast, KV retrieval methods retain the complete KV cache but select subset for computation. While preserv- ing accuracy, retrieval methods introduce significant efficiency challenges. First, applying selection across the entire context by scoring over every token leads to unacceptable overhead. To miti- gate this, most KV retrieval methods adopt page-wise selection, summarizing the keys within a page and scoring only these page summaries. For example, Quest uses min-max pooled keys, Ark Vale employs bounding volumes of keys within a page, and ShadowKV simply relies mean-pooled keys. Moreover, the handling of the complete KV cache poses significant challenges. Quest stores the entire KV cache in GPU memory with limited capacity, restricting support for long con- text lengths and large batch sizes. Furthermore, its inconsistent selection within head groups incurs G times memory access overhead. Vrle offloads the KV cache to CPU memory and recalls the selected KV pages during inference. While ensuring group-consistent using mean pooling over at- tention weights and maintaining a cache for selected pages on GPU, the recall process of Vale remains costly, severely impacting efficiency. ShadowKV takes different approach by leveraging the low-rank property the pre-rope key cache. retains only the low-rank key cache obtained through singular value decomposition (SVD). During inference, for selected pages, it reconstructs the key cache from the low-rank representations, while only recalling the value cache. This reduces memory transfer costs but requires additional GPU memory to store the low-rank key, consuming _ (15%-30%) of the original key cache size, where r 160 is the rank used by ShadowKV and dev des is the dimension of key cache. Moreover, ShadowKV KV does not support long- geggeeeeertion since the SVD is performed only once during prefill, leaving the low-rank key unupdated during decod- ing. Leveraging the high similarity of hidden states across adjacent layers, InfiniGen re-projects the current layer's ss hidden states using the skewed query weights of the next layer, to pre-select and prefetch the KV cache for the next layer. While this strategy enables partial overlap between recall and computation, InfiniGen's recall latency cannot be fully hidden due its inefficient token-wiss recall. In addition, InfiniGen incurs substantial computation and memory overhead. An extra de. (30%) skewed key cache must be computed and maintained for all tokens, and a skewed query vector must be computed at every decoding step.
The features of KV dropping and retrieval methods are summarized in Table 1. As illustrated, FreeKV ensures accuracy preservation through KV retrieval, while attaining high efficiency with fixed O(B) GPU memory usage, group-consistent selection, and fully overlapped, efficient recall.
3
ALGORITHM DESIGN
3.1 OBSERVATION
We sample the query vectors of generated tokens during inference of 1--ma-3.1-8B-Insttae on
Long-generation tasks and DeepSeek-Rl-Qwere 14B on long- oongronngnnig tasks. The cosine similarity
between the query vectors of adjacent generated tokens is calculated as C; Iq:l-q:-111 where
(q.9:-1).
9;-1.9 E Rlxd are the query vectors of tokens generated at step and i. We present the mean
4

---

Published as a conference paper at ICLR 2026
Uamo
3:1
layer
DeepSock 148.
layer
layar 42, lead 2
Decoding Stop
0.97
0.90
0.96
0.37
0.94
0:94
0.93
0.95
0.95
0.87
0.92
0.84 0.53
0.97
0:96
0.94
0.97 0.97
0.97
0:96
0.92
0.99
0.06
0
88
0.90
0.89 0.88
0.96
0.95
0.96
0.96
0.84
0.86
0.89
0.90
0.90
0.97
0.96
0.91
0.94
0.93
0.96
0.89
0.90
0.90
0.96
0.90
0.000
0.92
0.90
0.89
0.90
32
(a)
(b)
(c)
Figure 3: (a) Cosine similarities between query vectors of adjacent generated tokens, averaged over generation; each cell corresponds to attention head. (b) Attention map of SrapSeek-R--weer 14B on reasoning tasks. (c) Variations in C, during generation of DeepSeek- ---ssekk-R--wen.. --leen--e4...
g Fig. 3a. As shown, across various models and tasks, the mean similarity of all attention heads consistently exceeds 0.84, with most heads achieving similarity greater than 0.9. This high similarity is observed across different layers, models, and tasks, likely due to position embeddings (Su et al., 2023) and the semantic continuity of adjacent tokens (Yuan al., 2025). This observation aligns with the vertical line patterns in attention maps, as shown in Fig. 3b and supported by prior studies (Jiang et al., 2024a; Hu al., 2025; Wu et al., 2025), which show that adjacent decoding steps exhibit high attention scores on similar tokens. This insight motivates our speculative recall mechanism (Sec. 3.2).
similarity during generating g = 13000 tokens, calculated as Σi=1 C.
To delve deeper, we analyze the changes in similarity during generation for DeepSeek-R1-Q4 R--wwen- 14B on reasoning tasks. As shown in Fig. 3c, while the mean similarity remains high, certain decoding steps exhibit outliers with significantly lower similarity. Moreover, these outlier steps vary across attention heads, indicating head-specife variations in query similarity during decoding. These variations underpins the fine-grained correction mechanism (Sec. 3.3) employed by FreeKV.
3.2 SPECULATIVE RETRIEVAL
Speculative retrieval Based on the high similarity observed in query vectors and selected tokens between adjacent decoding steps, i.e., Sel(q:, )) Sel(q:-1, K). we propose speculative retrieval mechanism that shifts the selection and recall out of the critical path of inference. Specifically, the attention computation of step i bypasses the selection and recall, instead directly being launched by reusing the KV tuples recalled during step 1, as shown in Fig. 4a. This design enables the selection and recall operations to overlap with attention and FFN computations of the current layer, as well as the QKV projections of the next layer. The recalled KV tuples during step will then be reused in step + 1, continuing the process iteratively.
Compared to InfiniGen, speculative retrieval hides the latency of both selection and recall without in- curring additional overhead, eliminating the need for re-projection. We further compare speculative retrieval with direct recall using the last layer's 's query vector in Appendix I. which demonstrates that leveraging the last step's 's query vector in speculative retrieval yields superior accuracy.
Group-consistent selection FreeKV adopts mage-wise selection, utilizing the min- max pooled keys within each page as the page summary, similar Quest (Tang et al., 2024b). Let Nggge denote the number of KV pages. To ensure group-consistent selection, after computing the attention weights ph E IR "pe for query vector of attention head and the corresponding page summaries, FreeKV ap- plies mean pooling across the group softmax(P*)) For KV head m, the corresponding attention heads select consistent pages based on scores calculated as Σ softmax(P(m- -1mmm-i)xx++/// //G. Other alternatives for achieving group-consistent selection are evaluated in Appendix B.2, and the results show that mean pooling over the softmax of page attention weights yields the best performance.
3.3 FINE-GRAINED CORRECTION
While purely reusing KV pages recalled from the previous step maximizes efficiency, it can result in significant accuracy degradation. To mitigate this, FreeKV introduces correction mechanism
5

---

Published as a conference paper at ICLR 2026
Step/
Layer
Layar Lsk
Atn. Head
5
Ago
Cosine sim.
0.7
0.8
0.8
0.9
0.9
0.8
Agy
QKV Proj
Attertion
FFN
QKV Proj.
Altention
FFN
QKV Proj.
KV Head
3
Sclection
Fast
Mean Sim.
0.75
0.85
0.85
Correction Recall
Step
of KV Head
Layer
Layer 14)
QKV Proj. Sim.
Selection cv beadk)
Rec.
Attention
FFN
OKV
Attention
Attention
FFN
Pmy.
Selection
Fast KV Recal
Selection
Fast Recall
Recall of KV Head 2, for Reuse at Next Step
Recall
(a)
(b)
Figure 4: (a) Inference timeline with speculative retrieval, where the blue arrows represent the reuse KV pages recalled the previous step. (b) Timeline for fine-grained correction with query-based identification and head-wise correction recall.
that selectively recall KV pages for the current step. By employing query-based identification and head-wise recall, VreeKV minimizes the associated efficiency overhead.
Query-based identification A straightforward correction involves directly comparing the indices of selected KV tuples between step and - 1, i.e., Sel(q;, K) and Sel(q:-1 K). However, this approach incurs substantial overhead due index comparisons and hinders the overlap of selection with other operations. To address these limitations, FreeK employs correction mechanism based on the cosine similarity of query vectors, C.. Correction is triggered only C, < r, indicating significant deviation of Sel(q;, K) from Sel(qi-1, K), where is a predefined threshold. To en- sure group consistency, FreeKV performs mean pooling over C, across the group, and compares the pooled value with to determine whether correction is required for KV head. As illustrated in Fig. 4b, KV head 1, with mean similarity of 0.75 (below т = 0.8), is flagged for correction. In Ap- pendix B.3, we compare mean pooling with max pooling for achieving group-come consistent correction, and show that mean pooling is the more effective choice. We further evaluate different threshold settings and identify the optimal values for different scenarios in Appendix B.3.
Head-wise correction As shown in Fig 4b, once the KV heads requiring correction are identified, FreeKV initiates selection and recall for these KV heads before the attention computation at the current decoding step. For KV heads that do not require correction, recall is deferred and overlapped with other operations, retrieving selected KV tuples for the reuse at the next decoding step. To avoid the overhead and reduce GPU utilization caused by separately launching selection for corrected and non-corrected heads, FreeKV executes selection for all KV heads whenever correction is required. Then for non-corrected KV heads, the recall proceeds directly without repeating the selection.
4 SYSTEM DESIGN AND IMPLEMENTATION
Effective recall overlapping to minimize overhead demands high recall efficiency. FreeKV achieves this through a dedicated system design and implementation, featuring caching, hybrid layouts and streamed recall, as detailed in the following sections.
4.1 OVERVIEW
The system overview of FrecKV is illustrated in Fig. 5. In the data plane, FreeKV retains the query vectors from the previous step, page summaries and cache for selected KV pages in GPU memory. In CPU memory, FreeKV maintains complete KV cache pool for offloading KV pages. In the control plane, a controller on CPU manages the scheduling and synchronization of operations such correction, attention, selection and recall launched different CPU threads and GPU streams, following the timeline described in Sec. 3.
4.2 HYBRID LAYOUTS AND STREAMED RECALL

---

Published as a conference paper at ICLR 2026
NHD
HND
NHD GPU
Layout Conversion
Transfer
Contiguous
HND CPU
Tokan KV Head 1
Hoad Token
Taiion KV Head
during
when Rocall
Head Taken 1
Token 1. KV Head 2
KV Head 1, Token 2
Token 1, KV Head 2
Buffer
KV Head 1, Token 2
(Trarffrninal
Taken KV
Token
Taken KV
3
Taken KV Head 2
Head Token
Taken KV Head 2
Buffer (Converting)
Head Taken 1
Token 3, KV Head 1
KV Head 2, Token 2
Taken 3, KV Head
KV Head 2, Taken
Token KV Head 2
Token
Taken
2
Taken 3
Trans. Offloading Page
Step
(Oieerapp with Computation)
Figure 6: Left: KV cache pages under NHD and HND layouts with and T&k represent elements for given KV head. Right: Hybrid KV cache layouts across CPU and GPU memory, along with streamed recall enabled by doublc-buffering
2; the highlights
The KV cache layout defines the memory organization of the underlying key-value tensors. Two commonly used KV cache layouts are NHD and HND (flashinfer ai, 2025). The NHD layout organizes the KV cache in the shape of (L, nkg. d), while the HND layout uses the shape of (num: L, L,d). In practice, when managing the KV cache in pages, the shapes of NHD and HND layouts are (Neage: P. nkke, d) and (Neage, flke p. d), re- spectively, where is the page size. Since the key and value derived from projections over hidden states are K, V I Lx xd), NHD is the natural layout while the HND layout requires additional transpose operations. To eliminate this overhead, mainstream efficient inference frameworks adopts the NED layout (Dao, 2024).
GPU Computation
GPU Memory
Ath. + FFN
Quarry Ventoo
Page Summeriss
Selective Correction
Selection
Selected KV Pages
Scheotute & Synchronize
Streamed Recall Page Officad
Controller
Control Plane
Data Plane
Figure 5: System overview of FreeKV.
However, since the indices of selected KV pages differ across KV heads and recall is performed individually for each KV head, using the NHD layout results in inefficient fragmented data transfers. As shown on the left side of Fig. 6, under the NHD layout, for a given KV head, the memory of p = key/value vectors within a page is non-contiguous. When recalling a key/value page, the maximum transfer unit contains only d elements, equivalent to just 256 bytes for d = 128 and 1 precision. This extensive fragmented data transfers significantly degrade recall efficiency. In contrast, the HND layout ensures that p key/value vectors within a page are contiguous for each KV head, allowing a transfer unit of x d elements, or 8KB when = 32.
Hybrid layouts To avoid fragmented data transfer while minimizing transpose overhead, FreeKV adopts hybrid layouts on CPU and GPU memory. As shown in Fig. 6, FreeKV employs the NED layout on GPU eliminate the need for per-step transposes during decoding, and the HND layout on CPU to ensure contiguous and efficient CPU-GPU data transfers during recall. With the hybrid lay- outs, the NHD-HND transpose is only required when offloading a KV page. effectively amortizing the overhead. In addition, FreeKV utilizes an HND layout on CPU with a shape of (M'age 7kxv 2, p, d), enabling the transfer of 2 x p x d contiguous elements for both key and value vectors during recall.
Streamed recall While offloading and the associated transposes can overlap with computation, the conversion from HND layout to NHD layout during recall can block data transfers and subsequent attention computation. To avoid such blocking from sequential data transfer and layout conversion, FreeKV employs a double- bufle-buffering mechanism to achieve streamed recall. As shown in Fig. 6, after a selected KV page is transferred buffer 2, its layout conversion begins immediately, while the transfer of the next page is concurrently initiated into buffer 1. Both buffers and the conversion process reside in GPU memory, leveraging its high bandwidth enhance efficiency.

---

Published as a conference paper at ICLR 2026
Table Accuracy results of LongBench v2 and LongGenBench.
Benchmark
Metric
Razor
RaaS
Quest
ArkVale
ShadowKY
InfiniGen
FreeKV
Llama 3,1 1-8B Instruct
Overall
29.22
27.44 44
28.23
28.43
28.63
25.45
28.56
29.23
LongBench
Short
34.44
33.89
33.89
33.33
33.89
32,78
32.28
35.00
Medium
27.44
26.98
22.79
27.44
Long
23.15
21.30
22.02
22.22
23.15
18.52
24.07 4
23.15
LongGenBench
CR
80.03
35,90
76.63
78.03
39.36
79.28
76.68
78.03
Acx
26.82
12.20
26.00
10.36
30.66
26.21
27.62
Qwen-2. 5-78-nnsrtet
Overall
27.44
25.25
26.24
27.63
26.84
25.84
26.44
26.84
LongBench
Short
36.11
32.78
35,56
36.67
36.11
32.22
32.22
34.44
Medium
23.72
21.86
21.86
22.79'
22.33
20.00
23.26 M
22.33
Long
20.37
19.44
19.44
22.22
20.37
26.85
23.15
23.15
LongGenBench
CR 79.56
42.13
77.65
62.89
75.91
35.49
72.96
76.93
Acx
31.09
21.48
34.40
25.96
31.79
11.43
27.67
Qwen-2.5.
Overall
33.40
34.19
32.60
33.80
34.19
34,79
32.31
34.19
LongBench
Short
41.11
43.33
40.56
40.00
41.11
40.56
40.56
41.11
Medium
31.16
30.70
32.09
33.49
33.49
34,88
29.89
33.49
Long
25.00
25.93
20.37
24.07
24.07
25.00
23.95
24.07
LongGenBench
CR
65.84
26.48
62.29
45.49
45.31
21.25
63.02 Y
65.46
CRxAce
19.76
19.65
EVALUATION
5.1
EXPERIMENTAL SETUP
Datasets and models We evaluate FreeKV across various models and tasks. For accuracy evalu- ation, we select LongBench v2 (Bai al., 2024) and LongGenBench (Wu al., 2024) to cover general long-input and long-generation scenarios. In addition, we assess FreeK on long rea- soning tasks, including MATH500 (Hendrycks et al., 2021). AIME24 (Huggingface, 2025) and GPQA (Rein et al., 2023). For LongBench v2 and LongGenBench, we use general models including Llama-3. Ins-rust Qwen- 2.5- In-strutttt and Qwen-2.5 1-5-1-.B--11-ttet use DeepSSekkRR-LAAn
For reasoning tasks, we and DeepSeek-RI Reee---------------- 14B (DeepSeek-Al al., 2025). Detailed metries of each dataset are provided in the corresponding sections.
M1-Seee-RR-QWenn
Baselines We compare FreeKV against SOTA methods, including KV dropping methods such as RazorAttention and RaaS, as well as KV retrieval methods, including Quest, Ark Vale, ShadowKY, and InfiniGen. The sparsity of RazorAttention is set to 0.15, while the budget B for all other methods consistently set to 2048. Detailed settings of all methods are provided in Appendix A.
5.2 ACCURACY EVALUATION
LongBench v2 Improved from LongBench (Bai al., 2023), LongBench v2 covers realistic scenarios. It spans various difficulty levels and context lengths, ranging from 8K to 2M tokens. All problems of LongBench v2 are presented in multi-choices question format, with accuracy used as the unified metric. We report accuracy under the context length categories of short, medium and long, as well as the overall accuracy. For all methods, we truncated the inputs to 64K tokens
As shown in Table 2, for all models, the overall accuracy of FreeKV deviates by at most 0.6 com- pared the model with full KV cache, while FreeKV achieves the best or second-best performance across most metrics. KV dropping methods, although exhibiting moderate accuracy losses on this long-input benchmark, consistently underperform compared to KV retrieval methods.
LongGenBench Unlike traditional long-context benchmarks that focus on long inputs, LongGen- Bench designed to evaluate the model's 's ability to handle long generations, assessing the capabil- ity to generate coherent and high-quality long-form content. Each task of LongGenBench contains subtasks that prompt the model to generate specific content at designated points, within specific ranges in periodic manner. We report the completion rate (CR) of subtasks and the overall
8

---

Published as a conference paper at ICLR 2026
Table Accuracy results of long reasoning tasks.
Benchmark
Metrie
Razor
RaaS
Quest
Vrke
ShadowKY
InfiniGen
FreeKV
DeepSeek RJ Lloma-80
MATHS00
pass
78.00
72.00
74.00
72.00
76.00 Y
78.00
ag
67.25
60.50
62.50
62.00
62.50
60.25
66.50
66.75
AIME24
pass @
80.00
46.67
66.67
73.33
80.00
63.33
70.00
76.67
47.08
30.00
36.25
44.17
46.67
36.50
45.83
47.50
GPQA
pass @
82.00
60.00
64.00
76.00
72.00
78.00
46.00
86,00
avg
39.75
34.25
33.50
37.25
39.75
36.25
41.25
DeepSeek -uwen-nnn
MATH500
78.00
72.00
74.00
76.00
76.00
74.00
74,00
78,00
avg@
71.75
66.75
67.00
68.00
68.25
64.75
70.00
70.00
AIME24
pass
83.35
65.33
73.33
76.67
73.33
73.33
63.33
83.33
@
56.66
35.42
42.92
47.50
47.92
43.75
43.34
52.92
GPQA
pass k @vg@ 35.75
72.00
60.00
58.00
72.00
72,00
70.00
$8.00
74,00
32.50
38.75
34.25
39.50
DeepSeok-R R T---Qeeee-ie 148
MATH500
k
74.00
70.00
68.00
76,00
72.00
76,00 Y
72.00
78,00 Y
avg@ k:
70.25
59.75
64.75
67.25
66.25
63.00
67.00
67.50
AIME24
pass k 86.67
46.67
73.33
83.33
76.67
83.33
76.67
83.33
@vg@ 66.25
32.50
48.75
58.33
61.25
$7.25
60.00
64.17
GPQA
pass @ l 82.00
68.00
80.00
80.00
86.00
86.00
$8.00
86.00
arg@
53.25
38.50
44.25
51.25
53.75
51.75
56,00 Y
accuracy (CRxAcc). As LongGenBench relies on LLMSs for accuracy evaluation, we use Qwen-3- 32B (Qwen, 2025a) as the evaluator.
As shown Table 2, across all evaluated models, FreeKV maintains overall accuracy comparable to or exceeding that of the model with full KV cache. Compared other methods, FreeK V achieves the best or second-ber -bsst performance in terms of CR and overall accuracy. For long- oong-generation tasks, RazorAttention with static dropping suffers significant accuracy losses, while RaaS with dynamic dropping demonstrates strong accuracy, likely due to the relative simplicity of the tasks. In addition, observe repeated output and reduced accuracy for ShadowKV with Qwen-2.5 models, which can be attributed to errors in the reconstructed keys.
Long reasoning tasks Besides the general genggeenerttion tasks where users prompt generate long-form content, reasoning models like DeepSeek-R1 autonomously generate long thinking pro- solve complex problems. We evaluate reasoning tasks using problems from MATH500, AIME24 and GPQA datasets. For testing, we select 50 problems each from MATH500 and GPQA and the entire AIME24 dataset. Since the outputs of reasoning models are highly sensitive to random seeds (Hochlehnert et al., 2025), we generate k = 8 different samples for each problem. We report two metrics: pass@k, @ which measures the likelihood of at least one correct solution among the & samples, and avg@ @ k, which represents the average accuracy across all & samples.
As shown in Table 3, FreeKV delivers accuracy comparable models with full KV cache and outperforms other compression methods across most datasets. KV dropping methods such as Ra- zorAttention and RaaS exhibit significant accuracy losses, particularly on AIME24, which involves more complex problems. Moreover, FreeKV consistently outperforms other KV retrieval methods in most cases, demonstrating the effectiveness of its page summaries, softmax-based group consistent selection, and fine-grained correction mechanism.
5.3 EFFICIENCY EVALCATION
Setup Our experiments were conducted on an Nvidia A100 40GB GPU, connected with AMD 7302 CPUs via PCIe Gen4. The evaluation covers Qwen-2.5-7B and Llama-3. 1-8BBBBBBBBBBBBB models under both long-input (32K input, 512 output) and long-generation scenarios (600 input, 16K output). We set 7 to 0.8 for long-input and 0.9 for long-gen mon-ggneratiin scenarios, with = 2048 and = W = 512. We exclude Quest from the efficiency evaluation because it does not support offloading and fails to complete inference for large batch sizes and long contexts (e.g., batch size 4 with 32K contexts) due to GPU out-of-memoon -o-ooorro emrrr..

---

Published as a conference paper at ICLR 2026
NWVale
IntriSen
ShadowkY
Freeky (oun)
Rood
RacomMtenton
Quen- 5-78-nnirrd (Long Input)
Liama-3 1-88-instnud (Long Inpul)
5.76x
10.03x
70
3.29x
4.27x
4.85x
8.27x
Sum
Owen-2.5 Bween-25-7B-imee 5-7B-instnull (Long Gemeration)
Llama-3 (arra-3.1--B-iip 1-----nnttddd (Long Generation)
(8)
3000
7.92x
13.74x
5.32x
8.40x
3.200 26
3003
5.32x
Blachh
Figure 7: End-to-end latency comparison between KV compression methods.
End-to-end latency As shown in Fig. 7, FreeKV demonstrates significant efficiency gains SOTA KV retrieval methods, achieving up to 1 and 8.4x speedups compared to Ark and ShadowKV, respectively. Moreover, FreeKV attains efficiency comparable dropping methods like RaaS and RazorAttention, which do not involve offloading or recall. The speedups over Ark Vale are detailed in Fig. For InfiniGen, FreeKV achieves 3.2x and 5.4x speedups under long-input and gong-genrration scenarios on Qwen-2.5-7B, and 5.1 x and 8.5 on Llama-3.1-88 The improve- ments over Shadow KV are comparable to those over InfiniGen, reaching up 8.4x on Llama-3. 1- 8B in the long-generation scenario. The improvements become more pronounced for large batch sizes and genggeeeratiin scenarios, where more recall operations are required. In addition, the improvements amplified for Llama-3.1 3.1-BB, which has more KV heads and larger KV cache compared to Qwen-2.5-7B. Moreover, we present inference latency across different input and out- put lengths in Appendix C.I,, showing that VreeK consistently achieves substantial speedups under various settings. We also conduct ablation studies on the impact of our efficiency optimizations in Appendix C.2, which demonstrate their effectiveness.
DISCUSSION
While FreeKV achieves near-lossless accuracy, techniques such as adaptive budgets (Feng al., 2024) or dynamic budgets with top-p sparsity (Chen et al., 2024b; Lin et al., 2025; Zhou et al., 2025) can applied orthogonally further enhance accuracy. In addition, machine learning based methods have been proposed to predict attention patterns for KV cache compression (Yang et al., 2025: Akhauri et al., 2025). However, these methods introduce significant training and runtime overhead and are only effective for long-input scenarios. Moreover, although page-wise selection is found to be less effective for small budgets (Liu et al., 2024; Hooper et al., 2024), learnable block- wise sparsity techniques, applied during pre-training (Yuan et al., 2025; Lu et al., 2025) post- training (Gao et al., 2025b), show promise achieving native and optimal page-wise KV cache compression and retrieval.
CONCLUSION
We present FreeKV,, algorithm-systen co-optimization KV retrieval framework that integrates speculative retrieval and fine graingr correction on the algorithm side, as well as hybrid layouts and streamed recall the system side. FreeKV achieves near-lossless accuracy across various scenarios and models, delivering up to 13x speedup over SOTA KV retrieval methods.
ACKNOWLEDGMENTS
This work is sponsored by the National Natural Science Foundation of China (62472273, 62232015).
10

---

Published as a conference paper at ICLR 2026
REFERENCES
Joshua Ainslie, James Lee-Thorp. Michiel de Jong, Yury Zemlyanskiy, Federico Lebron, and Sumit Sanghai. GQA: Training generalized multi-query transformer models from multi-head check- points. In Houda Bouamor. Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 4895-4901, Singapore, December 2023. Association for Computational Linguisties. doi: 10. 18653/v w1w.11//1//22...... http://1212222mmm/// commlppmminn22) main.298. URL ptttps:/aacaanthclogee org//o///333
emnlp-main.22
Yash Akhauri, Ahmed F AbouElhamayed, Yifei Gao, Chi-Chih Chang, Nilesh Jain, and Mo- hamed S. Abdelfattah. Tokenbutler: Token importance is predictable, 2025. URL https:
/ //arxiv..or/////// org. abs 0.553.0751.1
Yushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu, Jiankai Tang, Zhidian Huang, Zhengxiao Du, Xiao Liu, Aohan Zeng, Lei Hou, Yuxiao Dong, Jie Tang, and Juanzi Li. Longbench: A bilingual, multitask benchmark for long context understanding. arXiv preprint arXiv:2308. 14508, 2023.
Yushi Bai, Shangqing Tu, Jiajie Zhang, Hao Peng, Xiaozhi Wang, Xin Lv, Shulin Cao, Jiazheng Xu, Lei Hou, Yuxiao Dong. Jie Tang, and Juanzi Li. Longbench v2: Towards deeper understanding and reasoning on realistic long-context multitasks. arXiv preprint arXiv:2412. /5204, 2024.
Renze Chen, Zhuofeng Wang, Beiquan Cao, Tong Wu, Size Zheng, Xiuhong Li, Xuechao Wei, Shengen Yan, Meng Li, and Yun Liang. Arkvale: Efficient generative Ilm in- ference with recallable key-value eviction. In A. Globerson, L. Mackey, D. Bel- grave, A. Fan, U. Paquet, Tomezak, and C. Zhang (eds.), Advances Neural In- formation Processing Systems, volume 37, pp- 113134-113155. Curran Associates, Inc., 2024a. URL htttps://proceeedings. www.ips. cc. wctpppprpppppppppp.p..peee. httpe/eerrrieeeeppe/eeeeeeee carffreeee .df..
fi
Zhuoming Chen, Ranajoy Sadhukhan, Zihao Ye, Yang Zhou, Jianyu Zhang. Niklas Nolte, Yuandong Tian, Matthijs Douze, Leon Bottou, Zhihao Jia, and Beidi Chen. Magicpig: Lsh sampling for efficient Ilm generation, 2024b. URL https:////// /tt//aaxvvv.rrg.g abcp//bs/22/ 41401000 61.9.
Robert Chew, John Bollenbacher, Michael Wenger, Jessica Speer, and Annice Kim. Llm-assisted content analysis: Using large language models to support deductive coding, 2023. URL https: //arxiv. wwtp://///aaaaaaaaeeaae /206. 14924.
Tri Dao. FlashAttention-2: Faster attention with better parallelism and work partitioning. In Inter- national Conference on Learning Representations (ICIRR, 2024.
Google DeepMind. 2025.
Gemini 2.5: Our most intelligent ai model, March http:/eehhoollygggege ogyo e- deepmi nd/ Searchh202555
URL https://bbl http://lllg..oggllloooooooo gemini- moniini-moeel-ee www.oootttiningguuneoooee
DeepSeek-AI, Daya Guo, Dejian Yang. Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang. Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, Aixin Liu, Bing Xue, Bingxuan Wang, Bochao Wu, Bei Feng, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai Dai, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin. Fucong Dai, Fuli Luo, Guangbo Hao, Guanting Chen, Guowei Li, H. Zhang, Han Bao, Hanwei Xu, Haocheng Wang, Honghui Ding. Huajian Xin, Huazuo Gao, Hui Qu, Hui Li, Jianzhong Guo, Jiashi Li, Jiawei Wang, Jingchang Chen, Jingyang Yuan, Junjie Qiu, Junlong Li, J. Cai, Jiaqi Ni, Jian Liang, Jin Chen, Kai Dong, Kai Hu, Kaige Gao, Kang Guan, Kexin Huang, Kuai Yu, Lean Wang, Lecong Zhang, Liang Zhao, Litong Wang. Liyue Zhang, Lei Xu, Leyi Xia, Mingehuan Zhang, Minghua Zhang, Minghui Tang, Meng Li, Miaojun Wang, Mingming Li, Ning Tian, Panpan Huang, Peng Zhang, Qiancheng Wang, Qinyu Chen, Qiushi Du, Ruiqi Ge, Ruisong Zhang, Ruizhe Pan, Runji Wang, R. J. Chen, R. L. Jin, Ruyi Chen, Shanghao Lu, Shangyan Zhou, Shanhuang Chen, Shengfeng Ye, Shiyu Wang. Shuiping Yu, Shunfeng Zhou, Shuting Pan, S. S. Li, Shuang Zhou, Shaoqing Wu, Shengfeng Ye, Tao Yun, Tian Pei, Tianyu Sun, T. Wang, Wangding Zeng, Wanjia Zhao, Wen Liu, Wenfeng Liang, Wenjun Gao, Wenqin Yu, Wentao Zhang, W. L. Xiao, Wei An, Xiaodong Liu, Xiaohan Wang, Xiaokang Chen, Xiaotao Nie, Xin Cheng, Xin I Liu, Xin Xie, Xingchao Liu,

---

Published as a conference paper at ICLR 2026
Xinyu Yang, Xinyuan Li, Xuecheng Su, Xuheng Lin, X. Q. Li, Xiangyue Jin, Xiaojin Shen, Xi- aosha Chen, Xiaowen Sun, Xiaoxiang Wang, Xinnan Song, Xinyi Zhou, Xianzu Wang, Xinxia Shan, Y. K. Li, Y. Q. Wang. Y. X. Wei, Yang Zhang, Yanhong Xu, Yao Li, Yao Zhao, Yaofeng Sun, Yaohui Wang, Yi Yu, Yichao Zhang, Yifan Shi, Yiliang Xiong, Ying He, Yishi Piao, Yisong Wang, Yixuan Tan, Yiyang Ma, Yiyuan Liu, Yongqiang Guo, Yuan Ou, Yuduan Wang, Yue Gong, Yuheng Zou, Yujia He, Yunfan Xiong, Yuxiang Luo, Yuxiang You, Yuxuan Liu, Yuyang Zhou, Y. X. Zhu, Yanhong Xu, Yanping Huang, Yaohui Li, Yi Zheng, Yuchen Zhu, Yunxian Ma, Ying Tang, Yukun Zha, Yuting Yan, Z. Z. Ren, Zehui Ren, Zhangli Sha, Zhe Fu, Zhean Xu, Zhenda Xie, Zhengyan Zhang, Zhewen Hao, Zhicheng Ma, Zhigang Yan, Zhiyu Wu, Zihui Gu, Zijia Zhu, Zijun Liu, Zilin Li, Ziwei Xie, Ziyang Song. Zizheng Pan, Zhen Huang. Zhipeng Xu, Zhongyu Zhang, and Zhen Zhang. Deepseek-rl: Incentivizing reasoning capability in Ilms via reinforce- ment learning, 2025. URL https:ttt///// // /atp://////iv..//. org/ abs///b///// /2011 12948.
Yuan Feng, Junlin Lv, Yukun Cao, Xike Xie, and S. Kevin Zhou Ada-kv: Optimizing kv cache eviction by adaptive budget allocation for efficient Ilm inference, 2024. URL WRLttt:/s//axi org/abs/ 2407 11550.
flashinfer ai. Kv-cache layout in flashinfer, 2025. URL https://dods http://dooc...aaahafffr...o tutorials/kv_layoee html.
Yao Fu. Challenges in deploying long-contex transformers: A theoretical peak performance analy- sis, 2024. URL httpps/s//aaw.....r//aa///////// 08944.
Wei Gao, Xinyu Zhou, Peng Sun, Tianwei Zhang, and Yonggang Wen. Rethinking key-value cache compression techniques for large language model serving, 2025a. URL https://arxiv. /arxi.. brg/abs/25033 24000.
Yizhao Gao, Zhichen Zeng. Dayou Du, Shijie Cao, Peiyuan Zhou, Jiaxing Qi, Junjie Lai, Hayden Kwok-Hay So, Ting Cao, Fan Yang, and Mao Yang. Secrattention: Learning intrinsic sparse attention in your Ilms, 2025b. URL https: /tt///axivvv.. org/vi.orr//aai www.orrgg/bb//ee.eeo.oooeeeeesss 2212/41000 01441...0 13775.
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. In J. Vanschoren and S. Yeung (eds.), Proceedings of the Neural Information Processing Systems Track Datasets and Benchmarks, volume 1, 2021. URL https: www.::tss-bbbbctaaassssssseeeooo
neurips cc/paper forrs_fiee//___e saales/pper// Paper-ap----e- round2... Pownr-roundd. pdf.
Andreas Hochlehnert, Hardik Bhatnagar, Vishaal Udandarao, Samuel Albanie, Ameya Prabhu, and Matthias Bethge. A sober look at progress in language model reasoning: Pitfalls and paths to reproducibilityy 2025. URL
07086.
Coleman Hooper, Sehoon Kim, Hiva Mohammadzadeh, Monishwaran Maheswaran, June Paik, Michael W. Mahoney, Kurt Keutzer, and Amir Gholami. Squeezed attention: Accelerating long context length IIm inference, 2024. URL https: aaaaiaiiivv.///.////aaaa//aaaava org/ aabs/2411 09688.
Junhao Hu, Wenrui Huang, Weidong Wang. Zhenwen Li, Tiancheng Hu, Zhixia Liu, Xusheng Chen, Tao Xie, and Yizhou Shan. Efficient long-decoding inference with reasoning-aware attention sparsity, 2025. URL WtLLhttss:////xxt:t http//xivvvrrr// orgo hbss//////a/ 2502 11147.
Huggingface. Maxwell-jia/aime__222 2024 February 2025. URL https: htt.ttt::/huggingggeeeee http://hgggnngaac.coooo
datasets (itaaatts//axxeell-ilaaa//aaaaaa
Huiqiang Jiang, Yucheng Li, Chengruidong Zhang, Qianhui Wu, Xufang Luo, Surin Ahn, Zhenhua Han, Amir H. Abdi, Dongsheng Li, Chin-Yer Lin, Yuqing Yang, and Lili Qiu. Minference 1.0: Accelerating pre-filling for long-context 1lms via dynamic sparse attention, 2024a. URL https: //t///aaxvvvv/// wwwoorgg///oooo alag///b//aa 2407 02490.
Juyong Jiang. Fan Wang. Jiasi Shen, Sungju Kim, and Sunghun Kim. A survey on large language models for code generation, 2024b. URL https:/ /ttp/aaivvv.../ abss//b/// /2002 00515.
Greg Kamradt. Needle in a haystack pressure testing Ilms, 2023. URL https: / / /github. com/gkamradt// wt..dtttt/............. Need1 T InAaay
12

---

Published as a conference paper at ICLR 2026
Wonbeom Lee, Jungi Lee, Junghwan Seo, and Jaewoong Sim. InfiniGen: Efficient generative infer- ence of large language models with dynamic KV cache management. In 18th USENIX Symposium Operating Systems Design and Implementation (OSDI 24), pp. 155-172, Santa Clara, CA, July 2024. USENIX Association. ISBN 1-9991333------3 URL ht.ts:/ /tt:://wwwww /www useniseeei.. .orgg/ conferencce/ http://ee.....eeeeeeeeeetetttttt on/lee.
Yucheng Li, Huiqiang Jiang, Qianhui Wu, Xufang Luo, Surin Ahn, Chengruidong Zhang, Amir H. Abdi, Dongsheng Li, Jianfeng Gao, Yuqing Yang, and Lili Qiu. SCBench: A KV cache-centric analysis of long-context methods. In The Thirteenth International Conference on Learning Rep- resentations, 2025. URL https: /oppereeiiee new. Serrrumii=ggkuyyYYCC
Yuhong Li, Yingbing Huang, Bowen Yang, Bharat Venkitesh, Acyr Locatelli, Hanchen Ye, Tianle Cai, Patrick Lewis, and Deming Chen. Snapkv: Lim knows what you are looking for before generation, 2024. URL https: www.ivvvorrgo...... absg///b///// 2tbb//444 2404 14469.
Chaofan Lin, Jiaming Tang, Shuo Yang, Hanshuo Wang, Tian Tang, Boyu Tian, Ion Stoica, Song Han, and Mingyu Gao. Twilight: Adaptive attention sparsity with hierarchical top-p pruning, 2025. URL https: // arrxiv... org/ abs 22002. 02770.
Guangda Liu, Chengwei Li, Jieru Zhao, Chenqi Zhang, and Minyi Guo. Clusterkv: Manipulating IIm kv cache in semantic space for recallable compression, 2024. URL https://arxi:: oxg/abs 2412. 03213.
Xiang Liu, Zhenheng Tang, Hong Chen, Peijie Dong, Zeyu Li, Xiuze Zhou, Bo Li, Xuming Hu, and Xiaowen Chu. Can lims maintain fundamental abilities under kv cache compression?, 2025. URL https: /tl::///axxae aawai.iiiv......//aaaaaaaaa org/ abs/2502. 01941.
Enzhe Lu, Zhejun Jiang, Jingyuan Liu, Yulun Du, Tao Jiang, Chao Hong, Shaowei Liu, Weiran He, Enming Yuan, Yuzhi Wang. Zhiqi Huang, Huan Yuan, Suting Xu, Xinran Xu, Guokun Lai, Yanru Chen, Huabin Zheng, Junjie Yan, Jianlin Su, Yuxin Wu, Neo Y. Zhang, Zhilin Yang. Xinyu Zhou, Mingxing Zhang. and Jiezhong Qiu. Moba: Mixture of block attention for long-context llms, 2025. URL https: wtaaaxivvv.. www.iii..rrg..ooooooe oww.org/gaa.o.o. /abs//202..1188/
Meta. The llama herd of models, 2024. URL httpp///aaaabb...rr//////0.1.110
OpenAl. Openai ol system card.
2024.
URL https://cdn.opente com/
ol o1-system-card-222455 .df..
Qwen. Qwen3: Think deeper, act faster, May 2025a. URL h.ll.tttt////eenn..giieeee blog/qwen3/.
Qwen. Qwq-32b: Embracing the power reinforcement learning, March 2025b. URL https:
// ////wllmm.mmmmmmmmmmm.../....mmm github........... fo/bllllloooooo logo//// ho/////-/wooooooooo/o//////o//// gwq-32b/
Qwen, An Yang. Baosong Yang. Beichen Zhang, Binyuan Hui, Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang. Le Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin Zhu, Rui Men, Runji Lin, Tianhao Li, Tianyi Tang, Tingyu Xia, Xingzhang Ren, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yu Wan, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, and Zihan Qiu. Qwen2.5 technical report, 2025. URL https:: //aaaaiiiii/i org/ abs /dss/2411 2222 15115.
David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R. Bowman. Gpqa: A graduate-level google-proof q&a benchmark, 2023. URL https: /ttp/aaxivvv.////e org/ /ttp//21...2.2...
Jianlin Su, Yu Lu, Shengfeng Pan, Ahmed Murtadha, Bo Wen, and Yunfeng Liu. Roformer: En- hanced transformer with rotary position embedding, 2023. URL WiR.hhttps://exxivcoom 21 2104. 69.
Hanshi Sun, Li- Wen Chang, Wenlei Bao, Size Zheng, Ningxin Zheng, Xin Liu, Harry Dong, Yuejie Chi, and Beidi Chen. Shadowky: Ky cache in shadows for high thoouhhouu long-context 1lm inference, 2025. URL wwt.hs:////aateieeee org abs / 21 65.
13

---

Published as a conference paper at ICLR 2026
Hanlin Tang, Yang Lin, Jing Lin, Qingsen Han, Shikuan Hong, Yiwu Yao, and Gongyi Wang. Razorattention: Efficient kv cache compression through retrieval heads, 2024a URL https: http://...rg/a////...ooo
Jiaming Tang, Yilong Zhao, Kan Zhu, Guangxuan Xiao, Baris Kasikci, and Song Han. Quest: Query-aware sparsity for efficient long-context Ilm inference. ICML, 2024b.
Huaijin Wu, Lianqiang Li, Hantao Huang, Tu Yi, Jihang Zhang, Minghui Yu, and Junchi Yan. HShare: Fast LLM decoding by hierarchical key-value sharing. In The Thirteenth International Conference on Learning Representations, 2025, URL http::/:: http///ppeeeee.enmreeeeee new./.ne/////eee...ee. forum?
id=Tb5PY5vwp6.
Yuhao Wu, Ming Shan Hee, Zhiqing Hu, and Roy Ka-Wei Lee. Longgenbench: Benchmarking long-form generation in long context Ilms, 2024. URL httpss:/aaxcvv.org/aa///// 02076.
XAI. Grok 3 beta - the age of reasoning agents, February 2025. URL grok-3.
Guangxuan Xiao, Jiaming Tang, Jingwei Zuo, Junxian Guo, Shang Yang, Haotian Tang, Yao Fu, and Song Han. Duoattention: Efficient long-context Ilm inference with retrieval and streaming heads, 2024a. URL htttp:://axii..../t////
http/aaxivvooog. org www.oorgg//////ee./e/....ee.e//e 2410 10819.
Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. Efficient streaming language models with attention sinks. ICLR, 2024b.
Qingyue Yang. Jie Wang, Xing Li, Zhihai Wang. Chen Chen, Lei Chen, Xianzhi Yu, Wulong Liu, Jianye Hao, Mingxuan Yuan, and Bin Li. Attentionpredictor:: Temporal pattern matters for effi- cient II inference, 2025. URL https: htt///axxvv.rrgg abtp//bb///22.2//. 04077.
Jingyang Yuan, Huazuo Gao, Damai Dai, Junyu Luo, Liang Zhao, Zhengyan Zhang, Zhenda Xie, Y. X. Wei, Lean Wang, Zhiping Xiao, Yuqing Wang, Chong Ruan, Ming Zhang, Wenfeng Liang, and Wangding Zeng. Native sparse attention: Hardware aligged and natively trainable sparse attention, 2025. URL https::ttp.... //t//xiivvv. http://wwww..gaao////o/ooooo / 11089.
Zhenyu Zhang, Ying Sheng, Tianyi Zhou, Tianlong Chen, Lianmin Zheng, Ruisi Cai, Zhao Song, Yuandong Tian, Christopher Ré, Clark Barrett, Zhangyang "Atlas" Wang, and Beidi Chen. H2o: Heavy-hitter oracle for efficient generative inference of large language models. In A. Oh, T. Nau- mann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (eds.), Advances in Neural Information Processing Systems, volume 36, pp. 34661-34710 Curran Associates, Inc., 2023.
Qihui Zhou, Peiqi Yin, Pengfei Zuo, and James Cheng. Progressive sparse attention: Algorithm and system co-design for efficient attention in Ilm serving, 2025. URL https: /arxivv....///////////////////v/ orrg/ abs/2503. 00392.
14

---

Published as a conference paper at ICLR 2026
Table Accuracy results recall using query vector of the last layer and the last step.
LongBench v2
MATH500
AIME24
GPQA
Query Vector of the Last Layer Query Vector of the Last Step (Speculative Retrieval w/o Correction)
26.44
68.00
33.00
30.50
26.63
69.00
50.00
36,00
Table 5: Accuracy results of alternatives for achieving group-consistent selection.
LongBench v2
MATH500
AIME24
GPQA
MaxQ
26.64
70.00
51.67
38.00
MeanQ
26.64
70.00
46.67
37.00
MaxQK
26.84
71.00
46.67
36.50
MeanQK
26.84
70.00
51.67
34.00
MaxS
26.64
69.50
47.50
36.50
MeanS
26.84
70.00
52.92
39.50
A DETAILED EXPERIMENTAL SETTINGS
Baselines We consistently set the page size to 32 for FreeKV, Quest, Vrke, ShadowKV and RaaS. Since the original implementations of RaaS, Quest and InfiniGen are not group-consistent, we adapt them by applying maximum pooling over scores within the group ensure consistent selection. For ShadowKV, which does not natively support long-generation scenarios, we modify to update the SVD results every W generated tokens. Following standard practice, KV cache compression is not applied to the first layer in any of the methods. Other hyperparameters of the baselines, such as the update threshold for RaaS, the SVD rank for ShadowKV, and the skewing retained specified in their original configurations.
rank for InfiniGen,
The sink size S, local window size W. correction threshold т and sampling policies vary depending on the task.
LongBench v2 We set S W 128 and 0.8, and apply greedy decoding.
LongGenBench We set = W 512 and r = 0.9, applying stochastic sampling with a temper- ature of 0.95, a top-p value of 0.95, and maximum generation length of 16K following the original setup.
Reasoning Tasks we set S W = 512, = 0.9 and the maximum generation length to 16K, and apply stochastic sampling with temperature of 0.6 and top-p value of 0.95, following the original DeepSeek-R1 setup.
B ABLATION STUDIES FOR MODEL ACCURACY
B. RECALL USING THE QUERY VECTOR OF THE LAST LAYER VS. THE LAST STEP
InfiniGen computes skewed query vectors by re-projecting hidden states of layer and recall KV tuples for layer 1. To demonstrate the superiority of speculative retrieval in FreeK V. which recalls using the query vector the last decoding step, we adapt InfiniGen to select and recall directly with the last layer's 'ss query vector under the same page-wise setting FreeKV.
select
Table reports overall accuracy on LongBench v2 and avg@k on reasoning tasks, evaluated with 1enn--2.--B--strruue and DeepSeek-R 1-Qwen-7B under the same setting described in Section 5.2. As shown, while using the query vector from the last layer achieves accuracy comparable to spec- ulative retrieval LongBench v2 and relatively simple reasoning tasks such MATH500, it in- curs substantial accuracy degradation on more challenging reasoning tasks, including AIME24 and GPQA.
15

---

Published as a conference paper at ICLR 2026
Table 6: Accuracy results of alternatives for achieving group-consistent correction.
LongBench v2
MATH500
AIME24
GPQA
Max Pooling over Group C, Mean Pooling over Group C,
26.84
70.00
53.33
39,00
26.84
70.00
$2.92
39.50
Table 7: Accuracy results of different correction thresholds.
LongBench v2
MATH500
AIME24
GPQA
7 0 (No Correction)
26.63
69.00
50.00
36.00
0.7
26.63
70.00
50.00
36.00
0.8
26.84
70.50
51.66
35.50
0.9
26.84
70.00
52.92
39.50
T (No Speculation)
26.84
70.00
53.00
39.50
B.2 ALTERNATIVES FOR ACHIEVING GROUP-CONSISTENT SELECTION
To achieve group-consistent selection, we evaluate max and mean pooling across heads within the same group for (i) query vectors (Q), (ii) attention weights between query vectors and page sum- maries (QK), and (iii) normalized attention weights after softmax (S).
Table 5 reports overall accuracy on LongBench v2 and avg@k on reasoning tasks, evaluated with 2-n--..----B-tt-ue and -aepSeee--R---en.. under the same setting described in Section 5.2. We observe that mean pooling over the weights after softmax (MeanS) achieves the best overall performance, which is the strategy adopted in FreeKV.
B.3 EFFECT OF CORRECTION IN SPECULATIVE RETRIEVAL
Group-consistent correction The query vector similarity (C;) of heads within the same group needs to be unified to enable croup-consistent correction. Table reports overall accuracy on Long- Bench v2 and avg@ & reasoning tasks, comparing max and mean pooling over C, with Qwen-2.5- 7B-Instruct and DeepSeek-RI RI--Qwnn-B.. As shown, both pooling strategies yield similar accuracy, but max pooling triggers corrections and incurs higher overhead. Therefore, FreeK V adopts mean pooling.
Correction threshold Since correction triggered when C, < т, higher threshold would trigger more correction, theoretically yielding better accuracy with higher costs. Table 7 shows overall accuracy on LongBench v2 and avg@k on reasoning tasks using different т. with Qwen- 2.5-7B-Instrus and DeepSeck-R R1-Qwen-7BB As shown, while using 0.8 works well for simple tasks such as LongBench v2 and MATH500, underperforms on complex reasoning tasks, perhaps due to accumulated errors during reasoning. Therefore, 0.9 for long-generation scenarios.
0.8 for long-input scenarios and
C ABLATION STUDIES FOR INFERENCE EFFICIENCY
C.I EFFICIENCY UNDER DIFFERENT INPUT AND OUTPUT LENGTHS
Fig. 8 reports the inference latency Ark Vale and FreeKV across different input and output lengths, using the Llama-3. 1-8B--Instan model under the same settings as Sec. 5.3. In the long-input sce- nario, FreeKY consistently outperforms Vale by 2.7x-533x The relative speedup decreases for longer inputs due to the increasing prefill cost, which is identical for both methods. In the long- generation scenario, FreeKV maintains a stable 5.3x speedup across all output lengths, benefiting from its fixed-size KV budget and efficient recall overlap.
16

---

Published as a conference paper at ICLR 2026
ArkVale
FreeKV
1-88-instuud (Long Inputy
11
Liama--3.1100 3.1-88-instruu (Long Generation)
080
102
4.9x
3.9x
5.3x
t
14.9x
10K
SGK
1aBc
13K
BAK
Imput Length
Output Length
Figure 8: Inference latency for different input and output lengths.
No Opt.
HL+38
HL+DB+SR
Liama-3 3. I-8B-Instruct [Long Input)
Liama-3 1-88 iintructtttttttttt...mm (Long Generation)
510
Latoncy (5)
200
1650
100
Batch Size
Batch Sizo
Figure 9: Ablation results for efficiency optimizations.
C.2 EFFECT OF OPTIMIZATION TECHINIQESS
We present the ablation results of efficiency optimizations applied in FreeKV, including hybrid lay- outs (HL), doublc-buffering streamed recall (DB) and speculative retrieval (SR), evaluated using (lama-3.1-8B-Inttiii
under long-input and long-generation scenarios. As shown Fig. 9, hybrid layouts, which eliminate fragmented data transfers, contribute the most to the improvements, achiev- ing up a 10.5x speedup. For a batch size of 4, streamed recall adds a further 1.2x speedup, while overlapping with speculative retrieval provides an additional 1.9x speedup.
While HL alone provides the largest gains, the roles of SR and system-level optimizations are equally critical in our algorithm-system co-optimization framework. The SR algorithm enables pre-selection and prefetching of important KV pages, making it possible to overlap recall operations with ongoing computation. However, illustrated in the latency breakdown in Fig. I (right), recall operations in SOTA KV retrieval methods can be overwhelmingly slow. Therefore, applying spec- ulative retrieval without system-side optimizations results in limited efficiency benefit, as the slow recall operations remain on the critical path and cannot be effectively overlapped. The sidte optimizations (HL and DB) dramatically reduce the latency of recall operations to the same order as other operations, enabling effective overlapping and full latency hiding and allowing practical speedup of speculative recall.
D GENERALIZATION OF HIGH QUERY SIMILARITY
To validate the generalization of high query similarity, we analyze the query similarity across tasks, model scales, architectures, and training stages, presented in Table 8.
Tasks: We first measure the query similarity of Dowen-2.5-77 2.5-7B--nntruuct across multiple tasks, including LongBench, LongGenBench, AIME24, MATH500, and GPQA. We observe that similarity in the first layer is relatively low, where FreeKV and other compression methods are not applied. Therefore, we report the average similarity across all other layers and decoding steps. As shown in the table, query similarity remains consistently high, around 0.9 for all tasks.
Model scales: To validate that high query similarity generalizes across model scales, we measure it for Qwen-2.5-1.5B, 3B. 7B, and 44s-rustt As shown in the table, query similarity remains consistently high across all scales and tasks, around 0.9, demonstrating that this property is largely model-scale agnostic.
17

---

Published as a conference paper at ICLR 2026
Table 8: Query similarity across various tasks, model scales, architectures, and training stages.
Model
LongBench v2
LongGenBench
MATH 500
AIME24
GPQA
Qwen-2.5-7B-Bl Instruct
0.91
0.89
0.90
0.90
0,90
Dwen-2.5-115B-IIssti
0.92
0.91
0.91
0.92
0,91
Qwen-2.5-33 10nn-2.5-3B-Inttteee I-Istrrtttttt
0.89
0.89
0.90
0.90
0.90
Qwen-2.5-
0.88
0.88
0.89
0.89
0.89
Gaama-3.1-8 1--8B-Innttrcct
0.87
0.89
0.88
0.88
0.88
Qwen-3-8B
0.82
0.85
0.85
0.84
0.85
Qwen-2.5-7B (Base)
0.90
0.91
0.90
0.90
0.90
DeepSeek-R1 R1-Diisill-Qwen
0.87
0.86
0.86
EK
611
614
1ww.
60
Figure 10: Per-head query similarity Qwenn225-7B-Insttee LongBench.
Model architectures: To validate that high query similarity generalizes across model archi- tectures, we measure it for Beenn-55-B--Inttreee 1.1ma-3.1188 --struutt and Qwen-3-8PP As shown the table, similarity remains consistently high across architectures, ranging from 0.85 to 0.9, The slightly lower similarity in Qwen-3 may be due to its mixed-thinking training recipes.
Training stages: To validate that high query similarity generalizes across different training stages, we measure & RLHF), and DeepSeek-R 1- 1--Disill-Qwen-- similarity remains consistently high across all stages, ranging from 0.86 to 0.9, indicating that this property is robust to variations in training.
Qwen-2.5-7B (Base, Pretrain-only), Dwwe-2.5-7B-Innstmul (SFT
(Long-CoT RL). As shown in the table,
In addition, we provide the detailed per-head query similarity of some models and tasks, in Figure 10 and Figure 14. We observe that similarity in the first layer is relatively low, where FreeKV and other compression methods are not applied.
These results demonstrate that high query similarity is consistent property across tasks, model scales, architectures, and training stages. This robustness supports the general applicability of FreeKV's speculative retrieval mechanism.
18

---

Published as a conference paper at ICLR 2026
Figure 11: Per-head query similarity of Llama-3.1 1amma-3.1-8B-mme 3.1-SB-Intttuct on LongGenBench.
Figure 12: Per-head query similarity of Qwen 10wen22.5-444 2.5-14B-Inntrree Instruct on MATH500.
E CORRECTION RATE ANALYSIS
We report the correction rate, i.e., the fraction KV heads corrected averaged over decoding steps, across various tasks, models, and thresholds
Following our original setup, we use Llama-3. 1lama-3.1-BB-nee --I--nnsttctt and Been-225-BB-Innae for Long- Bench v2 and LongGenBench, and their DeepSeek-RR 1 variants for reasoning tasks.
19

---

Published as a conference paper at ICLR 2026
61
38
Lay LOBOIe
69
13
411
32
16
6H
33 13
Figure 13: Per-head query similarity of DeepSeek-RI-Qwen 7B on AIME24.
Table 9: Correction rates across different models, tasks and thresholds. Fin stands for that only
finished examples are counted.
LongBench v2
LongGenBench
MATH500
AIME24
GPQA
AIME24 (Fin.)
GPQA (Fin.)
Llama-8B. 0.8
0.14
0,05
0.04
0.10
0,08
0.04
0.06
Llana-88 8B, 7= 0.9
N/
0.17
0,20
0.43
0.35
0.19
0.29
Owen 7B, 0..8
0.09
0.04
0.08
0.22
0.18
0.11
0.15
Qwen-7B, r 0.9
N/A
0.22
0.17
0.52
0.47
0.29
0.41
The average correction rates for each task are shown in the Table 9. The rates range from 0.04 to 0.52 and depend on task difficulty. Simpler tasks (LongBench v2, LongGenBench. MATH 500) exhibit lower correction rates, while harder reasoning tasks (AIME24, GPQA) trigger more frequent cor- rection. We also observe that correction disproportionately higher on unfinished hard examples; when considering only finished cases, correction rates drop substantially.
We also provide per-layer histograms of detailed correction rate distributions across tasks, models, and thresholds, shown in Figure 15 to Figure 19.
F
LLM USAGE
Large Language Models (LLMs) were only used to aid in the polishing of this work.
important to note that the LLM was not involved in the ideation, research methodology, or experimental design. All research concepts, ideas, and analyses were developed and conducted by the authors. The contributions of the LLM were solely focused on improving the linguistic quality of the paper, with no involvement in the scientific content or data analysis.
The authors take full responsibility for the content of the manuscript, including any text generated or polished by the LLM. We have ensured that the LLM-generated text adheres to ethical guidelines and does not contribute to plagiarism or scientific misconduct.
20

---

Published as a conference paper at ICLR 2026
Levi
Leyer Pu
Leyel S.eo
6-70
637
17t san
www. www.e.......
Laye Leneerr
wyw
Figure 14: Per-head query similarity of Qwen-3-8B on GPQA.
21

---

Published as a conference paper at ICLR 2026
723222223
'222299
2222222
11111117
Figure 15: Per-layer distribution correction rates Qwen-2.5-7B-Inssiee on LongBench with 0.8.
Figure 16: Per-layer distribution of correction rates 2.wen22.5-7B-Innii on LongGenBencc with = 0.9.
22

---

Published as a conference paper at ICLR 2026
222222223
12222799
12222222
272222297
Figure 17: Per-layer distribution of correction rates -wwen22.--7B-Instmeee on LongGenBench with = 0.8.
Figure 18: Per-layer distribution of correction rates of 1oma--.1-8B-Insiie on LongGenBench with = 0.9.
23

---

Published as a conference paper at ICLR 2026
1772777 77
Figure 19: Per-layer distribution of correction rates tasks with 7 = 0.9.
on three reasoning
24


Reviews:
[{"review_id": "tHPwtBhqE8", "reviewer": "Reviewer_tvZV", "summary": "The paper introduces FreeKV, a KV-cache retrieval framework that combines algorithmic ideas—speculative retrieval that reuses pages recalled at the previous step, group-consistent page selection with softmax-pooled scores, and fine-grained head-wise correction triggered by low query-similarity—with a systems design that uses hybrid KV layouts (HND on CPU, NHD on GPU) and double-buffered streamed recall to cut fragmented transfers and overlap CPU to GPU recall with compute. The system keeps page summaries and last-step queries on GPU, stores the full KV pool on CPU in a transfer-friendly layout, and converts layout on-the-fly during recall. Across LongBench v2, LongGenBench, and several long-reasoning tasks, FreeKV matches or nearly matches full-KV accuracy while delivering large end-to-end latency reductions versus prior retrieval methods. Reported speedups reach up to 13.7× vs ArkVale and up to 8.4× vs ShadowKV on A100-PCIe settings, with ablations attributing a major share to the hybrid-layout plus streamed-recall pipeline.", "questions": "* What is the average correction rate (fraction of KV heads corrected per step) across tasks? How often does correction cascade across layers? A per‑layer histogram and the incremental latency/bytes attributable to correction would clarify the practical cost of robustness.\n* In which scenarios does speculative reuse mis‑speculate?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* The paper offers a clear algorithm–system co-design in which speculative reuse moves selection and recall off the critical path and head-wise correction restores accuracy only when needed.\n* The hybrid HND/NHD layout and double-buffered streamed recall directly target PCIe fragmentation and enable effective overlap, which is a practical systems contribution likely to transfer to real serving stacks.\n* The empirical study spans multiple models and benchmarks with detailed settings, and demonstrates near-lossless accuracy relative to full-KV along with large end-to-end speedups, supported by ablations over lengths and components.", "weaknesses": "* LongBench v2 spans 8K to 2M tokens, yet the paper truncates inputs to 64K and caps generation to 16K, which leaves the very-long regime underexplored where offloading dominates; please add at least one ≥128K long-input case and one ≥32K long-generation case to validate scaling and stress the recall pipeline.\n* Since accuracy relies on an LLM-as-judge (Qwen-3-32B), consider strengthening the evaluation with a larger or multi-judge setup, report inter-judge agreement to calibrate the scores.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "tLov5cWxFL", "reviewer": "Reviewer_rpQv", "summary": "The paper proposes FreeKV, an algorithm–system co-optimization framework for efficient KV-cache retrieval in long-context LLM inference. The key idea is to leverage the similarity of query vectors across adjacent decoding steps to enable speculative retrieval — shifting KV selection and recall out of the critical path — combined with fine-grained correction when query deviation occurs. On the system side, FreeKV introduces a hybrid CPU–GPU KV layout (HND on CPU, NHD on GPU) and double-buffered streamed recall to overlap transfers and computation. Experimental results show up to 13× speedup over prior retrieval methods (Arkvale, ShadowKV, InfiniGen) with negligible accuracy loss.", "questions": "1. FreeKV’s speculative retrieval design always reuses the previously recalled KV pages for the next token. Intuitively, one might expect cumulative drift or stale-selection errors over long generations, especially in reasoning tasks. However, the reported performance on long-context reasoning excels significantly. Could the authors elaborate on why this drift does not appear to accumulate in practice?\n2. Since the KV recall happens right after the (t-1)th token selection, which is far before the KV is used by t, maybe we can even store the KVCache in disk?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Clear motivation and strong setup:\nThe paper provides a well-motivated problem statement supported by preliminary empirical analysis.\n2. Novel speculative retrieval mechanism:\nThe proposed speculative retrieval with fine-grained correction is a conceptually novel idea that effectively breaks the strict dependency between KV selection and query scoring, enabling computation–I/O overlap. \n3. Comprehensive and convincing experiments:\nThe paper evaluates FreeKV across diverse models and tasks (e.g., LongBench, and reasoning datasets) against a wide range of baselines. The results show substantial speedups and near-lossless accuracy, demonstrating both practicality and robustness.", "weaknesses": "1. Missing analysis of correction overheads:\nWhile the fine-grained correction mechanism is central to FreeKV’s efficiency–accuracy balance, the paper lacks a quantitative analysis of correction frequency and its impact on latency under different similarity thresholds. Such results would clarify the trade-off between performance and accuracy.\n2. Limited study on KV budget sensitivity:\nThe experiments fix the KV budget B but do not explore how varying B influences accuracy and throughput.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "S6tBkQJTmU", "reviewer": "Reviewer_9HjF", "summary": "This paper addresses the efficiency bottleneck of KV cache retrieval in LLM inference and proposes FreeKV, an algorithm–system co-optimization framework. It introduces speculative retrieval to move KV selection and recall off the critical path and fine-grained correction to maintain accuracy. On the system side, it uses hybrid CPU–GPU layouts and streamed recall to reduce data transfer overhead. Experiments show near-lossless accuracy and up to 13× speedup over prior retrieval methods.", "questions": "1. quantify how query similarity impacts recall accuracy or establish conditions where correction is triggered.\n2. Include ablation studies isolating algorithmic vs. system-side gains, to show which components (speculative reuse, fine-grained correction, hybrid layout) contribute most to the speedup.\n3. Strengthen novelty positioning: highlight what design principles or insights make FreeKV different from prior works like InfiniGen and ArkVale, rather than appearing as an engineering integration.", "limitations": "", "rating": 2, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 1, "strengths": "1. The paper tackles an important and practical problem in LLM serving: efficient KV retrieval under long contexts.\n2. FreeKV presents well-motivated system–algorithm co-design with comprehensive experiments covering multiple models and tasks, demonstrating impressive empirical speedups and accuracy preservation.", "weaknesses": "1. FreeKV improves runtime efficiency via engineering and overlap techniques. Its algorithmic novelty is incremental over prior work like InfiniGen and ArkVale.\n2. The speculative reuse in FreeKV depends on strong query similarity assumptions that may not generalize to all model architectures or reasoning tasks.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5bqmnony8P", "reviewer": "Reviewer_NYAv", "summary": "FreeKV introduces a unified algorithm-system co-optimization framework that dramatically accelerates LLM inference (decoding specifically) by eliminating KV cache retrieval latency. It does this through speculative retrieval, which reuses KV caches from the previous decoding step to fully overlap selection and recall with ongoing computation, and head-wise fine-grained correction, which selectively updates only attention heads whose query patterns change. FreeKV achieves up to 13 times faster inference compared with prior KV retrivial methods and better accuracy then KV dropping methods.", "questions": "What factors could affect the \"query similarity\" property the paper assumed?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper proposes FreeKV, a speculative KV retrieval framework built on the key insight that query vectors in adjacent decoding steps exhibit high similarity. This observation enables efficient reuse of previous KV selections, and represents a novel and meaningful advancement in accelerating LLM inference (if the observation is empirically validated).\n\nThe paper presents the approach with clear organization and technical depth, detailing both the algorithmic design (speculative retrieval and head-wise correction) and the system-level implementation (hybrid layouts and streamed recall). The figures and timelines are well-constructed, making the method intuitive to follow.\n\nThe experimental evaluation is comprehensive and convincing, covering multiple models and datasets. The results consistently demonstrate that FreeKV achieves a superior trade-off between efficiency and accuracy compared with existing KV retrieval and KV dropping methods, highlighting both practical impact and generality.", "weaknesses": "While the design of FreeKV is primarily motivated by the observed similarity of queries across adjacent decoding steps, the validation of this core assumption remains limited. Section 3.1 provides some preliminary evidence, but the analysis is neither comprehensive across diverse models and prompts nor sufficiently detailed to characterize the factors influencing query similarity. A deeper investigation would strengthen the paper, because of the worry that this method could perform bad on corner case scenarios, which is also interesting to know. For example, examining whether similarity levels correlate with specific tasks, semantic patterns, or model architectures, and how training or fine-tuning methods affect this property. Additionally, it would be valuable to quantify what degree of similarity is necessary to maintain decoding accuracy. Although the appendix includes relevant ablation studies, a concise summary and discussion of these findings in the main text would make the paper’s design justification more convincing.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)