LouisKV: Efficient KV Cache Retrieval for Long Input-Output Sequences
Authors: Wenbo Wu
Qingyi Si
Xiurui Pan
Ye Wang
Jie Zhang
Venue: ICLR 2026 Poster (2026)
Keywords: LLMs
KV cache retrieval
LLM inference acceleration
Primary area: generative models

Abstract: While Key-Value (KV) cache succeeds in reducing redundant computations in auto-regressive models, it introduces significant memory overhead, limiting its practical deployment in long-sequence scenarios. Existing KV retrieval methods attempt to mitigate this by dynamically retaining only a subset of KV entries on the GPU. However, they still suffer from notable efficiency and accuracy bottlenecks due to per-token retrieval and coarse-grained page-level KV management strategy, especially in long-output reasoning scenarios. With the emergence of large reasoning models, efficiently handling such scenarios has become increasingly important. To address this issue, we present two key observations: (1) critical KVs exhibit strong temporal locality during decoding, and (2) these KVs exhibit distinct distribution patterns across the input prompt and the generated output. Building on these observations, we propose \emph{LouisKV}, an efficient KV cache retrieval framework designed for various long-sequence scenarios. Specifically, LouisKV introduces a semantic-aware retrieval strategy that leverages temporal locality to trigger retrieval only at semantic boundaries, drastically reducing computation and data transfer overhead. LouisKV also designs a decoupled, fine-grained management scheme that tailors differentiated strategies for input and output sequences to create retrieval units that better match the model's attention patterns, thereby enabling the precise identification of critical KVs. Furthermore, to boost system efficiency, LouisKV incorporates several kernel-level optimizations, including custom Triton and CUDA kernels to accelerate the KV clustering and retrieval. Evaluation results show that LouisKV achieves up to 4.7$\times$ speedup over state-of-the-art KV retrieval methods while maintaining near-lossless accuracy across diverse long-sequence tasks, including long-input short-output, short-input long-output, and long-input long-output scenarios.

Published as a conference paper at ICLR 2026
LOUISKV: EFFICIENT KV CACHE RETRIEVAL FOR LONG INPUT-OUTPUT SEQUENCES
Wenbo Wu'; Qingyi SI;; Xiurui Pan', Ye Wang', Jie Zhang"!
Peekin University Huawei Technologies Ltd. *Chongqing University of Post and Telecommunications
ABSTRACT
While Key-Value (KV) cache succeeds in reducing redundant computations in auto-regressive models, it introduces significant memory overhead, limiting its practical deployment in long-sequence scenarios. Existing KV retrieval methods attempt to mitigate this by dynamically retaining only a subset of KV entries on the GPU. However, they still suffer from notable efficiency and accuracy bottle- necks due to per-token retrieval and coarse-grainee goasee-g page-level KV management strategy, especially in long-output reasoning scenarios. With the emergence of large reasoning models, efficiently handling such scenarios has become increas- ingly important. To address this issue, present two key observations: (1) crit- ical KVs exhibit strong temporal locality during decoding, and (2) these KVs exhibit distinct distribution patterns across the input prompt and the generated output. Building on these observations, we propose LouisKV, an efficient KV cache retrieval framework designed for various long-sequence scenarios. Specifi- cally, VouisKV introduces semantic-aware retrieval strategy that leverages tem- poral locality trigger retrieval only at semantic boundaries, drastically reduc- ing computation and data transfer overhead. LouisKV also designs a decoupled, fine-grained management scheme that tailors differentiated strategies for input and output sequences to create retrieval units that better match the model's 's attention patterns, thereby enabling the precise identification of critical KVs. Furthermore, to boost system efficiency, LouisKV incorporates several kernel-level optimiza- tions, including custom Triton and CUDA kernels to accelerate the KV clustering and retrieval. Evaluation results show that LouisKV achieves up to 4.7x speedup over state-of-the-art KV retrieval methods while maintaining near-lossless accu- racy across diverse long-sequence tasks, including long-input short-output, short- input long-output, and long-input long-output scenarios.
INTRODUCTION
Large language models (LLMMs) (Achiam et al. 2023 Guo et al.) 2025) have demonstrated remark- able capabilities both long-context understanding tasks like long document question answer- ing (Bai et al., 2023), and in long thain-of-thoughe (CoT) generation for complex reasoning tasks like mathematics (Hendrycks et al. 2021). The inference process of mainstream LLMs auto- regressive, with the KV cache (Shi et al. (024) stored in memory to avoid recomputation. However, the memory footprint of the KV cache grows approximately linearly with the sequence length, easily exceeding GPU memory capacity.
To address this challenge, recent studies have proposed sparse attention mechanisms to reduce KV cache usage. One line of work, known KV dropping (Xiao et al. 2023 Zhang et al., 2023 Feng ctt al., 2024), retains only important tokens and permanently discards those deemed less relevant. However, such methods fail to account for the dynamic nature of KV importance, that is, tokens ini- tially considered unimportant may become critical in subsequent steps, leading to reduced inference accuracy. To address this, another line of work, KV retrieval (Chen al. 2024a Tang al. 2024
"Equal contribution.
I "orresponning author.

---

Published as a conference paper at ICLR 2026
Prefil XV
Decode
Critical for Current Token
Generated RV of Current Token
Retrieve on this token
Prefill
Decode Token 0
Decode Token
Decode Token 6
GPU
Arkvale
x
[
Officod
Autrieve Page 0
1
Officee
Offoad
...
Officed D
Revrieve Per
CPU
Page
Page 2
Tolen
Paae
1 Parr
Pade
Pule
Page
2
GPU
LouisKV
Cluster
Retrieve Cluster 1,2
Offioed
Roase Collial
Offocd
Recrieee Cluster
Offloed
Retrieve
CPU
Officed
Lewt Takon
Segment
Claskr
Clustf
Claatr
Sez
Chesk
Chiuer
2
Se2
Figure 1: Comparison of Arkvale and LouisKV. Arkvale adopts page-level KV management and retrieves critical pages from the CPU for every decoding token, leading to high transfer overhead and potential accuracy degradation. In contrast, LouisKV reuses critical K by exploiting temporal locality to significantly reduce retrieval frequency. It also employs a decoupled KV management scheme, enabling precise retrieval to improve transfer efficiency while maintaining high accuracy.
Liu et al. 2024a Chen et al. 2024b: Pan et al. 2024). has emerged as a promising alternative. This approach preserves the full KV cache, typically in CPU memory to avoid GPU Out-Of-Memory (OOM) errors, whereas dynamically estimating a subset of important entries and retrieving them for the current token's 's generation.
Although existing KV retrieval methods achieve better accuracy than KV dropping approaches by retaining complete information, these methods still face significant bottlenecks in both inference efficiency and retrieval performance. As shown in Figure I
primary issue is that existing methods (Zhang et al. 2025; Lee et al.) 2024 Xu et al. 2025) trigger retrieval at every decoding token. This introduces two primary sources of overhead: the computational cost of selecting critical tokens and the data transfer latency of recalling KV entries from the CPU. This accumulated overhead becomes particularly prohibitive in scenarios involving long-output reasoning models (Guo et al. 2025). which have become mainstream paradigm. Meanwhile, existing methods (Chen et al. 2024a
Xiao et al. 2024 Yuan et al. 2025) typically adopt page-level KV management strategies. How- ever, this coarse-graings management strategy frequently transfers entire pages containing numerous non-critical KV entries, which either increase data transfer overheads or reduce inference accuracy. Therefore, enhancing inference efficiency while preserving accuracy across all long-sequence sce- narios has become a key challenge.
To address this bottleneck, we conduct an empirical analysis of critical KV access patterns in long- output reasoning models (Yang et al. 2025). Our two key observations bring new insights to well address two fundamental questions: how to trigger retrieval to reduce overhead, and how to man- age the KV cache to improve retrieval precision. First, observe that critical KVs access exhibits strong temporal locality. Specifically, the sets of critical KV entries accessed in adjacent decoding tokens show high similarity. For instance, during mathematical reasoning, the model produces inter- mediate reasoning steps wherein tokens within the same step consistently attend to the same math- ematical lemmas. This insight motivates triggering retrieval only at semantic boundaries. Second, we observe that critical KVs exhibit distinct distribution patterns across input and output sequences. Specifically, for the currently generated token, critical KVs in long input sequences are often dis- tributed sparsely throughout the context, while those in long output sequences of reasoning models tend to concentrate locally within some reasoning steps. This inspires the adoption of different KV management strategies for the prefill and decode stages.
Building on these observations, we propose LouisKV, an efficient KV cache retrieval framework that integrates semantic-aware retrieval strategy with a decoupled fine-grained management scheme, as shown in Figure [ To exploit temporal locality during decoding, LouisKV groups consecutive decoding tokens into coherent segments based on query vector similarity. It triggers retrieval only at segment boundaries to drastically reduce the retrieval overhead. Based on the attention distribution, LouisKV proposes decoupled KV management scheme tailored for distinct distribution patterns in the input and output sequence. Specifically, for the long input sequence, LouisKV employs clus- tering to group KV entries of semantically similar tokens into semantic clusters; for the long output
2

---

Published as a conference paper at ICLR 2026
Norm. Accuracy
0.8
SOTA Retrieval Methods
FullCache
LouisKV
Norm. Latenoy
1.0
0.6
0.4
0.5
Doo
WOO
0.0
0.0
Long-Input, Short-Ouput
Short-Input, Long Long-Ouput
Long-Input, Long Ouput
Long-Input Short-Ouput
Short-input. Long- Lnn-Oupput
Long- Long-Ouput
(a) Accuracy
(b) Efficiency
Figure Accuracy and efficiency comparison on various long-sequence tasks. (a) LouisKV achieves accuracy comparable to FullCache (the lossless baseline) and superior to state-of-the-art retrieval methods.
(b) LouisKV significantly reduces inference latency, substantially outperform- ing state-of-the-a art retrieval methods, while also avoiding the Out-Of-Memory errors that FullCache faces in Long-sequence and large batch size scenarios.
sequence, it leverages dynamic boundaries to partition the KV cache into temporal segments. By creating retrieval units that better match attention distributions. LouisKV enables precise identifi- cation and transfer of critical KV entries, achieving higher accuracy and lower transfer overheads than page-level methods. Furthermore, kernel-level optimizations, including custom Triton and CUDA kernels to accelerate KV clustering and retrieval. As validated by extensive experiments summarized in Figure [ LouisKV maintains near-lossless accuracy on various long-sequence tasks while achieving up to a 4.7x speedup in end-to-end latency compared to state-of-the-an retrieval methods.
boost system efficiency, LouisKV incorporates several
Our contributions are as follows: (1) We conduct an empirical analysis of critical KV access pat- terns, revealing two key observations: strong temporal locality and distinct distribution patterns. (2) Based on these insights, we propose LouisKV, a novel KV retrieval framework that incorpo- rates semantic-aware retrieval strategy and a decoupled management scheme. To our knowl- edge, LouisKV is the first KV retrieval framework designed to comprehensively cover diverse long- sequence tasks. (3) We extensively evaluate LouisKV on various LLM benchmarks, demonstrating that it achieves 4.7x speedup in end-to-end latency while maintaining near-lossless accuracy, as shown in Figure 2
2 RELATED WORK
Scaling LLMs to Long Contexts and Generations. The capability of LLMs to process long- sequence tasks is rapidly advancing. On the one hand, modern models have supported vastly longer inputs. For instance, Qwen3 (Yang et al. 2025) can handle contexts of 128K tokens, while Gemini (Team et al. 2023) models push the boundary to 2M tokens. On the other hand, a parallel trend involves generating longer outputs. Driven by test-time scaling laws, prompting models to generate more extensive, step-by-step reasoning at inference has proven to be cost-effective method to boost model performance (Guo al. 2025 OpenAl 2024 Yang al.] al.] (205). Our work, Louis V, targets KV cache optimization in both long-input and long-output scenarios.
KV Cache Dropping. To mitigate GPU memory pressure, several methods (Zhang et al. 2023 Hu et al. 2025) Chen et al., 2205)] Cai et al. 2026) employ strategies that permanently discard KV entries based on attention scores heuristics rules. However, these approaches lead to irreversible information loss, as discarded KV entries cannot be recalled for subsequent decoding steps. Even more advanced techniques like Scope (Wu al. 2025), which attempt preserve context by retain- ing compressed prefill tokens, still fundamentally rely on eviction mechanisms that risk discarding critical data. Consequently, while these methods are computationally efficient, they suffer from significant accuracy degradation in long-output reasoning tasks.
KV Cache Retrieval and Optimization. To avoid the irreversible information loss inherent in KV dropping methods, KV retrieval approaches preserve the full KV cache in CPU memory and dy- namically fetch a critical subset for computation. However, existing methods still suffer from severe efficiency and accuracy bottlenecks. First, as illustrated in Figure triggering retrieval operations for every single decoding token introduces prohibitive computational and data transfer overheads, particularly in long-oun outguutpuut scenarios. Second, traditional coarse- graseeegrnedddde management (c.g., page- level) results in the transfer of numerous non-critical tokens, which exacerbates bandwidth waste and can even degrade model accuracy (a detailed bottleneck analysis is provided in Appendix A).
3

---

Published as a conference paper at ICLR 2026
(a) Long Document Question Answer (QA) Sample
Background Information... At the university's graduste program, located an academic setting dedicated festering advanced research capabilities.. Most students jursuing high degrees olten find themseelves at the top these rankings... Group Als composed
Prompti
outstanding have consistently demonstrated both outstanding students and those
contributions
Chiiput
Okay, tackle conclude proportion of high-degree
barkground information... Group
remaining students.
Segmentti
Group
Group A.
Crifical KV Sparse In Long Input Sequence
Similarity
1.0-
0.8
Curron Segere
0.6
0.4
10
15
20
25
30
Token Index Current Soyment
Promut and Previnus Outpot
(b) Mathematical Reasoning Sample
Prompti
Convert the point (0,3) rectangular coordirates polar coordinates. Eriter answer in the form Str.Wthetaa...
Previons Outputi <think/s Okay, so need to convert the rectangular point V/0,31V to polar coordinate representationnlfrr ltheta/))... First, 3*2) (sert(0 9) (ogrt)9)
calculate VGl ...
angle Witheta))... position
corresonnts in rotation desrees whish equivelent Mumiiii redians
Current Segmenti
Therefore, the polar coordinates should be n/2).
Critical KV Dense In Long Output Sequence
Shaulority
1.0
Currout Segaeent
Token Index in Chrrreor Segmeet
Prompt and Previous Outpo
Figure 3: Access patterns critical KVs different long-sequence tasks. First, both tasks demon- strate temporal locality: during the generation of a coherent segment (Current Segment), the similar- ity of critical KV sets maintains high values. Second, they reveal a distinct spatial distribution: (a) in long- -occmeennnnt task, attention is sparsely distributed in the prompt, whereas (b) in mathematical reasoning task, attention is densely focused on some intermediate steps in the previous output.
While recent works such as ClusterKV (Liu et al. 2024b), Squeezed Attention (Hooper al. 2025 and SentenceKV V (Zhu et al.) 2025) attempt to improve retrieval granularity via semantic clustering or syntactic boundaries, they predominantly focus on static input sequences, neglecting the dynamic attention patterns inherent in long-output generation. In contrast, LouisKV addresses these funda- mental limitations through two targeted innovations: an adaptive semantic-aware retrieval strategy that mitigates per-token overhead by exploiting temporal locality, and decoupled fine fine-grained man- agement scheme tailored for both input sparsity and output density.
MOTIVATION
3.1 PRELIMINARIES
The rego-segressive inference of LLMs involves two phases: prefill and decode. During prefilling, the model processes the entire input prompt to generate initial KV cache for each layer, denoted Kinp E Pxd and Vinp E Pxd where P is the prompt length and d is the hidden dimension. Dur- ing decoding, the model generates tokens auto-regressively until encountering token. At decoding step t, the model generates a query vector E 1xd for each layer. Simultanc- ously, the corresponding key k; and value U, 1gd are computed and appended the historical KV cache, forming updated K, and V. The output og is then computed via the attention mechanism:
end-of-sequence
1 K? Vd
ot A.V. Softmax
V
Here, A, represents the attention weights indicating the importance of each historical KV entry to the current query qt-
KV retrieval methods usually select subset of KV entries under fixed budget B. Quest Tang et al. 2024) and Arkvale (Chen et al. 2024a) partition the entire key cache (kg, ,,...., kp4+) into fixed-size pages and construct index C₁ for each page i. At every decoding step they select the most critical pages by computing the similarity between the current query and all indices (Co...Cm-1). All KV entries within selected page retrieved for the attention computation. The

---

Published as a conference paper at ICLR 2026
selection process can be formalized as I, Sel(q,. K2), where I₃ is the set of KV indices selected in step t, with the constraint [Zs] I B. Ultimately, only the selected KV subset (Kz, Vz,) participates in the attention computation, significantly reducing the inference latency and memory footprint.
3.2 OBSERVATIONS
We conducted an in-depth analysis of critical KV access patterns on long-sequence tasks (Jia 2025 Ling al. 2025). leading to two key observations that directly motivate our design. These observa- tions provide insights into two fundamental questions: 1) When should retrieval be triggered, and 2) How should the KV cache be managed to enable precise and efficient retrieval?
Observation I: Critical KVs access exhibits strong temporal locality during decoding. The model tends to produce its output as a sequence of semantically coherent segments. For instance, one reasoning step in mathematical problem forms segment, illustrated in Figure Bb). We hypothesize that within each segment, the model continuously attends to a relatively similar subset of tokens to maintain semantic coherence.
To validate this hypothesis, we calculated the Jaccard similarity between the critical KV index sets, I, and I,+1, of adjacent decoding tokens within the same segment. The results provide clear idence for our hypothesis. As shown in Figure Fa) and (b), the similarity curves (bottom left) indicate that the Jaccard similarity remains consistently above 0.8 within Current Segment for both document QA and mathematical reasoning examples. This observation is also visually confirmed by the attention heatmaps (bottom right of Figure 3() and (b)). The bright, vertical bands in the heatmaps indicate that attention is persistently focused on the same regions of the prompt and previ- ous output across the iterations of the current segment's 's generation. The observed temporal locality renders per-token KV retrieval unnecessary, motivating semantic- strategy that amortizes the retrieval overhead across multiple tokens.
Observation 2: Critical KVs exhibit distinet distribution patterns across the input and output sequences. While temporal locality informs when to retrieve, the spatial distribution of critical K VVs informs how to retrieve critical KV entries both efficiently and precisely.
As exemplified by the long-document QA task in Figure Ba), we observe that critical KVs in the long input sequence are distributed sparsely. Specifically, the attention heatmap reveals that high- attention regions are sparsely dispersed across the long input sequence. This shows that the model must extract scattered information from distant parts of the context to generate the answer. In con- trast, critical KVs in a long output sequence are often concentrated densely. As shown in Figure ()). for tasks like mathematical reasoning, the model's attention focuses densely on the previously generated segments, such as intermediate lemmas. As discussed in Section existing KV retrieval methods typically employ fixed-size pages as the fundamental unit for management and selection. However, this approach ignores the distinct spatial distribution of critical KVs, leading to the re- trieved pages containing numerous non-critical KV entries. This distinct distribution pattern poses a fundamental challenge for conventional page-level management mechanisms. This strongly moti- vates the design of finer-grained KV management mechanism capable of adapting to this distinct distribution.
4 LOUISKV SYSTEM
4.1 SEMANTIC-AWARD ADAPTIVE RETRIEVAL
As established in Observation 1, the model tends to focus a highly similar KV subset across multiple consecutive tokens during decoding, rendering per-token retrieval redundant and inefficient. To leverage this temporal locality, LouisKV introduces semantic-awan adaptive retrieval strategy that operates at the granularity of temporal segments, thereby amortizing the retrieval cost over multiple decoding tokens.
The central challenge is to identify the boundaries of these segments accurately with low overhead. A naive approach is to compare the critical KV index sets of adjacent tokens. However, this approach incurs substantial computational costs, as the Sel function itself involves dense matrix computations.
LouisK therefore employs lightweight alternative (Xu et al. 2024 Liu et al. 2025). Specifically,
5

---

Published as a conference paper at ICLR 2026
Hidden
Prefill
Decode KV
Critical KV Current Token Ci Current Query and KV Vector
QKV Proj.
GPU
Calcoilate
GPU
Prefill Stage
Decode Stage
Calculate
GPU
Cricall RVs autfer
locoll Buffer
Chassring
E-Moos
Officeal Cheswe
Olker Seg
2
$ r
Agpond
Retrieval
Buffer
Centroide
Selvet Critical Kiy
Transfor Citiical AVS
Attn FFN
Controii
60
Controld
CPU
KV Cache Pool
CPU
KV Cache Pool
Hidden
(a) Semantic-Aware KV Retrieval
(b) Decoupled KV Management
Figure 4: The design LouisKV. (a) Retrieval triggered when the query similarity r drops below threshold т, loading critical KV entries from the CPU cache pool. (b) During prefilling, K-means clustering is employed to group semantically similar KVs into clusters. During decoding, consecu- tively generated KVs partitioned into temporal segments. These clusters and segments are then offloaded to unified cache pool on the CPU. The detailed algorithm is provided in Appendix C
for each transformer layer each decoding step LouisK computes the cosine similarity between the current query vector dr and the previous one - averaged across all attention heads:
Tt
where H is the total number of attention heads and q.¹ is the query vector for head f at step t. As shown in Figure a)), when this similarity score drops below a predefined threshold т, LouisKV identifies a semantic boundary and triggers KV retrieval operation. This operation selects the critical units that scored the highest by computing the similarity between the current query 4e and the centroids C' of all units, and subsequently loads the corresponding KV cache from CPU to GPU.
4.2 DECOUPLED FINE-GRAINED MANAGEMENT
After determining when to trigger the retrieval, the subsequent challenge how to manage the KV cache to improve retrieval precision. According to Observation 2, traditional coarse-grained page- level management failed to efficiently adapt to the distinct distribution patterns across the input and output sequences. LouisKV therefore designs a decoupled fine-grained management scheme (Figure 4(b) that tailors retrieval units separately for each phase.
Prefill Stage. To manage the sparse critical KVs in the input sequence, LouisKV introduces gratturrraaularrty strategy. Since the attention score for query vector 9e is primarily determined by its dot product with each key vector, key vectors that similar in semantic space tend exhibit similar attention scores (Liu et al., (2024b). Accordingly, LouisKV employs the k-means clustering algorithm to group similar key vectors into the same cluster. Subsequently, LouisKV calculates the centroid O; for each cluster by averaging all its key vectors. Unlike traditional page-level strategies, the key vectors within the same cluster are semantically similar, allowing this centroid a more precise index for subsequent retrieval. This KV clustering and offloading process occurs asynchronously with the prefill forward pass, avoiding blocking the forward computation.
as
Decode Stage. To handle the highly dense attention pattern in the output sequence, LouisK lever- ages the semantic boundaries identified in Section 4.1 to partition the generated KV cache into multiple temporal segments. Similar to the semantic clusters from the prefill stage, each temporal segment is represented by an index vector derived from averaging its key vectors. To prevent ex- cessive GPU memory consumption during decoding, LouisKV maintains a fixed-size local buffer on the GPU to cache the most recently generated KV segments. When this buffer becomes full, the oldest segment will be offloaded to CPU memory. These offloaded temporal segments from the decode stage, together with the semantic clusters from the prefill stage, form a unified KV cache pool in CPU memory.

---

Published as a conference paper at ICLR 2026
LouisV
FulCache @ Clusteryy
Z Arkoole
Quest RoeS M
H20
LouiKV
FulCCacee E Clustency Adisll || Quest Raas H20
100+
45
Accunscy (%)
Score
30
60
15
20
NarrtteeDA
Musique GouReport
Qosper MuBiieenDA HolpotQA
MATH500
GPOA
AIME24
LR-16K
LR-32K
LR. LR-64K
60 Lagginpt Undestanding Tasks.
000
Lint Ressoring Tacks
Figure 5: Performance comparison of V against six baseline methods under the fixed KV cache budget. Subplot (a) presents the results on six long-input understanding tasks using Llama- 3. -nsttuc-nttruct. Subplot (b) presents the results on six long-output reasoning tasks using Qwen3-8B.
H20
Quest -in
Arkuale
RaaS
LauiskV ... FullCache
11500
25
30
F1 5core 20
50
50
Scame
50
20
30
15
128
256
512
1024
123
256
512
1024
128
256
512
1024
128
256
512
1024
KV Cache Budget
KV Cache Budkgtt
KV Cache Budget
KV Cache Budget HotpotQA
NarrativeQAA
Rouge Score
Qasper
MultiiledQA
5.cor
90
32
Accuracy
0.8
0.6
20
0.6
0.4
10
24
0.4
0.2
128
256
512
1024
128
256
512
1024
512
1024
2013
4096
512
2048 KV Cache Budget
1024
4096
KV Cache Budget
KV Cache Budget
KV Cache Budget
Musique
(D) GovRepart
MATHE0
(1) GPQA
0.8
0.8
0.8
0.8
Accurosy
0.6
0.6
80.6
0.6
0.4
0.4
0.4
0.2
0.2
512
1024
2048
4096
512
1024
2048
4096
512
1024
2043
4096
512
1024
2048 4096
KV Cache Busget
KV Cache Budget
KV Cache Budget
KV Cache Budget
6) AIME
① LangRueann66KK
(6) LamsRRsoonS5KK
() LongResenn 16K
Figure 6: Performance comparison across various cache budgets. Subplots (a-f) present the results on six long-input understanding tasks. Subplots (g-l) present the results on six long-output reasoning tasks. For detailed experimental results on more models, please refer to Appendix D.I
4.3 SYSTEM OPTIMIZATIONS
To enhance overall system efficiency, LouisKV incorporates several system-level and kernel-level optimizations. First, we implement custom Triton (Tillet al. 2019) kernel to efficiently execute the clustering operation during prefilling. Second, we introduce group- consisten selection strategy that selects unified set of KV clusters and segments for an entire query group, which efficiently adapts to the Grouped-Quer (see Appendix [ for details). Finally, since these clusters or segments contain variable number of KV entries, selecting critical items under a fixed budget in standard frameworks like PyTorch is complex and inefficient. To address this, we design a highly optimized CUDA kernel that supports rapid, batched selection of critical KVs under given budget. Furthermore, to facilitate efficient transfer of selected KV entries between CPU and GPU, we use the DGL. library (Wang) 2019) to directly transfer specific rows from CPU tensors to the GPU device, rather than first gathering the rows on the CPU and then transferring them.
Attention (Ainslic et al. (023) by avoiding redundant data transfers
EVALUATION
5.1 EXPERIMENTAL SETUP
Datasets and Models. To comprehensively evaluate LouisKV, we select the following datasets: (1) Long-Input, Short-Ouiput. This type of task primarily involves document understanding. Thus, we select six representative and popular datasets from the LongBench (Bai et al., (2023) benchmark To further stress-test our system's performance on ultra-long sequences, we employ the RULER (Hsieh et al. 2024) benchmark with context lengths ranging from 32K to 128K. (2) Short-Input, Long-Oulput. We chose three challenging mathematical and scientific reasoning datasets, sampling
50 problems from each: MATH500 (Hendrycks et al. 2021 and AIME (Jia 2025), which require
'The selected datasets NarrativeQA, Qasper, MultiFieldQA, HotpotQA, Musique, and GovReport.
7

---

Published as a conference paper at ICLR 2026
multi-step mathematical problem-solving, and GPQA (Rein et al.
level questions demanding complex reasoning. (3) Long-Input 2024), which comprises graduate- We choose the Lon- gReason (Ling al., 2025) dataset. This dataset synthetically extends short-context reasoning prob- lems into complex versions requiring long-range dependency, and we sample 100 problems from each of its 16K, 32K, and 64K input-length subsets, denoted LongReason-16ks LongReason- 32K, and LongReason-6443 respectively. Our evaluation is conducted on two models: Qwen3-8B (Grattafiori al., 2024). We leverage the thinking mode of SwBn3-8B to handle long-output tasks requiring complex reasoning, while Llama-3.1-8B. Instruct addresses traditional long-input, short-output tasks.
Yang et al. 2025) and 1attaa-..--B-II-Sse
Baselines. We compare LouisKV against state-of ahe KV cache optimization methods, including KV dropping methods such as H20 (Zhang et al. 2023) and RaaS (Hu et al. 2025), KV retrieval methods such as Quest (Tang et al. 2024), Arkvale (Chen et al. 2024a) and ClusterKV V (Liu et al., 2024b). We set a uniform KV cache budget B for all methods. Following the setting of Quest, we preserve the complete KV cache for the first two layers and retain the sink tokens S and local buffer W on the GPU across all methods, where the values of and W are adjusted according to the specific task. To maintain fair comparison, and Arkvale to be group-consistent. For -ethod-specific parameters, the page size is set to 16 for Quest, Arkvale, and RaaS while the average cluster size is set 16 for LouisKV and ClusterKV on the input sequence. Our proposed I LouisKV employs varying similarity thresholds T for the output sequence, adapted to the specific model. We determine its optimal value via a one-time calibration process (detailed in Section 5.4) to ensure the best trade-off between efficiency and accuracy.
adapt the implementations of RaaS, Quest,
Environment. Our server is equipped with a single NVIDIA A6000 GPU (48 GB) and an AMD Ryzen 9 5950X 16-Core Processor. The CPU and GPU are interconnected via PCLe 4.0 x16 (32GB/s). All experiments run on Ubuntu 20.04 with Linux kemel 5.4.0 and CUDA 12.4.
5.2 ACCURACY EVALUATION
Long-Input Understanding Tasks. We set the system parameters to S 32, W 512, and T = 0.85. For a comprehensive analysis, Figure details the performance on individual datasets under specific budget of 512, whereas Figure a-f) presents the score across varying budgets. The results show that LouisKV consistently surpasses other page-level retrieval methods and dropping methods across nearly all long-input tasks. Specifically, with a budget of 512, LouisKV improves the average score by 1.1% over Arkvale and 3.3% over Quest. Although LouisKV is marginally lower than the cluster-level method, ClusterKV (a difference of approximately 0.4%), this slight gap represents a strategic trade-off in our system design. While both methods employ clustering for the input sequences, LouisKV distinguishes itself by incorporating a segment- retrieval mechanism that triggers retrieval only at semantic boundaries. This design choice yields significant enhancement in inference efficiency for a negligible cost in accuracy. In summary, the superior performance of LouisKV validates the effectiveness of our clustering management designed for the input sequence. Unlike coarse-grained page-level management, our approach more precisely identifies and retains the most critical KVs, leading higher accuracy under limited budgets.
Long-Output Reasoning Tasks. For long-output tasks that demand complex reasoning. we apply different parameter configurations. In short-input, long-output tasks, we set S 500, W 128. and = 0.7. In long-input, long-output tasks, we set S = 64, W = 256, and T = 0.7. The maximum generation length is 16K tokens, extended 32K for the AIME benchmark. Figure 6 g-I) show that LouisKV achieves the best or second- best accuracy across various budgets Meanwhile, Figure 5lb) highlights the performance on individual benchmarks, revealing that KV dropping methods suffer from severe accuracy degradation, particularly on challenging tasks like AIME and LengReason. Under a budget of 1024, LouisKV surpasses the page-level methods Arkvale and Quest by 2.0% and 4.8% in average accuracy, respectively, and outperforms the cluster-level method ClusterKV by 3.8%. These results conclusively validate the effectiveness of our decoupled KV management scheme, demonstrating its capability to precisely preserve critical information across diverse input and output distributions.
Robustness RULER Benchmark. To evaluate LouisKV's robustness in extreme long-context scenarios, we evaluate performance across 13 datasets from the RULER suite. As shown in Table of the Appendix D.I LouisKV demonstrates strong performance from 32K to 128K context lengths
8

---

Published as a conference paper at ICLR 2026
Arkvale
FullCache
LouiskV
Latency (s)
120
Latency (s)
2000
Latenoy
4000
80
200
1000
2000
DOMM
0
Batch Size
Batch Size
Batch Size
(a) Long-lupat, Shart-Oulput.
Short-Lnpot, Long-Oulput.
Long-Input Lony-Oulpat
Figure 7: End-to-end latency comparison under different batch sizes and long-sequence scenarios.
Selection
Transfer
Attention
FFN
Input Length
2K
4K
8K
16K
32K
64K
Adkuata
FullCache TTFT (8)
8.61
FulKache
LouisKY TTFT (8)
0.37
0.76
1.60
3,52
8.85
23.49
K-Means Latency (9) 0.06
0.11
0.16
0.27
0.78
2.34
Lodskv
0.0
0.5
10
1.5
0.0
0.5
1.0
1.5
2.0
Lalency (ms)
Latency (ms)
Table I: Comparison of TTFT between FullCache
Figure 8: Per-layer latency breakdown during the decoding phase. Left: short-input, long- output. Right: long-input, long-output.
and LouisKV, and the standalone K-Means latency across different input lengths.
while setting the KV budget to 2K tokens. At 32K and 64K, its accuracy is comparable to the lossless FullCache baseline and surpasses Arkvale by 2.6% and 4.7%, respectively. At 128K, FullCache hardware, highlighting the fundamental necessity of retrieval-beie based
fails with OOM error methods for such extreme contexts. contrast, LouisKV completes the tasks successfully and consistently outperforms Arkvale across all tested lengths. This performance gap underscores the effectiveness of our design accurately identifying and retrieving the most critical KV entries, even in ultra-long sequences.
5.3 EFFICIENCY EVALUATION
We evaluate the inference efficiency of VouisKV in comparison with FullCache and Arkvale. Specif- ically, the FullCache baseline, which retains the entire KV cache on GPU, is implemented using FlashAttention-2 (Dao 2023). The evaluation covers three long-sequence patterns: long-input, short-output (32K input, 512 output); short-input, long-output (500 input, 16K output); and long- input, long-output (32K input, 16K output). Following the experimental setup from the Section 5.2 we use Llama- -------nIstrtttt for long-input scenarios and Qwen3-8B for long-output scenarios.
End-to-End Latency. As illustrated in Figure 7 compared to Arkvale, LouisKV achieves up to 1.9x, 2.9x and 4.7x speedups across the three patterns, respectively. This efficiency gain stems from semantic- retrieval strategy, which significantly reduces computation and data trans- fer overhead, and highly optimized kernels that enhance the efficiency of importance estimation and data movement. Specifically, by triggering retrieval only at semantic segment boundaries, LouisKV drastically reduces the retrieval frequency (detailed analyses regarding segment lengths and per-layer boundaries are provided in Appendix Appendix[D.Sjande [Dd]and D.6). This architectural advantage becomes more pro- nounced as the batch size increases, because larger batches exacerbate the data transfer bottleneck. LouisKV exhibits comparable latency compared with FullCache. When processing long sequences with larger batches, FullCache encounters OOM errors due to its significant memory footprint. In contrast, L offloads the entire KV cache CPU memory, enabling support for larger batch sizes, thereby achieving higher system throughput.
Prefill Latency. We evaluate the Time to First Token (TTFT) of LouisKV compared to FullCache across input lengths ranging from 2K to 64K. As detailed in Table LouisK V exhibits performance comparable to FullCache, introducing only a marginal average latency increase approximately 3.5%. This negligible overhead attributes to Triton-optimized K-means kernel and the asyn- chronous execution pipeline. To quantify this overhead, we measure the standalone latency of the clustering process. While the clustering cost scales with input length, the asynchronous pipeline cessfully overlaps this computation with the inference process. Consequently, LouisKV maintains high prefill efficiency without imposing a significant bottleneck.

---

Published as a conference paper at ICLR 2026
Noor Accuracy 1%6
2
Fixed
Adaptive
Nom. Accuracy (%)
Coarse-grained
Fine-grained
80
M 1%
8 10
Laiency (ms)
60
Accureey
Latency
2000
40
20
20
8
50
1000
0
40
LS
SL
LS
5L
LL
0.5 0.6 0.7 0.8 Threshodd
No Opt. +5R
+GS
+CK
(A) Fludd vs. AdaptiveSStitt (b) Couse-gralnedd Vs. Fine-grained
(c) SinilarftyThreshldd
(c) Elfidenc Optimizations
Figure 9: Ablation studies. (a) Accuracy comparison of our adaptive stride retrieval against fixed stride baseline. (b) Accuracy comparison of our decoupled, fine-grained management against coarse-gra grasseegg baseline. (c) Impact of the similarity threshold 7 on accuracy and latency. (d) Effi ciency contribution of individual system optimizations.
Decode Latency Breakdown. We evaluate the latency breakdown for single transformer layer during decoding. As shown in Figure although Arkvale retrieves only a subset of the KV cache, the overhead associated with retrieval, including both selection and transfer, still accounts for 65% of the total latency, highlighting the necessity to reduce this overhead. LouisK drastically reduces this overhead, lowering its contribution to just 11 1% of the total latency. We also provide the analyses of memory efficiency in Appendix D.2
5.4 ABLATION STUDY
Effectiveness of Core Strategies. We independently validate the effectiveness of the two strategies in LouisKV. First, we evaluate the adaptive retrieval against a baseline that employs a fixed-stride retrieval. The results in Figure(Ja) demonstrate that our adaptive approach consistently achieves higher accuracy across all tasks, with gains of up to 8% in long-output reasoning tasks. This indicates that dynamically triggering retrieval semantic boundaries captures critical depen- dencies more effectively than rigid, fixed-stride retrieval. Furthermore, we assess the effectiveness of our decoupled, fine ane-ggriied management scheme by comparing to a coarse- gaarseeg page-level baseline. To strictly isolate the impact of memory management granularity from the retrieval policy, we configure both methods to use an identical per-token retrieval setting. As shown in Figure (h)), our management scheme consistently outperforms the page-level baseline, achieving improvements of up to 6.3%. A detailed analysis of this ablation study is provided Appendix D..
Impact of Similarity Threshold т. The similarity threshold serves as key hyperparameter that governs the trade-off between accuracy and efficiency. As shown Figure Cj), higher T leads to more frequent retrievals, thus improving inference accuracy. However, this comes at the cost of higher latency due to the overhead of retrieval. The results show that for Qwen3-8B, T = 0.7 represents an ideal balance, achieving significant efficiency gains while maximally preserving inference accuracy. Notably, we calibrate 7 as a one-time, model-specific parameter using a small validation set (e.g. 300 examples). Once determined, this optimal value generalizes to all long- context tasks, eliminating the need for per-task tuning. Further details are in Appendix D.4
Impact of System Optimizations. Finally, we quantify the efficiency contribution of the system optimizations in LouisKV, testing on the Qwen3-8B model in long-input, long-output scenarios. As shown in Figure d), we incrementally apply each optimization on top of baseline system. Semantic- Aware Retricval (SR) contributes the largest performance improvement, achieving a nearly 2.6x speedup by drastically reducing the overhead from redundant retrieval and data loading oper- ations. Building on this, Group-Consistee Selection (GS) and our Custom Retrieval Kernel (CK) provide additional performance gains of 13.1% 1.11 % and 15.7%, respectively.
CONCLUSION
In this paper, we propose LouisKV, an effective KV retrieval framework designed for various long- sequence scenarios. Our work is motivated by two key observations: the strong temporal locality of critical KVs, and their distinct distribution patterns across input and output sequences. Building on these insights, LouisKV introduces semantic-an aware adaptive retrieval strategy and decoupled management scheme. Evaluation results show that LouisKV achieves up to a 7 x speedup com- pared to state-of-the-art KV retrieval methods while maintaining near-lossless inference accuracy.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGMENTS
This work is mainly supported by the National Key Research and Development Program of China under Grant No. 2023YFB4502702, the Natural Science Foundation of China under Grant No. 62472007, 62332021 and 62306056.
ETHICS STATEMENT
Our research is conducted full compliance with the ICLR Code of Ethics. We are committed to upholding the highest standards of academic integrity and have addressed key ethical considerations as follows:
We have upheld research integrity by ensuring all data, methods, and results presented truthfully and without misrepresentation. Our methodology is described with sufficient detail to support independent verification and reproducibility.
Research Integrity:
The datasets utilized in our experiments are publicly available and were handled in strict accordance with their respective licenses and usage terms. Our study did not involve human subjects or personally identifiable information.
Data and Privacy:
Attribution and Contribution: Proper attribution has been given all prior work through comprehensive citations. All authors listed have made significant intellectual contributions this research and have collectively approved this submission.
Societal Impact: We have considered the potential societal impacts of our work. While the intended applications are beneficial, we acknowledge the possibility of misuse and advocate for the responsible development and deployment of our findings.
REPRODUCIBILITY STATEMENT
To ensure the reproducibility of our results, we have provided detailed descriptions of our exper- imental setup, hyperparameters, and implementation details. We plan make the source code publicly available upon publication to facilitate verification and future research.
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Ale- man, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, al. Gpt-4 technical report. arXiv preprint trXiv:2303.08777 2023.
Joshua Ainslie, James Lee- Thorp, Michiel De Jong, Yury Zemlyanskiy, Federico Lebrón, and Sumit Sanghai. Gqa: Training generalized multi- query transformer models from multi-head check- points. arXiv preprint arXiv:2205.1322 2023.
Yushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu, Jiankai Tang, Zhidian Huang, Zhengxiao Du, Xiao Liu, Aohan Zeng, Lei Hou, et al. Longbench: A bilingual, multitask benchmark for long context understanding. arXiv preprint arXiv:2308. 14508. 2023.
Zefan Cai, Wen Xiao, Hanshi Sun, Cheng Luo, Yikai Zhang, Ke Wan, Yucheng Li, Yeyang Zhou, Li-Wen Chang, Jiuxiang Gu, Zhen Dong, Anima Anandkumar, Abedelkadir Asi, and Junjie Hu.
R-kv; Redundancc--wware kv cache compression for reasoning models, 2026. URL 7
https:
24133
Guoxuan Chen, Han Shi. Jiawei Li, Yihang Gao, Xiaozhe Ren, Yimeng Chen, Xin Jiang. Zhenguo Li, Weiyang Liu, and Chao Huang. Seplim: Accelerate large language models by compressing one segment into one separator, 2025. URL /RRt:ttt::///ar
wtg/abs////eooooooooo..oooeceere 2412.11222.1 12094
Renze Chen, Zhuofeng Wang. Beiquan Cao, Tong Wu, Size Zheng, Xiuhong Li, Xuechao Wei, Shengen Yan, Meng Li, and Yun Liang. Arkvale: Efficient generative Ilm inference with recallable key-value eviction. Advances in Neural Information Processing Systems, 37:113134-11313 2024a.
11

---

Published as a conference paper at ICLR 2026
Zhuoming Chen, Ranajoy Sadhukhan, Zihao Ye, Yang Jianyu Zhang, Niklas Nolte, Yuandong Tian, Matthijs Douze, Leon Bottou, Zhihao Jia, et al. Magicpig: Lsh sampling for efficient Ilm generation. arXiv preprint 21X1172410.1619, 2024b.
Tri Dao. Flashattention-2: Faster attention with better parallelism and work partitioning. arXiv preprinl arXii:2307.088///3 2023.
Yuan Feng, Junlin Lv, Yukun Cao, Xike Xie, and Kevin Zhou. Ada-kv: Optimizing kv cache evic- tion by adaptive budget allocation for efficient Ilm inference. arXiv preprint urXiv:2407.11550 2024.
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The Ilama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024.
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12944 2025.
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. arXiv preprint arXiv:2103.03877 2021.
Coleman Hooper, Sehoon Kim, Hiva Mohammadzadeh. Monishwaran Maheswaran, Sebastian Zhao, June Paik, Michael W. Mahoney, Kurt Keutzer, and Amir Gholami. Squeezed attention: Ac- celerating long context length Ilm inference, 2025. URL htt.t:tt////t http://aaxvvvv.o / 09688
Cheng-Ping Hsieh, Simeng Sun, Samuel Kriman, Shantanu Acharya, Dima Rekesh, Fei Jia, Yang Zhang, and Boris Ginsburg. Ruler: What's 's the real context size of your long-context language models?, 2024. URLps:::://axxi org. 2404. 06654
Junhao Hu, Wenrui Huang, Weidong Wang, Zhenwen Li, Tiancheng Hu, Zhixia Liu, Xusheng Chen, Tao Xie, and Yizhou Shan. Raas: Reasoning-aware attention sparsity for efficient I1m reasoning. arXiv preprint a7Xiv 2025.
Maxwell Jia. htttps://huggiii
2024 February 2025.
Wonbeom Lee, Jungi Lee, Junghwan Seo, and Jaewoong Sim. (InfiniGen):: Efficient generative inference of large language models with dynamic [KV} cache management. In 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24), pp. 155-172, 2024.
Zhan Ling, Kang Liu, Kai Yan, Yifan Yang, Weijian Lin, Ting-Hau Fan, Lingfeng Shen, Zhengyin Du, and Jiccao Chen. Longreason: A synthetic long-conteds reasoning benchmark via context expansion. arXiv preprint arXiv:2501.15033 2025.
Di Liu, Meng Chen, Baotong Lu, Huiqiang Jiang, Zhenhua Han, Qianxi Zhang, Qi Chen, Chen- gruidong Zhang, Bailu Ding, Kai Zhang, al. Retrievalattention:: Accelerating long-context Ilm inference via vector retrieval. arXiv preprint arXiv:2409. 105/6, 2024a.
Guangda Liu, Chengwei Li, Jieru Zhao, Chenqi Zhang, and Minyi Guo. Clusterkv: Manipulating Ilm kv cache in semantic space for recallable compression. arXiv preprint arXiv:2412.03213, 2024b.
Guangda Liu, Chengwei Li, Zhenyu Ning, Jing Lin, Yiwu Yao, Damning Kc, Minyi Guo, and Jieru Zhao. Freekv: Boosting ky cache retrieval for efficient Ilm inference. arXiv preprint 13X1v 2025.
OpenAl. ol system card, 2024.
Xiurui Pan, Endian Li, Qiao Li. Shengwen Liang, Yizhou Shan, Ke Zhou, Yingwei Luo, Xiaolin Wang, and Jie Zhang. Instinfer: In-storage attention offloading for cost-effective long-context 11m inference. arXiv preprint 24X4422409.04992, 2024.
12

---

Published as a conference paper at ICLR 2026
David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Di- rani, Julian Michael, and Samuel Bowman. Gpqa: A graduate-level google-prood q&a bench- mark. In First Conference on Language Modeling, 2024.
Luohe Shi, Hongyi Zhang, Yao Yao, Zuchao Li, and Hai Zhao. Keep the cost down: review on methods to optimize I1m's kv-cache consumption. arXiv preprint arXiv:2407. 18003, 2024.
Jiaming Tang, Yilong Zhao, Kan Zhu, Guangxuan Xiao, Baris Kasikci, and Song Han. Quest: Query-aware sparsity for efficient long-context Ilm inference. arXiv preprint arXiv:2406. 10774, 2024.
Gemini Team, Rohan Anil, Sebastian Borgeaud, -ean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew Dai, Anja Hauth, Katie Millican, et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:23/2./18055 2023.
Philippe Tillet, Hsiang- Tsung Kung, and David Cox. Triton:
an intermediate language and compiler for tiled neural network computations. In Proceedings of the 3rd ACM SIGPLAN International Workshop on Machine Learning and Programming Languages, pp. 10-19, 2019.
Minjie Yu Wang. Deep graph library: Towards efficient and scalable deep learning on graphs. In ICLR workshop representation learning graphs and manifolds, 2019.
Jialong Wu, Zhenglin Wang. Linhai Zhang, Yilong Lai, Yulan He, and Deyu Zhou. Scope:
Optimizing key-value cache compression in long-context generation, 2025. URL https:
warxiv.org/abs//// 13649
Chaojun Xiao, Pengle Zhang. Xu Han, Guangxuan Xiao, Yankai Lin, Zhengyan Zhang, Zhiyuan Liu, and Maosong Sun. Inflm: Training- free long-context extrapolation for llms with an efficient context memory. Advances Neural Information Processing Systems, 37:1 19638-119661, 2024.
Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. Efficient streaming language models with attention sinks. arXiv preprint arXiv:2309.174 17453, 2023.
Fangyuan Xu, Tanya Goyal, and Eunsol Choi. Refreshkv: Updating small kv cache during long- form generation. arXiv preprint arXiv:2411.05787 2024.
Jiamiig Xu, Jiayi Pan, Hanzhen Wang. Yongkang Zhou, Jiancai Ye, Yu Wang, and Guohao Dai. Specontext: Enabling efficient long-context reasoning with speculative context sparsity in Ilms, 2025. UtLpttps::/ wtwp///aavvvv... www.vvorrg/ab...... 2252000022
An Yang, Anfeng Li, Baosong Yang. Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report. arXiv preprint MXXi:22005.09388, 2025.
Jingyang Yuan, Huazuo Gao, Damai Dai, Junyu Luo, Liang Zhao, Zhengyan Zhang, Zhenda Xie, YX Wei, Lean Wang, Zhiping Xiao, et al. Native sparse attention: Hardware-aligner and natively trainable sparse attention. arXiv preprint arXiv22502.11089, 2025.
Hailin Zhang, Xiaodong Ji, Yilin Chen, Fangcheng Fu, Xupeng Miao, Xiaonan Nie, Weipeng Chen, and Bin Cui. Pqcache: Product quantization-based kvcache for long context Ilm inference. Pro- ceedings of the ACM on Management of Data, 3(3)1-30,, 2025.
Zhenyu Zhang, Ying Sheng, Tianyi Zhou, Tianlong Chen, Lianmin Zheng, Ruisi Cai, Zhao Song, Yuandong Tian, Christopher Ré, Clark Barrett, al. H2o: Heavy-hitter oracle for efficient gen- erative inference of large language models. Advances in Neural Information Processing Systems, 36:34661-3471D 2023.
Yuxuan Zhu, Ali Falahati, David H. Yang, and Mohammad Mohammadi Amiri. Sentencekv: Ef- ficient Ilm inference via sentence-level semantic kv caching, 2025. URL https: :/appsi:///iii/ org/abs/2504. 2500000744000000 00970
13

---

Published as a conference paper at ICLR 2026
A DETAILED BOTTLENECK ANALYSIS OF KV RETRIEVAL METHODS
OffloadCach Cache
Selection Transfer
Accuarcy
0.8
Arlvvle LousKV FullCache
3 3000
Arkvale
0.7
2000
LousKV
Attention
0.6
1000
0
2
1024
4096
512
1024
4096
Latency (ms)
KV Cache Budget
KV Cache Budget
Figure 10: Bottleneck analysis of KV retrieval methods. Left: Per-layer latency breakdown com- paring LouisKV with other methods. OffloadCache serves as baseline that preloads the full KV cache of the next layer from CPU during inference, while Arkvale a representative state-of-the-art retrieval method. Right: Inference accuracy (middle) and end-to-end latency (right) of LouisK and Arkvale in long-output scenarios under different KV cache budgets.
While existing KV retrieval methods avoid GPU Out-Of-Memory errors by offloading the entire KV cache to CPU memory, they still face two fundamental bottlenecks regarding inference efficiency and retrieval accuracy.
First, the per-token retrieval strategy incurs prohibitive cumulative overhead. As illustrated in Figure 10 (left), although retrieval methods like Arkvale attempt to reduce KV cache transfer costs by loading less data, they introduce an additional computational overhead to select the critical KV cache pages. Meanwhile, due to the limited CPU-GPU bandwidth, the data transfer overhead continues to be primary performance bottleneck. While this overhead might seem minor for a single token generation, it accumulates linearly each decoding token in a long-output sequence requires retrieval operation. This cumulative effect becomes substantial in long-output scenarios, leading to dramatic increase in total inference latency, shown in Figure TO (right) for Arkvale.
Second, the graiin- page-level management scheme leads to suboptimal retrieval perfor- mance, creating dilemma between efficiency and accuracy. These methods partition the sequence into fixed-size pages and use the page as the minimum unit for retrieval, which introduces a difficult trade-off. On the one hand, it results in significant accuracy drop under tight KV cache budgets, as shown in Figure TO (middle). On the other hand, allocating larger budget to prevent this accuracy loss results in significantly higher end-to-end latency, as depicted Figure I0 (right). In contrast, by retrieving the critical KVs precisely, LouisKV maintains high inference accuracy even under a small budget while simultaneously achieving high efficiency.
B DETAILS OF SYSTEM OPTIMIZATION
Group-consistent Selection. Existing retrieval methods are primarily designed for Multi-Head At- tention architectures, where the number of query heads hy equals the number of KV heads htv- However, modern models often adopt query heads share the same KV head. To avoid excessive transfer overhead caused by retrieving dis- tinct KV heads for each query head, we propose group-consistent retrieval strategy. This approach selects a common set of KV indices for all queries within a group, guided by a group-aggregated score AJ, computed by averaging the softmax-normalizer attention scores from each query:
Query Attention, in which group of g = h2/ h
9
(CijT 4t
jeg
1
softmax
Vd
where of jegti the query of the j-th head within group i step f, and C* is the matrix containing the centroid vectors of all clusters and segments belonging to the i-th KV head, the one shared by the i-th query group. Based on this aggregated score, LouisKV ranks all KV clusters and segments and selects the top-scoring units. This strategy ensures that all requests from the group are unified into a single, coalesced data transfer, perfectly aligning with the computational pattern of the GQA architecture and fundamentally eliminating the transfer overhead. Notably, for conventional MHA architectures where the group size = 1, our group- aware strategy naturally simplifies to a standard per-head retrieval mechanism, thus providing unified approach for different attention structures.
14

---

Published as a conference paper at ICLR 2026
C ALGORITHM OVERVIEW
Algorithm The LouisKV Algorithm
1: Input: Model M, Input Sequence Xmpps KV Cache Budget B, Similarity Threshold r, A KV Manager kum
2: Output: Generated Token Sequence O
3: function WWW.STOOR..COO WWW.STT.....CCCC.,,.... CACHI((K, V, stage)
4: if' stage = prefill' then
5:
Kelusters f- KMeans(K)
6: for each cluster c in Kelusters do
7:
centroid &- ComputeCentroidd(c)
8:
Store centroid on GPU;
9:
end for
10:
Offload (K, V) CPU memory pool asynchronously
11:
else stage 'dcodde' then
12:
K KVoaca [ VVoVial (Re, 1x)]
13:
if VVocal is full then
K Keegy Veeg f- the oldest KV segment in K Vtoeat
15:
centroid &- ComputeCCetrooid (K Kegg)
Store centroid on GPU
17:
Offload (K Voeg) to CPU memory pool asynchronously.
end if
19:
end if end function
21:
function KVM. RETRIEEVIE,, B)
22:
Scores +- Compute similarity scores between query qe and all centroids TopInsices + Select the set of top-scoring clusters and segments based on Scores and budget
23:
24:
KVeritiont + Load the corresponding KVs of T Topl ndices from CPU memory pool
25:
return Varitieal
26:
end function
procedure LOUISK Xinp: B, r, kvm)
Phase 1: Prefill Stage
29:
Kinps Vimp <- Process Xinp with model M to get initial KV cache kum.sstore
30:
www.ssettrreeeeeeeeeeee _wwche(ecceeccecc.e Kipp: Vimps "prefill))
31:
O - Sprev f- None;
32:
// Phase 2: Decode Stage
33:
for = ..2.....T7 do
34:
[t. kt, U2 Generate query and KV cache from the previous token's 's state
35:
is nen new_ssemmett - (1 = 1) v (CosineSimilarityye (prev) < r); Spreo f- True then
36:
if is newgeseemeett
37:
VVeritical + kum.retrieve( gt, B) wom..store.cchhe/ee k,, Us, 'dccode') "); KV Vattn [K IKKVrrtte Veriticul K Viocal]
39:
else if is new segment == False then kemmsttoe cache kt, Ut, "decode"); ): KVattn +- (Vrtttie K VVocal
41:
end if
42:
0: f- Model M Generate token using qe and K KVarmi O O.append(o:):
43:
end for
44:
return
45: end procedure
15

---

Published as a conference paper at ICLR 2026
D DETAILED EXPERIMENTAL RESULTS AND ANALYSIS
D.1
ACCURACY EVALUATION
Long-Input, Short-Output Tasks
Methods
NarrativeQA
Qasper
MultifieldQA
HotpotQA
Musique
GovReport
Llama3.1-8 1-rmaa...----- 1-8--nntruct
28.26
44.38
55.27
55.97
31.68
35.02
H2O
20.90
24.50
34.82
31.29
12.70
26.69
RaaS
22.01
30.72
40,31
41.54
15.02
26.74
Quest
25.35
43.65
51.49
51.63
25.09
32,20
Arkvale
27.32
42.59
52.88
55.92
29.07
32.79
LouisKV
27.53
45.31
54.22
55.64
31.20
33.30
Owen2.5-7B-Instter
20.1
39.29
46.66
55.84
25.88
31.4
H2O
7.36
20.04
31.07
30.75
15.29
22.48
RaaS
10.7
25.21
35.90
39.43
15.83
26.2
Quest
15.41
34.56
44.75
45.58
22.51
29.21
Arkvale 17.64
38.78
44.40
53.16
25.81
30.6
LouisKV 19.39
37.67
46.63
52.14
28.51
30.57
Table 2: Accuracy comparison on Long-Input Understanding tasks with a KV cache budget of 512
tokens across methods. The best results for each model family (excluding the baseline) are in bold.
Short-Input, Long- Long-Output Tasks
Long-Input, Long-Output Tasks
Methods
MATH500
GPQA
AIME
LongReason-64k
LongReason-32K
LongReason-164
Qwen3-8B
0.86
0.66
0.70
0.70
0.74
0.76
H2O
0.66
0.24
0.26
0.43
0.53
0.54
RaaS
0.78
0.50
0.43
0.38
0.42
0.52
Quest
0.76
0.60
0.67
0.70
0.73
Arkvale
0.82
0.60
0.63
0.68
0.68
0.75
LomisKV
0.86
0.62
0.66
0.70
0.73
0.74
DeepSeek Diviill --ssiill-wwen-7
0.37
H2O
0.74
0.26
0.23
0.15
0.18
0.28
RaaS
0.82
0.30
0.40
0.17
0.23
0.32
Quest
0.86
0.38
0.43
0.16
0.34
0.34
Arkvale
0.84
0.44
0.53
0.20
0.38
0.38
LouisKV
0.86
0.22
0.39
0.42
Table 3: Accuracy comparison Long-Outpur Reasoning tasks with KV cache budget of 1024 tokens across all methods. The best results for each model family (excluding the baseline) are in bold.
Methods
S1
82
83
MK1
MK2
MK3
MV
MQ
FWE
CWE
QAI
QA2
FullCache (32K)
100
$2.00
Arkenlee
100
99.5
99.5
99.5
99.5
92
94.1
99.75
92.17
31.30
73.5
54.0
99.2
LouisKV
100
100
100
100
99.5
96.12
99.75
92.67
49.85
75.0
FullCache (64K)
100
100
100
99
95.5
97.62
99.75
86.17
12.05
72.0
98.6
Arkvale
100
98.5
99
99.5
96.5
57.5
94.50
99.75
78.17
2.35
71.5
49.0
95.2
LouisKV
100
100
98.5
92
96.88
72.0
96.5
FullCache (128K)
OOM
00M
00M
00M
00M
00M
0OM
0OM
00M
00M
0OM
0OM
0OM
Arkvaale
100
99.5
97.5
73
6.5
72.00
90.62
60.17 17
0.00
68.0
45.0
79.5
LouisKV
100
100
100
79
25
W1.50
96.62
62.33
0.10
69.0
44.0
80.7
Table 4: Accuracy RULER across context lengths from 32K 128K. In the header, S1-S3.
MKI-MK3, MV, and MQ represent the niah single. 1-3, niah multikey 1-3, niah multivalue, and niah multiquery tasks, respectively.
16

---

Published as a conference paper at ICLR 2026
Method
Memory
Parameters
FullCache
4Lhdn + +m)
Quest
4Lhdd (n + m atm
page size: p
P
Arkvale
4Lhdh (B + n+m
page size: p. budget size: B
LouisKV
4Lhdu(B + 2e 2
cluster size: c, segment size: 8, budget size: B
Table Theoretical analysis of GPU KV cache memory footprint for different methods, where the input length and m is the output length.
FullCache
Quest
Arkvale
LouisKV
KV Cache (GB)
40
30
20
10
0
0
8
16
24
32
Decode Token
Figure 11: Comparison of KV cache memory consumption during the decoding phase for different methods, with a prefill length 32K tokens and batch size of 4.
D.2 MEMORY FOOTPRINT ANALYSIS
In this section, we analyze the KV cache memory footprint of different methods. We assume the number layers is L, number of heads h, head dimension is da, the input sequence length is and the output sequence length is m. All KV cache is stored in FP16, leading to a base factor of 4Lhdh As detailed in Table 5 FullCache retains the entire KV cache on the GPU. Quest builds upon this by adding indexing overhead for each page of size p. Both Arkvale and LouisKV are of- floading methods that keep a budget of size on the GPU, but they differ their indexing overhead. Specifically, both Quest and Arkvale create index vectors for each key page, whereas LouisKV requires only one for each semantic cluster or segment. We empirically validate this analysis by comparing memory consumption in long-input scenario (32K prefill tokens), with results shown in Figure ID The memory usage of FullCache and Quest grows linearly with decoded tokens, making them susceptible to mem-oory errors. contrast, Arkvale and LouisKV maintain consistently low memory footprints. As future work, LouisKV's memory consumption could be further reduced by offloading KV cache indices to CPU DRAM.
D.3 EFFECTIVENESS OF CORE STRATEGIES
In this section, conduct ablation studies independently analyze the effectiveness of LouisK Vss two core designs, adaptive semantic-aware retrieval strategy and decoupled fine- fme-grained manage- ment scheme. to increase inference accuracy. We follow the parameter settings in Section 5 testing on long-input understanding tasks with the Llama3.1-88 1-8---nstttstttt model and on long-output reason- ing tasks with the Qwen3-8B model.
Fixed Stride vs. Adaptive Stride. To validate the superiority of our adaptive retrieval strategy, we compare against baseline that employs a fixed-stride retrieval. For a fair comparison, the baseline triggers retrieval every 5 tokens for long-input tasks and every 16 tokens for long-output tasks. These values are chosen closely match the average retrieval frequency of our adaptive strategy on the respective tasks. As shown in Figure 12 a), our adaptive retrieval strategy achieves higher accuracy across the majority tasks. This result strongly indicates that a fixed-stride retrieval method degrade accuracy by failing to retrieve information at critical points. In contrast, adaptive strategy more precisely identifies the critical decoding points where information must be updated, leading to better inference accuracy.
Coarse-grained vs. Fine-grained Management. Next, we evaluate the contribution of our decou- pled, fine-grained management mechanism, comparing it against baseline that applies uniform
17

---

Published as a conference paper at ICLR 2026
Fixed Stride
Adaptive Stride
60
Accuracy
1.0
0.6
Score
40
0.22
00
07
0.5
20
0
0.0
Narrative
Qasper MultifieldQA
HotpotO
Musique GovReport
Math500
GPQA
AIME
LR-64K
LR-32K
LR-16K
Figure 12: Ablation study on retrieval strategy, We compare the inference accuracy of our adaptive semantic-aware retrieval against fixed-stride retrieval baseline. The adaptive strategy achieves higher accuracy across most tasks by retrieving information at critical decoding points.
Coarse-grained
Fine-grained
Accuracy
1.0
Score
40
A72
0.5
20
NarrativeO
Qasper
MultifielOO
HotpotQA
Musique
GovReport
0.0
Math500
GPQA
AIME
LR-64K
LR-32K
LR-16K
Ablation study management scheme. We compare the inference accuracy of decoupled, tne-grained management against coarse-grained page-level baseline. The superior performance of our tine-grained approach validates the necessity of its tailored design: semantic clusters for inputs and temporal segments for outputs.
Figure 13:
page-level management strategy. To ensure a fair comparison, we force both strategy and the baseline to perform retrieval every decoding step. We set the page size of baseline to 16, aligning it with the average cluster and segment size used in approach. As depicted in Figure [3 b), fine-grained management strategy consistently outperforms the coarse-grained baseline. This result confirms the necessity of tailored design: semantic clusters better suited for the sparse at- tention patterns input sequences, while temporal segments effectively capture the dense attention patterns of output sequences. This tailored design is crucial for achieving high inference accuracy.
D.4 IMPACT OF SIMILARITY THRESHOLD
The threshold 7 is key hyperparameter that governs the trade-off' between accuracy and efficiency. defines the sensitivity for triggering a new KV retrieval operation. lower value of T imposes stricter criterion for identifying semantic boundary, thereby reducing the frequency of retrievals. While this enhances inference efficiency, it typically leads to accuracy degradation by potentially failing to update the set of critical KVs. Conversely, higher T results in better inference accuracy. However, the frequent retrievals incur additional computational and data transfer overhead, thus reducing overall efficiency. We experimentally analyze the impact of on both inference accuracy and efficiency.
Analysis of Inference Accuracy. To quantify the impact T, conducted series of ablation studies on long-input and long-output tasks using the Llama-3. 1-B and Qwen3-8B models, respec- tively. As shown in Figure 14, a consistent trend is observed: model accuracy improves all tasks with an increasing т. This is because higher leads to more frequent retrievals, ensuring that critical information is recalled promptly. More importantly, long-input tasks (Figure I4 a-c)) ex- hibit significantly higher sensitivity to the value r than long-output reasoning tasks (Figure 1-)). We attribute this to the difference in task nature: to generate an answer, long-input understanding tasks often require shifting attention between disparate parts of the context, thus demanding more responsive tan to capture these semantic changes. contrast, long-output reasoning tasks exhibit stronger local coherence, each reasoning step primarily attends consistent context. This results in naturally longer semantic segments, making the model's performance less sensitive to the precise value of tan.
Analysis of Inference Efficiency. To quantify the impact of 7 on efficiency, we measured the retrieval frequency and inference latency across different thresholds. As shown in Figure 14
18

---

Published as a conference paper at ICLR 2026
45
35
5core
Score
40
50
30
35
45
25
0.5
0.6
0.9
0.7
0.8
0.9
0.6
0.7
0.8
0.9
Similarity Threshold
Similarity Threshold
Similarity Threshold
(a) Qasper
MultifielDQA
GovReport
Accuracy
0.7
Accurac/
Accuracy
0.70
0.6
0.7
0.65
0.6
0.60
0.5
0.5
0.5
0.6
0.9
0.5
0.7
0.8
0.9
0.6
0.7
0.8
0.9
Similarity Threshold
Similarity Threshold
Similarity Threshold (i) Longfurrsom-6KKK
(d) GPQA
(c) AIME
Retrieve Numbber
Retrieve Numbeer
200
10000
Latency (ms)
100
5000
50
40
0.5
0.6
0.7
0.8
0.9
0.5
0.6
0.7
0.8
0.9
0.5
0.6
0.7
0.8
0.9
Similarity Threshold
Similarity Threshold (h) AIME
Similarity Threshold
(g) GovRepart
Long-luput, Lungu
Figure 14: Impact of the similarity threshold inference accuracy, retrieval frequency, and la- tency. Subplots (a-f) show the accuracy on various long-input and long-output benchmarks. Subplots (g-h) illustrate the corresponding increase in retrieval frequency, and subplot (i) shows the resulting impact on inference latency. The results demonstrate the trade-off between accuracy and efficiency governed by
0.4
0.4
Proportion
0.3
Proportion
0.3
0.2
0.2
0.1-
0.1
0.0
0.0
(0.8)
(8.16)
(16.24)
(24,32]
(32.4)
401
(40.49)
(48,56)
(56.64)
(64,128)
(0.8)
(8.16)
(16,24)
(24,32)
32!
(32,40)
(40. 48)
(48.56)
(56, 64]
(64, 124,
Segment Length Range
Segment Length Range
(a) MATHE00
(b) LangRessnn61KK
Figure 15: Distribution of semantic segment lengths. Histograms (a) MATH500 and (b) LongReason- 66K.. The long-tailed distribution indicates that reasoning tasks frequently form long stable semantic segments, which allows LouisKY to effectively amortize the retrieval overhead by skipping redundant operations during these stable periods.
h), the number of retrievals increases with for both task types. Since each retrieval introduces non-negligible overhead, the inference latency consequently increases as well. We measured the average per-token inference latency for different 7 values in long-input, long-output (32K+16K) scenario using Qwen3-8B, as depicted in Figure 14)). The results show that for Qwen3-8B, 0.7 represents an ideal balance, achieving significant efficiency gains while maximally preserving inference accuracy.
D.5 DISTRIBUTION OF SEMANTIC SEGMENT LENGHTS.
To substantiate our amortization claim with hard numbers, we perform a detailed analysis using the Qwen3-8B model with the default threshold 0.7.
We first measure the average semantic segment length across different tasks. Reasoning tasks like MATH500 and GPQA exhibit an average segment length of approximately 14-18 tokens. This implies that LouisKV reduces the retrieval frequency by more than 10x compared to por-token methods. Even for long-document QA tasks (c.g., NarrativeQA) which involve more frequent topic shifts, the average length remains around 3-6 tokens. This provides strong evidence that LouiskV significantly amortizes the retrieval cost across diverse scenarios.
19

---

Published as a conference paper at ICLR 2026
Avg. Segment Length
Similarity
1.0
60
0.8
40
0.6
0.4
20
0.2
0.0
0
10
15
20
25
30
35
0
5
10
15
20
25
30
Layer Index
Layer Index
Layer-wise Sogment Leugth
(b) Inter-Layer Boundary Similartyy
Figure 16: Per-layer analysis of semantic segmentation. (a) Average semantic segment length at each Transformer layer. Shallow layers (e.g.. 0-10) exhibit shorter segments while deep layers (e.g., 28-34) form longer segments.
(b) Jaccard similarity of boundary trigger indices between adjacent layers. The low similarity across most layers indicates that semantic boundaries are detected different points, validating the necessity of our per-layer retrieval strategy.
We further visualize the full distribution of segment lengths in Figure 15 of the paper. The his- tograms illustrate long-tailed distribution for reasoning tasks. While short segments present, a significant proportion of the generation steps fall into long segments (spanning 16 to 128 tokens). These long segments play crucial role in reducing retrieval overhead, thereby contributing sub- stantially to the overall end-to-end speed improvement.
D.6 PER-LAYER BOUNDARY ANALYSIS
To provide rigorous justification for layer-wise retrieval design, analyze the similarity of semantic boundary detection across different Transformer layers. The analysis is conducted using the Qwen3-8B model the MATH500 dataset.
Layer-wise Segment Length. We first measure the average length of semantic segments at each layer. As shown in Figure [6]a] of the paper, shallow layers (Layers 0-3) exhibit shorter average segments (approximately 4-6 tokens). This observation aligns with the intuition that lower layers capture lower-level, rapidly changing features such as syntactic structures lexical choices, thus identifying more frequent semantic boundaries. In stark contrast, deep layers (e.g.. Layers 28-34) exhibit substantially longer segments. This reflects their function processing high-level abstract concepts, which remain stable during extended reasoning chains.
Inter-Layer Similarity on Boundaries. To further quantify the similarity in trigger points across layers, we calculate the Jaccard similarity of the boundary- trrigerg token indices between adjacent layers. As illustrated Figure [6)b), the similarity notably low, especially in the shallow layers. This low similarity indicates that semantic shift occurring in lower layer does not necessarily result in a corresponding shift in a higher layer. This phenomenon validates the necessity of our per-layer segmentation strategy, which enables each layer to independently manage the retrieval and segmentation frequencies.
E
USE OF LLMS
In this research, we utilized Large Language Models (LLMs) as an auxiliary tool to enhance research efficiency and the quality of the manuscript. The authors maintained strict oversight throughout this process, ensuring adherence to the principles of academic integrity. We affirm that the authors bear full and final responsibility for all content, analyses, and conclusions presented herein. A detailed breakdown of our specific use cases for these models is provided below:
LANGUAGE POLISHING AND COPY-EDITING
We utilized LLMs to assist with language refinement in this manuscript. Its confined to improving grammar, clarity, and overall readability. The authors reviewed all suggested changes to ensure the integrity of our original arguments and that the scientific content remained unaltered.
20

---

Published as a conference paper at ICLR 2026
LITERATURE REVIEW AND BRAINSTORMING
In the preliminary stages of research, an LLM was used to summarize established concepts and survey related work, which helped in identifying potential research gaps. However, all final litera- ture citations and core research ideas were independently developed, verified, and validated by the authors.
21


Reviews:
[{"review_id": "stVJqJBOFm", "reviewer": "Reviewer_14Z4", "summary": "The authors analyze long-sequence inference and discover that critical KVs (i.e. important tokens) demonstrate strong temporal locality during decoding (i.e., critical KVs between neighboring decoding steps are similar), and that critical KVs are sparsely distributed in prefill, but during decoding are locally concentrated within certain reasoning steps. Based on these observations, separate prefill/decoding cache management strategies are leveraged to improve recall from the CPU to the GPU of relevant KV entries. Custom kernels are implemented resulting in 4.7x speedup against baselines with minimal performance drop.", "questions": "- Could you try more questions from MATH500 and GPQA? You do not need to try the entirety of these datasets, but perhaps try 2-3 50 question samples and report the error. \n\n - How quickly can $\\tau$ can be tuned? That is, could you study how many questions are practically needed per model to determine a reasonable selection of this hyperparameter?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- Long-sequence KV strategies are very focused on long prefill. With the advent of reasoning models, the long decoding phase is now common, and the authors' study of this scenario is timely. \n\n- The described approach is simple and easy to understand. Mixed CPU/GPU KV management strategies are relatively less common, but are becoming increasingly important as token eviction continues to prove less generalizable. \n\n - The method is both fast and maintains high performance on a wide variety of task types. \n\n - A variety of model families are tested thus demonstrating architecture generalizability.", "weaknesses": "- As with all mixed CPU/GPU KV cache strategies, the approach requires careful kernel level optimizations to mitigate data transfer latency, thus making this approach non-hardware/software agnostic. \n\n- Selection of $\\tau$ seems to vary by both model and task. Time spent tuning this hyper-parameter limits the overall efficiency of this framework.\n\n- Small sample sizes from the math reasoning benchmarks.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "uFA5UHoFGB", "reviewer": "Reviewer_dBjq", "summary": "This paper presents LouisKV, a framework to accelerate long-context LLM inference by optimizing KV cache retrieval.\n\nLouisKV uses a semantic-aware retrieval strategy that triggers KV retrieval only at semantic boundariesr to reduce computation and data transfer.\nDuring the prefill stage, LouisKV performs k-means clustering on the KV cache to group semantically similar tokens, using the centroids for retrieval. In the decode stage, it partitions generated KVs into fixed-size temporal segments, retrieving them as clusters when a semantic boundary is detected.\n\nExperiments demonstrate that LouisKV achieves up to 4.7× speedup over state-of-the-art KV retrieval methods such as Arkvale, while maintaining near-lossless accuracy across diverse long-sequence tasks.", "questions": "What are the unique contributions and technical novelties of LouisKV compared to prior KV retrieval frameworks, particularly ClusterKV.", "limitations": "", "rating": 2, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 1, "strengths": "The paper presents comprehensive experiments across diverse benchmarks, models, and task types to demonstrate that LouisKV achieves near-lossless accuracy while improving inference efficiency.", "weaknesses": "This paper exhibits **a severe lack of novelty** and raises potential concerns about overlap with prior work.\n\nIn particular, the core KV cache retrieval mechanism in LouisKV appears **nearly identical** to that of ClusterKV \\[1\\].\nBoth of them apply k-means clustering to the KV cache during the prefill stage, select retrieval units based on cluster centroids, and manage KV entries generated during decoding separately. \nFurthermore, system-level components such as the asynchronous clustering, offloading pipeline and the CUDA kernel for efficient selection have also been introduced in ClusterKV. \nAs a result, LouisKV does **not present clear methodological or engineering innovation** beyond what has been previously published. \nThe absence of explicit discussion or experimental comparison against ClusterKV further weakens the paper’s contribution.\n\nThe claimed semantic-aware adaptive retrieval also offers **limited novelty**. The notion of exploiting temporal locality in decoding has been explored in prior works \\[2, 3, 4\\].\nFor example, the specific strategy of triggering retrieval based on cosine similarity between consecutive query vectors seems essentially the same as in \\[3\\].\nConsequently, the contribution of LouisKV beyond existing literature is unclear.\n\n\\[1\\] ClusterKV: Manipulating LLM KV Cache in Semantic Space for Recallable Compression\n\n\\[2\\] HShare: Fast LLM Decoding by Hierarchical Key-Value Sharing\n\n\\[3\\] FreeKV: Boosting KV Cache Retrieval for Efficient LLM Inference\n\n\\[4\\] CaliDrop: KV Cache Compression with Calibration", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "KYhMRSWxjo", "reviewer": "Reviewer_cVfv", "summary": "The paper proposes **LouisKV**, a KV-cache retrieval framework for long input and long output generation. The core idea is to avoid per-token retrieval by exploiting temporal locality in decoding and to improve precision by managing input and output KV caches differently. Concretely, LouisKV (i) detects semantic boundaries using the cosine similarity of consecutive query vectors and only triggers retrieval at these boundaries, and (ii) applies k-means clustering over prefill keys for inputs while forming temporal segments for generated outputs, each represented by a centroid for fast similarity lookup. The system adds Triton and CUDA kernels and a group-consistent selection strategy to reduce transfer and compute overhead.", "questions": "1) **Trigger statistics.** Please report the distribution of semantic segment lengths across datasets, average r at boundaries, and per-layer agreement on boundary detection. This can justify the amortization claim with hard numbers beyond the ablation. \n\n2) **Generated KV retention in long-reasoning.** In short-input long-output tasks, do you offload all generated KVs to CPU after the local buffer fills so they remain recallable. If yes, what is the incremental CPU-GPU traffic per segment, and how does that compare with page-level Arkvale under the same budget. A per-token bytes-moved metric would help.\n\n3) **Cluster locality in text space.** Within a cluster, are source text spans mostly contiguous or scattered. If clusters often map to contiguous spans, would span-aware grouping or block-sparse layouts reduce transfers further. Any analysis of edit distance between token positions in the same cluster.\n\n4) **Citation coverage.** More semantic-based retrieval work could be added.\n[1] Chen, Guoxuan, et al. \"Sepllm: Accelerate large language models by compressing one segment into one separator.\" arXiv preprint arXiv:2412.12094 (2024).\n[2] Zhu, Yuxuan, et al. \"SentenceKV: Efficient LLM Inference via Sentence-Level Semantic KV Caching.\" arXiv preprint arXiv:2504.00970 (2025).\n[3] Hooper, Coleman, et al. \"Squeezed attention: Accelerating long context length llm inference.\" arXiv preprint arXiv:2411.09688 (2024).", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1) **Decoding efficiency by fewer retrieval triggers.** Triggering only at semantic boundaries reduces repeated estimation and transfers while preserving accuracy within a segment, which the paper motivates and supports with locality measurements and latency breakdown. \n\n2) **Covers multiple inference scenarios.** The study includes long-input short-output, short-input long-output, and long-input long-output, showing stable accuracy at fixed budgets and scale-up in batch size without OOM. \n\n3) **Solid systems work.** Custom Triton and CUDA kernels and group-consistent selection improve throughput in realistic deployments with GQA. The ablation indicates which component contributes most.", "weaknesses": "1) **Motivation scope.** The assumption that decoding naturally forms stable semantic segments holds on math and step-wise tasks but may not on dialog or code completion with frequent topic shifts.\n\n2) **Clustering cost and TTFT.** Input clustering is central, yet the paper reports end-to-end latency without an explicit time-to-first-token under very long prompts. Please add TTFT curves vs prompt length and show the amortization benefit when queries are short. Also report k-means wall time vs input length and number of clusters, and whether the asynchronous pipeline hides this cost under prefill compute. \n\n3) **Long-context stress tests.** Results include LongReason up to 64K. RULER is a standard stress suite for long-context retrieval and sequence length extrapolation. Please add RULER at 32K-128K with accuracy and latency, and clarify memory usage under those settings.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "mi6diIdlQB", "reviewer": "Reviewer_7kuj", "summary": "The paper introduces LouisKV, a KV cache management framework for long-context and long-generation inference. The design is motivated from the observation of temporal locality between adjacent tokens and distinct spatial distributions between the long inputs and long outputs. During prefill, LouisKV uses k-means clusters and triggers retrieval during decoding only at semantic boundaries detected by query vector cosine similarity. Offloading is also performed for efficiency, and additional system level optimizations using CUDA kernels is done to enable speed-ups. Experiments on long-input tasks like long bench and complex reasoning tasks like AIME show that LouisKV achieves a speedup and performs better than other retrieval methods like Arkvale and Quest.", "questions": "1. How sensitive are results to the local buffer W and sink tokens S?\n\n2. How was the number of clusters determined in k-means?\n\n3. The clustering on prefill KVs can be quite expensive. Could you quantify this latency overhead and how it scales with input length?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The decoupled management scheme which applies different strategies for both prefill and decoding stage, along with system level optimizations lead to strong performance compared to the other testes retrieval baselines.\n2. Experimental setup is very comprehensive, testing for not only long-context input, which is more common in KV cache literature, but also long-output using reasoning benchmarks like MATH500 and AIME.", "weaknesses": "The contribution of the paper is spread across multiple aspects, making it difficult to recognize the main contribution. LouisKV modifies multiple stages of the LLM inference pipeline: Prefill with k-means clustering and asynchronous offloading. Decoding with semantic-boundary detection, and asynchronous offloading. System level optimizations with triton kernels. While each piece makes sense, it is difficult to assess conceptual novelty.\n\nFor instance, the two observations noted in motivation has been reported before and already made use of in prior works. For example, ShadowKV [1] makes use of temporal locality to rebuild only necessary KV pairs. The second observation of distinct patterns has also been noted in SCOPE [2]. Similarly, the novelty of boundary trigger is limited, as FreeKV [3] measures cosine similarity between adjacent tokens' query vectors during long generation, and clustering has also been used during prefill stage (ClusterKV [4]). While the synthesis of these components is novel and the engineering effort is appreciated, it is difficult to isolate or have a view of what the main contribution of the paper is.\n\n\n[1] Sun, Hanshi, et al. \"Shadowkv: Kv cache in shadows for high-throughput long-context llm inference.\" arXiv preprint arXiv:2410.21465 (2024).\n\n[2] Wu, Jialong, et al. \"Scope: Optimizing key-value cache compression in long-context generation.\" arXiv preprint arXiv:2412.13649 (2024).\n\n[3] Liu, Guangda, et al. \"FreeKV: Boosting KV Cache Retrieval for Efficient LLM Inference.\" arXiv preprint arXiv:2505.13109 (2025).\n\n[4] Liu, Guangda, et al. \"Clusterkv: Manipulating llm kv cache in semantic space for recallable compression.\" 2025 62nd ACM/IEEE Design Automation Conference (DAC). IEEE, 2025.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)