SentenceKV: Efficient LLM Inference via Sentence-Level Semantic KV Caching
Authors: Yuxuan Zhu
Ali Falahati
David H. Yang
Mohammad Mohammadi Amiri
Venue: COLM 2025 (2025)
Keywords: Large Language Models
KV cache compression
Sentence-level semantic caching
Inference efficiency
Long-context inference
TL;DR: SentenceKV improves LLM inference efficiency by compressing and retrieving key-value cache at the sentence level based on semantic similarity.

Abstract: Large language models face significant computational and memory challenges when processing long contexts. During inference, efficient management of the key-value (KV) cache, which stores intermediate activations for autoregressive generation, is critical to reducing memory overhead and improving computational efficiency. Traditional token-level efficient KV caching methods overlook semantic information, treating tokens independently without considering their semantic relationships. Meanwhile, existing semantic-preserving KV cache management approaches often suffer from substantial memory usage and high time-to-first-token. To address these limitations, we propose SentenceKV, a novel sentence-level semantic KV caching approach designed to enhance inference efficiency while preserving semantic coherence. During prefilling, SentenceKV groups tokens based on sentence-level semantic similarity, compressing sentence representations into concise semantic vectors stored directly on the GPU, while individual KV pairs are offloaded to CPU. During decoding, SentenceKV generates tokens by selectively retrieving semantically relevant sentence-level KV entries, leveraging the semantic similarity between the prefilling-stage semantic vectors and decoding-stage queries. This ensures efficient and contextually accurate predictions, minimizing the loading of redundant or irrelevant data into GPU memory and significantly reducing memory overhead while maintaining stable inference latency, even for extremely long contexts. Extensive evaluations on benchmarks including PG-19, LongBench, Needle-In-A-Haystack, and RULER demonstrate that SentenceKV significantly outperforms state-of-the-art methods in both efficiency and memory usage, without compromising model accuracy.

Published as a conference paper at COLM 2025
SentenceKV: Efficient LLM Inference via Sentence-Level Semantic KV Caching"
Yuxuan Zhu
Ali Falahati
Department of Computer Science Rensselaer Polytechnic Institute Troy, NY 12180, USA zhuy27@rpi.
Department of Computer Science University of Waterloo Waterloo, ON N2L 575, Canada afalahat@uwaterlle ca
David H. Yang
Mohammad Mohammadi Amiri Department of Computer Science Rensselaer Polytechnic Institute Troy, NY 12180, USA mamiri@rpi. edu
Department of Computer Science Rensselaer Polytechnic Institute Troy, NY 12180, USA pangd13@rpi.c edu
Abstract
Large language models face significant computational and memory chal- lenges when processing long contexts. During inference, efficient manage- ment of the key-value (KV) cache, which stores intermediate activations for autoregressive generation, is critical to reducing memory overhead and improving computational efficiency. Traditional token-level efficient KV caching methods overlook semantic information, treating tokens inde- pendently without considering their semantic relationships. Meanwhile, existing semantic-preserving KV cache management approaches often suf- fer from substantial memory usage and high time-to-first-toke To address these limitations, we propose Sentence KV, novel sentence-leve seman- tic KV caching approach designed to enhance inference efficiency while preserving semantic coherence. During prefilling, SentenceKV groups to- kens based on sentence-level semantic similarity, compressing sentence representations into concise semantic vectors stored directly on the GPU, while individual KV pairs are offloaded to CPU. During decoding, Sen- tenceKV generates tokens by selectively retrieving semantically relevant sentence-level KV entries, leveraging the semantic similarity between the prefilling-stage semantic vectors and decoding-stage queries. This ensures efficient and contextually accurate predictions, minimizing the loading of redundant or irrelevant data into GPU memory and significantly reduc- ing memory overhead while maintaining stable inference latency, even for extremely long contexts. Extensive evaluations on benchmarks includ- ing PG-19, LongBench, Needle-In-A- A-Hyssack, and RULER demonstrate that SentenceKV significantly outperforms state-of-the of-tthe-rrtt methods in both efficiency and memory usage, without compromising model accuracy.
1 Introduction
Large language models (LLMs)) have achieved remarkable success across wide range of natural language processing tasks (Min et al., 2023; Zhao et al., 2023). As LLMs are increasingly applied to complex reasoning and multi-step mecision-making scenarios, the demand for longer context windows has grown significantly Chen et al. (2023). Recent advancements have extended the context window from the early GPT models the latest iterations, such as GPT-ol and beyond (OpenAI, 2023; Koubaa, 2023). However, this growth
*Code available at http://ghhhuuuboooorrirg//////// V

---

Published as a conference paper at COLM 2025
introduces substantial challenges in key-value (KV) cache efficiency, leading to memory bottlenecks and increased inference latency Shi et al. (2024).
Due to the transformer-basett architecture of LLMs Vaswani et al. (2017), the memory footprint of KV cache grows linearly with the context length, directly impacting GPU usage. For example, in the case of the LLaMA-3.1-8D model AI (2023), shown in Appendix A.1, processing a 32k-token prompt during the decoding stage requires approximately 16 GB (using float16 precision) of GPU memory, which can be prohibitive for users without high-end hardware. Furthermore, longer prompts significantly increase computational overhead, as each additional token requires computing attention weights across the entire sequence. This quadratic attention complexity leads to substantial compute inefficiency, making efficient inference particularly challenging in desource-consrringe environments.
To mitigate these issues, several methods have been proposed to compress KV cache Shi et al. (2024). Among them, token eviction strategies based on fixed selection patterns or attention- weight-based pruning have shown promise (Xiao et al., 2023; Zhang et al., 2023). However, these approaches suffer from key limitations: they fail to adapt to the dynamic nature of token importance during decoding and often discard tokens permanently, neglecting their potential relevance in the future decoding steps. This static behavior can degrade the performance of the model, especially in tasks requiring complex reasoning and long-range dependencies Liu et al. (2025a).
Another category of approaches, such Quest Tang et al. (2024) and ShadowKV Sun et al. (2024), dynamically retrieve portions of KV cache from CPU-offloaded storage back to GPU memory during attention calculation. Typically, these methods segment the input into fixed-size chunks and selectively retrieve them based on query relevance at each decoding step. However, fixed-size chunking can disrupt semantic coherence by arbitrarily breaking natural semantic boundaries. Some dynamic retrieval approaches, including ClusterKV Liu et al. (2024) and SqueezeAttention Hooper al. (2024), mitigate this issue by clustering semantically related keys into coherent chunks. Yet, these clustering-basee methods introduce significant computational overhead, resulting in increased time-to-first- token (TTFT) latency.
To address these limitations, we propose SentenceKV, a novel approach that dynamically manages KV cache based on sentence-level semantic information. Our key insight is that sentences inherently encapsulate richer semantic information compared to isolated tokens, making them more effective unit for caching and retrieval. For instance, as illustrated in Figure 1, sentences from different domains tend to form distinct semantic clusters within the representation space of LLMs, a phenomenon also observed in recent literature Chen et al. (2024). We further propose a novel multi nulti-query-awarr mechanism for retrieving relevant sentence-level KV cache entries during decoding.
SentenceKV introduces a sentence-level KV cache compression method that operates effec- tively during both the prefilling and decoding stages. During prefilling, our method stores the long input prompt at sentence granularity on CPU while maintaining compact seman- P
tic representation vector on GPU. During decoding, we maintain cache of multi-query vectors computed from the tokens of the most recently generated sentence. By averaging these vectors, we retrieve the most relevant prefilled sentences and dynamically load their corresponding KV pairs to GPU for attention computations. This approach ensures effi- cient recall of important tokens and preserves the semantic context, significantly reducing computational overhead while maintaining high inference efficiency.
Our method includes: (1) Semantic token grouping based on sentence chunking without complex clustering. (2) Adaptive decoding cache which maintains dynamic cache for KV pairs retrieval aligned with coherent semantic units (e.g., full sentences) sentences)duriin during decoding. (3)
Efficient recall of sentence-level KV entries to maintain coherence. Compared to existing KV compression methods, SentenceKV achieves superior semantic preservation and improves recall precision efficiently.
Extensive experiments on long-contex benchmarks, including LongBench Bai et al. (2023), Mondle-In-A-Hayseee (NIAH) Kamradt (2023), and RULER Hsieh et al. (2024) demonstrate that our approach not only improves inference efficiency but also maintains or even enhances
2

---

Published as a conference paper at COLM 2025
model accuracy across diverse tasks. Quantitative analysis shows that our method offers efficient solution for long-context processing.
2 Related Work
Transformer-basee models rely KV cache for efficient inference. However, context length increases, KV cache can consume substantial memory, imposing practical limitations Long-sequence processing. To address this, various techniques have been proposed to compress or optimize KV cache.
KV Cache Compression. StreamingL.LM Xiao et al. (2023) enables infinite-contex han- dling via attention sinks and moving window, allowing models to process extremely long sequences without growing memory footprints. SnapKV Li et al. (2025) preserves essential KV positions based on per-head attention patterns with an observation window. FastGen Ge et al. (2023) progressively drops tokens with low future impact, enhancing efficiency with minimal quality loss. PyramidInfer Yang al. (2024) and PyramidKV Cai et al. (2024) propose a layer-wise pyramidal KV cache retention pattern, preserving more tokens in upper layers of the model maximize memory savings without sacrificing context.
Dynamic Access KV. Dynamic access mechanisms adaptively manage KV retention based on task-specific requirements. H₂O Zhang et al. (2023) dynamically retains high-impact tokens based on accumulated attention scores. TOVA Oren et al. (2024) prunes tokens based real-time query relevance, but risks removing context needed for future steps. The quest Tang et al. (2024) partitions KV cache into pageable memory and predicts critical pages per query, in which case fragmentation issues can arise. InfLLM Xiao et al. (2024) stores distant contexts into additional memory units and employs an efficient mechanism to lookup token-relevant units for attention computation. ShadowKV Sun et al. (2024) offloads the value cache to CPU and stores a low-rank key cache on GPU, using a KV selection strategy to reduce memory footprint and maintain high-throughput decoding. SqueezeAttention Hooper et al. (2024) optimizes cache allocation across both sequence and layer dimensions, assigning larger budgets to important layers while pruning unimportant ones. DynamicKV V Zhou et al. (2024) implements task-aware adaptive retention, adjusting KV budgets per layer to maintain task-specific relevance while reducing cache size.
Semantic-Level Optimization. As LLMs tackle increasingly complex tasks, optimizing KV cache at semantic level is crucial to maintaining output coherence. ChunkKV Liu et al. (2025b) groups tokens into semantic chunks, retaining the most informative segments and discarding redundant ones to enhance long-context inference efficiency. ClusterKV Liu et al. (2024) introduces semantic clustering, which, unlike methods that permanently evict tokens or recall them based solely on textual positions, clusters tokens semantically and recalls them at the granularity of semantic clusters. Task-KV He et al. (2025) differentiates between heterogeneous and general aggregation attention heads, preserving full KV cache for critical heads while reducing it for less relevant ones. SepLLM Chen et al. (2024) introduces data-dependent sparse attention mechanism to mitigate the excessive attention allocated to seemingly uninformative separator tokens. By compressing segment-level information into these tokens, SepLILM reduces KV cache while preserving essential content.
While chunk-based compression better preserves semantics than token-wise eviction, it may still fragment full thought units, omitting crucial context. Prior studies show that ignoring natural boundaries, such as sentence breaks, can harm language modeling performance (Dai et al., 2019; Rae et al., 2019). SentenceKV addresses this by retaining entire sentences or semantically self-contained segments, ensuring holistic context preservation. By bridging local semantic retention from ChunkKV with global coherence, SentenceKV further enhances long-context inference while maintaining minimal memory overhead.
3

---

Published as a conference paper at COLM 2025
3 Motivation
Existing KV cache compression methods typically operate at the token-level, where tokens considered less important-often determined by heuristics such as attention scores- are evicted during the prefilling stage. However, recent studies suggest that the relevance of prefilled tokens is not static throughout the decoding process Tang et al. (2024). Evicting tokens prematurely based on their initial perceived importance can degrade inference accu- racy since tokens initially deemed irrelevant might later become crucial during decoding. To address this dynamic nature, recent studies propose retrieving essential prefilled tokens dynamically based on their relevance to the current decoding query: To mitigate computa- tional overhead associated with searching across all prefilled tokens, these methods typically segment the prefilling text into fixed-size chunks, retrieving and utilizing only the most relevant chunks during decoding.
However, these chunk-based strategies lack a principled justification for preserving semantic coherence, as fixed- size chunks often fragment semantic content arbitrar- ily. Intuitively, sentences encapsulate complete seman- tic units, making them natural candidates for maintain- ing context coherence. To empirically validate this intu- ition, we examined sentences from distinct categories in the Pile dataset Gao et al. (2020) and measured their embedding similarities in LLMs. As illustrated in Fig- ure 1, sentences from different semantic categories exhibit clearly distinguishable embedding patterns. Moreover, during the decoding phase, observed that the model- generated query tends to focus attention primarily on semantically related sentences rather than arbitrary token segments. This observation motivates our sentence-level caching approach, which segments the prefilling text into semantically coherent sentence units. During decoding, we leverage the sentence currently being generated dynamically identify and recall thereby efficiently preserving
Freelaw
GitHub
Math
PubMed
Wikipedia
FreeLaw
GitHub
Math-
PubMed
Wikipedia
Figure Similarities between sentence embeddings from dif- ferent categories in the Pile datasets Gao et al. (2020).
the most semantically relevant sentence-level cache entries, context and enhancing inference accuracy.
4 SentenceKV
To address the limitations of token-level KV cache compression, we propose SentenceKV, which reorganizes CPU cache at the sentence-level, leveraging semantic coherence rather than individual tokens. SentenceKV consists of two main phases: prefilling and decoding. Figure 2 illustrates our framework, with pseudocode provided in Appendix A.2.
4.1 Prefilling Stage
We first split the input text into sentences according to punctuation, creating multiple sentence buckets, each containing a varying number of tokens. For example, a 32K token input might be split into hundreds of sentence buckets of different lengths. While token importance changes dynamically during decoding, maintaining all tokens at full precision in GPU memory inefficient and unnecessary. To determine which tokens to retain, apply an observation window Li et al. (2025) for token importance calculation as follows:
Token importance measurement. For an input prompt of length L (e.g., = =32)), we designate the last N tokens (typically N = =32) as our observation window. We perform forward pass where each token in positions (L - N + 1) to calculates attention scores with respect to all preceding tokens in positions 1 to (L N). For each token i in the preceding positions, we calculate its importance score a; by summing the attention scores it receives from all tokens in the observation window, across all the attention heads.
4

---

Published as a conference paper at COLM 2025
affioad important KV pairs CPU
semantic GPU
sentence
Prefill
sentence
aimilarity retrieval
2
less simitar sontence,
7
mean query g
retrieve
Decoding Sentence
onloaded key cache
current query
sentence cache (queries)
2
attention
similarity retrieval
mean query ?
2
S
Decoding Sentence 2
onloaded key cache
query
sentence cache (queries)
attention
Figure 2: The figure illustrates the two-phase Se caching mechanism. (1) During the prefilling phase, the input prompt is segmented into sentences. Token importance scores are computed, and only the top r.. T| tokens are offloaded to CPU while the rest are discarded. Semantic vectors representing each sentence are computed and stored. (2) During the decoding phase, an empty sentence cache Q. is initialized. As new tokens are generated, their queries Q are appended to Qs. At each decoding step, a mean query q is computed for the current sentence cache. (3) The similarity between & and stored sentence semantic vectors are evaluated, and tokens from the most relevant sentence buckets are retrieved along with their corresponding KV pairs, subject to the token budget т. (4) Finally, attention outputs computed, and Qs is reset when the generated sentence ends.
Token selection within sentence buckets. We token budget (e.g., T 1024)
that represents the maximum number of tokens we can keep in GPU memory during decoding. Since tokens that appear unimportant during prefilling may become relevant at later decoding steps, we introduce the semantic keeping factor (typically r == 2 or 3), which controls how many tokens are initially retained before selective pruning during decoding. By setting > 1, we retain more tokens than final budget (T) allows, giving us richer semantic pool to select from during decoding. We select the top r T tokens with the highest a; values across all sentence buckets. This design ensures that the method works when sentence lengths are unbalanced; see Appendix A.7 for the analysis.
Sentence-level semantic representation. For each sentence bucket s, after identifying its important tokens, we compute a mean key vector for each attention head
1
kg,
kgp xES,
(1)
[
where Sy contains the indices of retained tokens in sentence and kg, is the key vector of token x in head h. These compact sentence-level semantic vectors remain on the GPU for efficient similarity calculations during decoding.
Memory management. Rather than permanently discarding tokens, we keep all [r T selected tokens along with their KV pairs, but offload them to CPU memory. During decoding, we will selectively retrieve only the most relevant sentence buckets, up the token budget t, based semantic similarity to the current generation context.
This approach maintains sentence-level semantic coherence while efficiently managing memory. For instance, in 32K context with 1024 and r = 2, might inítially retain 2048 tokens distributed across sentence buckets, but only load the most relevant subset (up to 1024 tokens) to GPU during each decoding step.

---

Published as a conference paper at COLM 2025
4.2 Decoding Stage
During the decoding (generation) phase, SentenceKV employs a dynamic mechanism to retrieve the most semantically relevant information from the prefilled tokens.
Sentence-level query aggregation. As the model generates new tokens, we aggregate
their query vectors into temporary sentence cache Qs. This cache accumulates query vectors 9i for each newly generated token until sentence boundary (e.g., period, question mark) is detected. Once a complete sentence has been generated, we compute a mean query vector that represents the semantic center of the generated sentence:
g
9: FEQ.
(2)
This aggregation captures the overall semantic intent of the generated sentence rather than relying on individual token queries, which may contain position-specific or token-specific noise.
Semantic similarity and token retrieval. Using this sentence-level query representation q, we compute semantic similarity scores between the generated sentence and each stored sentence bucket from the prefilling stage. For each attention head l and sentence bucket we calculate (q ks,)) q" kg, where ksh is the mean key vector of sentence for head h, as computed during prefilling (Equation 1). This calculation identifies which sentence buckets from the original prompt are most semantically relevant to the current generation context.
We then rank all sentence buckets by their similarity scores and retrieve tokens from the most relevant buckets in descending order of similarity. For example, the highest similarity scores are for sentences containing financial data, we prioritize retrieving tokens from those sentence buckets first. We continue this process until we reach our token budget T, ensuring we load only the most contextually relevant information back to the GPU.
Attention computation with retrieved tokens. CPU pairs associated with the retrieved tokens are loaded from CPU back to the GPU memory. For the current query vector 9, attention is computed using only these retrieved tokens:
qK]
O = softmax
V
(3)
Vd
where (K,, Vz) are the key and value matrices for the retrieved tokens (up to T tokens), and d is the head dimension. This focused attention computation significantly reduces computational overhead by considering only the most relevant context. After generating the next token, we append its query to Q₂ and repeat the process. When a new sentence boundary (e.g., period, question mark) is detected, queries for the next sentence. Compared to existing compression strategies, SentenceKV pre- serves richer semantic information by treating entire sentences retrieval units, mitigating fragmentation, and allowing dynamic reactivation of offloaded tokens when needed.
reset Qs and begin accumulating
5 Experiments
In this section, we present the experimental setup and evaluate our proposed method, Sen- tenceKV, on several benchmarks. We compare it with state-of-the-art KV cache compression methods using metrics such accuracy, perplexity (PPL), memory usage, and latency.
All the experiments are conducted using two NVIDIA H100 S0GB GPUs and two NVIDIA H100 NVL 96GB GPUs. We evaluate our method on four benchmarks: PG-19 (language modeling task) Rae et al. (2019), LongBench Bai et al. (2023), NIAH (retrieval "needle") Kamradt (2023), RULER Hsieh et al. (2024). Our experiments use the models AI (2023), -lama-3.2-3B-Insttu AI (2024b) and Longchat-vl.5-7B LMSYS (2024). We compare SentenceKV with several baselines,
hidden
Llama-3-8B Al (2024a),

---

Published as a conference paper at COLM 2025
Table 1: Perforrancc comparison between SentenceKV and baseline methods on LongBench tasks. Token budget is set to 1024, " indicates out-of-memory (OOM) errors, bold values are the maximum among non-Full KV methods.
Single-Docung Docuumen QA
Docum-Do QA
sho- Learning
Synthetic
Code
LLMs
NrtVA
Qasper
Mi-enn
HelpoQQA
2WikiMQA
Musique
TREC
TriviaQA
SAMSum
PCount
PRe
Lcc
RB-P
Full KV
29.59
47.52
53
53,76
46.12
28,36
7.5
89.41
7.47
6.25
99.5
13.52
11.72
H2O
47.89
12.5
98
SnapKY
52.66
45.67
99.5
Quest
22.49
45,2
48.72
51.94
44.02
26.78
13.5
88.63
10.31
4.21
97
14.79
16.64
InfLLM
27.64
45.32
50.6
51.47
44.59
23
12.5
90.92
6.79
95
13.33
23.79
SentenceKV
29.53
47.49
53.15
53.39
28.04
11.5
89.41
7.48
5.28
99.5
13:44
11.28
Full KV
22.68
33,99
46.32
39,22
26.95
14.57
85.71
84,93
22.34
0
18.5
52.94
56.89
Llama-3 3-330 Longcha
SnapKY
20.79
30.23
41.98
36.41
26.81
13.38
64.5
80,88
26.79
17.5
52,02
55.41
Quest
20.76
31.67
41.55
37.25
23.47
13.65
64
$1.83
25.73
3.5
17
49.1
51.2
InfLLM
13.59
9.42
49.0
SentenceKV
22.01
32.98
43.29
37.49
26.17
13.29
64.5
76.54
24.29
16.5
54.33
56.23
Full KV
23.53
41.69
49.82
49.96
41.8
20.96
2.0
9.17
5.78
78.5
26.31
25.2
H20
SnapKV
20.02
35.17
48.29
48.98
37.63
20.12
0.5
9.07
6.19
4.35
81.5
26.19
25.36
Quest
31.7
46.07
19.12
3.5
13.17
67.0
28.64
28.45
InfLLM
20.48
33.97
45,89
45.94
36.7
16.54
0.0
48.77
3.56
2.0
36.5
24,86
26.27
SentenceKV
24:12
38.38
49.38
48.98
40.01
20.95
6.0
8.82
6.06
0.0
84.0
26.64
23.46
including H2O Zhang et al. (2023), SnapKV Li al. (2025), Quest Tang et al. (2024), InfLLM Xiao et al. (2024), ShadowKV Sun al. (2024), as well as Full KV cache baseline. The evaluation metrics include model accuracy, GPU memory usage (in GB), and latency (in ms/token).
5.1 Language Modeling
Setup. We evaluate the PG-19 dataset to measure PPL. under context lengths of up to 32k tokens. A lower PPL indicates that the model is less surprised by ground truth tokens, reflect- ing higher confidence in its predictions. In our setup, the model is fed sequences exceeding 32k tokens from the dataset, followed by autoregressive decoding, during which PPL is computed at each step. Lower PPL values correspond to better predictive Derformance.
Results. Figure 3 shows PPL measured at a token budget of 1024 (3% of the full KV cache when the context length is 32k). Despite this significant compression, SentenceKV consistently achieves PPL nearly identical to the full KV cache baseline, even as the input length grows. This demonstrates that SentenceKV effectively retains essential contextual information, preserving generation quality while dra- matically reducing GPU memory usage. The minimal performance gap across all context lengths highlights the strength of semantic-level caching in maintaining decoding fidelity under tight memory constraints.
Perplexity flowerls is better)
8.4
sontencakv
7.6
-.44
5000 10000 15000 20000 25000 30000 Input Length
Figure 3: onL PG-19 with token budget т = 1024. Lower PPL indi- cates better performance.
5.2 LongBench
Setup. We select suite of tasks from the LongBench benchmark, including single- document QA, multi-document QA, and few-shot learning, as well as synthetic and code generation tasks. Each task features input sequences ranging from a few thousand to tens of thousands of tokens.
Results. Table 1 reports accuracy on LongBench tasks. The full KV cache achieves the best performance but is impractical for long inputs due to high GPU memory usage. In contrast,
7

---

Published as a conference paper at COLM 2025
SentenceKV achieves comparable accuracy across most tasks while operating under strict token budget, demonstrating strong performance-memoy trade-offs. consistently matches closely approaches full KV cache both Llama models and LongChat-vv.5-7B. Compared to other baselines, SentenceKV more stable: H2O frequently fails with OOM errors on long inputs (marked as in the table). SnapKV and Infî. Inf...Mpeooomms well but slightly lags behind SentenceKV on several tasks. Quest does not perform well on GQA- based models (including Llama-3.1). Overall, SentenceKV provides an effective solution for long-context inference with reduced memory usage.
5.3 Needle In A Haystack
Setup. In the NIAH benchmark, single critical sentence (the "needle") is placed within large, mostly irrelevant context. The task requires the model to retrieve this key sentence and answer a related question. We evaluate the method using the retrieval accuracy (%) metric, which indicates whether SentenceKY correctly identifies the crucial sentence.
(a) H2O Accuracy: 25.2%
(b) SnapKV Accuracy: 78.2%
(c) Quest Accuracy: 48.3%
(d) InfLLM Accuracy: 87.8%
ShadowKV Accuracy: 85.8%
SentenceKV Accuracy: 97.5%
Figure 4: Retrieval accuracy on the NIAH benchmark across context lengths up to 8000 tokens, with token budget T 128.
Results. Figure 4 shows the retrieval accuracy results on the NIAH benchmark, comparing SentenceKV against H2O, SnapKV, Quest, InfLLMand ShadowKVwith a token budget 128 and context lengths up to 8000 tokens, with Llama-3-8B model. SentenceKV achieves an outstanding accuracy of 97.5%, significantly outperforming other methods. SnapKV, InfLLM and ShadowKV shows reasonable accuracy (78.2%, 87.8 87.8% and 85.8%), whereas Quest provides moderate performance (48.3%). The reason for Quest's low accuracy analyzed in Appendix A.4. Notably, H2O exhibits poor performance (25.2%), primarily due to its limitations in handling extensive contexts, further highlighting the advantage of Se in accurately and efficiently retrieving critical sentences within lengthy texts.
RULER
Setup. Following the LongBench and NIAH evaluations, we additionally benchmark SentenceKV on the RULER long-context retrieval suite. Each example contains up to 64K tokens, pushing GPU memory pressure well beyond the typical setting. We use Llama- 3.1-8B-Instruct with flash-attention enabled and compare against representative KV-cache baselines.
Results. Table 2 reports retrieval accuracy (higher is better) eight RULER subtasks. SentenceKV matches or exceeds every offloading baseline and trails the full kv cache upper- bound specific tasks. ShadowKV attains comparable accuracy, but incurs longer prefill

---

Published as a conference paper at COLM 2025
time owing to its per-sentence SVD reconstruction; in contrast, SentenceKV keeps the pipeline simple, light- and easier to integrate.
Although fixed seman- tic keeping factor already performs well, tice task-dependent vari-
Method
52
MQ
QA1
FulKKV
1,00
1.00
1.00
0.98
0.96
0.85
0.94
0.55
ation. An adaptively cho-
SnapkV
1.00
sen semantic keeping fac-
H2O
OOM
OOM
OOM
OOM
0OM
OOM
OOM
OOM
Quest
1,00
0.98
1.00
0.82
tor may increase the ac- curacy; we leave a full ex- ploration this adaptive strategy to future work. Overall, these RULER re- sults confirm the scalabil- ity and robustness of Sen- tenceKV in extreme long- context scenarios and motivate its deployment in real-world applications where very long contexts are becoming the norm.
InfLLM
1,00
0.20
0.14
0.17
0.22
0.81
0.12
ShadowKV
1.00
0.98
1.00
0.87
0.74
0.78
0.79
0.70
SentenoeKV
1.00
0.98
0.98
0.77
0.78
0.61
0.86
0.63
Table 2: Retrieval accuracy on the RULER benchmark (64K to- kens). "OOM" indicates out-of-memory on a single H100 GB GPU.
Latency arfson
5.5 Efficiency
RV Sentancedy
We evaluate SentenceKV against Full KV cache, SnapKV, and SqueezeKV, focusing on GPU memory usage and inference latency (Figure 5). SnapKV per- forms KV eviction only during the prefill phase with- out dynamic retrieval during the decoding, resulting in minimal latency. Therefore, we focus our end-to-end comparison primarily with Full KV and SnapKV.
Latuncy (metakknn)
70
60
40
16K
64K
120K
266K
Context Length
Using the Llama-3.1-80 1.aaa...---B--nstuuttt model, SentenceKV achieves substantial memory savings as context length increases. At 256k tokens, it uses 52.71 GB of memory, compared 89.7 GB for Full KV. Its inference latency remains stable at around 17.8 ms, significantly lower than Full KV's 84.9 ms. Overall, SentenceKV supports longer contexts under constrained GPU memory while maintaining low latency, making it a practical and efficient solution for long-context decoding, See Ap- pendix A.3 for results of the LongChat-vll .5-7B model. A detailed efficiency and accuracy tradeoff analysis is shown in Appendix A.5, and a detailed GPU profilling shown in Appendix A.6.
(a) Latency Comparison
Memory Usage Compartson
KV
80
Snapky 1024
Cemen Unagle (Gm)
26
16K
32K
64K
128K
238K
Context Length
(b) Memory Usage Comparison
Figure 5: Efficiency comparison of KV caching methods (T 1024).
6 Ablation Studies
To further validate the design choices behind SentenceKV, we perform ablation studies focusing on two key components: sentence chunking and the cache strategy.
6.1 Sentence chunking
An alternative strategy to our sentence-level chunking is to perform equal-sized chunking based on the total number of sentences, dividing the text uniformly. However, this approach consistently yields worse results on LongBench (see Table 3), likely due to the loss of coher- ence and topical alignment within chunks. When sentences are split arbitrarily, important semantic relationships and narrative flow can be disrupted, making it harder for the model to maintain context and generate meaningful outputs.

---

Published as a conference paper at COLM 2025
In contrast, approach leverages seman- tic segmentation to group related informa- tion together, preserving logical boundaries and thematic continuity. This experiment demonstrates that chunking based mantic structure leads to better contextual integrity and improved downstream perfor- mance.
Table Equal-size chunks perform worse than SentenceKV V on LongBench tasks.
Equal Chunks
SentenceKV
NrtvQA
27.02
29.59
HotpotQA
53.19
53.76
TREC
8.5
11.5
PCount
4.81
5.28
6.2 Semantic keeping factor and cache strategy
Figure 6 illustrates how the semantic keeping fac- tor-which defined as the ratio r between the number of tokens retained per sentence during the prefilling stage and the fixed decoding token bud- get-influences retrieval accuracy on the NIAH bench- mark. As the semantic keeping factor increases, the model retains broader context, leading to improved uccuracy-up to a point. However, overly high values may introduce redundant noisy information, high- lighting the importance of careful parameter tuning to balance recall and precision.
Semantic Factor
Sentence Cache
Current Query
1.0
NIAH Accuracy
3
4
Semantic Keeping Factor
The figure also compares two retrieval strategies: us- ing the current token's 's query versus the mean query aggregated from the sentence cache. The latter con- sistently outperforms the former, demonstrating that sentence-level aggregation yields more stable and semantically representative signal for selecting relevant sentence buckets.
Figure Effect of semantic keep-
ing factor and caching strategy.
7 Conclusion
In this work, we introduced SentenceKV, sentence-level semantic KV caching framework that improves inference efficiency for large language models on long contexts. SentenceK integrates sentence-level processing into token-level transformer computations by grouping tokens into coherent sentences. It compresses the KV cache into compact sentence embed- dings on GPU and offloads full KV pairs to CPU. During decoding, only the most relevant sentences are retrieved within a fixed token budget, preserving coherence and reducing data transfer. Evaluations PG19, LongBench, NIAH and RULER show that SentenceK matches full-cache performance in perplexity and task accuracy using as little as 3% of the cache, while reducing GPU memory usage by over 30% and maintaining low latency up to 256K tokens. In retrieval-heavy settings, it improves retrieval accuracy by over 20% compared to prior methods, effectively identifying critical information in long inputs.
Future Work. Future work could explore several promising directions. First, relying on punctuation for sentence segmentation can lead to inaccurate boundaries, especially in texts with sparse or irregular punctuation, which harms semantic coherence. Using NLP- based segmentation tools, such as those in the Natural Language Toolkit Bird (2006), could improve accuracy. Second, the fixed semantic factor may reduce adaptability to texts with varying sentence lengths and structures. Third, many recent reasoning models generate cfain-of-thought (Col) traces with each sentence representing a self-contained logical step, making SentenceKV natural fit. Future evaluations could target dedicated CoT datasets such as DeepSeek-Ri Guo et al. (2025) and GSM-Infinite Zhou et al. (2025) to assess this potential. Lastly, incorporating lightweight on-device similarity search or mixed precision sentence embeddings could reduce CPU-GPU round-trij overhead under long-context scenarios.
10

---

Published as a conference paper at COLM 2025
Acknowledgments
We thank the anonymous reviewers for their valuable suggestions, particularly on eval- uating additional benchmarks, as well as their constructive comments on analyzing the efficiency of our method. We also thank Zhenyu Liu from the Department of Electrical, Computer, and Systems Engineering at Rensselaer Polytechnic Institute for his guidance on the implementation details.
References
Meta AI Meta llama 3-1. https://ai. wwwa. htmmbbloggmmttallaamae 2023. Blog post.
Meta AI. meta-llama//ettt-LLam--- 3-BB (model). https:: co/metaa.ccoomoaaaaaaaaa
Meta-Llama- 8-8 2024a. Accessed: 7 August 2025.
Meta AI. meta-Llama/LLama-33 2---:33.--33-----r--- Instruct (model). https::/huggingfaace
meta- 1ata-llmmm/LLaaa3..222 Insssuuutt 2024b. Accessed: August 2025.
Yushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu, Jiankai Tang, Zhidian Huang, Zhengxiao Du, Xiao Liu, Aohan Zeng, Lei Hou, et al. Longbench: A bilingual, multitask benchmark for long context understanding. ar Xiv preprint ar Xi::23308..1500 2023.
Steven Bird. Nltk: the natural language toolkit. In Proceedings of the COLING/ACL 2006 interactive presentation sessions, pp. 69-72, 2006.
Zefan Cai, Yichi Zhang, Bofei Gao, Yuliang Liu, Tianyu Liu, Keming Lu, Wayne Xiong, Yue Dong, Baobao Chang, Junjie Hu, et al. Pyramidkv: Dynamic kv cache compression based pyramidal information funneling. arv preprint ar arXi::2406.00 02066,, 2024.
Guoxuan Chen, Han Shi, Jiawei Li, Yihang Gao, Xiaozhe Ren, Yimeng Chen, Xin Jiang, Zhenguo Li, Weiyang Liu, and Chao Huang. Seplim: Accelerate large language models by compressing one segment into one separator. ar Xiv preprint ar arXio:2412.1209. 2024.
Shouyuan Chen, Sherman Wong, Liangjian Chen, and Yuandong Tian. Extending con- text window of large language models via positional interpolation. arXiv preprint arXiv:2306.1559) 2023.
Zihang Dai, Zhilin Yang, Yiming Yang, Jaime Carbonell, Quoc V Le, and Ruslan Salakhut- dinov. Transformer-xl: Attentive language models beyond fixed-length context. ar Xiv preprint arXic:1901.000 00211100,.,00,,.... 2019.
Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, Shawn Presser, and Connor Leahy. The pile: An 800gb dataset of diverse text for language modeling. https://pile. eleuther. ai/, 2020. EleutherAT.
Suyu Ge, Yunan Zhang, Liyuan Liu, Minjia Zhang, Jiawei Han, and Jianfeng Gao. Model tells you what to discard: Adaptive kv cache compression for 1lms. arXiv preprint arXiv=2310.01801, 2023.
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song. Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint ar XXio:25001.12453 2025.
Xingyang He, Jie Liu, and Shaowei Chen. Task-kv: Task-aware kv cache optimization via semantic differentiation of attention heads. arXiv preprint arXiv:2501.15113, 2025.
Coleman Hooper, Sehoon Kim, Hiva Mohammadzadehh, Monishwaran Maheswaran, June Paik, Michael W Mahoney, Kurt Keutzer, and Amir Gholami. Squeezed attention: Accel- erating long context length llm inference. XrXiv preprint ar Xiv::444 2tt::2411.00688 2024.
11

---

Published as a conference paper at COLM 2025
Cheng-Ping Hsieh, Simeng Sun, Samuel Kriman, Shantanu Acharya, Dima Rekesh, Fei Jia, Yang Zhang, and Boris Ginsburg. Ruler: What's the real context size of your long-context language models? Xiv preprint ar arXio:2404.06654, 2024.
Gregory Kamradt. Needle in a haystack pressure testing Ilms. https://githubb.cm com/
SiaarratttLLL--Veeeee-tttttttaa 2023. GitHub repository.
Anis Koubaa. Gpt-4 vs. gpt-3.5: concise showdown. Preprints. org, 2023030422, 2023.
Yuhong Li, Yingbing Huang, Bowen Yang, Bharat Venkitesh, Acyr Locatelli, Hanchen Ye, Tianle Cai, Patrick Lewis, and Deming Chen. Snapkv: Llm knows what you are looking for before generation. Advances in Neural Information Processing Systems, 37:22947-229971 2025.
Guangda Liu, Chengwei Li, Jieru Zhao, Chenqi Zhang, and Minyi Guo. Clusterkv: Ma- nipulating I1m kv cache in semantic space for recallable compression. arXiv preprint arXi::2412.0322 2024.
Xiang Liu, Zhenheng Tang, Hong Chen, Peijie Dong, Zeyu Li, Xiuze Zhou, Bo Li, Xum- ing Hu, and Xiaowen Chu. Can Ilms maintain fundamental abilities under kv cache compression? ar arviv preprint arXiv:2502.01941, 2025a.
Xiang Liu, Zhenheng Tang, Peijie Dong, Zeyu Li, Bo Li, Xuming Hu, and Xiaowen Chu. Chunkkv: Semantic-preservingg kv cache compression for efficient long-context Ilm infer- ence. arXiv preprint ar arXiv:2502.00299 2025b.
LMSYS. Longchat-7b-w1.5-32k. 2024. Accessed: 2025-03-26.
http://///gggiiiiieeeoooo//////m 5-3k,,
Bonan Min, Hayley Ross, Elior Sulem, Amir Pouran Ben Veyseh, Thien Huu Nguyen, Oscar Sainz, Eneko Agirre, Ilana Heintz, and Dan Roth. Recent advances natural language processing via large pre-trained language models: A survey. ACM Computing Surveys, 56 (2):1-40, 2023.
OpenAL. O1, 2023. URL
com/////... from httt:s//opeeaa.ccom////
Matanel Oren, Michael Hassid, Nir Yarden, Yossi Adi, and Roy Schwartz. Transformers are multi-state rnns. ar Xiv preprint ar Xxxx:2401.06104, 2024.
Jack Rae, Anna Potapenko, Siddhant M Jayakumar, and Timothy P Lilliccap.. Compressive transformers for long-range sequence modelling. arviv preprint ar arXiv:1911.05500 2019.
Luohe Shi, Hongyi Zhang, Yao Yao, Zuchao Li, and Hai Zhao. Keep the cost down: A review on methods to optimize Ilm's kv-cache consumption. ar preprint ar arXio:2407.18003, 2024.
Hanshi Sun, Li-Wen Chang, Wenlei Bao, Size Zheng, Ningxin Zheng, Xin Liu, Harry Dong, Yuejie Chi, and Beidi Chen. Shadowkv: Kv cache in shadows for high-throughput long-context Ilm inference. Xrv preprint ar arXio:2410.21465, 2024.
Jiaming Tang, Yilong Zhao, Kan Zhu, Guangxuan Xiao, Baris Kasikci, and Song Han. Quest: Query-aware sparsity for efficient long-context Ilm inference. arXiv preprint ar 4rX1::2406.10774, 2024.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances neural informa- tion processing systems, 30, 2017.
Chaojun Xiao, Pengle Zhang, Xu Han, Guangxuan Xiao, Yankai Lin, Zhengyan Zhang, Zhiyuan Liu, and Maosong Sun. Infllm: Training-fre free long-contert extrapolation for Iims with an efficient context memory. Advances in Neural Information Processing Systems, 37: 119638-11966 119661, 2024.
12

---

Published as a conference paper at COLM 2025
Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. Efficient stream- ing language models with attention sinks. arvio preprint ar Xrio:2309.174553 2023.
Dongjie Yang, XinoDong Han, Yan Gao, Yao Hu, Shilin Zhang, and Hai Zhao. Pyramid- infer: Pyramid kv cache compression for high-throughput Ilm inference. ar Xiv preprint ar arXiv:2405.1253/ 2024.
Zhenyu Zhang, Ying Sheng, Tianyi Zhou, Tianlong Chen, Lianmin Zheng, Ruisi Cai, Zhao Song, Yuandong Tian, Christopher Ré, Clark Barrett, et al. H2o: Heavy-hitter oracle for efficient generative inference of large language models. Advances in Neural Information Processing Systems, 36:34661-34710, 2023.
Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yinggian Min, Beichen Zhang, Junjie Zhang, Zican Dong, al. A survey of large language models. Xiv preprint ar XrXo:2303..8822, 122), 2023.
Xiabin Zhou, Wenbin Wang, Minyan Zeng, Jiaxian Guo, Xuebo Liu, Li Shen, Min Zhang, and Liang Ding. Dynamickv: Task-aware adaptive kv cache compression for long context Ilms. Xiv preprint arXiv:2412.14838 2024.
Yang Zhou, Hongyi Liu, Zhuoming Chen, Yuandong Tian, and Beidi Chen. Gsm-infinite: How do your Iims behave over infinitely increasing context length and reasoning com- plexity? ar Xiviv preprint ar arXin:2502.05252, 2025.
13

---

Published as a conference paper at COLM 2025
A Appendix
A.1 Memory Usage Calculation at Decoding Stage
In modern transformer-based large language models (LLMs), each input token typically produces a key (K) and value (V) vector for every attention head in each layer. As the context length grows, the dimension of the KV cache stored in the GPU memory also grows linearly. Formally, let L be the initial prompt length, M the number of Transformer layers, and H the number of attention heads, each with dimension At each decoding step f, the GPU memory cost, denoted by Cost(t), increases as follows:
Cost(f) (M H (L (L+
where (L + f) indicates the total number of processed tokens (L tokens in the prompt plus t tokens generated). When is very large (e.g,, thousands of tokens), this memory cost can quickly exceed the available GPU capacity, leading to performance degradation or even inference failure.
A.2 SentenceKV; pseudocode
Algorithm
SentenceKV Attention Mechanism
Require: Prompt tokens, token budget semantic keeping factor observation window
size
1: Prefilling Phase:
2: Split prompt into sentences based on punctuation or linguistic cues
3: for each sentence do
4: Compute token importance a; for tokens in s using the last tokens
5: Retain top [r- T] tokens based on a; and discard the rest
6: Compute mean key vector ks,, for each head h (Eq. 1)
7: Offload small subset (r 7) of tokens to CPU for retrieval
8: end for
9:
Decoding Phase:
11: Initialize an empty sentence cache Qs
for 1,2,... do
13:
Generate next token x and compute query 9:
14:
Append q: to Q₂
15:
Compute o for the current sentence (Eq. 2)
16:
Compute similarity (q, ks,n) for each sentence bucket
17:
Retrieve top-T tokens from most relevant sentence buckets by similarity Load keys and values of the corresponding T tokens to GPU from CPU
18:
O = softmax qK_
19:
V (Eq.
if a sentence boundary reached then Reset Q, for the next sentence
21:
22: end if
23:
end for
A.3 Latency and Memory Footprint
For the Lon5chat-v1.5-7B LMSYS (2024) model, SentenceKV similarly demonstrates consid- erable efficiency gains. The peak memory footprint at the maximum context length (256k) is reduced from the OOM (out-of-memory scenario encountered by Full KV caching to manageable 48.31 GB. Latency also remains stable around 16.9 ms, whereas Full KV caching becomes infeasible beyond 64k tokens due to excessive memory demands. The details of the results are shown in the Appendix Table and Table 5.
14

---

Published as a conference paper at COLM 2025
Table 4: Latency and memory usage, Results are from the a token budget of r 1024.
model with
Latency (ms/token)
Method
16K
32K
64K
128K
256K
Full KV
16.8ms
20.2ms
29.6ms
47.9ms
84.9ms
SnapKV 1024
16.5ms
16.5ms
16.5ms
16.5ms
16.5ms
SentenceKV 1024
17.8ms
17.8ms
17.7ms
17.8ms
17.8ms
Memory Usage (GB)
Method
16K
32K
64K
128K
256K
Full KV
19.78GB
24.45GB
33.77GB
52.42GB
89.71GB
SnapKV 1024
17.47GB
19.82GB
24.52GB
33.92GB
52.71GB
Se 1024
17.47GB
19.83GB
24.54GB
33.92GB
52.71GB
Table 5: Latency and memory usage. Results are from the Longchat-v1.5-7B model with
token budget of T 1024.
Latency (ms/token)
Method
16K
32K
64K
128K
256K
Full KV
29.2ms
47.3ms
84.2ms
OOM
OOM
SnapKV 1024
16.5ms
16.5ms
16.5ms
16.5ms
16.5ms
SentenceKV 1024
16.6ms
16.6ms
16.5ms
16.8ms
16.9ms
Memory Usage (GB)
Method
16K
32K
64K
128K
256K
Full KV
27.96GB
43.26GB
73.88GB
OOM
OOM
SnapKV 1024
15.18GB
17.73GB
21.52GB
30.45GB
48.30GB
SentenceKV 1024
15.18GB
17.73GB
21.53GB
30.45GB
48.31GB
A.4 Quest Sensitivity to Chunk Size
The Quest Tang et al. (2024) cache- cache-offloading baseline divides the context into fixed-length chunks and retains only a subset per chunk. Prior work typically uses a chunk size of tokens, but this granularity differs markedly from SentenceKV, where one bucket naturally corresponds to a full sentence (~-5--30 tokens on our corpora). To ensure a fair comparison NTAH Kamradt (2023), we therefore re-evaluated Quest with larger chunk sizes that better match the average sentence length.
Method
Quest (Chunk Size 32)
Quest (Chunk Size 16)
SentenceKV (Adaptive)
NIAH Acc. %
48.3
96.1
97.5
Table 6: Quest results on NIAH under different settings. Model: Llama-3.1-8B with token budget - = 128.
Quest proves highly sensitive to chunk granularity: reducing the chunk size from 32 to 16 tokens nearly doubles accuracy (Table 6). In contrast, SentenceKV employs adaptive buckets (median length 25-30 tokens) and attains 97.5 % accuracy with no manual tuning, highlighting the robustness of sentence-aligned grouping.
A.5 Efficiency/Accuracy Trade Off
Profiling 64 k-token niah-single input on H100-96 GB NVL shows: prefill bucket con- struction costs 4 once; each decoding step adds 0.019 s (GPU-CPU 2048 tokens), 0.0038 s (CPU~>GPU 1024 tokens), and 0.050 for similarity ranking, totalling 0.073 These
15

---

Published as a conference paper at COLM 2025
overheads are negligible relative to the > 60 ms gap between SentenceKV and Full KV, con- firming favourable securacy-laeency-mmmmoo balance for real-world long-context serving. Fixing T=128 and semantic- keeping factor r=3 on NIAH, SnapKV decodes in 15 ms per token; SentenceKV needs only +1 ms extra yet boosts accuracy by 25% (Table 7). Such minor latency growth is often preferable to the sharp accuracy drop that accompanies more aggressive compression.
SnapKV
SentenceKV
Increase
Latency (ms/token)
15
16
6%
NIAH Acc. (%)
78.2
97.5
+25%
Table 7: Trade-off on NIAH (T = 128, -= 3).
A.6 GPU Profiling Details
To illustrate how SentenceKV achieves reductions in both memory and latency, we present fine-grained GPU profile showing (i) the set of KV cache blocks retained in GPU memory during decoding, and (ii) the breakdown of decoding time 64K-token context length.
Memory composition at 256 K context (Llama-3.1-8B), Peak usage reported by torch. cuda. max memory
m allocated() drops from 89.71 GB (Full KV) to 52.71 GB with Sen- VV The residual footprint consists of:
(1) KV cache (subset only): 33.55 GB > 0.10 GB after retrieving just ~ 1K top tokens per head.
(2) Sentence-level semantic vectors: tens of MB; negligible relative to the model.
(3) Temporary tensors: short-lived attention buffers, now smaller because attention com- puted on far fewer keys/ /value.
(4) Model weights: feed-forward and projection matrices dominate the remaining 50+ GB; they stay on-GPU for all methods.
The reported peak includes allocator fragmentation and transient buffers, yet the relative 37 GB reduction clearly evidences SentenceKV's compression efficacy.
Latency breakdown at 64 K context (RULER niah.single). Profiling on an H100-96 GB NVL with = 1024, к yields:
(1) Prefill bucket construction (one-time): 4 s for sentence grouping and top-k token selection.
(2) CPU+ transfer (per step): 0.0191 offload 2048 tokens + 0.0038 s onload 1024=0.0229 s.
(3) Sentence ranking (per step): 0.050 (CPU dense similarity search).
Even if the 0.073 per-step overhead is conservatively added to the computation latency shown in Figure 5, SentenceKV remains markedly faster than Full KV because drastically fewer keys and values participate in attention.
In summary, the large KV-cache shrink (33.55 GB- CBB->0.10 GB) alleviates the memory bottleneck in long-context attention, simultancously cutting decoding time while staying within modest GPU capacity.
A.7 Effect of Sentence Length
SentenceKY ranks sentences by relevance until the remaining token budget T is exhausted. One possible drawback is that an unusually long but highly relevant sentence could consume a disproportionate share the budget, thereby excluding other useful context.
To address this, apply SnapKV-style Li et al. (2025) prefiltering globally across all input tokens during the prefill stage. Specifically, instead of selecting top tokens from each sentence individually, SnapKV selects the top r. most important tokens from the entire
16

---

Published as a conference paper at COLM 2025
document, regardless of sentence boundaries. As result, the number of retained tokens per sentence is determined by its relative semantic contribution, rather than its raw length. Even if a sentence is long, it will retain fewer tokens if it is not highly informative.
In practice, this global filtering strategy significantly reduces token retention from long but unimportant sentences. Our empirical analysis of LongBench shows that sentence lengths are generally concentrated around the mean, with very long sentences being rare. After filtering, token counts per sentence are typically modest and broadly distributed, helping to avoid scenarios where a single sentence dominates the token budget.
Future work could include detecting abnormally long retained token spans ("buckets") during prefill and applying simple mitigation strategies. For example, one could compute the mean and standard deviation of sentence lengths in the input, and if a sentence exceeds mean + n x std, split it into smaller sub-spans. Such lightweight check would add mini- mal computational overhead while preventing outlier buckets from skewing the retrieval process.
17


Reviews:
[{"review_id": "6iYQy66aLv", "reviewer": "Reviewer_pn6t", "summary": "The paper proposes a sentence-level KV caching method to improve inference efficiency while maintaining semantic coherence. During the pre-filling stage, this paper groups tokens according to sentence-level semantics and then compresses the tokens within a sentence into semantic vectors. These semantic vectors are directly on the GPU. By offloading individual key-value pairs to the CPU, during the decoding stage, attention operations are performed through compressed vectors on the GPU, thereby accelerating attention operators while reducing memory overhead.", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper is well-written. The idea is simple and effective.", "reasons_to_reject": "Although the experiments can prove the effectiveness of the proposed method, the paper lacks a comparison with some typical offloading methods, such as InfLLM and KTransformer. Adding more offloading baselines can make the paper's results more solid.", "questions_to_authors": "Please refer to 'Reasons To Reject'.", "extra_scores": {}}, {"review_id": "ZwbngfhkG6", "reviewer": "Reviewer_W49T", "summary": "The paper proposes to use sentence-level semantic KV cache to improve the efficiency of LLM decoding. Particularly, the paper organizes buckets of sentence-level KV cache during the prefilling phase. During decoding, the paper proposes to decode one sentence at a time, and for each sentence, only related KV cache is loaded into the GPU to assist the decoding. By doing so, the decoding of each sentence only requires limited context, and the efficiency can be improved. On long-context tasks, the paper shows that the proposed method has no significant increase in latency as the context length grows, while preserving the performance of the LLM.", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper proposes a novel idea for LLM decoding where KV cache is handled at the sentence level.\n2. By the design of the method, the context required for decoding could be better controlled and restricted by using only most relevant tokens.\n3. The paper shows a good efficiency boost as context length increases as well as maintaining the performance.", "reasons_to_reject": "1. A more detailed analysis of the overhead could be added. I.e., what is the cost of handling the buckets during the prefilling phase, what is the latency of loading KV cache from CPU to GPU, and what's the cost of ranking the retrieved tokens from CPU?\n2. As also mentioned in the paper, the semantic factor is now a fixed value for the proposed method, limiting the ability to generalize to highly variable sentence lengths or semantic meaning diversities.\n3. The proposed method has a different efficiency-peroformance trade-off compared with SnapKV. It is not completely clear that the proposed method's tradeoff is dominating over SnapKV.", "questions_to_authors": "1. The algorithm in Appnedix is a bit confusing. It seems that Ot is only computed when the end of the sentence is reached.\n2. Is the proposed method applied to all attention layers?", "extra_scores": {}}, {"review_id": "HLRg7SxRJs", "reviewer": "Reviewer_tRUR", "summary": "This paper proposes SentenceKV, a framework for sentence-level KV cache compression aimed at improving decoding speed and reducing GPU memory usage during LLM inference. In the prefill stage, the input sequence is segmented at the sentence level. Token-level importance scores are then computed based on attention weights, and less important tokens are discarded. The sentence-level representations of the retained KV caches are computed and kept on the GPU, while the full retained KV caches are offloaded to CPU memory. During decoding, SentenceKV operates at the sentence level as well: for each sentence being decoded, a mean query vector is computed by averaging the query vectors of the decoded tokens in the current sentence. This vector is used to retrieve relevant sentence representations from the CPU back to the GPU. The authors conduct extensive experiments on PG19, LongBench, and NIAH datasets. Results demonstrate that SentenceKV preserves task performance while significantly reducing GPU memory consumption and improving inference speed. SentenceKV outperforms several baselines, including H2O, SnapKV, and Quest, across a variety of tasks.", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Here are my reasons to accept this paper:\n\n---\n\n1. The paper is written in a clear and elegant style. The problem formulation is well-motivated, and the methodology is explained in a way that is easy to follow. The experimental section provides sufficient detail to support reproducibility, which adds to the soundness of the authors' claims.\n\n---\n\n2. Benchmark results demonstrate that SentenceKV consistently outperforms classical baselines such as H2O, SnapKV, and Quest. It maintains competitive performance on LongBench with no significant degradation, and it clearly surpasses all baselines on the NIAH benchmark. The ablation studies convincingly justify the use of sentence-level KV cache optimization.\n\n---\n\n3. While not emphasized as a primary contribution, the integration of KV cache eviction with sentence-level offloading is a novel design. In the prefill stage, KV cache entries with low token importance are discarded, while during decoding, sentence-level retrieval determines which caches are retrieved to GPU memory. This combines sequence-level and sentence-level optimization: less relevant caches are discarded with the awareness of the complete input sequence, and retrieval is dynamically conditioned on the decoding sentence. This hierarchical strategy likely contributes to SentenceKV's strong performance, and represents an innovative approach to memory-efficient inference.", "reasons_to_reject": "Here are my concerns about this paper. With more thorough discussion and additional experimental results, I would be inclined to increase my overall rating.\n\n---\n\n1. Although H2O, SnapKV, and Quest are classical baselines, there exist more recent methods targeting similar scenarios that should be considered for comparison. For example, ShadowKV [1] is mentioned only in the related work section but is not included in the experimental comparisons. I suggest the authors include at least one modern KV cache offloading framework as a baseline for a more comprehensive evaluation.\n\n---\n\n2. The selected benchmark datasets are limited. LongBench, in particular, is relatively short—most of its entries are under 16K tokens—making it less suitable for evaluating performance in truly long-context scenarios. It would be more convincing to evaluate SentenceKV on longer and more challenging benchmarks, such as RULER [2], InfiniteBench [3], or SCBench [4]. Including an additional experiment that compares SentenceKV and baseline methods on one of these datasets would help strengthen the paper’s empirical claims.\n\n---\n\n3. The size of the test models are limited (7B and 8B). It would be better to conduct experiments on smaller models (1.5B or 3B) or larger models (14B or 32B).\n\n---\n\n4. Writing issues. (See questions.)\n\n\n---\n\n[1] ShadowKV: KV Cache in Shadows for High-Throughput Long-Context LLM Inference\n\n[2] RULER: What's the Real Context Size of Your Long-Context Language Models?\n\n[3] ∞Bench: Extending Long Context Evaluation Beyond 100K Tokens\n\n[4] SCBench: A KV Cache-Centric Analysis of Long-Context Methods", "questions_to_authors": "Here are my questions:\n\n---\n\n1. From my understanding, conducting KV cache optimization at the sentence granularity could be particularly beneficial for long chain-of-thought (CoT) decoding, as each sentence typically focuses on a different aspect of the previously generated content. For example, a simple test using DeepSeek's distilled versions of the Qwen models [1] in a long-CoT setting could provide useful insights. For both long and challenging reasoning tasks, GSM-Infinite [2] may serve as an appropriate benchmark.\nWhile this is not a mandatory request, I encourage the authors to at least include a discussion on this direction. Providing additional experimental results along these lines, if feasible, could significantly broaden the scope and impact of the work.\n\n---\n\n2. When retrieving sentences from the CPU, SentenceKV follows a strategy of selecting sentences in order of relevance until the token limit is reached. However, this design overlooks the variation in sentence lengths and the number of retained tokens per sentence. For instance, if a highly relevant sentence is particularly long and contains many retained tokens, it may occupy most or all of the available space, leaving no room for other relevant but shorter sentences.\nThis raises the question: Does sentence length or the number of retained tokens per sentence affect the overall performance of the retrieval strategy? It would be valuable for the authors to analyze or discuss how this design choice impacts retrieval quality and task performance, especially in cases with imbalanced sentence lengths.\n\n---\n\n3. Minor writing issues. In line 28, the citation should use \"\\citep{}\". Similar citation formatting issues appear multiple times throughout the paper. The top of Figure 1 exceeds the upper margin of the page. The font size in Figure 1 and Table 1 is too small to read.\n\n---\n\n\n[1] DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning\n\n[2] GSM-Infinite: How Do Your LLMs Behave over Infinitely Increasing Context Length and Reasoning Complexity?", "extra_scores": {}}, {"review_id": "JAlY2uO9fG", "reviewer": "Reviewer_ecqc", "summary": "This paper proposes SentenceKV, a sentence-level semantic caching method to improve long-context LLM inference efficiency. \n\nSentenceKV groups tokens into sentences during the prefilling stage, computes mean key vectors for each sentence, and keeps these semantic vectors on GPU. During decoding, it retrieves relevant sentence KV pairs from CPU based on the similarity between the current generation's aggregated query vector and the stored sentence vectors. \n\nWhile the idea of leveraging sentence-level semantics is intuitive and potentially valuable, I found several critical issues in both the system evaluation and the correctness of the experimental setup that, in my opinion, make the current version not ready for publication.", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The core insight that sentences are natural semantic units is sound. Compared to fixed-size token chunks, sentence-based grouping can potentially lead to more coherent context retrieval.\n- The method is relatively simple to implement and avoids heavyweight techniques like clustering or learned memory routing. The use of mean query vectors and semantic similarity scoring makes the approach interpretable.\n- Even if some aspects need refinement, the idea of abstracting KV cache to higher-level units could spark further research in retrieval-based inference.", "reasons_to_reject": "- The reported latency reduction is surprisingly large. In Figure 5, SentenceKV reportedly brings down decoding latency from 84.9ms to 17.8ms at 256K tokens, while GPU memory usage goes from 89.71GB to 52.71GB. This does not align with our understanding of decoding in LLMs (memory bound). Memory usage is reduced by ~40%, but latency is reduced by almost 5x?  This undercuts the claimed efficiency gains. A deeper breakdown (e.g., profiling results, CPU-GPU communication time, kernel latency) is needed to make these results convincing.\n- Figure 4 shows that Quest performs poorly on the NIAH benchmark with only 48.3% accuracy, while SentenceKV hits 97.5%. This is very surprising. In prior work such as [1], Quest performs reasonably well on needle-style retrieval tasks (RULER Single Needle Retrieval). It is a bit unclear that what setup was used here?\n- To validate the effectiveness of retrieval-based caching, a single benchmark like NIAH isn't enough—especially when Quest is performing unexpectedly poorly. I strongly recommend including RULER or similar retrieval-intensive tasks that are better established in the literature.\n- In Line 184, the paper mentions selecting top-$r\\times \\tau$ tokens by attention importance. I am a bit confused: Are these head-specific selections (i.e., each attention head has its own important tokens) or is the importance aggregated across KV head groups or is it shared for all heads?\n- The sentence bucket mechanism is supposed to save memory by storing only compact sentence-level vectors on GPU and offloading full KV to CPU. However, in Figure 5, the memory savings from FullKV to SentenceKV is from 89.71GB to 52.71GB. So what exactly remains on GPU during decoding? Are the semantic vectors that large?\n\n[1] ShadowKV: KV Cache in Shadows for High-Throughput Long-Context LLM Inference", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper presents SentenceKV, a sentence-level semantic KV caching method to improve inference efficiency for long-context LLMs. Overall, the idea is novel and addresses a practical challenge in memory-efficient LLM inference. The reviewers acknowledge the paper’s strong technical merits and improvements made during the rebuttal phase.

Pros:
- Innovative Methodology: All reviewers appreciated the sentence-level abstraction for KV cache management, which aligns well with semantic structures and enables more efficient inference.
- Reviewer W49T commended the consistent performance and strong memory savings with modest latency cost.
- Reviewer tRUR appreciated the integration of semantic abstraction and cache eviction, noting it as a strong design innovation.
- Reviewer ecqc initially had concerns, but ultimately acknowledged that the authors’ detailed clarifications resolved all doubts.
- Reviewer Concerns & Resolution:

Concerns:
- Lack of comparison with InfLLM, KTransformer, shadowkv	
- Overhead and latency analysis incomplete	
- Evaluation on limited benchmarks
- Questions on sentence-level granularity trade-offs	
- Lack of results on other model sizes	

The authors were responsive and addressed every major concern. SentenceKV is a well-executed and timely contribution to LLM inference research. With the promised additions in the camera-ready version, the paper is ready for publication. However, the authors have promised the following updates, which should be included in the final version:
- Full experimental results for InfLLM and ShadowKV.
- Comparisons on Larger models.
- GSM-Infinite benchmark.
Decision: Accept