Dialogue Without Limits: Constant-Sized KV Caches for Extended Response in LLMs
Authors: Ravi Ghadia
Avinash Kumar
Gaurav Jain
Prashant J. Nair
Poulami Das
Venue: ICML 2025 poster (2025)
Keywords: Large Language Model
Key-Value Cache Compression
Primary area: deep_learning->large_language_models
TL;DR: An effective KV compression technique for long-response benchmarks

Abstract: Autoregressive Transformers rely on Key-Value (KV) caching to accelerate inference. However, the linear growth of the KV cache with context length leads to excessive memory consumption and bandwidth constraints. Existing methods drop distant tokens or compress states in a lossy manner, sacrificing accuracy by discarding vital context or introducing bias.

We propose ${MorphKV}$, an inference-time technique that maintains a constant-sized KV cache while preserving accuracy. MorphKV balances long-range dependencies and local coherence during text generation. It eliminates early-token bias while retaining high-fidelity context by adaptively ranking tokens through correlation-aware selection. Unlike heuristic retention or lossy compression, MorphKV iteratively refines the KV cache via lightweight updates guided by attention patterns of recent tokens. This approach captures inter-token correlation with greater accuracy, which is crucial for tasks like content creation and code generation. Our studies on long-response tasks show 52.9\% memory savings and 18.2\% higher accuracy on average compared to state-of-the-art prior works, enabling efficient deployment.

Dialogue Without Limits: Constant-Sized KV Caches for Extended Response in LLMs
Ravi Ghadia
Avinash Kumar
Gaurav Jain"
Prashant Nair
Poulami Das
Abstract
KV cache (GB)
64K
10K
H100 HBM CAPACITY
Autoregressive Transformers rely on Key-Value (KV) caching to accelerate inference. However, the linear growth of the KV cache with context length leads to excessive memory consumption and bandwidth constraints. Existing methods drop distant tokens or compress states in lossy man- ner, sacrificing accuracy by discarding vital con- text or introducing bias.
1K
70B
100
10
32
64
128
1K
4K
16K
32K
Tokens Processed
Figure 1. KV cache sizes for the Llama 3.1 70B and 405B models across varying sequence lengths with a batch size of 256.
We propose MorphKV, an inference-time tech- nique that maintains constant-sizer KV cache while preserving accuracy. MorphKV balances long-range dependencies and local coherence dur- ing text generation. It eliminates early-token bias while retaining high-fidelity context by adaptively ranking tokens through correlation-awar selec- tion. Unlike heuristic retention or lossy com- pression, MorphKV iteratively refines the KV cache via lightweight updates guided by atten- tion patterns of recent tokens. This approach cap- tures inter-token correlation with greater accuracy, which is crucial for tasks like content creation and code generation. Our studies on long-response tasks show 52.9% memory savings and 18.2% higher accuracy on average compared state-of- the-art prior works, enabling efficient deployment.
Normalized Scores
KV cache (GB)
10K
OUT OF MEMORY
1.2
SnapKV
MorphKV
1.0
SnapkV MorphiV
0.8
100
0.4
5K
15K
25K
Long-Contekt Song-Response
Response Length
Benchmark
(a)
(b)
Figure 2. (a) Despite compression, the state-of-the-art SnapKVV memory footprint increases with response length and exceeds available HBM capacity even high-end GPUs. This study uses Qwen
model on an NVIDIA H100 and Long- Writer benchmark. (b) Even at lower memory capacity, MorphKV achieves higher accuracy than SnapKV for long-response tasks.
However, this progress hindered by the memory overhead of Key-Value (KV) caches. KV caches store the key-value pairs to enable attention mechanisms for auto-regressive de- coding during LLM inference. Unfortunately, shown in Figure 1, the KV cache size grows with sequence length, of- ten exceeding the memory capacity of even high-end GPUs.
1. Introduction
Large Language Models (LLMs) have become indispens- able for tasks requiring extensive context retention (e.g., document summarization) and prolonged text generation (e.g., code synthesis). As model architectures become phisticated, their ability to process nuanced inputs and pro- duce coherent, long-form outputs has improved dramatically.
The distinction between long-context and long-response tasks lies in the phase where token processing dominates. Long-context tasks, such document summarization and prompt comprehension, primarily process large volume of input tokens during the prefill phase, where the model ingests and encodes the initial prompt. In contrast, long- response tasks, such as essay writing and code generation, generate substantial number of output tokens during the decode phase, requiring sustained attention over growing sequences of self generaedde tokens.
'Dpaarmment of Electrical and Computer Engineering, Univer- sity of Texas Austin, Texas, USA 'D-Matrix University of British Columbia, USA. Correspondence Ravi Ghadia <rgha- dia@ utexas.cdu>..
Proceedings of the 4nnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Dialogue Without Limits: Constant-Sized KV Caches for Extended Response in LLMS
Prefill
Decode
Recent
Token
Current
Token
Evicted
Only most recent tokens
Limited initial (old) lokens
Bias toward early tokens
Bias toward early tokens
Seleet old tokens
All output
over time (d) Keyformer
tokens
Relevant old tokens retained
(a) Scissorhands
(h) Streamingt.I.M
(c) H₂O
(e) SnapKV
(f) MorphKV
Figure 3. Illustrative comparison of KV cache reduction methods tokens are processed. (a) Scissorhands retains only window of recent (shown in green) tokens, (b) StreamingLLM also stores few initial tokens (old shown in yellow) from the prefill step, and (c) HzO stores even more old tokens (all prompt tokens) and only relevant recent tokens. (d) Keyformer stores only important old and recent tokens but remains biased towards the early tokens. (c) SnapK V retains selected old tokens from prefill and all decode (more recent)
tokens (I) MorphKV identifies and stores only those old tokens that correlate with recent tokens.
Numerous approaches have been proposed in the literature to minimize the impact of growing KV cache sizes. In stud- ies like FlashAttention (Dao, 2023) and vLLM (Kwon et al., 2023), the authors propose techniques to either materialize only partial caches at time use paging techniques by fragmenting KV caches into smaller blocks, thereby avoid- ing the need to reserve memory for the entire cache at once. Beyond this, prior works like FastGen (Ge et al., 2023) and MiKV (Yang et al., 2024) compress KV caches by retaining only subset of KV pairs from recent and older tokens, pri- oritizing those deemed important based attention scores and discarding the rest. However, this creates a trade-off: while memory savings increase as more KVs are discarded, the accuracy depends on the retained KVs effectively captur- ing context for future tokens. Consequently, these methods often sacrifice accuracy for reduced memory usage.
of-the-art, achieves high accuracy in long-context tasks by retaining the most attended tokens from the input prompt. However, shown in Figure 2(a), it retains all generated output tokens from the decode phase, causing the KV cache size to scale with response lengths, making it unsuitable for long-response tasks. Overcoming these limitations is crucial for improving L.MM performance in applications like scripting and content creation (long-response tasks).
Our Proposal - MorphKV: MorphKV achieves a constant- size KV cache by retaining only limited number of old and recent tokens. However, to achieve higher accuracy, MorphKV employs more dynamic KV selection algorithm that analyzes the attention patterns of the current token to- ward retained KVs. Unlike prior methods that independently identify important tokens, MorphKV retains only those old tokens that correlate strongly with recent tokens.
For example, as shown in Figure 3, Scissorhands (Liu al., 2024b) retains only the KVs of recent tokens, sacrificing accuracy by discarding past context. StreamingI.LMM (Xiao et al., 2023) improves accuracy slightly by preserving KVs of a few initial tokens (attention sinks) alongside recent tokens but struggles when early tokens fail to capture suffi- cient context. H2O (Zhang et al., 2023) retains KVs from the entire input prompt and the most attended output tokens, achieving high performance but reduced memory savings due to the large number of retained KVs. It also suffers from selection bias during decoding, preserving unimportant past KVs, which hinders performance in long-response tasks.
To better capture context, MorphKV prioritizes the attention scores of relevant recent tokens rather than relying histor- ically most-attended tokens, addressing bias issues observed in methods like H₂O and Keyformer. As shown in Fig- ure 2(b), MorphKV achieves better scores than SnapK for Phi4, up to 8% higher for long-context task (VCSum) while saving 56% on KV cache memory, and for Qwen2.5, up to 21 higher score for the long-response task (LongGen- Bench) while saving 83% on KV cache memory. This shows the impact of retaining a compact set of high-qualit, KVs and an improved attention mechanism in MorphKV.
MorphKV improves accuracy by 9.4% and 18.2% on av- erage compared to SnapKV and H₂O while reducing the KV cache footprint by 88.1% and 52.9% respectively for long-response tasks.
Keyformer (Adnan et al., 2024) selects top old and re- cent tokens for memory savings and uses Gumbel noise to reduce selection bias. While somewhat effective, it cannot entirely eliminate selection bias because Gumbel noise by itself introduces new forms of biases (Mussmann et al., 2017), reducing accuracy for long-con ann-cootext and long- response tasks. SnapKV (Li et al., 2024), the current state-
MorphKV is now open-source, and accessible at https: //gi //github. com/
2

---

Dialogue Without Limits: Constant-Sized KV Caches for Extended Response in LLMS
2. Background and Motivation
3. MorphKV
2.1. Large Language Model Inference
This paper proposes MorphKV, a KV compression technique that reduces the KV cache size without compromising accu- racy. MorphKV partitions the context into two components: recent context (R) and distant context (D). The recent con- text R corresponds to the last R tokens that preserve local coherence, while the distant context D captures long-range dependencies. By attending to both R and selective subset of DD MorphKV ensures that the generated text remains contextually coherent and semantically meaningful.
LLM inference begins with the prefill step, where the model processes the input prompt and generates Key- Value (KV) pairs for each token in the prompt. Next, in the decode phase, the model generates output tokens auto-regressively such that each output token attends to the KV pairs of all preceding tokens in the sequence while creating its own KV pair. This attention mechanism enables the LLM to maintain context and produce coherent responses. The KV pairs are stored in memory structures known as KV caches.
Figure 4 presents an overview of our proposed design. A key insight in MorphKV is that tokens in R have already attended to tokens in D during their generation. Therefore, rather than retaining all or a subset of older tokens based on aggregated patterns, MorphKV leverages the attention profiles of recent tokens to select only the most relevant dis- tant tokens. In this way, MorphKV constructs a compact yet accurate KV cache of size C+ R. where C' is the number of distant tokens retained and R is the number of recent tokens. Specifically, MorphKV ranks older tokens based their relevance to the recent tokens using a specialized algorithm that performs element-wise transformations, denoted by f(x) in Figure 4. As attention scores inherently quantify how strongly past tokens were attended to during prior gen- erations, using the attention scores of the recent tokens helps surface the most contextually relevant older tokens. Next, MorphKV selectively retains only the most correlated old tokens in the KV cache, evicting those deemed irrele- vant. This approach ensures an optimized memory footprint while preserving essential long-range dependencies.
KV caches scale with the number of tokens processed, becoming prohibitively large for long-context and long- response tasks and posing significant challenges in deploy- ing LLMs. Long-context tasks, such as creating diet plans from medical histories or summarizing documents like man- uals, loan agreements, or papers, involve long prompts with many input tokens. contrast, long-response tasks such as crafting lesson plans, providing step-by-step instructions, or writing scripts generate numerous output tokens from short inputs. While both types of tasks require large KV caches, they differ in when the KVs are produced. Long- context tasks generate most KVs in the prefill step, unlike long-response tasks that create most KVs during decoding.
2.2. Limitations of KV Cache Compression Methods
KV cache compression addresses their growing memory footprint through several strategies, such as quantization, algorithmic optimizations, cross-layer and cross-head ap- proaches, and pruning. Quantization-base methods store KV pairs using lower precision (Kang al., 2024; Zhang et al., 2024), whereas algorithmic methods modify attention- layer computations (Chang et al., 2024; Saxena et al., 2024). Cross-layer optimizations leverage inter-layer similarities, selectively retaining KVs from layers with significant con- tributions (Yuan et al., 2024; Saxena et al., 2024; Cai et al., 2024). Cross-head optimization (Fu et al., 2024; Feng et al., 2024) reduces the KV cache footprint by retaining KVs only from the most impactful attention heads, different heads contribute unevenly to model performance. Prun- ing strategies selectively retain a subset of important KV pairs (Adnan et al., 2024; Xiao et al., 2023; Li et al., 2024; Zhang et al., 2023; Liu et al., 2024b) and are more effective than other approaches because they capture specifce context more accurately.
Rank old tokens using attention score of recent tokens
Identify and retain most- correlated older tokens
Current
Token
Old Tokens
Recent
Retainod
Evicted
Tokens
Figure 4. Overview of MorphKV. (1) MorphKV uses the most recent window tokens to capture neighboring context and their attention scores to rank the older tokens. (2) To capture relevant distant context, MorphKV only retains old tokens maximally cor- related to the recent window tokens by consulting the attention scores aggregated using a fusion function f(x).
However, compressed KV caches are either limited by ac- curacy or scalability. Constant-sizee KV caching methods, such as Scissorhands, StreamingLLM, and Keyformer, have limited accuracy. On the other hand, more accurate methods like SnapKV are not scalable as they fail to address the growing KV cache size for long- responss tasks.
3.1. Mathematical Formulation
Let Qi, Kü and V; be the query, key, and value vectors for the token being generated at timestep i. Let G, denote the KV cache storing (K3, V2) pairs for all the previously generated tokens j < i. The standard attention mecha- nism (Vaswani et al., 2023) computes the attention weights
3

---

Dialogue Without Limits:
Constant-Sized KV Caches for Extended Response in LLMS
AW,, as shown in Equation (I):
MorphKV proposes two choices, the and max fusions,
as shown in Equations (5) and (6) respectively:
Qi QiK4 K
AW; = Softmax
O: = AW, - V,, (1)
Vah
i- R
Sum Fusion: F[[M]
AW,[k]
(5)
where K = [K1, K2. Ki- E ( V [Vi. Ve,..., V......VV11]] E Riii)xdd, and d,, is the hidden di- mension. The attention output O E Rdd encodes the infor- mation from all the previous tokens.
r=--1
i-R max r=--
Max Fusion: F([R]
AW,
(6)
The Sum Fusion prefers tokens consistently attended to across multiple recent tokens, whereas the Max Fusion selects tokens strongly attended by at least one recent token. The intuition that tokens frequently or strongly attended to by recent tokens are likely critical for maintaining long- range coherence. For example, recurring entities, such as characters story, often receive sustained attention across multiple decoding steps. Algorithm shows the dynamic token selection process using the auxiliary score vector.
3. PROBLEM STATEMENT
Retaining every key-value pair (K₁, V2) for j < i can in- crease memory usage as i grows. Let GŽ be an optimal reduced cache of size C + R that minimizes the change in the attention output as denoted in Equation (2):
G arg min G -++
|10, 0:
Algorithm MorphKV; Dynamic Token Selection Input:
where O is the attention output, as shown in Equation 1, computed using only the tokens in G. We want ||O; - 0/ S e for a small 2 0, ensuring minimal error despite reducing the KV cache to C C+A R entries.
KV cache G; (keys/values of older tokens)
Tokens Tis (sequence generated so far)
3.1.2. A APPROXIMATING OPTIMAL KV CACIIE
Num of Query-Heads, Key-Heads: {')
Although solving Equation 2 directly is intractable because of its combinatorial nature, MorphKV adopts two intuitive heuristics (HI and H2):
Per-head attention weights (AW") m==1
Window size R, fusion function f. capacity C
Local Coherence (HI): Always retain the last R to- kens to preserve continuity.
Output: Updated cache Gi+1
W; + (ii-1, Ri-R for w, W; do
Recent tokens
Distant Relevance (H2): Retain only the C' most in- formative older tokens, as measured by fused attention scores with respect to the R recent tokens.
S, (M/jM) AWM
Aggregate scores if GQA
end for
F2 < f(Si- Si-,, S
Fuse recent tokens'
Concretely, we define an approximate policy P' Equa- tion (3) as shown below:
scores
G + (FP)(FF) U W. recent
> Retain top-C' distant + R
G = P',, R.), where |G
C + R.
(3)
Return: G
Here, G² S G; contains (1) the R most recent entries G. and (2) the top-C older entries selected based on an auxiliary score vector F..
1.4. SELECTION OF KV PAIRS
After computing F,, we pick the top-C' entries (older tokens) according to F; and combine them with the most recent tokens, as shown in Equation (7):
3.1.3. DEVELOPING THE AUXILIARY SCORE VECTOR F;
Let W. = [w,-1, 2/- w R denote the window of R most recent tokens at timestep i. MorphKV inspects the attention weights of these tokens to build F,, using Equation (4). Specifically,
G = (TTp U R recent tokens)
(7)
Hence, Gi41 contains C + R tokens in total, satisfying heuristics (HI) and (H2). By updating G, + G++1 at each timestep, MorphKV prunes the KV cache incrementally, ensuring that memory usage remains fixed at C + R while preserving essential local and distant context.
F [k] (AW, [k), AW,-2(k), ... A(k) (4)
where AW, [k] is the attention weight that w. assigned to the I--th older token, and f(-) is a fusion function. For f(),,

---

Dialogue Without Limits: Constant-Sized KV Caches for Extended Response in LLMS
Tell me today's weather
Attention Profile (AP)
To
The
loday's 's
weather
The
today's
0.1
0.4
weather
0.3
0.3
0..
The
AP[R]
0.1
0,6
0.0
0.4
0.3
0.
T
6
The sun is our and the bright weather
sun
out
the
bright weather
0.6
K
K
K
K
out
0.4
0.2
the
bright
0.5
0.4
weather
0,3
0-4
0.3
02
0.4
T 12
swn
weather
perfect
for
outdoor sun
0.6
The sum is out and the bright weather makes it perfect for outdoors
K
R
K
V
K
weather
0.3
perfect
for
0.2
10:2
0.2
0.1
Yay!
outdoor
0.0
0.1
Figure 5. Illustration of the Key Value (KV) caching mechanism in MorphKV. MorphKV uses the insight that recent tokens naturally capture some distant context from old tokens due to the auto-regressive nature of token generation. For example, at decoding step To, MorphKV consults the Attention Profile of recent tokens weather and The to learn that these tokens have attended considerably the old token today's 's than me. MorphKV V uses this information and evicts the latter.
3.2. Intuition with A Walk-Through Example
(Vaswani et al., 2023). In MHA with M heads, each head maintains a separate KV cache, resulting in M distinct caches and significant memory overhead. Modern LLMs like Llama3.1 (Dubey et al., 2024) use Crouped-Query At- tention (GQA) (Ainslie et al., 2023) with M' grouped-key heads (M' M), where every M/M' query heads share a single key/value head, reducing the number of KV caches to M' and cutting memory usage by a factor of M M/M".
Figure 5 demonstrates the operations in MorphKV with C' = 2 and R = 2, using sum fusion as the fusion function.
At timestep To: Suppose the recent tokens are [weather, The], while the old tokens are [me, today's 's. The attention profiles (AP) of weather and The both strongly point to today's 's (e.g., 0.3 each) and weakly to me (e.g.. 0.05 each). By summing these attention scores, MorphKV deems to- day's to be more relevant (score 0.6) than (score 0.1). Consequently, it retains today's 's in D and evicts me.
MorphKV is compatible with both MHA and GQA archi- tectures. For MHA, we independently apply MorphKV to each of the M heads, pruning their caches in parallel. For GQA, we first aggregate attention weights from the M/M grouped query heads, then compute fusion scores F, for the shared key-value heads. Our experiments show that MorphKV performs equally well with both approaches. By default, we choose GQA due to its memory efficiency. This flexibility distinguishes MorphKV from prior works like SnapKV (Li et al., 2024) which are only limited to MHA.
At timestep Ti The most recent tokens shift to [perfect, for), while older tokens include [sun, out). MorphKV re- calculates the fused attention scores for sun and out. If out receives lower combined score than (as illustrated), it is evicted, preserving only in the distant context.
At timestep T2: The most recent tokens are now [for, bright], and the older tokens still include sun. If sun contin- ues to receive relatively high attention from the new recent tokens, remains in the cache despite being one of the old- est tokens. This shows how MorphKV can preserve distant tokens of continued importance (e.g., sun) while evicting those that have very likely become less relevant over time.
4. Evaluation Methodology
Models: We evaluate MorphKV across four state-of of-the-art LLMs chosen based on their complementary strengths:
Llama-3.1 8B Instruct (Dubey al., 2024): model
3.3. Handling Multiple Heads
optimized for long-context tasks (128K token window) and coherent multi-ture dialogue.
For transformer-based LLMs, attention is performed across multiple heads, called Multi-Headed Attention (MHA)
Mistral-v0.2 7B Instruct (Jiang et al., 2023):
5

---

Dialogue Without Limits: Constant-Sized KV Caches for Extended Response in LLMS
lightweight architecture designed for efficient deploy- ment consumer hardware.
5. Results
5.1. Long- Respoeses:
LongWriter Tasks
Qwen2.5 7B Instruct (Qwen et al., 2025): A model with multi-lingual English and Chinese proficiency.
We evaluate MorphKV on open-ended, long-response text generation using the Long Writer (en) benchmark. Long- Writer covers tasks such as writing emails, blog posts, es- says, and novels, with 60 prompts requesting responses rang- ing from 100 to 12000 words. For comparison, SnapKV retains 600 prompt tokens plus all decoded tokens across all attention heads, while H2O stores 600 decoded tokens per head. In contrast, MorphKV maintains a recent window of 30 tokens, a total KV cache capacity of 600 tokens, and supports two fusion strategies: sum() and max().
Phi-4 14B (Abdin et al., 2024): A model specialized in STEM reasoning through high-quality training data and curriculum learning.
This diversity ensures rigorous validation of MorphKV's robustness, scalability, and cross-architectural consistency.
Setup: We run experiments on an NVIDIA Grace Hopper node with an H200 GPU (96GB HBM3) and Grace CPU (116GB LPDDR5) interconnected via NVLink. We imple- ment MorphKV using HuggingFace Transformers (Wolf, 2020) with FlashAttention-2 (Dao, 2023) for hardware- aware optimization, mirroring the configuration of prior KV cache works (Adnan et al., 2024; Li et al., 2024; Wang et al., 2024; Zhang et al., 2023).
5.1.1. PERFORMANCE
Performance is assessed using LLM-based Judge (Mistral-Large-122) 123B), with assigned scores across several criteria aggregated into a final metric. Table 1 shows that MorphKV outperforms H2O, and SnapKV on Llama, Mis- tral, and Phi4, while achieving comparable performance for Qwen. MorphKV consistently excels over H₂O in rele- vance (see Appendix A.4), demonstrating its ability retain critical context, even under memory constraints.
Benchmarks:
LongWriter (Bai et al., 2024) and LongGenBench (Liu et al., 2024a), which require synthesizing structured outputs (e.g., diaries, floorplans) based on input prompts.
Long-Responss generation:
Model
Llama Mistral Phi4 Qwen
Long-Context understanding: LongBench (Bai et al., 2023), for tasks like code repository navigation and document summary with 16K-128K token contexts.
H₂O
68.5
80.0
61.5
63.8
SnapKV
67.7
81.1
63.8
68.4
MorphKV
69.5
81.1
64.7
64.9
Baseline: We compare the performance and memory ef- ficiency of MorphKV against SnapKV (Li et al.. 2024), H₂O (Zhang et al., 2023), and Full-Attentio. SnapKV is the state-of-the-art for KV cache compression, while Full- Attention provides an upper bound on accuracy. However, SnapKV does not perform token eviction during the genera- tion phase, making it less efficient for long-response tasks where cache management is critical. H2O applies KV cache pruning and is thus a more meaningful baseline for evaluat- ing MorphKV in long- oongoees settings. Prior works retain KV pairs across all attention heads, while MorphKV V com- patibility with Grouped-Ques Attention (GQA) enables more memory-efficient approach as it only retains KV pairs across grouped-key heads, allowing us to assess trade-off's between KV cache size and retention of relevant tokens
Full-Attention
66.5
81.3
62.9
66.2
Table 1. LongWriter Comparison of LLM Judge Scores shows that MarphKV outperforms other methods by up to 4.5%, retaining important older tokens at much smaller memory footprint.
5. 1.2. KV CACHE SIZES
Figure 6 shows the normalized KV cache sizes relative to Full-Attentio.. On average, MorphKV reduces memory usage 0.25x that of Full-Attention, while H₂O requires 1x and SnapKV incurs significantly higher usage, up to 4x.
KV cache size relative to Aztn. [Lower is Better?
H20
6
Snapky
KorphKV
across different models and benchmarks.
4
We implement MorphKV in Hugging- Face rans formers library (Wolf, 2020), integrat- ing into the existing attention mechanism and leveraging FlashAttention (Dao, 2023) for efficient inference. To ex- tract attention scores for older tokens, we compute partial attention weights for window queries within FlashAtten- tion and store them as a lightweight KV cache extension. We update this cache during generation by appending new attention profiles and discarding the oldest ones.
Implementation:
2
Liama 3.1
Mistral
Phi4
Qwen
Model
Figure 6. LongWriter: KV cache usage relative to Full-Attention On average, Morph V requires only 0.25x the KV cache size, while 0LO0 and SnapKV consume 1x and 4x respectively.
6

---

Dialogue Without Limits: Constant-Sized KV Caches for Extended Response in LLMs
5.1.3. IMPACT OF INCREASING RESPONSE LENGTH
Accuracy (%)
To evaluate the robustness of MorphKV for long responses, compute the LLM Judge Score against increasing sponse lengths, shown in Figure 7 for Mistral-7B. As the response length increases, performance declines across all methods due to the inherent challenges of generating extremely long text (Bai ct al., 2024). However, VorphKV degrades more gradually: 4x increase in length reduces performance by 15%-18% for SnapKV and HzO, whereas the performance only reduces by 10% for MorphKV. No- tably, MorphKV maintains constant KV cache size regard- less of the response length, contributing to its efficiency and robustness over extended text generations.
Model
CR (%)
Once
Range Periodic Avg.
HgO
45
60
27
44
LLama
SnapKV
64
50
55
26
44
MorphKV
64
50
61
24
45
H₂O
71.2
57
60
32
50
Mistral
SnapKV
71
55
57
36
49
MorphKV
71.2
57
62
36
52
H₂O
55
46
51
28
42
Qwen
SnapKV
53
44
46
28
39
MorphKV
51
43
68
30
47
4.50
Judge Score
4.25
Table 2. LongGenBench: Performance comparison of MorphKV, against SnapK V, HzO. MorphKV achieves better across all evaluation metrics i.c., Completion Rate (CR), and all grouped accuracy metries (Accuracy Once, Range, Periodic, and Average).
4.00
3.75
H2O
Full-Attention
3.50
SnapKV
MorphKV
400
600
800
1000
1200
1400
1600
Response Length
to assess response quality. This includes information re- call about singular instances (Accuracy Once), range of instances (Accuracy Range), periodic instances (Accuracy Periodic), and their average (Average Accuracy), while Completion Rate (CR) quantífies the percentage of tasks successfully completed. MorphKV generally outperforms or matches SnapKV, and H2O on all models and metrics. Notably, SnapKV retains all prompt tokens due to ample cache budget. It also keeps track of every decoded token, effectively replicating Full-Attention for these tasks.
Figure 7. LongWriter: LLM Judge Score versus response lengths. MorphKV is more robust against increasing response lengths com- pared SnapKV 0₂O it algorithm, discarding those KVs which don't contribute signifi- cantly the contextual flow. Notably, MV maintains fixed KV cache size regardless of response length.
adaptive KV selection
5.2. Long-Respons:: LongGenBench Tasks
5.2.2. KV CACHE SIZES
LongGenBench contains structured long-respn responss tasks for temporal and spatial categories. The temporal category is divided into Diary Entry and Menu planning tasks, while the spatial category includes Skyscraper Design and Urban Planning. The original dataset has 400 samples (100 from each sub-category). For fair comparison under limited resources, we select 40 samples, with from each sub- category, and use greedy decoding capped at 8K tokens. SnapK employs all attention heads with 32-token win- dow and a total KV cache capacity of IK tokens per head, whereas O₂O keeps 4K tokens in the cache, also maintain-
Figure 8 shows the KV cache sizes for H2O, SnapK and MorphKV relative to Full-Attentio.. On average, Mor- phKV achieves significant memory savings, requiring only 0.55x the memory usage with Full-Attention, while H₂O and SnapKV require 1.22x, and upto 5.01 the cache size of Full-Attentinn, respectively.
Norm. KV cache size w.r.t Artn. [Lower Better)
H2O
6
SnapKV
MorphkV
ing all attention heads. In contrast, MorphKV uses a 200- token recent window and a 4K-token total capacity across GQA heads, adopting max() fusion due its consistently higher performance than sum(). The larger window enables MorphKV to retain critical distant tokens during generation (refer to Appendix A.2 for further information).
4
2
Liama 3.1
Mistral Model
Qwen
5.2.1. PERFORMANCE
Figure 8. LongGenBench: KV cache usage relative to Full- Attention. SnapKV incurs up 13x higher due the extensive retention of KV pairs, while MorphKV maintains constant cache.
Table shows the performance of MorphKV over prior works. LongGenBench uses a rigorous evaluation suite

---

Dialogue Without Limits: Constant-Sized KV Caches for Extended Response in LLMs
Model
2wmqa
drdr
hpqa
mnews
mfqaen
mfqazh musq
nqa
pent
prt
qsp
qms
sams
tqa
Vcs
SnapKV
16.0
22.0
14.9
25.6
25.4
18.7
10.7
32.2
7.6
98.4 11.7 23.1 42.9 91.7 14.2
Llama
MorphKV
14.9
22.5
15.9
26.6
25.7
19.9
10.7
31.9
7.5 97.8 11.9 23.6 42.9 91.5
15.2
Full-Attention
16.5
30.0
16.7
26.8
27.4
20.1
11.4
32.0
6.9
97.7
13.2
23.6
43.7
91.6
16.1
SnapKV
26.6
23.7
40.5
26.0
48.8
41.3
18.3
25.6
2.5
88.6
31.0
23.8
41.9
86.3
13.5
Mistral
MorphKV
26.7
23.9
40.8
26.6
48.4
43.0
16.7
26.7
3.0
85.9 30.9
23.6
42.3
86.3
13.7
Full-Attention
27.1
30.4
43.0
27.1
49.2
48.3
18.8
26.7
2.8
87.0 33.0
24.2
42.8
86.2 15.2
SnapKV
22.3
24.2
19.5
25.0
38.0
47.2
5.2
20.5
12.6
63.9
32.4 22.1 47.2
90.5 11.4
Phi4
MorphKV
22.6
24.1
19.3
25.5
38.2
46.4
6.2
21.0
12.6 64.3
31.2 22.4
47.6 90.6 12.3
Full-Attention
22.2
29.0
19.6
25.9
38.2
48.9
6.0
20.7
11.6 63.3 33.3 22.9 48.2 90.4 13.4
Table 3. LongBench: Performance comparison of MorphKV, Snapk V. and full attention across different models. MorphKVV achieves
higher accuracy in most micro-benchmarks, its KV selection algorithm minimizes redundancy and noise the attention profile.
KV cache size relative to
Full Attn. (Lower is Better)
KV cache size relative to
Full Attn. (Lower is Better)
0.6
SnapKV
MorphKV
SnapKy
MorphKV
1.5
0.4
1.0
0.5
0.0
0.0
qsp
mfqa en
hpqa
govr
trec
musq
qms
mfga zh
sams
pent
mnews
Isht
tga
VCs
nga
drar
Liama 3.1
Mistral Model
Phi4
Benchmarks
Figure LongBench: (a) Llama3.1 I-BB Instruct KV cache sizes SnapKV, and MorphKV relative full attention. On an average, S has a KV cache size of 0.42x, whereas MorphKV is 0.15x compared to Full-Attention (b) Average KV cache sizes of SnapK V and MorphKV relative to full attention across different models. MorphKV yields comparable performance to SnapKV at roughly 50% lower KV cache budget in long-context setting, where the prompt is significantly larger than the response.
5.3. Long-Context: LongBench Tasks
Moreover, for tasks like Phi4-2WikiMQA, Phi4-Passage- Count, and Phi4-TriviaQA, MorphKV even exceeds Full- Attention performance while using only 20% of the memory capacity. This suggests that larger models (c.g.. Phi4 with 14B parameters) can better leverage the dynamic token se- lection MorphKV to capture essential information.
Besides being memory-effiiient for long-response tasks, MorphKV also offers competitive performance the state- of-the-art prompt KV compression for long-conter, tasks. We evaluate MorphKV, SnapKV, and Full-Attention across benchmarks LongBench. LongBench is a comprehensive, bilingual, multitask benchmark suite used to evaluate LLMs for processing extended contexts. It comprises datasets across six task categories in English and Chinese, with average prompt length of nearly 6K tokens. For MorphKV, we set the recent window to 32 tokens and fix its total cache capacity at 2K tokens, and use the sum() fusion. In contrast, SnapK preserves 1024 tokens from the prefill phase and all decoded tokens across all attention heads.
5.3.2. KV CACHE SIZES
Figure 9 compares the average KV cache memory usage of SnapKV and MorphKV relative Full-Attention across all LongBench datasets. MorphKV achieves up to 2 memory savings over SnapKV and up to 5x over Full-Attention. Notably, for datasets like MultiNews, SnapKV requires 2x more memory than Full-Attention because retains KV pairs across all heads, whereas MorphKV operates at just 0.4x the memory of Full-Attention, benefiting from dynamic eviction and GQA compatibility. Designed for GQA, MorphKV supports 2 more tokens while using only half the KV cache capacity of SnapKV.
5.3.1. 1. PERFORMANCE
Table 3 shows that MorphKV generally matches or outper- forms SnapKV across most datasets. Notably, MorphKV consistently surpasses SnapKV V for NultiNews on all models.
8

---

Dialogue Without Limits: Constant-Sized KV Caches for Extended Response in LLMS
6. Runtime and System-Level Trade-offs
in output quality, or degeneration (Holtzman et al., 2019). We examine the N-gram repetition rate in responses from the Llama3. Instruct model to measure this degeneration in generated responses. A higher repetition rate indicates higher degradation in output quality. Table 5 shows that MorphKV has the lowest repetition rate as it continues to prune the KV cache during token generation. In contrast, SnapKV and full attention exhibit greater redundancy as they retain many more tokens in the KV cache.
MorphKV incurs runtime overhead during response- generation as it introduces additional computation steps that depend on the attention profile of the recent window tokens, necessitating additional memory accesses. To minimize this overhead, the implementation of MorphKV employs dedi- cated CUDA streams to prefetch the attention profile ahead of time. By default, MorphKV performs token-eviction at each generation step to precisely manage KV cache memory usage, crucial for long-context and long-response tasks.
Metric
MorphKV SnapKV Full-Attention
Table 4 shows that the memory savings with MorphKV significantly outweigh the runtime overhead, improving the overall system throughput. Furthermore, this runtime cost can be amortized by evicting tokens less frequently (see Appendix A.5), albeit at higher memory usage. These trade- offs highlight that KV cache compression techniques must navigate several system-level considerations, and optimizing for single metric alone is inadequate for practical adoption.
Repetition Rate (N=10)
68%
89%
89%
Table 5. I LongWriter: Degeneration in Llama3. 1-8BB responses measured using N-gram (N=10) repetition. MorphKV reduces rep- etition by dynamically evicting tokens, limiting contextual noise.
8. Future Work
In addition to token selection, prior work has shown that KV cache can further be optimized across attention layers (Cai et al., 2024), and attention heads (Fu et al., 2024; Feng et al., 2024). MorphKV is complementary to these works, and in- tegrating them would yield further memory savings. While detailed evaluation of integrating MorphKV across other optimization axes is deferred to future work, we present preliminary analysis with layer-aware considerations in Appendix A.6. Furthermore, to minimize the runtime cost with MorphKV, we plan to integrate MorphKV within the Flash Attention kemel, improving inference efficiency while preserving accuracy and memory benefits.
Metric
SnapKV
H₂O
MorphKV
Runtime (lower better)
1x
1.62x
1,50x
Memory (lower is better)
1x
0.26x
0.14x
Accuracy (higher is better) 1x
0.94x
1.01x
Throughput (higher is better)
1x
2.40x
4.68x
Table LongBench: Despite introducing runtime overhead, Mor- phKV achieves substantial memory savings, leading to higher overall system throughput, while maintaining response accuracy
7. Discussion
9. Conclusion
7.1. Variation with Design Parameters
The growing memory footprint of KV caches in LLMs poses critical bottleneck for long-context and long-response tasks. In this paper, we propose MorphKV that addresses this challenge by introducing a dynamic, correlation-an aware token selection mechanism that maintains a constant-sized KV cache while preserving contextual coherence. Mor- phKV leverages attention profiles of recent tokens to retain only the most relevant distant tokens. Our studies on long- response tasks show 52.9% memory savings and 18.2% higher accuracy on average compared to sfate-of-the-ar prior works. Our experiments demonstrate that MorphKV scales efficiently with response length, degrading only 10% in performance even as outputs grow 12K tokens, com- pared to 15-18% degradation for state-of-the-an prior works. Furthermore, MorphKV's compatibility with GQA enables 4x greater memory efficiency than MHA- approaches, making it practical for real-worid deployment. These ad- vances position MorphKV as a practical inference- -iie solution. balancing accuracy and memory usage without sacrificing the ability to capture long-range dependencies.
The design of MorphKV consists of three key design param- eters, namely, fusion function used to create the attention profile, recent window size, and total KV cache budget. These hyperparameters collectively influence the overall performance. MorphKV shows minimal sensitivity to vari- ations in its hyperparameters, highlighting its robustness across various configurations. We refer the reader to Ap- pendix 1, A.2, and A.3 for detailed discussions on the sen- sitivity of MorphK V's performance the choice of fusion function, window sizes, and KV cache budget respectively.
7.2. Outperforming Full Attention
KV cache compression methods, despite relying on a subset of context tokens, often outperform full attention in prac- tice (Li et al., 2024; Cai et al., 2024). This stems from their ability to focus on the most relevant tokens, reducing the influence of less useful ones. This is especially beneficial for
scenarios, where noise due to irrelevant to- kens accumulates over time, leading to noticeable decline

---

Dialogue Without Limits: Constant-Sized KV Caches for Extended Response in LLMS
Acknowledgments
points, 2023. URL https://arxiv..r
This research has been supported by computing support on the Vista GPU Cluster through the Center for Generative Al (CGAI) and the Texas Advanced Computing Center (TACC) at the University of Texas at Austin. We thank the gener- ous support from the Cockrell School of Engineering and the AMD endowment at the University of Texas at Austin. This work is supported in part by the Cisco Research Award. Prashant J. Nair is supported by Intel Transformation Server Architecture (TSA) and the Natural Sciences and Engineer- ing Research Council of Canada (NSERC) [funding refer- ence number RGPIN-20199 -2119-05099) Grants. We acknowledge Won Joon Yun for editorial feedback and Sourish Wawdhane for the code structuring effort.
Bai, Y., Lv, X., Zhang, J., Lyu, H., Tang, J., Huang, Z., Du, Z., Liu, X., Zeng. A., Hou, L.,, et al. Longbench: bilingual, multitask benchmark for long context under- standing. arXiv preprini arXiv:2308. 14508, 2023.
Bai, Y., Zhang, J., Lv, X., Zheng. L., Zhu, S., Hou, L., Dong, Y., Tang, J., and Li, J. Longwriter: Unleashing 10,000+ word generation from long context Ilms, arXiv preprint arXiv:2408. 07055, 2024.
Cai, Z., Zhang, Y., Gao, B., Liu, Y., Liu, T., Lu, K., Xiong, W., Dong, Y., Chang, B., Hu, J., et al. Pyramidkv: Dy- namic kv cache compression based on pyramidal informa- tion funneling, arXiv preprint arXiv:2406.022 02066, 2024.
Impact Statement
Chang, C.-C., Lin, W.-C., Lin, C.-Y., Chen, C.-Y., Hu, Y.--,, Wang, P.-S., Huang, N.-C., Ceze, L., Abdelfattah, M. S., and Wu, K.-C. Palu: Compressing kv-cache with low- rank projection. arXiv preprint arXiv:2407.21118, 2024.
This work aims to advance the field of Machine Learning by reducing the memory footprint of large language model (LLM) inference while maintaining accuracy. By lowering hardware constraints, our technique potentially broadens the applicability of LLMs to more users and domains, includ- ing creative writing, data-intensive research, and education. However, also amplifies existing concerns: easier access to powerful LLMs may exacerbate issues such as misuse of misinformation, large-scale automation of personalized or sensitive communication, and environmental impact due to increased computational usage. While the proposed method does not introduce new ethical risks beyond those already associated with LLMs, we encourage developers and prac- titioners to deploy it responsibly, carefully considering the contexts in which LLMS are employed and the societal im- plications of further democratizing access.
Dao, T. Flashattention-2: Faster attention with better par- allelism and work partitioning, 2023. URL https:
/ //axiivv..o org/ aas//ab///// / 22002337.0000 08691.
Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang, A., Fan, A., et al. The llama 3 herd of models. arXiv preprint 2mXhv:240.21178. 2024.
Feng. Y., Lv, J., Cao, Y., Xie, X., and Zhou, S. K. Ada- kv: Optimizing ky cache eviction by adaptive budget allocation for efficient I1m inference. arXiv preprint arXiv: /550, 2024.
Fu, Y., Cai, 7., Asi, A., Xiong. W., Dong, Y., and Xiao, W. Not all heads matter: A head-level kv cache compression method with integrated retrieval and reasoning. arXiv preprint arXiv:24100 19258, 2024.
References
Abdin, M., Aneja, J., Behl, H., Bubeck, S., Eldan, R., Gu-
nasekar, S., Harrison, M., Hewett, R. J., Javaheripi, M.,
Kauffmaan, P., Lee, J. R., Lee, Y. T., Li, Y., Liu, W., Mendes, C. C. T., Nguyen, A., Price, E., de Rosa, G., Saarikivi, O., Salim, A., Shah, S., Wang, X., Ward, R., Wu, Y., Yu, D., Zhang, C., and Zhang, Y. Phi-4 techni- cal report, 2024. URL pittps://arxiit org/ abs/ 2412. 08905.
Ge, S., Zhang, Y., Liu, L.,, Zhang, M., Han, J., and Gao, J. Model tells you what to discard: Adaptive kv cache compression for llms. arXiv preprint 1rXiv:2310.01801, 2023.
Holtzman, A., Buys, J., Du, L., Forbes, M., and Choi, Y. The curious case of neural text degeneration. arXiv preprint N0Ki://944007555 2019.
Adnan, M., Arunkumar, A., Jain, G., Nair, P., Solovey- chik, I. and Kamath, P. Keyformer: Kv cache reduction through key tokens selection for efficient generative in- ference. Proceedings of Machine Learning and Systems, 6:114-127,203
Jiang, A. Q., Sablayrolles, A., Mensch, A., Bamford, C., Chaplot, D. S., de las Casas, D., Bressand, F,, Lengyel, G., Lample, G., Saulnier, L., Lavaud, L. R., Lachaux, M.- A., Stock, P., Scao, T. L.,, Lavril, T., Wang, T., Lacroix, T., and Sayed, W. E. Mistral 7b, 2023. URL https: / //aaiiiv.. wwo. ab//a/bs/33100 06825.
Ainslie, J., Lee- Lee-Thorp, J., de Jong, M., Zemlyanskiy, Y., Lebrón, F., and Sanghai, S. Gqa: Training generalized multi-query transformer models from multi-head check-
10

---

Dialogue Without Limits: Constant-Sized KV Caches for Extended Response in LLMS
Kang, H., Zhang, Q., Kundu, S., Jeong, G., Liu, Z., Krishna, T., and Zhao, T. Gear: An efficient kv cache compression recipefor near-lossless generative inference of Ilm. arXiv preprint arXiv:2403.05527, 2024.
Saxena, U., Saha, G., Choudhary, S., and Roy, K. Eigen attention: Attention in low-rank space for kv cache com- pression. arXiv preprint arXiv:2408.0500 2024.
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., and Polosukhin, I. Attention is all you need, 2023. URL https::/arxxiv. abs 0376..0766......
Kwon, W. Li, Z., Zhuang, S., Sheng, Y., Zheng, L., Yu, C. H., Gonzalez, J., Zhang, H., and Stoica, I. Efficient memory management for large language model serving with pagedattention. In Proceedings of the 29th Sym- posium on Operating Systems Principles, pp. 611-626, 2023.
Wang. Z., Cui, B., and Gan, Squeezeattention: 2d man- agement of kv-cache Ilm inference via layer-wise opti- mal budget. arXiv preprini Stti::2044044999 2024.
Li, Y., Huang, Y., Yang, B., Venkitesh, B., Locatelli, A., Ye, H., Cai, T., Lewis, P., and Chen, D. Snapkv: Llm knows what you are looking for before generation. arXiv preprint ar 2rv4224404. /4469, 2024.
Wolf, Transformers: State-of-the-ar art natural language processing. arXiv preprint arXiv: (etXii:9910007711 2020.
Xiao, G., Tian, Y., Chen, B., Han, S., and Lewis, M. Ei- ficient streaming language models with attention sinks. arXiv preprint 2339. 17453, 2023.
Liu, X., Dong, P., Hu, X., and Chu, X. Longgenbench: Long-context generation benchmark. arXiv preprint arXiv:2410.04199, 2024a.
Yang. J. Y., Kim, B., Bae, J., Kwon, B., Park, G., Yang. E., Kwon, S. J., and Lee, D. No token left behind: Reliable kv cache compression via importance-aware mixed precision quantization. arXiv preprint arXiv:2402. 18096, 2024.
Liu, Z., Desai, A., Liao, F., Wang. W., Xie, V. Xu, Z., Kyril- lidis, A., and Shrivastava, A. Scissorhands: Exploiting the persistence of importance hypothesis for Ilm kv cache compression at test time. Advances in Neural Information Processing Systems, 36, 2024b.
Yuan, J., Liu, H., Zhong, S., Chuang, Y.-N., Li, S., Wang, G., Le. D., Jin, H., Chaudhary, V., Xu, Z., al. Kv cache compression, but what must give in return? a compre- hensive benchmark of long context capable approaches. arXiv preprint arXiv:2407.0133
Mussmann, S., Levy, D., and Ermon, S. Fast amortized inference and learning in log-linear models with ran- domly perturbed nearest neighbor search. arXiv preprint arXiv:1707.03372 2017.
2024.
Zhang, Y., Hu, Y., Zhao, R., Lui, J., and Chen, H. Unifying kv cache compression for large language models with leankv. arXiv preprint arXi::2412.03131 20244
Qwen, :, Yang, A., Yang, B., Zhang, B., Hui, B., Zheng, B., Yu, B., Li, C., Liu, D., Huang, F., Wei, H., Lin, H., Yang. J., Tu, J., Zhang, J.. Yang. J., Yang, J., Zhou, J., Lin, J., Dang, K., Lu, K., Bao, K. Yang, K., Yu, L.. Li, M., Xue, M., Zhang, P., Zhu, Q., Men, R., Lin, R., Li, TT, Tang, T., Xia, T., Ren, X., Ren, X., Fan, Y., Su, Y., Zhang, Y., Wan, Y., Liu, Y., Cui, Z., Zhang, Z., and Qiu, Z. Qwen2.5 technical report, 2025. URL https: // //axxiv.org abs/ wta.s//444 2515.5
Zhang, Z.., Sheng, Y., Zhou, T., Chen, T., Zheng, I.., Cai, R., Song, Z., Tian, Y., Ré, C., Barrett, C., et al. H2o: Heavy-hitter oracle for efficient generative inference of large language models. Advances in Neural Information Processing Systems, 36:34661 1116661-4710, 2023.
15115.

---

Dialogue Without Limits: Constant-Sized KV Caches for Extended Response in LLMS
A. Appendix
A.1. Impact of Fusion Function: sum() Versus max()
MorphKV considers two fusion functions for deriving the final attention profile, namely, Fj: sum() and max(). In this subsection, we discuss their impact on the performance.
A.1.1. ON LONG-CONTEN, TASKS
We compare both sum() and max() fusion using the LongBench suite on the Llama3. 1-8B model, with recent window configuration of 32 tokens and KV cache capacity of IK tokens. Table 6 shows per-dataset performance. On average, max() fusion outperforms sum() by about I %, and up to 2.7% on QMSum. Datasets such as 2WikiMQA, MultiNews, Passage Count, Passage Retrieval (En), QMSum, and TriviaQA often demand sharply focused retrieval or reasoning. A single strongly attended token in these tasks can suffice to link crucial context, making max() advantageous, In contrast, sum() tends retain past tokens which are preferred by majority of the window tokens. Consequently, max() better captures a small set of pivotal tokens spread over large distance (Ionger sequence of tokens).
Llama3.1
2wmqa drdr hpqa Isht mnews mfqaen mfqazh musq nqa pent prt qsp qms sams trec tqa
max fused
14.9 21.0
14.9 33.5
25.6
24.2
17.9
9.3
32.0 8.0 97.6 10.0 23.5 42.1
91.8 14.5
sum fused
14.6
22.0 15.0 35.5
25.6
25.4
19.2
9.9
32.2 7.9 96.9 10.5 22.9 43.1 49 91.8 14.8
Table 6. LongBench: Llama3.
comparison of MorphKV under different fusion functions with the same cache budget.
A.1.2. ON LONG-RESPONTE TASKS
We similarly compare the sensitivity of sum() and max() for LongWriter tasks which contain essay-style long response prompts. For our studies, we fix the recent window be 30 tokens, and KV cache capacity to total of 600 tokens. As shown in Table 7, sum() fusion tends to be more effective for most models, except Qwen2.5 where max() excels in certain metrics. LongWriter tasks are typically open-ended, causing max() to emphasize specific tokens that are not always universally relevant. Conversely, sum() aggregates attention across the recent window, providing a broader (though slightly noisier) context that suits open-ended generation.
Fusion
Relevance Accuracy Coherence Clarity
Llama
max
83.8
81.3
57.1
64.2
sum
89.2
81.7
63.3
71.3
Mistral
max
91.7
86.7
82.9
82.1
Sum
92.5
89.2
84.2
85.4
Phi4
max
62.9
79.6
68.3
72.1
sum
62.5
80
70.4
75.0
Qwen
max
83.3
70.8
58.3
60.4
sum
85,4
70.4
58.3
59.1
Table 7. LongWriter: Sensitivity to sum() versus max() fusion across different models. scores are aggregated to make the final attention profile.
These fusion functions dictate how attention
A.2. Impact of Window Size on Long-Response Tasks
We evaluate the impact of window size on MorphKV's performance. Intuitively, recent-window tokens determine which tokens to retain from the distant past. Thus, a larger window allows capturing of more diverse information from the past.
Particularly, for LongGenBench. this effect is evident since the prompts contain lot of information which might be needed at a much later point in the generation. Hence, we run the LongGenBench suite for Llama3. 1-8B, Mistral-7B and Qwen2.5
12

---

Dialogue Without Limits: Constant-Sized KV Caches for Extended Response in LLMS
models for variants of MorphKV, both using max fusion and a total capacity of 4K tokens. The first configuration uses 32-token window, while the second uses 200-token window.
As shown in Table 8, changing the window size significantly impacts evaluation metrics for all Llama, Mistral, and Qwen models. This is due to the fact that a very small window does not suffice for capturing extensive amounts of distant information present in a typical LongGenBench prompt (such as specific details about different floors in a building, specific Menu items etc.), because very small windows tend to capture local context, while failing to capture more distant context. Therefore, larger window more effective since it allows the model to retain diverse pieces of information even at very large response sizes. For instance, a window of size 200 lets L recall accurate instructions regarding the 96th floor of building, in spite of already generating extensive descriptions of the previous 95 floors On the other hand, with a window size of 32 tokens, the model struggles to maintain consistency with the input request, and generates generic responses after certain number of floors, thereby losing on accuracy.
Note that the Completion Rates for both window sizes are comparable. This is because the Completion Rate measures the number of times the model was able to generate what it was expected to (for example, how many floors did the model generate the floor plan for out of the requested 100 floors). This relatively simpler task, and smaller window can keep track of such information (for instance, by simply retaining the floor number of the last floor it generated the plan for). Consequently, do not observe substantial differences in the Completion Rate metric.
Accuracy (%)
Config
CR(%)
Once Range Periodic Avg.
Llama
(32, 4K)
64
43
56
27
42
(200, 4K)
64
50
61
24
45
Qwen Místrall
(32, 4K)
71
57
60
32
50
(200, 4K)
71
57
62
36
52
(32, 4K)
52
41
43
34
40
(200, 4K)
51
43
68
30
47
Table LongGenBench: Sensitivity of evaluation metrics with window size across different models (CR: Completion Rate). A small window is insufficient for capturing large-amounts of distant information in long-response tasks, leading to degradation in model accuracy for all models. However, Completion rate remains comparable across window sizes as it only measures the model's ability complete the response as expected, without considering the relevance or correctness of the generated output.
A.3. Robustness Against KV Cache Compression
To assess the impact of KV cache compression MorphKV versus SnapKV, we ablation studies on subset of benchmarks within the LongBench suite. We record the resulting performance across Llama3. and Mistral-7B models. For both MorphKV and SnapKV, the KV cache budget is varied from 1 1% to 7% with respect to full attention KV cache size. MorphKV uses a window of 32 tokens, with sum() as the fusion function. SnapKV also uses the same window size of 32 tokens, but maintains KV cache across all attention heads. This enables MorphKV to store 4x as many tokens as SnapKV at the same cache capacity.
Figure 10 shows the mean and individual benchmark scores for both models under varying compression scenarios. At very low KV cache budgets, we observe drop of more than 50% on average between MorphKV and SnapKV, for benchmarks like NarrativeQA, this difference reaches upto 88%. This disparity indicates that MorphKV is significantly more effective at retaining crucial context information compared to SnapKV under tight memory constraints. Even with larger budgets, MorphKV consistently outperforms SnapKV, demonstrating the robustness and reliability of its design.
The input prompt for resgores tasks is typically very small, and SnapKV does not evict KV cache during decoding, hence we exclude a similar analysis for these tasks.
13

---

Dialogue Without Limits: Constant-Sized KV Caches for Extended Response in LLMS
25
10
Llama3.1 Eval
Score Score
30
4%
20
-11%
-8%
-6%
-6%
44%
-15%
13%
-9%
25
-8%
15
-14%
6-
20
27%
20
SnapKV MorphKV
10
SnapKV
SnapKV MorphKV
SnapkV MorphKV
4-
10
-51%
884%
MorphKV
-75%
-88%
2%
4%
6%
2%
6%
2%
4%
6%
4%
6%
KV Cache % of Full Attn. (a) Average
KV Cache % of Full Attn. (b) MultifieldQA
KV Cache % of Full Attn. (c) Musique
KV Cache % of Full Attn. NarrativeQA
Mistral Eval Score
8%
-1%
1%
Mistral
Eval Score
25
20
4%
-4%-4%
80
60
-13%
2%
20
25
-8%
-8%
-8%
-5%
20
-29%
10%
15
21%
SnapKV MorphKV
SnapkV MorphKV
40
SnapKV MorphKV
15
SnapkV MorphKV
-75%
10
20
-84%
5
2%
4%
2%
6%
4%
6%
2%
4%
KV Cache Full Attn. (e) Average
Cache % of Full Attn. (f) NarrativeQA
Cache % of Attn. (g) PassageRetrieval-en
KV Cache % Full Attn. (h) 2WikiMQA
Figure 10. Comparison of MorphKV versus SnapKV for Llama3.1-8B, and Mistralv0.2-7B. (a)-(d) show results for Llama3. 1--B on Average, MultificldQA-en, Musique, and NarrativeQA respectively, while (c)-(h) show results for Mistralv0.2-7B Average, NarrativeQA, PassagRectrievll-nn, and 2wikimqa respectively. MorphKV consistently beats SnapKV across varying KV cache budgets.
A.4. LongWriter: LLM Judge Scores for Different Models and Metries
Table 9 provides detailed LLM Judge Scores across various criteria. All configurations use KV cache capacity of 600 tokens, with recent window size of 30 tokens for both SnapKV and MorphKV. MorphKV consistently outperforms HzO on Relevance, highlighting its effectiveness in retaining imporatant tokens. Qwen2.5 differs from other models as employs GQA with 7x less heads than the default MHA configuration, potentially introducing partial information loss when MorphKV operates under fewer heads, explaining why MorphKV performs poorer compared to SnapKV for this model.
Model
Relevance Accuracy Coherence Clarity Breadth and Depth Reading Experience
Total
Mistral-71 - Llamu3. 1-8B
H₂O
84.6
81.7
63.3
71.7
$4,2
55.4
68.5
SnapKV
85.8
80.4
63.3
73.3
48.8
54.6
67.7
MorphKV
89.2
81.7
63.3
71.2
57.1
54.6
69.5
Full-Autnttoo
86.2
81.7
$7.9
71.2
49.2
52.5
66.5
H₂O
91.7
89.6
81.2
83.3
60.4
73.8
80.0
SnapKV
90.4
89.6
84.6
84.2
61.7
76.2
81.1
MorphKV
92.5
89.2
84.2
85.4
60.4
75.0
81.1
Full-Attention
93.8
88.8
85.0
84.6
60.4
75.0
81.2
H₂O
59.2
77.9
63.8
70.8
40.4
57.1
61.5
Phi4
SnapKV
66.7
78.3
68.3
71.2
42.5
55.8
63.8
MorphKV
62.5
80
70.42
75
41.2
58.75
64.65
Full-Auention
65.0
78.3
63.8
72.9
43.8
53.8
62.9
H₂OO
85.0
67.1
54.6
56.2
67.1
52.5
63.8
Qwen2.5
SnapKV
87.7
72.5
60.6
63.6
68.2
57.6
68.4
MorphKV
85.4
70.4
58.3
59.2
63.8
52.5
64,9
Full-Attention
86.4
69.5
59.8
61.0
64.4
56.4
66.2
Table 9. LongWriter: LLM Judge Scores by model across multiple metrics.
A.5. Coarse-Grained Token-Eviction
MorphKV performs a dynamic token eviction at every generation step to maintain a constant sized KV cache. However, this fine-grained approach incurs runtime overhead as the algorithm must run at every timestep, necessitating memory accesses
14

---

Dialogue Without Limits: Constant-Sized KV Caches for Extended Response in LLMs
to load the attention profile. While MorphKV reduces this overhead by employing dedicated CUDA streams prefetch the attention profile, further reductions could be achieved by invoking the algorithm at certain time intervals. Table 10 presents the performance impact when the token-selection policy is applied at every 8th generation-step (eg. 8, 16, 32 etc.). Adopting a coarser loken-eviction results in runtime improvement, albeit at reduced accuracy. This trade-off suggests that the balance between latency, and accuracy can be dynamically tuned to suit task-specific requirements at runtime.
MorphKV
Completion Rate Avg. Accuracy Runtime
default
72.1%
47.0%
1x
coarse- guusse-rained eviction (step=8)
71.9%
46.1%
0.82
Table 10. Lonc:: Performance of MorphKV with different token eviction policies on Mistral-7B. Coarse-grained selection incurs a slight degradation in long-response generation but yields better runtime compared default (per-timestep) eviction.
A.6. Selective Layer Retention with MorphKV
MorphKV only preserves those distant tokens which are heavily attended by the recent window tokens. However, KV cache also be optimized across attention layers. Prior works (Cai al., 2024; Wang al., 2024) have shown that early layers capture more critical information than later ones. Building on this insight, we perform preliminary experiment where we disable MorphKV for the first three layers i.e., the first three layers are retained for all tokens, to emulate a layer-sensitive strategy. Table 11 shows that this configuration outperforms the default MorphKVV configuration, although with an additional 10% memory footprint.
While integrating MorphKV with orthogonal techniques which optimize the KV cache across attention layers (Cai et al., 2024; Wang al., 2024), and attention heads (Fu al., 2024; Feng et al., 2024) further improve performance, such integration necessitates comprehensive understanding of their trade-offs, and careful parameter tuning. We defer this exploration to future work.
MorphKV)
Completion Rate Avg. Accuracy
default
70.51%
44.2%
layer-aware optimization 71.96%
46.1%
Table 11. LongGenBench: Performance of MorphKV improves with selection (first three layers) layer retention on Mistral-7B, although at an additional 10% memory overhead. Default configuration MorphKV has a capacity of 2000 tokens.
15


Reviews:
[{"review_id": "B8uKJNHqI0", "reviewer": "Reviewer_fibk", "summary": "This paper presents MorphKV to reduce the KV cache in long LLM context. Its dynamic KV selection algorithm improves the accuracy by 18.6 and 13.6 compared to previous SnapKV and H2o, while reducing KV by 88.1 and 51.6.\n\n## update after rebuttal\nDuring the rebuttal, the authors add experiments on larger (24B, 32B) to demonstrate the effectiveness of the method. Though I did not further increase my score (my initial score is 4), I am very supportive of the proposed methods and paper.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer's feedback on both the structure of our paper and the results. We have addressed the points raised in the review below:\n\n**Q-1) Analysis on Larger Models** \n\nR-1) MorphKV remains effective even for larger models. We evaluate MorphKV using long-response tasks from the LongGenBench suite on an Nvidia H100 GPU equipped with 80 GB of HBM-2e memory on two models (Mistral-24B-Instruct [1] and Qwen2.5-32B-Instruct [2]). Even for a single task, SnapKV consumes significant KV cache sizes: 6.16 GB (50.26 GB including weights) and 9.7 GB (71.14 GB including weights), respectively, nearly exhausting the GPU's available HBM capacity. In contrast, MorphKV requires only 0.80 GB and 1.06 GB, respectively. The tables below summarize these results. The evaluated model sizes are consistent with prior works, like SnapKV (7B), H$_2$O (30B), and Keyformer (7B).  \n### **Table-1**: Mistral-24B-Instruct on LongGenBench\n| KV Cache Compression Method | Completion Rate | Average Accuracy |  KV cache size  |\n|:--:|:--:|:--:|:--:|\n| SnapKV  |  61.40% | 57.8% | 7$\\times$ |\n| H$_2$O  |  61.35% | 58.4% | 3.6$\\times$ |\n| **MorphKV** | **61.64%** | **58.4%** | **1$\\times$** |\n\n### **Table-2**: Qwen2.5-32B-Instruct on LongGenBench\n| KV Cache Compression Method | Completion Rate | Average Accuracy |  KV cache size  |\n|:--:|:--:|:--:|:--:|\n| SnapKV |  71.59% | **54%** | 9.1$\\times$ |\n| H$_2$O |  71.39% | 53% | 4.6$\\times$ |\n| **MorphKV** | **71.68%**  | 53% | **1$\\times$** |\n\n[1] Mistral-24B-Instruct: https://mistral.ai/news/mistral-small-3\n\n[2] Qwen2.5-32B-Instruct: https://qwenlm.github.io/blog/qwen2.5/", "claims_and_evidence": "The paper claims to achieve lower KV cache usage and higher accuracy, which is well supported by Table 1 and Figure 6.", "methods_and_evaluation_criteria": "Yes, it runs on llama3-8b, Mistral-7b, qwen-7b, phi 14b which are leading LLMs. It evaluates on LongWriter, LongBench; it compares against SnapKV and H2o, which all make sense.", "theoretical_claims": "There are no theoretical claims.", "experimental_designs_or_analyses": "The experimental designs make sense. It evaluates against the two baselines and the above mentioned LLMs and benchmarks. It shows the method reduces the KV cache size with improved accuracy.", "relation_to_broader_scientific_literature": "Related work in compressing KV are either not scalable or has limited accuracy (as discussed in the background section). This paper introduces a scalable method with high accuracy after KV compression.", "essential_references_not_discussed": "The paper discusses the majority of related works.", "other_strengths_and_weaknesses": "The paper is well written, and the results are significant.", "other_comments_or_suggestions": "I am wondering whether the author can include more analysis on larger models. The current models are 7-14B, where the efficiency issue in larger models is more severe. However, the reviewer votes for accept due to the clear effectiveness of the method.", "questions_for_authors": "Please see the above comments.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "BkZ4vEDdS5", "reviewer": "Reviewer_bnHa", "summary": "This work designed and developed an efficient KV cache management technique to keep constant KV size while achieving higher accuracy  for long context and long response tasks. The author has compared with relevant works, e.g., SnapKV, H2O and full attention, etc, using different model and different benchmarks comprehensively. The evaluation shows that MorphKV achieved start of art performance on KV size controlling and benchmark task accuracy.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their feedback. We have addressed the points raised in the review below:\n\n**Q-1) Effect of Window Size on MorphKV**\n\nR-1) The window size indeed affects the performance of MorphKV. In practice, **all** KV cache pruning methods rely on hyperparameters: examples include total cache capacity in H$_2$O and Keyformer, window sizes in StreamingLLM and SnapKV, and the number of layers for KV cache projection in SwiftKV. Careful tuning of these parameters is essential for practical adoption and achieving substantial memory savings [1,2].\n\nSimilarly, MorphKV also benefits from tuning its window size parameter. Our experiments show that a window size of 32 consistently offers high performance across all LongBench suite tasks, including diverse scenarios such as code generation and literature review. Although increasing window sizes can slightly improve performance for tasks requiring information recall over longer spans, it typically yields diminishing returns, as described in **Table 3** and **Table 4**. Hence, the window size can be coarsely approximated based on the nature of the task in terms of the information span to achieve good performance. \n\n\n\n### **Table 3**: Mistral-24B-Instruct performance on LongGenBench\n| MorphKV Configuration | Completion Rate | Average Accuracy | \n|:--:|:--:|:--:|\n| window size: 32 | 61.55%  | 54.3% |\n| window size: 200 | 61.64%  | 58.4% |\n\n### **Table 4**: Qwen2.5-32B-Instruct performance on LongGenBench\n| MorphKV Configuration | Completion Rate | Average Accuracy | \n|:--:|:--:|:--:|\n| window size: 32 | 71.68%  | 51% |\n| window size: 200 | 71.68%  | 53% |\n\nNOTE: each scenario above has a total KV cache capacity of 4000 tokens\n\n[1] Accelerating Enterprise LLM Workloads: https://www.snowflake.com/en/engineering-blog/swiftkv-llm-compute-reduction/\n\n[2] StreamingLLM Integration into TensorRT-LLM: https://github.com/NVIDIA/TensorRT-LLM/tree/main/examples/llama#run-llama-with-streamingllm\n\n**Q-2) Ease of Usage in Real-World Systems**\n\nR-2) Currently, MorphKV is implemented in the widely used transformers library from HuggingFace (see **setup in Section-4**). MorphKV is integrated inside both standard Attention and FlashAttention modules and uses cache offloading to accommodate demanding memory-intensive use cases. Consequently, MorphKV is compatible with a vast spectrum of models hosted on HuggingFace. Our approach is consistent with prior works and facilitates seamless adoption across LLMs for real-world scenarios.", "claims_and_evidence": "Yes. The claims and evidence are clear and well backed by numbers and detailed step by step illustration.", "methods_and_evaluation_criteria": "Yes. The method makes sense to the problem, i.e., keeping relevant context in the older tokens is helpful for the performance. \nThe evaluation process is comprehensive, i.e., different models and benchmarks are evaluated to compare MorphKV with existing works.", "theoretical_claims": "Yes, the theory and math are correct. The key part is to calculate the attention weights, which makes sense and seems correct.", "experimental_designs_or_analyses": "Yes. The experimental design and analysis are done on a H200 with NVLink.", "relation_to_broader_scientific_literature": "Yes. The key contribution of this paper is it identified a better attention mechanism related to the coherence and semantic meaning of all the tokens, i.e., history token and recent token. Such that, more research can be inspired from this work on how to retain older tokens for better performance and optimized KV cache management.", "essential_references_not_discussed": "Yes.", "other_strengths_and_weaknesses": "NA", "other_comments_or_suggestions": "NA", "questions_for_authors": "How easy to use in real world system.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "xpc5zkC75j", "reviewer": "Reviewer_vs1e", "summary": "The paper introduces MorphKV, a novel method for efficiently managing key-value (KV) caches in Large Language Models (LLMs) while maintaining memory efficiency and model accuracy. The method overcomes the problem of growing memory requirements for KV caches during inference by employing a dynamic, correlation-aware token selection process. MorphKV reduces memory usage by up to 88.1%, providing higher accuracy and scalability compared to current state-of-the-art methods such as SnapKV and H2O.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their appreciation of our work. We have addressed the point raised in the review below:\n\n**Q-1) Paper Could Explore PyramidKV, Ada-KV, HeadKV Advancements**\n\nR-1) Thank you for the excellent suggestion. MorphKV is orthogonal to methods like PyramidKV, which optimizes the KV cache across layers, and Ada-KV and Head-KV, which optimize the KV cache across attention heads. Thus, integrating MorphKV with these methods can improve its efficacy even further. In fact, we have already conducted some preliminary studies wherein we simulate a layer-wise optimization by naively deactivating MorphKV for the first three layers based on prior studies that describe the importance of information in the early layers [1,2,3]. Our studies show that this design is even more effective (with roughly 10% additional memory requirements) than the default implementation of MorphKV, which prunes the KV cache uniformly across all model layers. The results are summarized in **Table 5** below.\n\nIntegrating MorphKV with methods like PyramidKV, Ada-KV, and Head-KV as a generalized solution requires a deeper understanding of how the methods complement each other, careful tuning of the design parameters, and development of a robust policy that dynamically combines them for optimal performance. Such an extensive study is beyond the scope of our current paper, and we reserve it for future work.    \n\n### **Table 5**: MorphKV with basic layer-wise optimization for Mistral-7B on LongGenBench\n|Task|Completion Rate|Average Accuracy|\n|-|-|-|\n|MorphKV (default) | 70.51 | 44.2%|\n|MorphKV (layer-wise optimization) | 71.96 (2.05% better) | 46.1% (4.29% better)|\n\nMorphKV configuration has a capacity of 2000 tokens\n\n[1] PyramidKV: https://arxiv.org/pdf/2406.02069\n\n[2] SqueezeAttention: https://openreview.net/forum?id=9HK2rHNAhd\n\n[3] Layer-condensed KV cache: https://aclanthology.org/2024.acl-long.602.pdf\n\n**Q-2) Trade-Off between Computational Complexity and Memory Efficiency**\n\nR-2) By default, MorphKV prunes the KV cache after each token is processed to attain a constant-sized cache, which incurs additional computational overheads. The overheads can indeed be lowered by adopting a *lazy* token-selection policy, at the cost of slight performance degradation. The computational savings scale in the number of pruning steps skipped by the lazy token-selection policy. For example, if MorphKV prunes the cache after every 10 tokens processed, the computational overheads reduce to 0.1$\\times$ of that required in the default MorphKV.\n\nTo study this further, we implement a variant of MorphKV, which allows the KV cache to exceed the pre-allocated capacity before pruning it back. **Table 6** compares the performance of the default and lazy variant of MorphKV. We will include these results in the revised manuscript. \n\n### **Table 6**: MorphKV with lazy token-selection for Mistral-7B on LongGenBench\n|Task|Completion Rate|Average Accuracy|Runtime|\n|-|-|-|-|\n|MorphKV (default) | 72.10 | 47.0%|1$\\times$|\n|MorphKV (with lazy token-selection) | 71.86 (0.34% worse) |46.1% (1.9% worse)|0.82$\\times$|\n\nMorphKV configuration has a capacity of 4000 tokens\n\n**Q-3) Inference Time**\n\nR-3) We request the reviewer to consult R-5 in response to **Reviewer Jsuc**.\n\n**Q-4) Generalizability of MorphKV**\n\nR-4) We already evaluate MorphKV using the LongBench [1] suite that encompasses tasks such as question answering [2,3] based on academic, legal, government, literature and financial reports, reasoning [4-6] from articles and reports from wikipedia, and summarization [7] on documents from academia and product development (please see **Section-5.3**). MorphKV delivers robust performance across these tasks, showing its generalizability across various domains\n\n[1] LongBench: https://arxiv.org/pdf/2308.14508\n\n[2] NarrativeQA: https://ar5iv.labs.arxiv.org/html/1712.07040\n\n[3] Qasper: https://paperswithcode.com/dataset/qasper\n\n[4] HotpotQA: https://paperswithcode.com/dataset/hotpotqa\n\n[5] 2wikiMQA: https://aclanthology.org/2020.coling-main.580/\n\n[6] PassageCount: https://arxiv.org/pdf/2308.14508\n\n[7] QMSum, VCsum: https://arxiv.org/pdf/2308.14508", "claims_and_evidence": "MorphKV claims to offer substantial improvements in both memory efficiency and model performance. Through experimental evaluations on various LLMs (e.g., Llama, Mistral, Qwen), the authors show that MorphKV achieves 86.5% memory savings with a 13.5% increase in accuracy over previous methods. These results are demonstrated across multiple benchmarks, including long-response generation and long-context understanding tasks.", "methods_and_evaluation_criteria": "The authors evaluate MorphKV's performance on tasks such as long-response generation (LongWriter), long-context understanding (LongBench), and structured long-response tasks (LongGenBench). These tasks are compared to SnapKV and H2O, using key metrics like model accuracy, KV cache sizes, and task completion rates. The method's memory efficiency is also tested under varying response lengths, with MorphKV demonstrating stability even as outputs grow larger.", "theoretical_claims": "The paper introduces the theory that the selective retention of KV tokens based on correlation with recent tokens helps to preserve long-range dependencies while minimizing memory usage. This claim is supported by the experimental results, where MorphKV's selective KV cache management outperforms prior methods that either discard or retain uncorrelated tokens.", "experimental_designs_or_analyses": "The experimental design includes comparative studies using different state-of-the-art methods for KV cache compression. The results indicate that MorphKV performs favorably in both memory efficiency and task accuracy. The experiments are robust, spanning a variety of LLMs and task categories, and also consider different fusion strategies (sum and max fusion) for selecting KV tokens.", "relation_to_broader_scientific_literature": "The work builds on existing research in KV cache compression for LLMs, extending previous approaches like SnapKV, H2O, and Keyformer. By dynamically selecting relevant tokens, MorphKV advances these methods, offering better scalability and efficiency. The authors also relate their findings to memory management strategies in other machine learning models and systems.", "essential_references_not_discussed": "While the paper cites numerous relevant works, it could have explored more recent advancements in KV cache optimization and memory-efficient model architectures. For example, methods that combine KV cache compression with layer-specific optimizations might provide additional insights for further improvements. (i.e. PyramidKV, Ada-KV, HeadKV)", "other_strengths_and_weaknesses": "Strengths: The paper proposes a practical, real-time solution for LLM inference tasks that need memory efficiency without sacrificing performance. MorphKV's ability to scale with response length is a notable advantage for tasks involving extended outputs, which are increasingly common in applications like content generation and interactive assistants.\n\nWeaknesses: The paper could have benefited from more detailed discussions on the potential trade-offs between the computational complexity of the token selection process and its memory efficiency. The impact of MorphKV on inference speed is not discussed in detail, which could be a critical factor in real-time applications.", "other_comments_or_suggestions": "Further work could explore the integration of MorphKV with other model architectures, particularly those that focus on multi-modal data. The efficiency of MorphKV could also be tested on more specialized benchmarks, such as those involving medical or legal text generation, to assess its generalizability across domains. Additionally, future versions could investigate the computational overhead introduced by the dynamic KV cache selection process.", "questions_for_authors": "No", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Nz0qRvwR1C", "reviewer": "Reviewer_Jsuc", "summary": "This paper introduces **MorphKV**, a method that dynamically selects caching tokens in pre-trained language models during inference. Unlike prior approaches such as **streamingLLM** and **SnapKV**, MorphKV employs two metrics—*sum fusion* and *max fusion*—to identify and retain tokens most closely attended to by recent tokens during both the **prefill** and **decoding** stages. By leveraging this dynamic caching strategy, MorphKV effectively supports generation tasks involving long contexts and extended responses.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their feedback. We have addressed the points raised in the review below:\n\n**Q-1) Analysis on Larger Models**\n\nR1) MorphKV remains effective for larger models, as demonstrated by our recent evaluations with larger models. We request the reviewer consult **Table-1** and **Table-2** in the response to **Reviewer fibk**.\n\n**Q-2) Comparison with Prior Works**\n\nR2) For a detailed comparison with prior works, including StreamingLLM and SnapKV, please refer to **R-1** in response to **Reviewer-FtVG**.\n\n**Q-3) Choice of Benchmarks**\n\nR-3) Both long-context and long-response generation tasks cater to widely prevalent applications of LLMS and are key areas of focus [1-3]. Long-context generation not only represents common use cases of LLMs like text summarization and passage retrieval but can also be used for innovative training techniques such as multi-shot learning, where the model can learn from hundreds of training examples provided directly in the prompt.\n\nIn contrast, long-response generation is crucial for tasks such as story generation, paragraph completion, comprehensive question-answering, content creation, etc. While prior works, such as SnapKV, are optimized explicitly for long-context tasks, they are inefficient for long-response tasks. MorphKV addresses this critical bottleneck. \n\n[1] Google: https://cloud.google.com/transform/the-prompt-what-are-long-context-windows-and-why-do-they-matter\n\n[2] IBM: https://research.ibm.com/blog/larger-context-window\n\n[3] Amazon: https://aws.amazon.com/blogs/security/context-window-overflow-breaking-the-barrier/\n\n\n**Q-4) Effect of KV cache budget on MorphKV**\n\nR-4) Thanks for the detailed feedback. We have conducted additional studies using LongGenBench tasks for the Mistral-24B-Instruct [1] model to evaluate the impact of varying cache capacities on performance. We observe that the performance of both MorphKV and H$_2$O increases with the number of cached tokens. At the same KV cache capacity, MorphKV remains more effective than H$_2$O. We will revise the Appendix to include these results. \n\n### **Table 7**: Performance with increasing KV cache capacity on LongGenBench\n|KV cache compression method (number of cached tokens) |Completion Rate|Average Accuracy|\n|-|-|-|\n|H$_2$O (1000)  | 56.46% | 35.0%|\n|H$_2$O (2000)  | 61.35% | 52.0%|\n|H$_2$O (4000)  | 61.35% | 58.4%|\n|MorphKV (1000) | 61.64% | 40.0%|\n|MorphKV (2000) | 61.64% | 54.0%|\n|MorphKV (4000) | 61.64% | 58.4%|\n\n[1] Mistral-24B-Instruct: https://mistral.ai/news/mistral-small-3\n\n**Q-5) Is dynamic token selection for caching computationally expensive? If so, how does the runtime of MorphKV compare to SnapKV and H$_2$O?**\n\nR-5) KV cache compression methods must navigate complex trade-offs encompassing accuracy, inference time, throughput, and memory footprint. Optimizing for a single metric alone is insufficient for practical adoption. \n\nMorphKV prunes the cache at every token processing step, which increases the computational overhead. Our implementation employs several optimizations to reduce these overheads, such as CPU offloading and prefetching with dedicated CUDA streams that minimize the time overhead associated with loading attention weights. \n\nCompared to a similar dynamic token eviction policy, such as H$_2$O, MorphKV achieves **8%** faster inference time (while remaining more accurate). MorphKV’s runtime is higher than SnapKV’s, which is expected because SnapKV is tailored for long-context tasks, not long-response ones. \n\nHowever, the reduced memory footprint of the KV caches enables MorphKV to deliver a much higher throughput (up to 4.68$\\times$) due to a larger batch size, effectively compensating for the degradation observed in the runtime of each request. In conclusion, MorphKV outperforms existing KV cache compression techniques by achieving the best overall balance: it reduces KV cache memory usage by up to 85%, increases throughput by 4.68$\\times$, and achieves an inference time improvement of 8% over state-of-the-art methods (such as H$_2$O) – all while preserving or improving accuracy. **Table 8** summarizes these results.\n\n### **Table 8**: Comparison of key metrics across KV cache compression methods for Mistral-7B on LongBench\n|Task|SnapKV|H$_2$O|MorphKV|\n|-|-|-|-|\n| Runtime | 1$\\times$ |  1.62$\\times$ | 1.50$\\times$ |\n| Throughput | 1$\\times$ | 2.4$\\times$ | 4.68$\\times$ |\n| Accuracy | 1$\\times$ |  0.94$\\times$ | 1.01$\\times$ |\n| Memory | 1$\\times$ | 0.26$\\times$ | 0.14$\\times$ |\n\n**Q-6) MorphKV on Retrieval Tasks**\n\nR-6) MorphKV dynamically evicts tokens based on attention weight scores of the recent window tokens. Hence, it is not biased towards retention or eviction of initial tokens during generation. We evaluate MorphKV on the LongBench suite, which includes many retrieval tasks (single-doc QA, multi-doc QA, Passage Retrieval etc.), and MorphKV shows robust performance across all these tasks.", "claims_and_evidence": "The paper is well-written, and most of the claims are supported by evidence. \nIn lines 381-384, “this suggests that larger models can better leverage MorphKV…” seems to be not supported by existing experiments.", "methods_and_evaluation_criteria": "Standard benchmark like LongBench is utilized in the paper for evaluation. Also proposed a long-response scenario for evaluation.", "theoretical_claims": "Not applicable.", "experimental_designs_or_analyses": "In the paper, the author conducted experiments in both long response and long context scenarios.", "relation_to_broader_scientific_literature": "In the paper, there are two major concepts that are prefilled token selection and decode token selection. For prefil,l it is closely related to SnapKV, while MorphKV utilizes slightly different metrics. For decode head, it is more closely related to H2O.", "essential_references_not_discussed": "Not applicable", "other_strengths_and_weaknesses": "### Strengths\n\n1. MorphKV addresses an important challenge in LLM inference, particularly relevant to scenarios involving long-context processing and long-response generation.\n2. The paper is well-organized, clearly written, and easy to follow.\n\n### Weaknesses\n\n1. The largest model evaluated in the experiments is Phi-4 (14B parameters). The authors' claim in lines 381–384 that \"larger models can better leverage MorphKV\" is speculative and not sufficiently supported by their current experimental results.\n2. MorphKV appears to be a relatively minor modification to existing caching methods. The novelty could be better highlighted by clearly differentiating it from previous methods, such as streamingLLM and SnapKV.\n3. Long-context and long-response generation present distinct caching challenges. The paper lacks detailed analysis or justification for why these particular tasks are the most appropriate to demonstrate the efficacy of the proposed strategy.", "other_comments_or_suggestions": "N/A.", "questions_for_authors": "1. In the Appendix, only an ablation regarding the number of recent tokens and fusion strategy is presented. Do you expect performance differences when varying the total number of cached tokens beyond just recent tokens?\n2. Dynamically selecting tokens for caching appears computationally expensive. Could you clarify whether this is indeed the case? If so, could you provide runtime comparisons between MorphKV, SnapKV, and H2O?\n3. Previous methods typically retain older tokens and their all associated information (i.e., full column). In MorphKV, are initial tokens progressively evicted as generation length increases? If tokens are evicted, could this negatively impact performance in tasks that heavily rely on information retrieval?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "JUtVDx47iZ", "reviewer": "Reviewer_FtVG", "summary": "The authors introduce MorphKV, a KV cache compression method for large language models (LLMs). MorphKV is an inference-time technique that maintains a fixed-size KV cache in autoregressive Transformers, addressing the issue of memory expansion as sequence length increases. Unlike traditional approaches that rely on truncation or lossy compression, MorphKV employs a correlation-aware selection mechanism to dynamically prioritize tokens, ensuring high-fidelity context retention while mitigating early-token bias. By continuously refining the KV cache based on recent attention patterns, MorphKV surpasses state-of-the-art methods such as SnapKV, achieving 86.5% memory savings and 13.5% higher accuracy. This makes it particularly effective for long-form tasks like content creation and code generation, enhancing the efficiency of LLM deployment in real-world applications.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their feedback.\n\n**Q-1) Comparison with Prior Works**\n\nR-1) MorphKV’s superior performance mainly stems from its more accurate token selection policy.\n\nStreamingLLM retains the KVs of the first few initial tokens called *attention sinks* and a sliding window of tokens, evicting all intermediate tokens between them. This creates a “context gap”, leading to partial capture of distant context as noted by prior works [1]. In contrast, MorphKV retains distant tokens (including important intermediate tokens) actively attended by recent tokens, enabling essential context capture throughout token generation.\n\nMorphKV eliminates the token selection bias dominant in H$_2$O and Keyformer. MorphKV exploits the insight that due to the auto-regressive nature of token generation, each token attends to all its past tokens and thus, is naturally aware of useful distant tokens. MorphKV retains only those distant tokens identified as relevant by preceding recent tokens, allowing the current output token to focus only on a curated set of useful distant tokens. \n\nSnapKV assumes that critical tokens from the input prompt remain critical throughout token generation. Based on this insight, it does not prune the KV cache during token generation. However, LLMs often suffer from *degeneration* while generating large responses [2] as each token attends to all past tokens, compounding noise from early decoding steps. As SnapKV retains KVs of all generated tokens, it is significantly more vulnerable to degeneration and performance degradation for long-response tasks. In contrast, MorphKV only retains *important* tokens in the KV cache enabling it to generate more coherent responses. \n\nDegeneration is quantified using the rate at which phrases are repeated in the LLM response. **Table-9** shows that SnapKV has 1.3$\\times$ higher repetition rate than MorphKV on LongWriter.\n\n### **Table 9**: Text Degeneration via N-Gram Repetition\n|Llama3.1-8B-Instruct|MorphKV|SnapKV|Full-Attention|\n|-|-|-|-|\n|Repetition Rate|68%|89%|89%|\n\n[1] Attention-Gate: https://arxiv.org/pdf/2410.12876\n\n[2] Text Degeneration: https://arxiv.org/abs/1904.09751\n\n**Q-2) Why MorphKV outperforms Full-Attention (FA)**\n\nR-2) Although we intuitively expect FA to perform superior to KV cache pruning techniques, for the same reasons as described above (degeneration), it may actually perform worse for certain tasks. Recent works, like SnapKV, and PyramidKV also make similar observations. SnapKV argues that FA often introduces noise to the attention mechanism by retaining all tokens, hindering its ability to attend to only the most relevant tokens more strongly. This trend is heavily task-dependent and KV cache compression does not always guarantee outperforming full-attention. For example, while MorphKV outperforms FA for few long-response tasks, FA outperforms MorphKV by 1.8% on LongGenBench for Qwen2.5-32B-Instruct.\n\n**Q-3) Statistical testing**\n\nR-3) Our results do not exhibit any statistical variance because MorphKV performs greedy decoding, and therefore generated responses remain same across runs.\n\n**Q-4) Optimal Value of KV cache size**\n\nR-4) We can obtain the optimal values through Grid-Search with validation on the end task. For our evaluations, we perform a coarse-grained search and determine that a KV cache size of 4K tokens is a practical starting point for most tasks.\n\n**Q-5) Generalizability of MorphKV**\n\nR-5) Please refer to **R-4** in response to **Reviewer vs1e**.\n\n**Q-6) Inference Time**\n\nR-6) Please refer to **R-5** in response to **Reviewer Jsuc**.\n\n**Q-7) Comparison of MorphKV against combination of SnapKV and H$_2$O**\n\nR-7) MorphKV differs substantially from a naive combination of SnapKV and H$_2$O. This primarily stems from the token selection policy used in MorphKV. Although both H$_2$O and MorphKV prune the KV cache dynamically during the decode phase, they use different policies to identify the *important* KVs they must retain. Consequently, even for the same memory budget, the set of KVs retained by the two approaches is significantly different. Please refer to R-1 for a detailed discussion regarding how MorphKV compares with SnapKV.\n\n**Q-8) H$_2$O Illustration**\n\nR-8) We use Figure 3 only as an illustrative example to highlight that H$_2$O’s token retention policy introduces a bias towards early tokens due to aggregated attention scores, similar to *Figure 2 (a)* in NACL [1]. We will update the figure to highlight the dynamic token-eviction in H$_2$O.\n\n[1] NACL: https://arxiv.org/pdf/2408.03675\n\n**Q-9) Compatibility with FlashAttention**\n\nR-9) Yes, MorphKV is already compatible with FlashAttention (please see **Section 4** of the paper). While fused kernels do not provide attention scores directly, It is still possible to tweak the FlashAttention kernel to return partial attention matrices alongside the final attention output, which can be consumed by MorphKV. An in-depth discussion of this is beyond the focus of this work.", "claims_and_evidence": "The paper validates its claim of **constant-size KV cache inference while preserving performance** through benchmark results.  \n\nHowever, beyond overall scores, there is little supporting evidence. The authors do not provide theoretical or empirical justification for why MorphKV should outperforms existing KV compression methods.  \n\nAdditionally, since the KV cache size is a hyperparameter, it is unclear how to determine its optimal value in practice while ensuring performance preservation.", "methods_and_evaluation_criteria": "The evaluation criteria overall make sense. \n\nHowever, I found that the full attention model performs worse than KV cache pruning methods, including MorphKV, in LongWriter and LongGenBench. The authors do not provide sufficient explanation for these abnormal results. (In contrast, in LongBench, one of the most widely used benchmarks, the full attention model maintains top performance.)  \n\nThe paper primarily presents overall benchmark performances, which does not help in gaining a fine-grained understanding of the method. For better clarity, the authors could include example-level or task-level (e.g., retrieval, reasoning) analyses to offer deeper insights into their approach.", "theoretical_claims": "The paper does not contain theoretical claims.", "experimental_designs_or_analyses": "Please check Methods And Evaluation Criteria above.", "relation_to_broader_scientific_literature": "This paper's evaluation is limited to NLP, but the inference algorithm could also be applied to Transformer models in other domains.", "essential_references_not_discussed": "I do not find any essential references that need discussion.", "other_strengths_and_weaknesses": "Strengths\n- The paper is generally easy to read.  \n- The paper is self-contained.  \n\nWeaknesses\n- [**Important**] The paper does not include an analysis of inference time. A key issue is that the proposed method is not compatible with fused attention kernels, such as FlashAttention, since it requires attention scores at each decoding step, which are unavailable for fused kernels.  \n- The approach appears to be a combination of SnapKV (prefilling phase) and H2O (decoding phase). I find little novelty or new insights in the proposed method.", "other_comments_or_suggestions": "In Figure 3, it appears the authors have **misunderstood the H2O baseline**. This method dynamically evicts tokens from the KV cache during generation, rather than retaining all early tokens. As presented, Figure 3 is misleading. (For reference, please see their paper: https://arxiv.org/pdf/2306.14048)", "questions_for_authors": "- Could you provide statistical testing on LongWriter and LongGenBench? I am not convinced that the full attention model performs worse than KV cache pruning methods. I suspect there may be significant variance in performance.  \n- Is MorphKV compatible with FlashAttention?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "vb3R2ZZcHx", "reviewer": "Reviewer_vKki", "summary": "The authors consider the problem of maintaining a fixed size KV cache during autoregressive generation with large language models. The key idea is to retain recent tokens and a limited number of old tokens according to a dynamic selection algorithm that uses the attention patterns of future tokens on past tokens. Morph KV achieves similar performance compared to strong baselines while using a significantly smaller KV cache size.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their feedback. We have addressed the points raised in the review below:\n\n**Q-1) Selection Bias in H$_2$O**\n\nR-1) H$_2$O retains tokens based on aggregated attention scores. This introduces a selection bias because it retains early tokens even when they do not significantly attend to newer tokens. Various prior works, like NACL [1], Keyformer, and EMS [2], have made similar observations. In NACL, the authors extensively study this problem using LongBench tasks on the Llama-2-7B model. While generating the last token, 96% of H$_2$O’s KV cache comprises entries from the first 200 tokens, while only 4% is derived from the subsequent 1199 tokens. Please refer to the NACL paper (particularly **Figure 2 (a)**) for details about this study.\n\n[1] NACL: https://aclanthology.org/2024.acl-long.428.pdf\n\n[2] EMS: https://arxiv.org/pdf/2412.08521\n\n\n**Q-2) SnapKV for Long-Form Generation**\n\nR-2) Indeed, SnapKV’s overwhelming memory requirements make it unsuitable for long-response tasks. While H$_2$O is more memory-efficient for these tasks, its performance is limited due to the problem of selection bias towards early tokens. MorphKV efficiently navigates this trade-off and achieves robust performance and high memory efficiency for both long-context and long-response tasks. \n\n**Q-3) H$_2$O for Long-Context Benchmarks**\n\nR-3) SnapKV is the state of the art for long-context tasks and already outperforms H$_2$O [1]. Hence, we only compare against SnapKV because outperforming it naturally implies outperforming H$_2$O (please see **Section-5.3** of their paper).\n\n[1] SnapKV: https://proceedings.neurips.cc/paper_files/paper/2024/hash/28ab418242603e0f7323e54185d19bde-Abstract-Conference.html\n\n\n**Q-4) Prompt for LM Judge**\n\nR-4) We use the publicly available LongWriter framework in our studies. LongWriter uses a single-answer grading approach, where an LLM judge is asked to assign a score to a single answer (please refer to Section 3.1 in [1] for further details). The prompt used for the LM judge can be found here: https://github.com/THUDM/LongWriter/blob/main/evaluation/judge.txt\n\n[1] Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena: https://arxiv.org/abs/2306.05685\n\n\n**Q-5) Design Choices in H$_2$O Baseline**\n\nR-5) The performance of a KV cache pruning method depends on the quality of the retained KVs and how these are utilized for future tokens. In contrast, the memory savings depend on the number of KVs evicted. Naively reducing the memory capacity of H$_2$O also lowers its performance due to the limited accuracy of its token selection policy that exhibits a bias toward preserving early tokens (please see our earlier response). \n\nOur studies using LongGenBench on the Mistral-24B-Instruct model show that MorphKV has 1.67$\\times$ better accuracy when both H$_2$O and MorphKV operate within the same cache budget. To achieve comparable accuracy for the same task, H$_2$O must overcome the limitations of its token selection policy and retain a lot more KVs, increasing the memory footprint. MorphKV is 3.6$\\times$ more memory-efficient compared to H$_2$O in this case.\n\n\n**Q-6) Local coherence and Distant Relative Heuristics in MorphKV**\n\nR-6) Local and distant heuristics enable MorphKV to capture useful information. Discarding recent tokens severely degrades performance, critical to maintaining the text’s local coherence. In contrast, distant heuristics are crucial to generating an apt response overall, as they capture information that establishes the global pre-text and relevance. \nMorphKV leverages the insight that due to transformers’ auto-regressive nature of output token generation, each token attends to all its past tokens and naturally holds awareness of the useful distant tokens. MorphKV retains only those distant tokens identified as relevant by preceding recent tokens, allowing the current output token to focus solely on this curated set of useful distant tokens. \n\n\n**Q-7) Adversarial Cases where Heuristics Overlook Important Tokens**\n\nR-7) The ability to retain *useful* or *important* distant tokens depends on the accuracy of the heuristics and the choice of the appropriate design hyperparameters. For example, the default implementation of MorphKV uses a window size of 32 because it works well across most representative tasks. However, for certain benchmarks that involve recalling information over very long time spans, such as LongGenBench, while this window size captures *most* of the useful distant tokens, the performance can be improved slightly by increasing the window size (which would allow capturing all useful distant tokens). Please see **Table-3** and **Table-4** in response to **Reviewer bnHA** for the detailed results. However, the probability of encountering such scenarios is very low because we carefully tune them, and the most appropriate window size can be well approximated based on the nature of the task and its information span.", "claims_and_evidence": "The claims are well supported by extensive experiments. morph KV maintains a smaller KV cache than prior methods while retaining what seems like good performance as measured by LM judge scores, completion rates, and accuracy on benchmarks like long gen bench.  \nThis is more of a presentation note but it seems like this notion of of selection bias during decoding hinders methods like H2O but I'm not totally clear what the selection bias is or how it was evaluated, at least in a more explicit way\nSome of the baselines felt a bit like red herrings. For example, Snap KV is obviously not a good fit for long-form generation, although I appreciate that there might not be other good long-form generation KV cache techniques", "methods_and_evaluation_criteria": "The authors use standard long generation benchmarks. Regarding long-form generation quality I'm not necessarily super convinced by LM judge scores so it would have been better to have a human evaluation but I realize this is tricky. One question I have is why was H2O not compared against in the long bench tasks in experiment 5.3? Is it because H2O maintains too large of a KV cache?", "theoretical_claims": "NA", "experimental_designs_or_analyses": "I checked the soundness of all experiments, they seem reasonable and standard benchmark/baselines. I would have liked to see the prompt used for the LM judge and maybe some discussion of whether or not there's a standard prompt or LM judge framework like alpaca eval for long form generation.", "relation_to_broader_scientific_literature": "I think the discussion of prior methods is clear although I'm not totally clear on some of the design choices of these baselines with respect to this work. for example for H2O, why isn't there a hyperparameter that can help you reduce the number of retained KVs?", "essential_references_not_discussed": "Regarding clarity I thought the math notation was Overkill in section 3.1.3 I mean basically you're looking at the sum of attention weights or the max attention weight over some window from recent tokens. I think the other presentation issue was that the problem statement for finding a reduced cache size was good but it's not clear how the local coherence or distant relative heuristics really optimize that objective or approximate that objective in any meaningful way.", "other_strengths_and_weaknesses": "NA", "other_comments_or_suggestions": "NA", "questions_for_authors": "Regarding clarity I thought the math notation was Overkill in section 3.1.3 I mean basically you're looking at the sum of attention weights or the max attention weight over some window from recent tokens. I think the other presentation issue was that the problem statement for finding a reduced cache size was good but it's not clear how the local coherence or distant relative heuristics really optimize that objective or approximate that objective in any meaningful way.\n\nBeyond the heuristics you've proposed in this paper for defining notions of what KV entries are important do you have any adversarial examples of when these heuristics miss out tokens are actually important?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: **Summary:**  
This paper presents MorphKV, a dynamic KV cache management method that reduces memory usage during inference in large language models by selectively retaining high-utility tokens based on attention patterns. The method demonstrates strong results across multiple benchmarks and models, achieving notable memory savings while maintaining or improving accuracy, though questions remain regarding novelty, inference efficiency, and compatibility with fused attention implementations.

**Pros:**  
1. Proposes a practical and effective KV cache compression technique that improves upon prior methods (e.g., SnapKV, H2O) in both memory efficiency and task accuracy.  
2. Empirically validated on multiple models (up to 14B) and long-context/long-response benchmarks, with ablation studies included.  
3. Well-written, well-organized, and shows potential for scaling to more complex inference scenarios.

**Cons:**  
Some reviewers questioned the novelty of the approach, raised concerns about its compatibility with fused attention and computational overhead, and noted limited evaluation on larger models and diverse long-context tasks.
Decision: Accept (poster)